The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) - Airbnb Is Building a ‘Company Brain’ for AI Agents with Ahmad Al-Dahle - #781

0:01Security teams don't need another list of alerts. They need to know which vulnerabilities are actually reachable, exposed, and safe to fix. That's what hyperscale remediation from Blitzi does. Blitzi understands your entire software state, then finds known and previously undiscovered vulnerabilities in your codebase. It traces each one through the execution paths, checks the dependencies, and validates the fix. Your engineers approve the plan, and Blitzy delivers batches of tested pull requests ready to review. Less alert noise, faster remediation, and confidence that fixes are validated as safe before you review them. Visit blitzy.com to see it in action. That's blitzy.com. Your company knows a lot more than what's written down. Experienced employees know why a system works a certain way, which decisions apply, and how to ask when the documentation falls short.

1:00giving AI agents access to that knowledge is a much harder problem than just connecting them to a code base. Ahmed Aldale helped build the original iPhone at Apple and led the team behind Llama and Metta. Now, as Airbnb's CTO, he's tackling that problem by building what he calls a company brain. The system, Everest, captures institutional knowledge from employees' workflows and makes it available to agents. But As that shared knowledge grows, how do you keep agents from learning the wrong lessons? I think if you aren't careful, markdown files could be exactly the kind of problem that I was referring to where you have, like, it's difficult to find and there's a lot of, like, context poisoning that's happening. One person uses it, we extract a thing, we put it down a markdown file and then the model's like very good at finding the thing that's old or wrong. Because the biggest mistake you can make is you reveal too much context too quickly to the model.

1:55versus you try to give it enough context to move one step forward to then ask for more context. I'm Sam Charrington, and this is the Twimel AI podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in. When you arrived at Airbnb, talk a little bit about the landscape that you encountered and how you've seen that evolve since you joined. The good news is there was a lot of momentum already. There was a lot of pretty decent foundation, I would say. I think it's both the timing that I joined because really, agentic coding had just begun to become something that people could really leverage. The moment and timing of the capability was right, but also.

2:59I think the Airbnb culture, Airbnb has had a long lineage of working on AI and MLOps and all sorts of things. So it had a strong sort of infrastructure investment. So what I've been able to help the company do is just really accelerate the adoption of AI across maybe three frontiers or three boundaries. The first one is AI for productivity. How does the average engineer use AI to move faster?

3:27That turns out to be like, you have to sort of engineer your platform to make it easy to facilitate AI agents in your infrastructure. And then the second one is, you know, how do we deploy AI agents to the marketplace? These are things that maybe the user doesn't see that really add a lot of accretive value to our platform. It either builds more trust, more reliability, quality, et cetera. These are things that you do in an offline fashion.

3:54And then how do we deploy AI agents to improve both the guest experience or the host experience? So these are kind of like the three frontiers we pushed on. And each one's made like a tremendous amount of progress in a really short amount of time. When you think about that first pillar in AI for engineering productivity, you mentioned you kind of came on just as agent decoding was taking place.

4:22When you think about like how to help engineers be productive today with AI coding tools, like there are lots of different ways that you could come about that. I see lots of different philosophies about AI coding in the marketplace today. I'm not sure we've like fully nailed this is the right way to do it or the one way to do it. How do you describe like the Airbnb way to do or even is there an Airbnb way to do it or an Ahmed way to do it or is it team by team?

4:50Well, I mean every company Has a culture set of principles they're a bit curious to so you know other you know Some companies will say like move fast, you know, that's kind of the mantra So what you'll find is the way that AI is being deployed for activities really about velocity I think Airbnb like really focuses on craft Which I think is actually really really good. It's a person. It's aligned to how I think about things And so because we're very focused on craft, it's not just a velocity or speed question. Velocity actually has a heading, which is good. It's really also about quality. So we kind of have to balance the equities of both. And so when we look at productivity, we don't just look at PR throughputs or token rates or things like that. We also look at defect ratios, reliability metrics. So we try to guardrail the velocity with

5:46craft and judgment to make sure that we are continuing to stay on track. That way, when you get exponential use of AI in your system, you're also pretty much reinforcing a culture that you need to maintain the quality of your system as well. So I think that's one of the very high-level things that stays true everywhere. But I think there's some common truths in how to deploy AI agents for your own productivity that are consistent at every company.

6:14So how do you basically MCP your entire company so that an agent can access what it needs based on any generic task? How do you construct context in your code base or documentation within your code base so that agents are able to specialize on the work that they need to do? So if I am a product engineer and I need to work on a PR that lands in the payment stack, which is complicated and regulated and gnarly, Is there enough documentation to inform the agent so that your PR is actually kind of reasonable for the payments team to review? These are all things that you kind of need to build out in order to get velocity at the end, to actually get more productive and also maintain some level of quality. And then the last one is institutional knowledge, which isn't embedded in your organization, really figuring out how to extract institutional knowledge and code it so that

7:09Agents can learn from it. This is like an underexplored but something that we've been working on. So we have something that we call, you know, Project Everest, which is effectively kind of like the company brain. So as people use AI and as they use Everest, it's able to sort of collect a lot of institutional knowledge from everybody using it. So we have a shared context pool of how people do things that maybe isn't in the documentation. What's the organizing principle there? Is it primarily for engineering and engineers?

7:38workflow based? Is it a very free form? No, it's for engineers. I started actually, let me be clear. It started for product engineers, but it is now like dispersing across the company. So we even have product managers, which we call product marketers at Airbnb that are actually also using it. Eventually design will use it and we'll kind of like bring this as like the company, company context graph that is, that is like more about everybody's workflows and learning from everyone's workflows. Interesting. That makes me think about in many moons ago, I worked in the knowledge management field. Knowledge management and collaboration are these perpetual challenges within large enterprises. People create a lot of documents, but there's a lot of implicit knowledge that never gets written down.

8:34Certainly AI and agents change all that because the agents can do some of that documenting for us and they can pull it from places where the work happens as opposed to being told to do it per se. But at the same time, the agents can create a lot of documentation and too much documentation or documentation about the wrong things.

8:59It poisons the well. Talk a little bit about how you've managed that challenge and how Everest approaches curation or prioritization. How do you think about that? Ultimately, the best way to keep your system on track is to have a really strong evals. One of the things that we do with Everest is we have consistent tasks that we look at doing.

9:29Are we trying to benchmark those on... If you use Everest, are you able to complete the task in a shorter number and is it more accurate? So, is Everest having value? What's an example of one of those tasks? Take what a product engineer would do in our pricing domain. If they land that, need to land a PR in our pricing domain with a prompt and they're like, apply a new pricing rule change to the pricing domain. Is it able to complete that task given that pricing engineers have been working with Everest, giving it context and doing their work with it? Is it able to land a PR that is faster and more accurate? As an example, it's complicated because there's a lot of policies and a lot of very specific ways of working with pricing. It's probably one of our most complicated domains.

10:28So is Evers able to let people work in the pricing domain, encoding a lot of the institutional know-how of how to work with something that's as complicated as pricing? As an example of it. So the idea in that particular domain, you've got a lot of existing code that illustrates how the pricing aspects of your system work, but the code isn't everything. There's also regulation.

10:53policy and other things and the idea is to make that available to agents as well. That's right. We want to get there. I think it's early, but I think this is to be clear. It's not that Everest has completely solved everything for us, but I think it is the beginnings of something that could eventually become empowering to every engineer who isn't an expert in everything to basically move faster. Do you think of that as a database of some sort or a collection of markdown files of some sort or something else? It's a database of sorts. Yeah. There's a question of what's the right way to structure it. I think markdown files are really good because frontier models have been trained to navigate file systems. So actually, you can get pretty far. There's a question of like, what's the best structure for something like memory? I mean, this is kind of what it is. It's a little bit of a memory structure. And you know, there's a question. So we were experimenting with the right data structure. It's a database.

11:52Is a better approach or if actually a markdown file would like and a harness kind of deciding what the directory structure needs to be is the right approach. So I don't think we've like I don't I'm not convicted that we have the right architecture for it yet. Do you think of it as primarily collecting the context and letting the agent figure out what to do with the context or do you believe that kind of extracting some degree of structure and making that structure available to the agents is something that is a promising direction. I guess I'm wondering, you know, you kind of creating vector embeddings on it and using it in beddings to allow the agents to find context. Are you doing like graph kinds of things to allow the agents to navigate this or is it, you know, here's a bunch of files or a way to like list.

12:45a bunch of files in an area and a database and you think the agents as they get more and more powerful will be able to figure out what they need to do with it. The short answer on this, I think, is I think we're going to end up looking at all of these to figure out which one works best. I find it hard to believe a vector search is going to scale to where we need it to work, where the whole company is using it.

13:15I think if you aren't careful, markdown files could be exactly the kind of problem that I was referring to where you have like, it's difficult to find and there's a lot of context poisoning that's happening. One person uses it, we extract a thing, we put it down in a markdown file and then the model is very good at finding the thing that's old or wrong. So I think we're going to end up finding some hybrid where actually there's some intelligence in between that is figuring out how to search.

13:44how to do progressive disclosure of context. Because the biggest mistake you can make is you reveal too much context too quickly to the model versus you try to give it enough context to move one step forward to then ask for more context. Kind of like a progressive disclosure approach. And I think that is still kind of open. In terms of the kind of tech stack or platforms that you use for agent decoding, like have you talked about what What that is, like, are you an X shop or a Y shop or does everybody use their own thing? We started as one shop at some point, but we've quickly learned, like, given the pace of the industry, it's a little crazy to be an X shop. At least my strong inclination was, like, we need to be a multi-model shop.

14:41So actually, if you look across the company, you know, for internal productivity, we have something called Air Chat, which basically wraps other harnesses with our MCPs, but we're able to sort of select which one we want to use. And it's kind of nice because Codex is also open source. It's nice of a book now to open source stuff. So generally, we're like pretty open and flexible with the engineer preference, like people can kind of pick which, which a jet to come.

15:10tools they want to use. And then for product, we actually also have our own set of harnesses that we deploy. And we also use a lot of open source stuff that has our own modifications to it to make it work for our systems. What are you doing around evals? That's an area that a lot of companies really struggle with to ensure that they're able to kind of get that data flywheel.

15:37going, one of the systems that I mentioned earlier, Zipline was feature engineering. Feature engineering plays a similar kind of data flywheel role in the ML Ops platform as eval tooling does today. How is the way Airbnb thinks about eval tooling and kind of closing the loop from interactions to...

16:06to kind of intelligence. How has that evolved? For agentic tasks, we've worked with a set of engineers. We have a council of senior engineers and we have a set of tasks or prompts that people want to complete. The challenge with the evals, you could argue there's a very binary outcome. Did it build the thing or did it not? But there's also a lot of subtleness to the correctness of the solution and the quality of the solution.

16:36And so, we have built agentic evals that look at engineering, Airbnb engineering tasks. I can't think of step off the top of my head, but there are a set of tasks that we've worked across various teams to engineer. And then we run those evals against all of the models, both open source and closed source.

17:01to have a sense of the Pareto between the costs of the model and the quality and the performance. And the hardest part is like really trying to push the Pareto down. So like you're trying to not saturate, because everything clutters around, clusters around like 92, 93% or something like that, which a virtual judge is giving you the score. So we calibrated a virtual judge to basically say, given this task, given the execution, like what's the quality.

17:30So we get a percentage. And so we have this chart that we use that basically shows cost versus accuracy. And then we look at all of the open source models and we decide. It sounds like without a lot of care, what you're seeing is that the model will generally say, oh, that's about 90% good. Yeah, yeah. That's the hard part. Exactly. And getting enough separation between them is like...

17:56Sometimes a chore which actually either says one of two things either evals aren't good or the models are actually probably good enough For a lot of stuff that we're doing and then it becomes like a cost and efficiency trade off which and speed trade off really But but I think Most of the time I'm very suspicious that evals if we're not able to get some separation between frontier and non frontier Are these hard enough? Are we like really calibrating the judge to look for?

18:24enough signal in the implementation or the completion. Let's take into the model costs versus capability a bit deeper. There have been some famous examples in, well, Uber, for example, told all their engineers to go to town on agent decoding and then some period of time later was like, whoa, you guys really went to town on agent decoding. This is expensive. How do you think about kind of cost versus value versus for obvious reasons, the topic of routing has gotten very popular nowadays, which is essentially like, how do we get the right tasks to the right model? What have you done to try to optimize there? I fundamentally think that engineers are problem solvers and you just have to inspire the objective, not constrain the execution.

19:22So, like things that I adhere to is like, we didn't have token leader, like we didn't really focus on token leaderboards. The only thing that tokens tell me is if someone's not using any tokens, that is probably something to work on. And if someone's using a lot of tokens, we should try to understand what they're doing. And either by the way, we found like two cases, one case where people needed help to better understand how to build efficiency into their workflows.

19:51And others were actually they're doing really great migrations and it's actually really valuable and like go hand and so I like didn't apply a draconian token limit to everybody I didn't really say token max We didn't do any of those fall into any of those pit holes What we did was try to apply nuance to understanding the shape of adoption Everything is like my theory on the right way to approach it is you know people aren't using it needed need help I need support People are using a lot of it We need to understand and people who are using and you look for other signals like you know people who are using one model pinned Imagine you're using you know name your favorite frontier at maximum are yeah, and it's like it's like these large, you know 10,000 tool calls and every sub agent is like your frontier model and like at some point you're like hey like

20:49That's slow, you know, it's slow right and also like are you sure you need that much? And so, you know, we kind of look for those outliers and help bring them into the shape that we're actually going for In terms of like impact, I mean, this is like what the whole industry is trying to figure out I don't know that we've like completely nailed it Which is like, okay, so you're using all these tokens. Are they a creative? Are they actually adding, you know growth or velocity?

21:18What we try to do is not anchor on usage. We try to anchor on the downstream metrics. We have a pretty rigorous way of looking at, okay, for software engineers, do we see PR throughput increasing? Because that shows that the output is increasing, which is a good sign that they're producing more. But it's not a sign that they're producing more useful stuff, it's just they're producing more. And then I think we look at actual road maps.

21:47The best indicator that you're using AI well is your roadmap is shifting to the right and compress it. We track that, and I think we actually shared stats around it. There are so many confounding factors in everything you're saying, though. It is 100%. PR granularity, roadmap distance. Yeah, totally. I think what matters is that we feel that we're able to do more in shorter amounts of time.

22:17And I think that has been true. But to your point, there is no perfect metric on whether your AI use is resulting in business impact. But there is trend lines that you can look at that, again, the confounding factors almost don't matter. If you're able to shift more faster, AI has contributed. I also think, by the way, we underestimate how much there's been a shift in how we work. It's not just about, oh yeah, I'm using lots of tokens.

22:44But like now, you know, when you think about product development, the old days used to be like have like an MLE and the MLE would work with a product engineer who would then work with a designer who would then work with a product manager and then the legal and then a policy. And, you know, each one's producing like a different artifact that needs to be cascaded across each other. But now you can just jump straight to the project. Everyone's just like right in code.

23:14including the designers, including the PMs, including policy, including legal, like everybody could just jump in to better understand what is the implementation versus trying to pre-negotiate what might be at the end and then like slowly kind of going through that. Every time I hear you say the roadmap pull in, like I feel like the roadmap should be expanding because you can be bolder and think bigger about what you want to build. And if it's pulling in, That is true of companies that don't have more things to do than people. Our road map is so deep. The most I can do is pull it in and try to get it done. There's a lot to do. There's a lot to do. You're right. There's two ways to look at it. Are we doing more or are we pulling things in? I think we have a large ambition.

24:11wants to reinvent itself. Our roadmap is like pretty insane. We're trying to move from, you know, a core marketplace of stays to like, you know, everything that you can do in the physical world is now on our platform to kind of reinventing the entire tech stack and the like platform to support all of that, but also to become more AI native. And also there are things that we will introduce hopefully soon.

24:39that kind of almost shape what the company can become. So I don't know how much more ambitious we can possibly be than that. So yeah, for us, it's really like we got so much to do. Can we do it faster? Can we do it faster? And that's like our main energy. But yeah, I mean, some companies should be asking themselves questions of like, are we ambitious enough in the age of AI? So we've talked quite a bit about AI for your engineers and then helping them be more productive. Certainly, there's a lot more detail we can dig into there, but I'd also like to hear about some of the other ways you mentioned that you're applying agents in particular to the marketplace and the guest host experience. What comes to mind in those domains in terms of concrete things that you've been able to achieve with agents? Most people see the guest side of Airbnb.

25:34It's kind of like a, my best description of it is like an iceberg where at the top you see like the booking engine and the like UI that most people use. It's just the guest experience. But then underneath that surface is an incredibly complex like two-sided marketplace where like the main thing that we trade in is trust and payments and quality. And there's just like this very complicated operational problem, customer support.

26:03insurance, there's a lot that goes behind supply and demand. You know, there's like a whole host side. There are millions of hosts that we have to support and we have to like, they're almost, you know, they're a core part of our ecosystem. And so, you know, we build a whole bunch of stuff for them. So there's a whole very, very complicated like stack that lives below what most people see. And so, you know, when we think about AI deployment, A lot of what we've been able to do even in offline fashion is help support the core tenets of what Airbnb is, which is a trusted marketplace. And so examples, payments. Could we, can we use AI agents to help identify payment disputes or fraud? Or can we use AI agents to support

27:01better guest host experiences, can we detect quality issues, bad on-trip experiences? And these are like can be done at scale in an offline fashion to help the platform improve its service. But it's kind of like behind the scenes can run large scale, you know, in a large scale, large scale fashion without actually getting in front of the user and trying to render anything to them. And then for customer support, I mean, It is a massive operational challenge, right? We have to bring people, we have to train them, we have to develop some level of consistency. And so most people can easily say, well, custom support can be an AI problem. But actually, we don't see it that way. We see it as, how can it be a quality improvement? And so we do want to make sure that AI is actually, if you have a very simple task.

27:54I don't want to call wait for 45 minutes just to ask, can I move my reservation? I just want the answer. AI has a really powerful place to solve for that. Actually, I would argue, there's a whole bunch of reasons for why that's probably one of the hardest problems. It is actually a frontier in a really interesting way to basically build for a two-sided marketplace customer support problem because we don't own the product if you are unhappy with something.

28:22It tends to be a mediation versus like you dealing with a company You're actually dealing with another person through a company and like two-sided mediation is literally having you know a person in between and Two people two very unhappy people and you're trying to get to a very happy outcome Which ends up being like a very complicated problem to deal with and so You know that that tend turns out to be one of the hardest problems that we're working on that is AI related What makes it so hard besides the kind of the broader context that you outlined of like a lack of control? Like do you think of it as fundamentally like an experience or user interface type of problem? Is it like a game theoretical kind of algorithmic thing? Is it something altogether? Like what makes it so gnarly? It's all the above. It's very adversarial. It can be very adversarial, right? So you have like two parties with almost like

29:19bounty that they're optimizing for. It's an outcome they're optimizing for. You have a difficult evidentiary track record, so you can ask each entity to produce its evidence or artifacts, but you can't really verify the accuracy of them. I can easily take a picture of a table, work with AI to modify the table and say, well, I came to this place and look at the table, it's missing a leg.

29:45But is it missing a leg? Is it not missing a leg? I can't go inspect it physically. So it turns out to be like a really hard adversarial problem. Did you saw that leg off? Yeah. Exactly. And so, you know, it's a really, if you think about it from like a machine learning perspective, you have an AI agent and you have these two parties that you're trying to resolve. But we also have your own policies, right? So if you, if you're, you know, if the company, if you, if you bought a Stay or experience and it had a policy of like non refundable and you're trying to find a reason to refund it Like you also have to stick to the rules that everybody plays on the marketplace. It has to be a fair Equitable relationship that you have with the company so we can't take sides. We have to be very Deliverate to applying our policy and you have to adversarial like entities that you're trying to move forward and so it is actually a very complicated

30:42How do you solve it? It's a frontier problem. The models today can't really do a great job of mediating between me and you. And then if I'm Airbnb and then Airbnb and the host, and let's say you're the guest, it is quite a complex multi-party mediation problem. People talk about multiplayer. We're all putting agents inside of group chats, and it barely works. Now imagine if you and I had a financial dispute.

31:10and we stick an AI agent between our chat. Is the AI agent going to really resolve it or is it going to not? It's a physical interaction. You cannot observe it. Right. You're not saying that you're not using agents there. You're saying you're using agents or at least thinking about using agents and it's hard. How are you approaching it? We're in research and development to deal with the hardest use cases.

31:40that I'm referring to but we still apply a lot of human judgment to those use cases where we can train like we can train humans to have the human touch to have the interaction to like really like make sure we get again Airbnb is all about like making sure that we give the best quality experience that we possibly can. So we don't feel like it's ready for the hardest use cases. There is a large tale of use cases where it is incredibly useful. So we've already deployed you know AI agents into production for customer support.

32:10And we resolve about 50% of all chats with AI. So our agenda solution is already resolving a lot. 50% of support chats in particular? Yes. All tickets. Is the user, guest or host interacting directly with an agent? Or are they interacting with someone that appears to be a person, but that person is really an agent or supported by an agent?

32:37We have a philosophy of being very transparent. We're not, we're not putting an AI in front of a person and saying it's a person. We're putting an AI in front of a person and saying it's an AI. So we're pretty, we're pretty transparent about that. And, but the 50% is purely AI, excuse me, resolved tickets. So like we're not, this isn't like a human assisted AI. And I'm talking to a human and the AI is assisting the human to close a ticket. This is 50% AI to human.

33:07Resolution rate so and it's a hill climbing problem, right? So like there's a set like informational queries turn out to be really easy That's not the full fourth 50 cent like those are pretty easy and actually it's way more convenient like if I'm calling Airbnb and I'm asking like an informational query Just give me the answer and I'll come my merry way Don't put me on hold. I don't want to wait for a person like I just want the answer and by the way We also give like cards action cards for people to resolve directly so you can take action. It's not just information queries, but it's also agentic, which covers a lot of use cases. But there's still some portion of use cases that are extremely hard that as we hill climb to the frontier, that is where we really need to push the boundaries and we really need to be quite innovative to solve. It's interesting. My personal experience as a guest and host with Airbnb, I don't feel like I've run into

34:03AI usage that has shifted significantly over the past like two three years like if I need to go to The support site it'll I feel like the flow is the same like what stays it about You know is it a guest issues at a host issue and it kind of walks me through that But it's been doing that for a long time and I've never had an agent answer or get involved in the chat. I have noticed the In the chat like in chat card moderation. Oh, it seems like you're trying to do like a change request Here's the card click the button which is cool. I'm wondering like You know one says that's a really good thing like because most of the time what I think about like a company forcing me to use AI for support It's like not a very good experience. I see that as a success metric the goal isn't to shove AI in front of people it is to meet people where they are but there is a chat

34:57Customer support agent that you can initialize so if you say contact, you know Customer support via chat I think what you're referring to is like our help center Which again people who are using our help center are actually looking for a particular experience They're not looking to just call Airbnb or text Airbnb with a problem, right? And so those workflows still exist and there's still a lot we should be doing with the help center So that's gonna happen, but yeah the volume of our tickets actually happen via chat. And if you do, by the way, if you get into a customer support chat and you say like, give me a human, they will like escalate and we'll route it to human. So we don't force people. I don't know if you've ever used those like terrible call centers where like it just like traps you. Sorry, you have to press like zero 18 times before I get you to a human.

35:49Or maybe if you yell at me enough, I'll get you there. We're pretty careful about getting you to a human, if that is what you prefer. I was just talking to someone about how difficult it is now, because I've been trained to say, agent with increasing urgency when I want to talk to a human, but now like, what's the agent? Yeah, exactly. Exactly. Oh, you want an agent? Well, let me introduce you to this new agent. I was like, human, human, human. Human. Yeah.

36:15Yeah, and if you look at the call volume, if you actually look at the number of people, there's people who have zero interest. And it's not an insignificant. We're talking about 10% of people who will immediately look. That might change as we get better systems, we get more natural systems, as we get more capable systems, people might shift. But there is a non-insignificant number of people that just have no interest in talking to an AI and just one human.

36:42even if their ticket, even if their item is just informational, even whatever, they're just, get me to a cursor. And those are going to be the hardest ones to resolve for because it is really, it's like us, it's psychological. It's not capability based. Since we veered into like personal experiences with Airbnb, I think one of the domains where I'd be most excited to see interesting AI things is like in terms of a genetic search. And I know you've, you've worked a lot historically on like trying to capture user preferences, but it seems like I often have to do like the same mental filter for what I really care about. And it'd be great to be able, and there are also, there is a lot of information that you have about properties that you don't expose via amenities or search filters or something like that. It'd be great to be able to- Yeah, it's deeply embedded in the imagery. It's also embedded in the reviews. It's embedded in a lot of-

37:40a lot of context. I mean, we have a part of why you trust to book these homes is because we have all these additional signals that make you comfortable to book them. And those signals are just as useful to a person to absorb or an AI to help match to a person. And you know, like search is such an important part of how people experience our platform. And we're really very deliberate. So we've experimented a lot with what to do here.

38:09And despite your personal preference, there is like a broader kind of non-consistent user-based behavior that we've been trying to better understand. So people don't want a blinking box where it's like chat. The text back and forth is like...

38:38It's friction, you know, I know where I'm gonna want to stay I know how many people are going give me my options And I don't want to type like a 45 sentence query, you know, we've run tests for people I don't want that either. I agree. Yeah. Yeah, so we've So so something we've we've like we're recently I think it just announced it or like we're testing but we're testing a lot of variants But one of the things that we're testing is like just a natural language box where you can just type where you're going We're going with and like why you're going what you're looking for And we should be able to return the results, but it's still very multimodal. You still have a lot of options. You're still in control. I'm not getting an AI to go pick the one property and sorry, that's like the one or the two or three. I'm still giving you the ability to sort of transpose and like look at different cuts. One of the things you see in how people operate these platforms is they don't, you know, it's not like a straight line to a book actually. It takes like weeks for people to book and they like get inspired. They come, they search.

39:37They click around a lot and then they like don't come back for like three more days because I guess they go talk to their partner and they're like, Hey, I don't know. Right. Yeah. Cause these trips are like social and they're like multiplayer. So, you know, you end up like going on our platform, finding some stuff, maybe you heart some stuff and then you go back. You got to figure out some other things like logistics, timing, travel. Like there's so much to figure out. And then you come back and then you click on some more and then you end up like.

40:05you know, maybe removing some hearts, adding some new ones. And it takes weeks before someone's like, all right, I'm going to spend $10,000 on a trip with the company. So you don't want AI to sort of remove that agency and exploration process. You want to enhance it. So part of what we're trying to do with our agent research is like accommodate for that journey that people go through when they actually decide what they want.

40:33For some trips at least because I think one of the things that you're Identifying is that you know even for me not all trips are the same and so for one type of trip I'm just gonna put in the city and find the first thing that meets the criteria and bounce and then for others It's like this wishlist process and like yeah. Oh, hey check these out, babe You know and yeah, you know weeks later. We settle on something For some kind of trips, you know, I might not even want to interact with Airbnb experience. I'd want Airbnb to be like a, an MCP server to a world of places I could stay and like have my agent sift through all of the things. And, you know, historically, like this has been like whether Airbnb exposes an API of, of, uh, you know, properties, for example, like.

41:26Historically, this has been not a technical issue, but like a political business issue. Do we want to lose control of the customer? Do we want to let people scrape our information, all these kinds of things? But it does strike me that argument is starting to disintegrate because of computer use. And I can still have my agent do it. It's just harder and takes a lot more time. So why not just give me the MCP access? How do you think about that?

41:53Come back to like consumer companies and their DNA and what they need to focus on like fundamentally you need to focus on like what is the best user experience to experience our platform and I think you kind of need to evaluate on a case per case basis and like what makes sense and what doesn't and so I think that is the lens upon which we will make all of our decisions I do want to I do think though That like travel is kind of a unique animal in a lot of ways relative to a lot of other things that you could be doing in the context of like convenience isn't key, trust is. And like a booking isn't just a booking. It's like you have, first of all, you have the inspiration question, which obviously, you know, AI with its world knowledge and its web hooks and like fact that it can search the internet and aggregate all this content. It's like useful, but it is not like it's not the only thing they use for inspiration. Lots of people get inspired by going to like Instagram and watching videos. And there's like all kinds of ways people get inspired about where to go.

42:53Yeah, including talking to friends or events or there's all agents or agents. Oh, yeah, AI is definitely huge and I think 40% or something like that of like humans in the US use AI for inspiration. But when you come to book, like there's the booking and then there's like going on trip and then there's like everything that happens in between and the trust that you have when you do book.

43:21we think that that's going to be something like the best user experience is going to be the one that you trust the most. And because it's such a high judgment, high value purchase, you really want to trust what you're booking and who you're booking with. And so that's kind of, I think one of our main strengths as a company, it's a platform that you can trust. If you, you know, we have a insurance, we have like payments and a payment stack you can trust. We have a customer support.

43:49will deal with issues versus a random MCP server somewhere that you're booking a $10,000 trip with. Yeah, I don't think that I'd want to... I don't think I'd want some agentic commerce thing where the agent just books to stay. I think I'd want the agent to help with screening just the same way your natural language textbox would, but presumably...

44:16The agent can know and store my preferences and kind of get through that more quickly. Yeah. Yeah. Yeah. And I think that's what I meant by like case by case is like, we have a bar for what we want people to feel experientially when they see what we're providing. And again, a home, for example, is like very personal to you, which is why the preferences thing really matters. But also finding the right SKU that works for you is like a deeply multimodal thing. I wouldn't want our product to be a bunch of text.

44:46that a user picks, I would want them to be able to experience it in the right way. So when we feel like there is a way to deliver the right experience to people, we'll work with them. But I think that's kind of very important to us. We talked a bit about some of the tooling and tech stack on the engineering productivity side of things. When you are deploying agents for kind of marketplace guest hosts, is that all?

45:15What are some of the platforms or tech stacks that you're using in building? Are you big consumers of open source? Are you using commercial agent platforms? Have you built everything from the ground up? It's a mix. Let's see. We are users of open and closed source models. We have definitely leveraged open source frameworks.

45:43And we've also built where we feel we need to, where we feel we can innovate and where it's actually valuable for us to build. So take customer support, for example. You know, some of the agent frameworks there, we started building before there were a lot of open source frameworks available. So it was more out of like, we were ahead of where the industry is, and we built what we think we needed to build because there wasn't anything there. You know, arguably, like we always evaluate whether we whether the open source community has built something that's like further ahead than what we have. If it did, we have no qualms like rebasing on the open source and then or migrating to the open source solution and then contributing actually to open source. So my stance on open source is like, well known. I am a big proponent to basically build in the open where we can. And then on the model side, you know, there's a strong

46:37We're an experiential company. There's a strong Pareto that we've been really focused on, which is the speed of our interactions. We have very, very tight latency requirements. And given our scale, for certain use cases, it makes no sense to use large models for very specialized tasks. And for that, we have our own post-training infrastructure, our own RL environments, et cetera, so that we can take more efficient, smaller, faster models that could really beat the frontier at the narrow benchmark that we care about. If you have use cases where you have a narrow thing you're trying to do, you're dealing with customer support and adherence to a policy, I don't have to worry about regression on mathematics or regression on some other domain that I don't care about. Then you can actually post train a model to be really good at it. I would actually say that's

47:34that was why my hot take was like the models are good enough for a lot of consumer experiences. Because right now we're kind of focused on the jagged edge of like, I have now solved a millennium prize problem versus can I do reliable computer use on a on a travel website or whatever, like that feels less, you know, you need a 10 trillion parameter model to achieve. And actually that applies at very like like various scales. And so the more you're able to, if you have narrow use cases, which we have examples of, like our search is a good example, where we take like the smallest model possible because our QPS is so high, that we like really post train and specialize and distill that model to be really good at what it needs to be good at doing. I mean, we're already starting to see a lot of people train different types of models that are just like focused on very narrow things. Like the Jeff example is a good one where, yeah, where people basically like

48:33trained a model for decision-making, which is awesome, but it's a very simple model that focuses on something really, really high leverage for a lot of places. And so we've learned that, and that's why we have our own investments in post-training open models. And do you have a favorite small model that's kind of a go-to? There's a lot of good ones. There's like Neumotron from Nvidia. There's a bunch that we look at.

49:00We're pretty open. We evaluate everything. I'm just imagining same as the case for the larger open weights models You just kind of have a stable that you you know eval everything against and you choose the one that makes the most sense at the time Yeah, that's right. That's right. Any thoughts on kind of where you see this all going at Airbnb? Like what are your what are maybe some of the big challenges ahead for you? And and how do you think AI will help you solve them of the three things we're doing? You know, I think we did a good job Sort of reshaping the company to be a native. I mean like Brian down everybody's using a Brian famously I think just shared at a conference that he has a Second brain that helps him run the company where it's been trained on he has built a context graph Which is local to his machine that is almost like it's been trained on like it has like every document that's ever come to him

49:53It has every vision document, manifesto, email, et cetera, piece of communication he's ever written. And he uses that to help reason through his day, which he's using. So if the CEO and the founder is doing that, you can imagine the company is very excited about it. So I think we're on the right clip. I don't know if it's a challenge, but there's a lot of...

50:17complexity that comes from a lot of AI use like Everest like is Everest going to be useful or is it going to become problematic? Are we going to corrupt it as we scale it? We don't know. Do we even have the right architecture for it? We have to rewrite it. But this is all work, not when you're trying to do it from zero to one, it's much harder than going from where we are to just keep pushing to stay at the bleeding edge. And then I think for the marketplace, I also feel really comfortable.

50:46I think we're working on it, but I think there's still a lot of degrees of freedom on how we shape the user experience that people have with Airbnb using AI while staying true to what we are as a company. I think that is actually, like I said, it's a frontier of interaction. And one of the things that's exciting to me is travel is very complicated. If I was watching this, I could go try to vibe code something. It looks great.

51:13try to use it for a few days and you realize like, oh my god, I can't use it. I want it, I want it, you know, itineraries. I want to be able to share with other people. I want to be, it turns into this like bloated mess. Um, so actually the interaction model and the, and the design of how that comes to life is actually really, really hard. And then, you know, for, for customer support and for this like multi-agent problem, multi- multi-person problem. Uh, how do you deploy agents that can actually resolve things in a constructive way. How do we get to 100%? That sounds like a meaty challenge to dig into there. Yeah. Awesome. Well, Amit, thanks so much for jumping on and sharing a bit about what you're up to there. Yeah. Thanks for having me, Sam. This was fun. Thank you. Thank you.