The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) - Your Ranking Model Is Optimizing the Wrong Thing with Zoe Weil - #780

0:00TikTok is known for getting you to buy things you never knew you wanted. Online retailers would love that kind of impact, but they don't have TikTok's data, so they try to show the shopper what they're looking for and hope that sales will follow. Optimizing for what you can measure is not unique to e-commerce. AI teams often settle twice. They lean on signals that are easy to collect like a click or a thumbs up, and they optimize for metrics that are easiest to evaluate like relevance or benchmark accuracy.

0:29then they hope it all adds up to what they actually care about. If you're building a RAG system, a co-pilot, or an agent, you may be making the same bet. Zoe Wall's answer, at least in retail, is to look between the clicks. The company she co-founded and leads, Sequin, is built on signals most systems ignore, how long a shopper hesitates, how fast they scroll, and how many times they go back and forth between two products before adding one to the cart. And it judges the results not by what people click, but by what they buy. I pushed back on two things, whether this is just age old upsell and cross sell with better data and where it starts to look like surveillance. But she made a point about the playbook most teams follow that I hadn't heard put this bluntly. The biggest innovators in the space are social media companies and they publish a lot in this, right? But they have a wealth of data that none of our customers have.

1:27So if you're a scientist at Wayfair or a price line, you're going to read these papers and you're going to try to implement it. And you know what? Oftentimes you're successful at implementing it, except guess what? It doesn't drive the same outcomes for you as it does for them because you don't have the data that they have. And SQL's big unlock is that we're finally able to get you that unlock without you having that data.

1:56I'm Sam Charrington, and this is the Twimble AI podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in. So anything that you see on the internet is ranked, right? Whether it's a pair of pants that you're looking for at a retailer, hotels that you're looking for for your ex travel, content, media.

2:34even the ads you see online are powered by an auction. And these seem maybe to most people to be unrelated, but every dollar of online spending consumption goes through a ranking model somewhere. And we power those ranking models for the world's largest consumer companies. Walk me through kind of the highlights of what you all are doing.

3:02make your approach unique and, you know, with an eye towards or ear towards the kind of technical innovations. Unsurprisingly, when people think about ranking, they think about information retrieval. In fact, I went to graduate school for information retrieval, which is literally the study of search systems, right? How do I retrieve the most relevant items or documents for a given query, whether that's in a search bar or some kind of relevance module somewhere out there. And historically, and again, quite I would say intuitively, the wisdom here has been that utility should be measured by whether or not I'm able to retrieve documents that are most semantically relevant to a user's query, right? That's a very technical side of it. And that works incredibly well. And I would say,

3:58by and large, maybe it's not a solved area of research, but there have been so many advances on it. It's been pretty remarkable. I mean, the frontier labs have it, you know, pat down or down pat, I forget what the idiom is, but they're pretty good at it. It's not really on its own and interesting, I would say, area of research. However, in a commercial context, you know, modern recommender systems, if you think about TikTok, Instagram and the like, they're not really recommending content anymore, and they're definitely not trying to help people find things. Instead, they're bending the end user's will in subtle ways over time to make them want things that they otherwise wouldn't want, right? So that destination vacation, you know, the latest fashion that you might not need.

4:54Whatever it is that you can think of that people end up just buying on Instagram or TikTok, they do that not because Instagram and TikTok are really good at helping you find what you're looking for. No, they're very, very good at influencing your behavior. And the reason that they're able to do that is that billions of people user apps and are frankly addicted to those apps. They open their apps, you know, 10 to 21 times a day. And when billions of people do that, you have an insane amount of data on them. And so you're able to create these incredibly immersive, I would even say addictive experiences that are hyper-personalized and dynamic. And in fact, the apps react in real time to the user's feedback, right?

5:46So even you, Sam, if what you see on the site or the app in the day, throughout the day will vary based on your interactions with the app. But the truth is that consumer companies, regardless of how large they are, regardless of how many customers they have, wafer.

6:08as a good example. Priceline is another really good example. These are very large companies that have incredibly large user bases, do tens of billions of dollars of online transactions. But again, people don't use Wayfair the same way that they use Instagram or TikTok. And so regardless of how big you are as a consumer business, you are going to be data-starved. And so in an environment, where you don't have as much data on the user, where people are not addicted to your app. How do you influence their behavior in a moment of high intent to increase their basket size, increase their average order value, to increase their engagement time on your site? Again, these aren't like big changes. So on Wayfair, for example, if you're going to buy a $100 couch, it's not like Sequelink can come in and get you to buy.

7:05a $500 couch and also a $1,000 coffee table. The point is, I can get people to buy $102, but I can do it consistently and at scale. And when you do that, that translates to forexing their target profit for the year in less than a month. Upsell, cross sell, like we've been doing this from a data science perspective, pre data science.

7:34You know, we've been doing this for a long, long time, trying to get people to buy more. What's different about the approach? The TLDR, as I was saying earlier, and in fact, with the entire history of this field of research, people have been trying, the wisdom has been, if I can make it easier for people to help to find things, then by extension, I can get them to convert better.

8:02That has been the prevailing wisdom forever. Every company that's ever existed in this domain has been trying to solve search and relevance from the perspective of search and relevance. Make it easier for people to find things faster. That is not what Sequin does at all. We are not at all trying to help people find things. We're trying to influence their behavior in a moment of time when they have high intent. How do we do that?

8:31How do we go about doing that? Which is, I think, the meat of the matter. So we started the company with a black box solution that was powered by what we call large event models. So you can think of it as, first of all, it's not a large language model. We actually don't use large language models anywhere. That's not our business whatsoever. And we don't use generated AI either.

8:53And in any case, so we started with large event models, and the idea behind large event models is that the same way that large language models are able to generalize to text, including text that they haven't seen before ever, large event models are able to generalize to human behavior and streams of events all in real time.

9:20Creating something like that is incredibly difficult to do for a number of reasons. One, you need to be able to create extremely ultra low latency systems that are able to keep track of data at a very, very fine grained level. And not only talking about clicks and conversions and the traditional good stuff, although we use that as well, but I'm talking about hovers, scrolls. I'm actually talking even deeper than that.

9:50of a scroll, the dwell time on a hover, how many times a user goes back and forth. Creating systems that are able to digest data in, frankly, nanoseconds, transform this data also in nanoseconds, put it through a model, several layers of ranking models in milliseconds, and serve dynamic, rebanked items to a user as they're browsing a site.

10:19all in under 50 milliseconds or thousands of requests per second is incredibly difficult to do. In fact, we created what we refer to internally as SQL ranking clusters to be able to even make this feasible at all. And so there was a lot of work done there, not only from an Algo perspective, but also from a data engineering perspective, future engineering perspective specifically for ranking.

10:50and a ton of MLops work and infrastructure work that was done. It's hard because you need to do three things that are very, very difficult to do on their own. You need to have high fidelity leakage-free data pipelines, which is a really difficult problem on its own. You need to have obviously models that are personalized to every user and continue to improve on those models. And then you need to be able to serve them fast enough.

11:19such that rankings are able to react to live user feedback and signals. So building this system at all requires you to do these three things really, really, really, really well. And there are very few people on the planet that can do this for the explicit objective of generating outcomes for businesses, right?

11:47And then, so that's one part of the problem that sequence solves. Two, even if you manage to build that system at all, improving anything within that system is really, really difficult because it's gated not only by how fast your scientists can do experiments and do research, but the ML Ops component, right? The business component of it, right? Like how much am I trading off? Diversity.

12:16or freshness of products or bookings or media content for direct impact to my bottom line. And so the ability to have this unified ranking interface that is focused and brings all these disparate parts of the organization together is incredibly helpful on its own.

12:45But the real secret sauce and where we started the business was with a collection of proprietary ranking algorithms and harnesses that are designed specifically, again, not to help people find things. And I know I keep repeating this, like I'm aware that I keep repeating this, but that is really the golden thread. Because once you realize it and you see, hey, Even though you're operating in the search space, you're not trying to help people find things. You can't unsee it. You said that earlier, and I think I had this like, and I maybe expressed it a little bit, a degree of dissonance and that like, that doesn't sound new to me. And certainly you've lived a space more than I have, but.

13:36You know, we've been talking about kind of next best offer and propensity to buy and like basket size optimization and all these things have been around forever and so is it like is it like old habits die hard and like, you know, they approach the folks for approaching this in a traditional way approach it like a search problem and really the the key is to you know, be like maniacally focused on business metrics, which are kind of harder to do than, you know, focus on like traditional search metrics or it or what. Yeah. Yeah. Yeah. So actually the first part of it is, is, is true that, you know, they're really old habits very hard and they continue to see it as a search problem and try to solve it from a search perspective. Let me help people find things more easily. That is definitely a part of it. The other part of it is that like.

14:35The biggest innovators in the space are social media companies, and they publish a lot in this, right? Amazon used to publish quite a bit about this too, but they have a wealth of data that none of our customers have. So if you're a scientist at Wayfair or at Priceline or at any of the companies that we work with, you're going to read these papers and you're going to try to implement it. And you know what? Oftentimes you're successful at implementing it, except guess what? It doesn't drive the same outcomes for you as it does for them because you don't have the data that they have. And sequence big unlock is that we're finally able to get you that unlock without you having that data. And is that a kind of aggregating customers across? We don't. We don't.

15:31we do not share any customer data between our customers. And that is what makes us truly, truly innovative is that by relying so much on true in session live signaling. I get it. So within the same customer interaction, you're extracting 100 more data than the typical. So we've made ourselves very, very independent of that. How do you respond to folks who kind of see that as like you know going deeper into this like surveillance capitalism kind of rabbit hole and like extracting more data about folks you know as they're trying to do pretty pedestrian things like you know that whole I'm sure you've encountered that. Oh yeah I hate it.

16:22I hate it. I'm one of the people that's probably the most vocal opponent of that kind of... I love what you call that surveillance capitalism. No, I absolutely hate it. We don't do that. And for several reasons, one, this is what distinguishes us from social media companies. We're not trying to create addictive experiences whatsoever. We're not trying to get people addicted to anything. We don't store people's information. It's all in a live transaction.

16:52And I cannot tell you the number of times, even I've been online. And I'm just trying to find, well, now I have a kid and I'm trying to find toys for my kid. But before I tried to find toys for my kid, my friend's kids. And it's insanely difficult to do that, right? And I can actually, I'm good at verbalizing what I'm looking for. And it's still difficult to find. That thing that inspires you.

17:21What makes us different is that we're not surveilling people. We're not chasing them from site to site to site. We're not doing any of that. You already have intent, which is why you're on Wayfair.com. You're on it already because you want to do something. We just make it easier for you to actually get to the thing that maybe you can't even verbalize, to create moments of inspiration for you, spontaneous inspiration for you.

17:50And so it does, yes. You know, price line ends up making a lot of money. Wavefront ends up making a lot of money. But the customers end up walking away happy. We're not doing anything like price gouging or like weird stuff. If anything, the difference is like maybe cents on the dollar per transaction that we're able to increase.

18:13the data that's available through that interaction like a salesperson might in a store. I noticed you were looking at that sofa a little bit longer. I'm going to kind of steer you to things related to that. And maybe you actually, you know, you're looking at the cell phone. What about this case also, you know? But we actually don't care about the identity of the user. We don't care about your name. We don't care about your last name. We don't care about your location. We don't really care about like...

18:41We really care about that interaction, the live interactive component. You mentioned ML Ops several times when you think about the evolution of ML there and ML Ops. What are some of the big things that you and your team have learned in trying to put this platform or environment in place? I think we got lucky in a sense because maybe six, seven years ago.

19:10Rust didn't have a ton of support in like open source AI libraries like hugging face That began to change rapidly in the past. I would say three to four years and You know, this is why I always say whenever you make a successful startup like there's quite a bit of luck there as well Yeah, timing is everything we got very very lucky because we started building this company at a time when support for Rust and many, many open source frameworks. I am offering you think you could not have done it in Python. No, I don't. I genuinely don't. I don't think that we could have because we, you know, ultra ultra low latency feature transformations serving are so core to what we do. I do not think we could have done it in Python. Certainly we do a lot of R&D in Python.

20:10But the core of our backend is all in Rust. So no, I can definitively say we could not have done it in Python at all. And we are very, very lucky to benefit from a lot of the open source contributors that have, you know, for free devoted their time and efforts to bringing support for languages like Rust. I'm imagining that another...

20:33you know, timing tailwind is the availability of coding agents now. You know, you may have needed to build a lot of, you know, supporting infrastructure before because it just wasn't available for what you're trying to do. But now you can just ask an agent to build that for you. For sure. I mean, a thousand percent. You mentioned that Wayfair was an early adopter of Transformers. Are your models Transformer based?

20:59We have some that are transformer-based. We have, I would say, all of our models certainly use self-attention, self-attention mechanism. I can say that without a doubt. All of them do. We have many mixture of experts models now, but they all rely on self-attention. I couldn't say that they use the transformer architecture across the board because now we have...

21:26so many proprietary models that we have built. It's kind of like an orchestra of many, many, many, many models. It's like a mixture of experts, but for ranking. Is the idea at a system level that you've got all this data coming in and you are applying a bunch of different models? Is it routing or it's like you're simultaneously passing the same data through a bunch of different models to kind of come up with new features? Actually, both, right? So there's definitely a routing component to it because if you're in the advertising space, things are going to be different for you than if you're just in traditional commerce. But if we're operating within an ads module, let's just say Wayfair, there are things that the more specialized advertising models that we would use to power our

22:25the bidding platforms might be helpful there. So in that case, there's some bit of routing involved, but also there's a simultaneous kind of rumation sharing and multitasking that we also do. The thing that I was trying to get at is a theme that comes up on the podcast quite a bit and it's this tension between like, we're going to take a bunch of data.

22:52and train some big Mongo model end-to-end and it's going to like take in everything and spit out the one answer versus we've got a bunch of data. We're going to apply a bunch of models to it that, you know, in some ways are, you know, creating features and then those features are going to be used by some other model, you know, or in, you know, or some set of models or independently. It's like this kind of, you know, one big end-to-end thing versus.

23:22We kind of know a lot about this problem. We're going to model individual aspects of the problem the way we might think about it as data scientists. That has some advantages around explainability, model size, et cetera, et cetera. We're closer to the second because we do have these mega models for advertising domain for different domains.

23:52they tend to be on the whole much smaller than what you would see as large ginormous models in the industry. Because the mammoth models nowadays are so, so big. Our largest models are a tiny, tiny, tiny fraction of that. What are some examples of a model input output transformation? I'm trying to get a sense for the granularity.

24:17I'll, I can give you a couple of examples here. So, you know, creating, even if you look at like creating embeddings, right? For queries, right? Very, very simple text. Queries tend to be smaller size. Queries in different contexts and travel and travel hospitality and media and in retail.

24:45can mean completely different things. And so using kind of generalized language, these giant language models to be able to create these holistic representations are not as useful to us in our context. Because again, keep in mind we're a ranking business. We do one thing and we do it incredibly well, and that's re-ranking.

25:08So that's an example of where a lot of people would just use one kind of embedding model, a really, really great embedding model for text that is able to contain the most amount of latent knowledge about intent, et cetera, et cetera. But queries are like what, one, two words maybe? There is a lot, it's very information-starred. So our ability to narrow that to specific contexts.

25:38and see just like two or three token queries in a very specific dimension within the user's overall journey because the user is the first class that is in its equal. And by the way, when I say user, I don't mean the user identity. I mean, the user is a collection of events in a session. When I think about a user, think about it as a collection of events.

26:06in a session, I don't care if you're Dawn Smith or Adam, whatever, to seek when you're just a collection of events in a live session. And how I'm able to best represent you in that context is what sets me aside from every other provider out there. And so we take these very, very raw streams of events, but where that gets routed to, in other words, how I end up representing that information to what then goes on to the final re-ranker is a big part of our secret sauce on what makes us really, really special. Is there a re-ranker or are there re-rankers for... I'm envisioning like, you know, we've talked about Wayfair. You go to Wayfair, you search for a blue sofa.

27:00You've got your primary list and that needs to be re-ranking, re-rank, but then there's like, you may also be interested in this. There's like a bunch of other stuff on the page that... We power all of it by the way. Okay. And they are different. They're very different. So I'll give you an example. Even a very tangible example that I think you'll be able to, that any listener technical or less technical will be able to follow.

27:25Think about like complimentary products, right? So let's say you bought a tennis racket and I'm not going to show you other tennis rackets. You just added it to your cart, right? In fact, that will most likely drive down conversion. Well, you would think, but then we also have the classic, I just bought a refrigerator on Amazon and now they're showing me refrigerators. Like why is that? Exactly. And that's why you would use sequin because we would never do that. That's exactly. No, I'm serious.

27:54But anyway, you go on the add to cart page and we show you complimentary items. So we don't show you other tennis rackets. We show you, hey, what about this like sweatband or this tennis ball? But which tennis band and which, sorry, which sweatband or ball we show you is going to vary from user to user. But the point I'm trying to get at is that that complimentary module exists throughout many of these sites. So it'll exist on a product display page, but it'll also exist on an active cart page. And how you rank items on a product display page, even though it's still a complimentary module, will vary greatly from how you rank items on a complimentary module on an active cart page. And that's because by the time the user gets to the active cart page, they've already committed to the tennis racket, right?

28:48whereas I need to be able to show greater diversity on the product display page because they haven't necessarily committed to that tennis racket yet. Does that make sense? Yes and no. So you have greater diversity in... That's just one example, but the point is that they're both complementary modules. Right, but I'm saying if I search for a tennis racket, I mostly want to see tennis rackets. I don't want diversity in that list.

29:12You don't, but once you click on it, you click on a specific tennis racket, you scroll down. Then you don't know what I'm gonna need next. Exactly. Well, we're betting that we have an idea before you do, or we can get you to click and purchase. Well, so this is an interesting thread, like how, so I'm, you know, I'm in this journey, I search for a tennis racket, bunch of tennis rackets have come up, I choose a tennis racket.

29:41I add it to my car and then you show me a bunch of other stuff. How does any of the information you've collected about dwell time and All these other things while I'm looking for tennis rackets apply to you know, which headband which set of balls which You know anything else like a ton of things. Oh my god You give you give the site so much information. You might not even realize so like how How did you end up deciding between the blue tennis racket? Let's say in the pink tennis racket How long does it take you to decide that? And how does that compare to every other sequence of events that every other user has ever had? Like, is it classic, like collaborative filtering, you know, folks who... No, because in this case, keep in mind, the identity of the user actually doesn't matter, right? And it's all in real time, right? So I look at the sequence of events that I've collected in real time, when people usually, let's just say...

30:38a hover for X amount of time, go back and forth X number of times before settling on this 10 in a live session. It almost sounds like an implicit personality modeling. Like this is an overly analytical, anally retentive person. He keeps flipping back and forth between these two things. He's scrolling over these lists.

31:03kind of, you know, he's like scrolling over the specs or something like that. So that he probably wants, you know, something different from the person who like is flipping through all the pictures. No, it's the difficult thing to do again, more than anything else is like, keep in mind the data, the environments you operate in are incredibly data-starved, right? So with as little information as possible, what can I do? And most of the reliance comes from streaming in session events.

31:32and getting all of that to a model. First of all, the data comes raw in a live session. So you have to clean up all that data in literally nanoseconds. And it has to go into a model, which has to then spit it out in milliseconds. So it's a lot when you think about the entire journey that the raw data has to go through. But yeah, the idea is based on how you got to finally adding that pink or blue or whatever color tennis racket.

32:01to your cart, I'll show you just the tennis ball or a sweatband or bag that you're going to buy. And in fact, one of the things that we are very, very, very good at is increasing basket size, like regularly. The other thing that we're really, really good at is typically historically, you know, ad revenue and organic revenue are considered competing metrics, which makes sense. Typically, usually when ad revenue goes up, it goes at the expense of organic because at the end of the day, they have a list of items and either organic goes to the top rods or whatever. And what we've been able to consistently show is that we're actually able to drive both up together so that one doesn't cannibalize the other, which allows us to compound these massive, massive gains on incredibly

33:01not sure relevant stacks. Elaborate on that for me. When you talk about ad revenue, are you talking about ad spend that's getting the user to the site, or are you talking about like at an Amazon where there's promoted items versus... The second. Second. Okay. Right. And I guess I think about that from an Amazon perspective because it's very...

33:27it's surfaced, it's more or less transparent, a little bit more than less. Whereas I don't know that when I think about Wayfair, I think about sponsored items versus non-spiders sponsored items, but a side of that scale is probably doing that whether or not I know about it is kind of the... I think they're fairly transparent about it. I just think that you see it more on Amazon because Amazon has a much, much larger ads business.

33:55And they're a huge, huge company. And so you were saying that, you know, typically when an organization is kind of, they optimize ad revenue as a... As the primary objective, conversion tends to go down, yeah. Yeah, okay. Because people will, because they're optimizing for people clicking on stuff. As opposed to buying things. As opposed to buying things. I would say one has proven that we can do time and time again.

34:24is we can get people to click significantly more and get them to also buy after the fact. Why is that? Is the same thing we're talking about just more data? Like is it that you're individually optimizing these things and so they both go up or is it some cross? No, they both go up. We're just creating like much more. We're showing people things that they actually want to see based on what we call microfeedback that we get.

34:54Things like scroll speed and stuff. Yeah, it's funny you you you say that because I was thinking just a bit ago about how You know, typically the the big kind of aha moment for Relevance is if you can find, you know, some feedback like any feedback and usually it's like Or you know historically has been in I'm thinking about like rag systems for example like you know, did the user click? Like, you know, that's a, did the user click on the thing that we showed them? Or did the user thumb up or thumb down? Like, you know, there's teams have kind of been starred for like, you know, any kind of feedback and like they, you know, have very few levers for this explicit feedback. And now, like you're just creating way more, way more feedback, implicit feedback. Yeah.

35:54Yeah, no. We rely, I would say, overwhelmingly on implicit feedback data. Overwhelmingly. It's crazy. I was talking to, I was talking to, and I'm not going to name because I don't want to embarrass them, but I was talking to C-level executives from a very, very major bank. So they're like, you'd be lucky to work with us. We're this giant bank, and we have all this transaction data and blah, blah, blah. I was like, actually, that's the least.

36:22interesting data you could give me, because you're only showing me a sliver of human behavior. That transaction data is useless because 99% of people don't actually convert. I'm interested in the 99% of data you actually don't have. And that's what I have. And you don't have that. We didn't get very far in stack. Like we mentioned, Russ, like...

36:48The things that come to mind as you describe the system or questions are like analytical databases no sequel You know cues confluence kind of thing like what are some of the? Infrastructure role the data infrastructure elements that allow you to do this. Yeah, so we definitely use a lot of no sequel for sure Distributed databases we definitely use rest you rely heavily on We rely very, very heavily on open source technology. We use a lot of hugging face, huge fans of hugging face here. I can tell you also what we don't use much of. We almost do not use LLNs at all. It's just not relevant to our business. We're not trying to do gender of anything. We're not a lab. We're not trying to reinvent the wheel there.

37:47So yeah, honestly, it's just a lot of ranking models. It's actually like the thing is that getting these models to work requires you to have an innate understanding of what features drive value. And that is not written in any research paper. And so what we have done is we've created a harness library with proprietary institutional knowledge that we've codified.

38:18what we call residual knowledge that meta has never published about, that Google has never published about, that Pinterest has never published about. They're biggest money makers. Even the papers you see them publish, they're not. The scientists that work on them have to wait for months to get legal approval, and the version of those papers that actually ends up getting published is a very, very watered down.

38:49version of that line of work. And so what we've done is that we have created a harness library and a knowledge base of some of the most effective proprietary methodologies to drive outcomes in ranking. And some of it has come from hiring folks from these institutions.

39:15you know, gathering residual knowledge that's not published anywhere. And then second of all, we have four deployed engineers embedded in all the customers we work with. So we actually don't need feedback data from the customers. But residual information that we gather about what works, what doesn't work is all codified. You mentioned or you referred to is the harness and recursive like the same concept?

39:42or is the harness a sub-concept or? The harness is one component of the entirety of our recursive stack. So you can think of the recursive stack as like being this engine with many, many different components. We have a proprietary knowledge drawer. We have a harness library. We have what we call frontier ranking models, all of which work together to power the recursive stack. And so our core IP...

40:11is the recursive stack itself that produces these insane ranking models that are able to, you know, four, five, 10X, fortune 500 companies revenue in just a matter of weeks. And that's why we are able to allow them to then own the ranking models, right, because.

40:33the real meat of the matter is the ability to continuously build ranking models, better and better and better and better ranking models that continue to drive revenue. And so, when you talk about this harness, what makes it a harness? Is it just a library and like harness is the cool word or is there something harnessy about it in the same way that, you know, Codex or, you know, Codex or...

41:00Quad code is a harness. Yeah, so it's actually a collection of harnesses that we have. Definitely not, certainly not a cool word at the time that we started building it in. We just needed a way to specifically have what we call eval harnesses to find what we call the commercial ground truth.

41:28of interactions. And so that's why we started to build these harnesses. And we quickly realized in short order that even in the space of evaluating these ranking models, how we're able to do so offline in, let's say, the travel space, which is considered, obviously, higher ticket items has a ton of overlap.

41:57with the furniture space, for example, which also has a higher ticket items. But guess what it doesn't have a lot of overlap with with the ad space because the user is just clicking on things and that translates directly to revenue, but it's not coming at a cost to a user. So how you evaluate these different kinds of ranking models requires you to have a very, very rigorous approach. And so we came up with a collection of these evaluation harnesses. And that was the start of our harness library. You call the tool or the platform recursive. What's recursive about it? Recursive ranking intelligence is kind of the first name. Is recursive just a name or is recursion a part of what you're doing? Actually, if I were to be very, very specific about it, it's most similar to tail recursion.

42:52Meaning that you're able to, instead of having to rebuild the entire ranking iterate on the entire ranking algorithm and rebuild it from scratch, which I would consider iterative and not recursive, you don't have to rebuild the entire ranking stack every time you're learning. So you're able to learn continuously and make minor modifications here and there.

43:20And in doing so, you're able to compound gains at a much faster rate at significantly less at a fraction of the compute. So that is, if I were to get very technical, that is why we call it recursion is that we don't have to rebuild ranking models. Previously, when we started the business, we actually did. So we would train the ranking models.

43:46A daily sometimes hourly, but it would be like from scratch and It took us quite a bit of time actually to figure out a way. How can I have how can I? What I quote iterate on this without having to rebuild it from scratch every single either day or hour or whatever cadence and And insert that recursive element in there We talked about this a little bit earlier, but kind of with the context of the full conversation thus far What is large event models? Is that just an instance of one of your ranking models or is that a particular? architecture or it's one kind it started as the thing that we did let me be very clear that in itself was Insanely difficult for us to do to create these models that were able to

44:42to generalize, to stream of events. It sounds extremely intuitive, and it is, but getting it to generalize across customers and domains is insanely difficult. Intuitively, it makes a lot of sense. But to get it to work in travel and media and retail at different crosspoints, different geographies without relying on any end user identity, is really, really, really difficult to do. So that was one of the hardest things that we began to work on. What's the input? Is it a stream of events? It's just a stream of events. And each event is kind of has a label and a value, for example. Yeah, so it would be like hover combinations, right? So it would be the query, the embedding for the query. And that query could either be literally the

45:33the source item or it could be an actual query of words that user searched for or whatever. That was the first very, very first thing that we worked on and it was the black box model on top of which we built Rankers. Then we built what we called large mesh models, which were like large event models, but they're much more relationally focused. So now I'm looking not only at streams, but I'm looking at may stream simultaneously.

46:03and looking at the relationship between those streams together and trying to create generalized representations out of that. Then we started creating different classes of what we called frontier ranking models, which were independent of large event models and large mesh models, but which large mesh models and large event models contributed to. And so we built these collection of many, many, many now.

46:31in the hundreds of thousands of frontier ranking models that we have that are proprietary, that are not published anywhere. A lot of them with very novel architectures. And that then became the first building block of the recursive stack. Over time, we started to develop harnesses and knowledge store, et cetera, et cetera. To what degree?

46:58Does the idea of time series models or time series modeling play into what you're doing? Do you think of these events as time series or do you just kind of capture a frame of events and kind of process it independent of, you know, longer time sequences? Earlier, I will tell you that earlier it played a much, much bigger role. Earlier in our journey, now I would say it's a small part of it.

47:27growing stock. And I'll tell you why. I think I can give an example that is perhaps intuitive for people to catch on to. Time series and sequence-based modeling works really, really well for languages and generalized reasoning because you can show A follows, sorry, B follows from A.

47:51And there is a general order to things. And even though we're in the ordering business and all our whole business is ranking, I'll take you back to the fundamental concept that the first class citizen at Sequin is the user. And the user does not behave as cleanly or logically as you would expect. There's a lot of bad illness in people's behavior.

48:19People aren't like languages. Language, despite how, however creative that you can get, has a certain repetition to it that human beings simply do not have, right? You could be in the middle of buying a couch, right? And your dog just comes barking in and completely ruins your train of thought. And then you suddenly realize, oh, I'm going to buy a dog bed too. You know, there is something inherently unpredictable.

48:48About human behavior to the degree that we've talked about like a big unlock being like capturing more information from the user Capturing me the events of the browser is certainly complex, but doable, but then you talked about like the infrastructure to manage those events at scale, it's a lot of, it's a lot. And then transform all of those in real time. And then know what to transform in it. Yes. Which by the way, we have dedicated models to do that. So you'll literally have streaming data that's going in, not to one model, but it's going into like hundreds of models, which all have to then output that data, reconstruct it, put it into the final riker and have it all be served.

49:35to the user in sub 20, sub 50, sub 100 milliseconds. It's an enormous infrastructure and modeling undertaking. With that in mind, when you think about kind of the technical horizon or the innovation opportunities, like what are the big opportunities to go after that will help you kind of push your system and approach further. There are several routes that we're looking at right now.

50:05without revealing too much. I think one of the biggest debates we've had internally is how much do we invest in traditional representation learning of images or texts and the like versus actually continuing to innovate on the ranking models themselves and the nuances there.

50:34And I was talking about how our evil harnesses and the library there is absolutely critical to our success. And so I would say right now, at this point in time anyway, we're very, very focused. I would say on the ladder rather than focusing on things like that we think are going to be very commoditized if not already.

50:57you know, in terms of having multimodal representations of like hero images on company websites or whatnot. So I would say that is probably what we're most focused on are on things that actually make the ranking, the core ranking model itself better rather than the auxiliary components that are, that involve more traditional representation learning. We're going to leave that to the big image or video or other frontier labs to do and open source projects and we'll continue to benefit from that research as it comes out. Well Zoe it's been great chatting with you and learning a bit about you know what you're up to and how you're approaching ranking. It's been super interesting. Thank you so much for having me. It's been a huge pleasure and I'm a huge fan of your podcast so. Awesome thanks. It's an honor to be here.