The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) - How AI Learns to Smell with Alex Wiltschko - _771

0:00AI has advanced primarily by learning from the digital world, text, images, audio, and increasingly video. By many of the problems people want AI to solve live outside these modalities in the physical world. Smell is one of the most interesting examples. It's how animals detect disease, identify food, navigate environments, and communicate through chemistry. Yet, scent has remained largely outside the reach of computing because Unlike language or images, there has never been a practical way to digitize it at scale. Alex Wilczko, founder and CEO of Osmo and a formal Google DeepMind researcher, is working to change this. His team is building what they call olfactory intelligence, AI systems that can model, predict, and design sense while creating the datasets and infrastructure needed to bring smell to the digital world. In this conversation, we explore what it takes to give computers a sense of smell, why scent is such a difficult AI problem,

0:57and what it teaches us about the next generation of foundation models. Here's Alex. 99 percent of species on this planet can only speak with chemistry, right? Thinking of bacteria and fungi and plants and insects, like they just only can talk with molecules. And I think that it's really worth adding other kind of alien forms of intelligence to our models. And the way to do that is to train it on the intellectual output of those other intellects, which is.

1:25That's chemistry. That's the sense that's in the air. They're produced by living things for reasons to talk to each other. I'm Sam Scharrington, and this is the Twimble AI podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in. When I think about giving computers a sense of smell, there's kind of two angles to this. One is...

2:01You know, there's some scent out in the world and I want my computer to be able to recognize it the same way I do. And the other, which is I think more along the lines of what you're working on at Osmo, at least initially, is to have the computer kind of grok the idea of scent so that it can create new ones. Any scent that's been given to computers, there's three kind of broad steps. You got to read the world to like turn atoms into bits and information.

2:27you have to map it, like understand it. So, you know, be able to manipulate it, digitally encode it, send it. And that's like JPEG and RGB, right? And then you have to be able to write it back out again, right? So a printer or a display or a speaker. And so the thing we focused on at Google Brain was the missing piece, which is for sent is the map. So color has had a map for representation of...

2:52What to what, though? Exactly, exactly. So like, let's approach it from the side. Like, how did this work for vision? How did this work for hearing, right? We've had maps for a long time, right? So the map for sound is just one dimension. It's low to high frequency, really simple to say. And then for color, it's three numbers. It's RGB, right? Or whatever your preferred color space is. But those three numbers, like, tell you how to deal with color. There's three channels of color information in our eye.

3:18But of course, you're simplifying a lot because for each of those other modalities, there's lots of different maps, just examples of their examples and they can kind of be translated into each other. But I'm like, I'm papering over like centuries of psychophysics here and anybody who knows anything about those things is going to come screaming at me. But you'll have to forgive the simplifications. I'm going to simplify sense stuff, too. And if people talked the way that I'm going to talk, I would come after them, too.

3:44So yeah, there's CMYK, there's lab, there's HSV, there's many different maps, and then there's more complex maps. I was scarred by a DSB class in Christ. It all came back in. Exactly, you know. There's filter sets, there's Gabor filter sets. There's all kinds of ways of representing images, and I'm super simplifying it, right? But like maps, they for certainly they're there, and we know them, we've known them for a while.

4:08in the notion that like we can map color has been instrumental in building like CCDs and CMOS and therefore like digital imaging, exactly. And then also the printers, right? So like the ink in the inkjet cartridges, we know we can combine them to make, you know, millions of colors. There's three channels of color information, roughly, roughly, RGB in the eye, but there's over 300 channels of olfactory information in the nose. And it's still a mystery exactly what they code for, but it's certainly Much higher-dimensional at least in terms of channel count. So when you say channel count in the nose is that? Mapping to physical structures, I'm gonna butcher this but for the eye I'm like is it like rods and cones and stuff like that exactly Yeah, so there's rods and cones and together. There's like four channels there like roughly RGB grayscale again. There's a lot of details there

5:00Those are specific receptors that are encoded in your genes, expressed in cells in your eye that are sensitive to light. So the equivalent for smell are those cells are called olfactory sensory neurons, and those cells actually start in the brain and they poke through your skull and they actually touch the world. It's one of the two parts of your brain that actually leaves the skull.

5:25And so when you smell something, your brain is physically touching another living thing that's like let off a little bit of some of itself for you to smell. So there are millions and millions of olfactory sensory neurons in the part of your nose, the inside that actually does the smelling that's sensitive to smell called the olfactory epithelium. Does the nomenclature here that these are sensory neurons imply that they're...

5:51More fundamental than a rod or a cone which I'm imagining are more like superstructures like bigger things I'm probably gonna mess this up but rods and cones are cell types and So those are names for types of cells and there's many other types of cells in the retina that'll help kind of compute the raw information that these cells get So the equivalent of those like primary sensory cells like without these cells light doesn't turn into awareness, right? So it's the front line, right?

6:18So the front line cells for smell are called OSNs or olfactory sensory neurons. And these cells at the tip, they basically shove part of their cell membranes into a little mucous membrane that then touches the world. And those little tips are just chock full of proteins called olfactory receptors. And those are the things that actually sense the chemical world. And so how many of those olfactory receptor types are there? Over 300.

6:45So that's where that number comes from is each of those receptor types are differently sensitive to the chemical world, right? Just like some radopsins, those are the proteins that actually sense light, some are sensitive more bluey, and some are sensitive more greeny, some are more ready. And there's hundreds times, there's a hundred more different types in olfaction. So what does that say about...

7:11the resolution of the human olfactory system, like, is there like a number of smells that the typical person can smell like? I don't think we have a good estimate for that. There is a very famous paper, which has a trillion, which has been pretty thoroughly debunked. But I don't know if we can really estimate that.

7:34There's a few things though when you talk about sensitivity, right? So think of, so I had laser eye surgery, so I had glasses for most of my life. I couldn't really see that well. So I couldn't resolve really subtle differences, but I wasn't blind. I could perceive all the light. So I didn't, light was getting in, but I just wasn't being focused properly. So with smell, if as long as your airways are working, you can smell and detect.

8:04things. You might not detect subtle differences, but you definitely are sensitive to it. And some people aren't and they have their, you know, airways closed for various reasons, like anatomical or an injury or something like that. But like, you know, this is an amazing experiment by Noam Sobel, who showed that actually people can do scent tracking. So if you put on a blindfold and you get somebody down on the ground and you leave a little trail of like chocolate or cinnamon or whatever.

8:31People if they concentrate can actually centric just like a dog. They're super slow, but they can do it That just goes to show that like there's this there's this I think a myth that like we're not good at smelling We're freaking amazing at smelling like we can smell the equivalent for some molecules of like a one little teardrop in an Olympic-sized swimming pool We are so sensitive to some things I mean that's how like natural gas has a smell It's just the tiniest amount of these molecules called more captains are added to it and we can smell it like super sensitively like parts per billion or trillion is crazy. So I mean we're good like humans are good at smelling. So we're trying to build this map. We have this structure that we know about from biology. Do we know like how do we you know what's next I guess like how do we represent that structure or what's the next step from. I'll walk you through how we thought thought about doing this or how we have done it. Step.

9:31Number one for us was there's a really basic version of this problem Which is like I assume you're only smelling one molecule and you know it's structured like you can draw it on the board like you're in high school chemistry class with like atoms and bonds and all that and You know what it smells like so this particular sets of you know combinations of carbon and nitrogen and oxygen Smells like sweet smells like vanilla smells like phenol like smells like chocolate like those are maybe four descriptors drawn from a set of a hundred that applied to that molecule Well, okay, go get thousands of those pairs and then train a neural network to predict that relationship from structure to odor. That problem called the structure odor relation problem had been unsolved for 100 years. And actually in some cases people thought it was unsolvable. And so what we did is one of our first kind of outputs at Google Brain was we just trained a relatively new at the time.

10:28kind of neural network called a graph neural network, which was specialized for chemistry. It's actually very related to the transformer. It's kind of been subsumed by the transformer in the intervening years. And we were able to predict what things smell like very well. In fact, so well that we said, why don't we set up an odor-turing test? Let's go find molecules that nobody's ever smelled before. Some have never been made before, like nature has not seen them. Let's predict what they smell like ahead of time and keep our prediction secret.

10:58Let's go get those molecules physically, send them to another location, have a great collaborator that I worked with on this. His name is Joel Mainland at Monal. Let's train people to smell and describe smell. It doesn't take a ton of training, maybe like eight hours to do okay at like, hey, this smells grassy or phenolic or vanilla or cucumber, you know, and maybe 50 terms you can be trained on. Let's double blind, have these people smell and describe completely new molecules and then compare.

11:27how well this panel of people does to, first of all, some individual panelists, because one person is always worse than the average of the panel. So let's compare that to our model. So the question is basically, you know, the odor-turing test is, if you want to make your panel better, would you rather add another person or would you rather add the predictions of a model, right? And it turned out our model predictions were better than any one individual panelist on average in the panel.

11:56Meaning that we've passed a noted during test like our model predictions were human quality, which was pretty cool That was the first thing and what we did with the neural network as we cracked it open and we looked at what's called the embedding layer which is a part of the neural network that Basically turns the inputs into a vector that's that is the map right and then that map is what and we kind of can slice up in regions and use for classification so this region of the map is vanilla, this region of the map is redberry, etc. Without that, you actually can't do that classification problem. So that embedding turned out, if you do the engineering right, it just kind of needs to be around 300 dimensions to work really well, which is like suspicious, but just suggestive, right? I can't make any claims. But that embedding had a ton of beautiful structure in it. And that seemed to be a first candidate for a continuous predictive.

12:51map of smell and we called it the principal odor map and that's been foundational to what we've built at the company since then. Before we get to that structure, you mentioned a graph neural net was the fundamental architecture here. What are the nodes and the edges in the graph represent? Yeah, great question. So like if you were doing machine learning on a social network graph, the nodes would be people and the edges would be relationships between people like friendships.

13:19And then it might be one very large graph of like Facebook or Twitter. In our case, every graph is a molecule. And the nodes of the molecule are the atoms. So it might be a carbon. It might be a nitrogen. It might be a sulfur or an oxygen. And the edges in the graph are the bonds. And that might be a single bond, a double bond, a triple bond. And the graphs aren't very big because molecules that have a smell aren't very big. If they were huge, they actually wouldn't make it into the air and fly away.

13:48And if they were also huge, they wouldn't fit inside the binding pockets of the olfactory receptors that we talked about. So they're not too small, they're not too big. They tend to be less than 20 atoms, but more than like three or four. And so those are the graphs, those are the inputs. And the graph neural network is able to basically process that and propagate information and basically gather the macro structures in that molecule. And then eventually you turn that into a fixed-length vector that describes how that molecule actually might smell, and that's what the principal odor map is. And so now the structure that you observed in the embedding space, talk a little bit about that. Sure. So we took that almost 300-dimensional vector, and we took a two-dimensional shadow of it so we could look at it. And we used something called principal components analysis to do that, or PCA.

14:40And what we did is when we plotted this two-dimensional map, we made every dot on that plot be a molecule. And so there were about 5,000 dots on that plot. What we did on that plot is we circled regions of molecules that all had the same smell. So the sweet neighborhood, the cucumber neighborhood, that kind of thing? Exactly. And they didn't have to be neighborhoods, right? Like all the cucumber molecules could be completely spread out, in which case every region might actually be all overlapping with each other.

15:05But what was really beautiful is if we circled the floral region, it was this pretty big part on the left. We can share the images with the listeners if you'd like, but the floral region was pretty big and was on a side of the left. But then if you draw like Jasmine or Rose or Violet, they actually ended up being subregions inside of floral. And we didn't tell it that. We didn't tell it that there was this nested relationship of nature there. And similarly, the region for fermented...

15:34and alcoholic was actually shaped like a bottle during their first model train. We haven't touched it since because it's so funny. But things like fermented and whiny and like rum, those all fermented alcoholic scents were all in the same region as well. And that this pattern continued for basically all of scent. I mean, that's, I thought that was very beautiful. One thing that really struck me with that map, which we're still working out frankly is These sense actually if you if you blur your eyes, it almost looks like how it's a story of how nature makes those sense, right? So fermented sense like from wine or rum, etc Those are made by yeast that are from actually it's a biological process that's producing these molecules, right? So there's a story of how the molecules produced and same same with flowers Those are all genetically much more related to each other than they are to say

16:28other species that produce nuts or bark or whatever, and that they're all clustered together. So in a way, there's this story of actual biological life and scent that seems to be very tightly woven together. And are there confounding examples there? I'm imagining both fermentation and florals are kind of these natural scents, but are there other scents that are...

16:53like completely unnatural, maybe the thing that we put in natural gas to give it a smell like, and what are the shapes of these non-organic smells like differ in some fundamental way? Actually, that was our first line of business that we started was like, can we somehow make new molecules that smell great and that are safe and that we can produce at scale and that are affordable, all that.

17:22And it turns out that's a very interesting business to be in because if you can make some molecules smell great but they're not safe or they're being removed because of new regulatory action. So we need replacements because we want our products to smell great, like our laundry to smell great or home to smell great. So we actually use these models which we've developed over many years to actually find new fragrance molecules that have never existed in nature before and then we make them.

17:49and we could bring them to market, which we're in the process of doing right now. And so when you're doing that, is your objective like, I'm imagining that somewhere in some prior step, you've got a classifier of like, smells great, doesn't smell great. And you're trying to map to that as opposed to, you know, or you could be, well, I want it to be like, floral or wanted to be fermented or wanted to be lemony or something like that. How do you guide the process? It's always really specific, right? So in our industry, it's pretty clear what the molecules are that need to be made. So, hey, we're missing a citrus molecule that is long lasting or we're missing a vanilla note that is optically clear because vanilla is typically brown. And so, you know, people want...

18:45clear fragrances so that they can change the color of the product. And there's many others like this, but you know, we kind of know what we need to make. And so we typically focus the team, because it's ultimately a team that's doing all this with, you know, physical infrastructure of all these models and a lot more chemo informatics on the kind of highest priority items. And of course, we discover some interesting things by accident along the way. And we don't ignore those.

19:12The initial set of molecules were these These are just kind of known like written down. You didn't have to collect anything and like I don't know like put it through like PCR or something like we just we knew what these are We didn't have a we didn't have any labs at that point in time And so the data that we collected partly through we like did some creative data licensing We also found some stuff on the internet. I mean I've been in the world of scent for like 15 years when we started that project. So I was like, okay, I know where I'm going to go to get this stuff. And we were successful. And since then, we've, we've dramatically scaled up our data collection abilities. So, you know, we, we had I think 5,000 molecules in our first data set. We've digitized 6 billion molecules at this point. What does that mean? Walk us through that process. Yeah. So we've enumerated all the molecules that could possibly be made.

20:07So as in the real, actually realizable physical molecules that could possibly have a smell. And we basically ran predictive models on all of them and we've made a huge number of them. And so we've made more new. And so this is that the initial part of that statement is like there's some physics that governs, you know, what the ways that molecules can form and you can like filter that based on some set of criteria the three atoms to 20 atoms and maybe like these bonds work these bonds don't work and then you you get some like starting place this list of potential and there's a manufacturability filter as well so you ultimately end up with this list that you big list a big list that you kind of derive

21:01like from first principles, like the way that these things work together, is that the idea? Yeah, that's right, exactly. So those are real molecules. We can make any of them, and they're more likely than not to be able to have a smell. And then the rest of, like, whether they're useful or interesting or beautiful, we have AI models to predict all that stuff, as well as safety. Safety's really critical. And that's kind of one of our core data sets. The other...

21:30things we've smelled a lot. So we train people to smell. There's many different protocols for how to smell something. Hey, does this smell good or bad or intense or not intense? Is this better than the other one? There's many different ways of doing this. And we've gotten really good at dialing that in. And we have people, basically internationally, that smell. And so I think I want to get the number right. It's probably shifted since I last looked this up a week ago. But we've digitized 5.43 million sniffs.

21:59So that's like the largest olfactory dataset for, you know, the purposes of training AI models, I think ever. And that's, we had to make all of that from scratch, right? Like there's no scale AI or there's no mechanical Turk for smell. Like we've had to make that internally, consume it ourselves and generate huge amounts of olfactory data for olfactory intelligence. And that dataset ultimately looks like a molecule, however you want to represent that and a set of labels.

22:28that the human might label that smell. It can be broader than that, right? So that's a part of our data set is like, we know the molecular structure and then we smell it and label what it smells like. But it also might be like a cucumber you buy from the grocery store that we smell, right? Or we might, and that's the case of, you know, analytical annotations, it might be a product, like a market product. Okay. So a thing and a smell. A thing and a smell. Not necessarily a mom. Exactly. And sometimes we don't just smell it, we put it through analytical machinery, right? So that gets to how you actually

22:58where does the real-world data come from? Like you need to use chemical sensors and so we put huge amounts of data through chemical sensors and then we also can align that with human sensory labels so that we can begin to actually relate sensors to human perception which in like that's kind of core to what we do. And when you're in this part of the process where you're manufacturing these molecules Like is there, are there known toxicity screens that you can? Oh yeah, you have to go through a very rigorous process in Europe, in the US, and worldwide. And there's a binder of tests you have to submit. And they're really thorough and they're the right test, right? So, you know, is this safe on your skin? Is it safe to breathe in? Is it safe for your eyes? Is it safe for fish? Because you might flush some of it down the toilet or in the shower drain right after you wash yourself with a shampoo. The question that I'm curious about is...

23:50Can you derive that from molecular structure or do you have to do it empirically? You can predict it, which is super important for how we're so efficient at what we do. But then you have to test it physically. It's just the law. And it's the right thing to do. Like you just check, right? Do the experiment. And we do over and over for all the products that we're taking through regulatory review. This is maybe a digression from the kind of technical conversation I want to have is like the business side of this thing. Like, are you validating that?

24:20You know, the scent is what the client wants and then you license it to them and they find someone to manufacture it. Are you like making the sense at scale or like where we do most of our business is we actually blend molecules and ingredients that already exist and are already approved. And so like if somebody comes to us and they want to launch an air freshener or a shampoo or a fine fragrance, we need to in order to get that out quickly, we need to be able to blend molecules together that we can get. And we actually stock many of these ingredients, and we've taught olfactory intelligence, which is really a fleet of different models, how to convert a customer's request, like, I want a scent that smells really fresh and clean, and is going to be useful or liked by Gen Z men. That's enough of a specification to kind of begin to make a scent against that.

25:13And then we have master perfumers and perfumers. The two-thirds axe. Just mix them together. I mean, you're not too far from the truth. Like, generally, new scents are like existing scents, right? Because, you know, scent is art and like art evolves. It doesn't like take these crazy jumps, right? So, you can always tell what the lineage of a scent is and like, yeah, axe is one of those, right? Axe is actually really famous.

25:38because it was the first time that fine fragrance perfumes were actually brought to that mass market price point. So some of the scents in Axe Body Spray had never been like accessible to the average person because those... I never associated Axe with fine anything. So yeah, exactly. It's like it's built its own rep over the years, but like the way those scents were actually chosen and designed were super interesting from like a marketing and kind of scent development perspective.

26:05But we've taught OI to collaborate with perfumers and also just work directly with our customers where you basically prompt it. You're like, hey, I'd like to design a scent. Our perfumers use it as inspiration and our customers sometimes design their own scents for themselves. In fact, I had a conversation earlier today where I sent somebody an OI design scent and they loved it. They picked it. They're going to launch it. Just right out of our software system. So you can think of that as a Canva.

26:28Right or or a figma percent where you you and me can can make sense that that wins in the market And so that's what we've put it all together with so the foundation of this is the you know this embedding space that you've framed and I'm envisioning the the task that you described as being akin a little bit to the word to back embedding space math like King minus, I always get the King minus man is equal to Queen minus woman or something in the text. Yeah, exactly. So no, it's really similar. Right. So the core business for us is, you know, you want to launch a brand that has some smell in it, like a shampoo or fine fragrance, you got to get the scent made.

27:14So we actually have a factory where we make it, but the core algorithmic pipeline is we take your text or image or audio description and then we embed it into a perceptual space. And there's lots of steps involved in getting that right and having that be commercially viable and acceptable. But then we then have to decode from that space back to a formula. And that formula is a set of instructions for a formulation team in the factory behind me.

27:42and a big robot that's the size of a school bus that can make a new fragrance every hundred seconds. And that's basically how all this is strung together. And the algorithms are, of course, driven by data. And every single time we make a new scent for a customer, like, yes, they're happy. We're happy because we get paid. They're happy because their brand is working. But we also get data, right? So we're always sniffing those those scents. We're always putting them through our chemical sensors. And so it creates this really nice, like self-perpetuating machine where We found a business where we can serve the world and actually help people be happy and become wealthy and grow their businesses. But also it helps us on our mission, right? So the last thing that we do is not going to be fragrance design, but it is certainly the first. And so it's teaching us and it's also funding us. And I think that's a really powerful... I'm glad we found it, because I think it's really, really powerful for what we're trying to do. But there are a couple of things in there.

28:39that I wanted to dig into, you know, essentially the, the encode and decode parts. Like you mentioned multimodal and that's like super interesting. Like, so I can describe this thing, but also say like, I want, you know, I want this sent to evoke this image, like, or this image to evoke the sent, whatever the directionality is like. And so the first question is like, How did you go from like this kind of molecule map thing to this multimodal embedding space, right? And then on the decode side Okay, you know, you found the neighborhood of some you know some thing if it's if we're still like in the regime where this embedding space is fundamentally embedding like representations of molecules Like how do you get from that from that?

29:35like a single point to, well, that's a mix of these five things. Is that also AI, or is that more deterministic or experience-based? Great questions. There's like a few things buried in there, right? So the first is the map that we discovered. Like we've evolved the system a lot. So what I'm going to say is like the gist, but the implementation and engineering details have kind of diverged from what like the simplest way to describe it. But like here goes that map that roughly 300 dimensional principle order map. The first model that we trained does indeed take single molecules as inputs. But there's no reason that you can't target that embedding with other inputs. There's no reason you can't target it with a... So, what about a chemical sensor? Could you just chop off the head of that neural network and put it on another neural network that took in chemical sensor readings? And the answer is, yeah, you can for sure do that. So, the question...

30:29is not so much like, how do you crunch everything down to the original input space? It's more like, how do you map new inputs into that same space? And we've been able to do that. And then there's a question of how do you deal with mixtures? And like, that is the holy grail. And we've spent a ton of energy and time and science on how to do that. And I'm super proud of what we've done, what we're doing, what's still ahead.

30:52But the special sauce is really there. It's like, how do you reason about how these molecules interact? That's a good question, but I'm not going to get the answer. Not a detailed answer. Yeah, come work at Osmo and push back the frontiers. It is the special sauce. Ultimately, truly, the data that drives it is actually the moat. And so I think we have the largest olfactory data sets in the world ever.

31:18But more importantly, the rate at which we're generating data is just far outstrips anybody, right? Like even the companies have been around for a hundred years, like they say they have data, but they have is a lot of Excel spreadsheets that don't actually kind of add up too much. And so everything that we do from the very beginning has been designed to be collected for analysis and for model training. And that makes all the difference. The data and model training suggested that.

31:44Mapping is indeed a learn thing. It's another model as opposed to, you know. the relationship's not that complex or it's business rules or definitely not business rules. I mean, like, you know, as with any like AIML project, you always start with the dumbest models first and see how far it gets. And then in the most interesting cases, like you're like, that's actually not millennial equals. Yeah, exactly. Exactly. Yeah, just, you know, keep keep pushing the same sense, like still polo blue, it's still ever-crumping pitchfierce.

32:15And like that's not that wrong, right? It's just you'd like to be a little bit more nuanced with it And those are great sensitives to this test of time truly but It's yeah, it's really an exercise in collecting very very large data sets tailored for AI and then what you do algorithmically on top of that like when we started the company like whatever almost four years ago Knowing how to build specific kinds of models on this data modality was super important And that's actually like been an accelerant for us. But really like a lot, if you just have great data, a lot of stuff kind of falls out from it. But you have to be careful with every step of the way and have great, great people who are thinking about every step too. And does graph models still figure?

33:06significantly into what you're doing or if you have a volatile transformers or... In the right applications, I mean, we're really not dogmatic about the modeling approach that we take. We're really dogmatic about data size and data quality. Does the implication then that you spin up new models for individual projects as opposed to what I envisioned was that like you had like the olfactory foundation model and that thing changes infrequently and...

33:31You just use it for a new task. So when we say olfactory intelligence, we kind of mean like the suite or the fleet of predictive models on all aspects of smell. And there's dozens. There's dozens of models. And if you go look at like Neolabs or life sciences companies, some of them are pretty upfront that like their core model or foundation model is actually a fleet of models. And that's how self-driving cars work too. Self-driving cars are driven by like a fleet of models that all connect together along a spine.

33:58to coordinate sensing with action with planning and all that stuff. So we're, I think we're close. The way we think about it is closer to autonomous vehicles, which is actually where my CTO is kind of background is. So Rich ran the data architecture for NVIDIA's autonomous vehicles program. We met at Twitter when we were working there and then he went to Spotify and did Brexit. But, you know, that in of itself is interesting and opens up this question, which is like NAV, there's this tension that I've explored quite a bit through interviews between models that, you know, are somehow faithful to the way we understand at least the physical world or the physical process or like classical control in the case of AV versus end-to-end. I don't care about any of that stuff. I'm not going to like human tinker or subsystems, whatever. I'm just going to end train on data. It sounds like you're more in the

34:52like we've got this physical understanding of olfactory systems or some process for developing these sense. And we're going to have subcomponents or submodels. Yeah, I would say one part of that's true and one's not. So the one that is not true is we don't necessarily strive for physical understanding. We strive for predictive accuracy and helping our customers and building, being able to predict the right thing to keep the organization going.

35:22And sometimes we use physics. That's just a modeling choice. It's like using a physics-based model. And I would say we actually cannot do one full unified model because there's regulatory work that has to be done. So you kind of need one prediction for, is this safe in this specific way? So you got to have, at least it could be one model with 12 heads.

35:46But you do have to have those specific outputs and there are specific data sets that need to be used to like train those and they effectively become Independent, you know models in that regime. Yeah, got it. So the the sub models are like Is it fair to say the core is present predicting a smell but then you know, you've got these other heads or attributes that you're also trying to predict which are toxicity, maybe manufacturer ability, maybe cost to manufacture, maybe whatever. Regulatory safety, you know, all that stuff. And like you can't really skip any steps, right? They're all required. But yeah, the core is like, what does it smell like? And does it smell good? And does it smell strong enough? Like those are kind of the core things that you need. And then everything else you can figure out. Does it smell good? Is that a derived

36:42property of what does it smell like, or do you also predict that independent of what it smells like? It's a derived property of the scent tupled with the target consumer. So, you know, an example is like, you could either choose Parmesan cheese or kimchi or strawberry. Like, depending on who you show it to, they may or may not like it. And in the case of strawberry, actually, the kind of strawberries that people think of or want in Japan, actually are pretty different from the ones that we think of or want in the US. And so, well, those are the gift ones, but just the flavor of them, it's a different sweetness profile. It's almost borderline different fruit in terms of how you construct it in a product. But what you like is heavily, heavily driven by what you've been exposed to before and what did you feel like when you got exposed to it?

37:39And if you're in a Korean household being exposed to kimchi, you do that under warm family conditions. And if you're exposed to kimchi, it could be literally whatever. If you're a non-Korean person who's never experienced before, you're like, what did I just open? I happen to love kimchi, but it wasn't acquired taste.

38:01And similarly, like, from my culture, there's all kinds of, like, you know, horseradish and gefilte fish and noodle kugel and all this stuff. So, like, I love those things, but that's, like, my people's food. Like, that's what we eat, and not everybody likes it, but I'm into it. Which raises the question about taste, like, which is kind of, I guess, you know, it's more adjacent to smell than it is to... Physically, it's pretty close, yeah. And so does that...

38:29you know, is that something that you dabble in? Is it far enough away that you don't, you know, think about it? We think about it a lot. We don't work in taste today because we're really, really focused on our kind of first vertical and like we got to focus. So there's kind of three aspects to this. So there's smell, which is for sure just like what comes in your nose, but also like you can smell things that go the other way. It's called retronasal affection. So when you're eating something, you're actually creating this chimney effect that like kind of has scent basically vent the reverse way. And so you, this is how flavor is produced, right? So if you ever eat a jellybean and hold your nose, you actually can't, it just tastes sweet. You can't tell if it's like lemon or lime or grapefruit or whatever. So flavor is 90% smell, 10% taste. And taste is just what happens on your tongue, like sweet, savory, sour, salty, umami, all that. And a bitter,

39:25But it's in terms of dimensionality and richness. Again, like I'm sure the taste neuroscientists would kill me for saying this, but like it's just not as complex. Like there's just, there's less diverse. They've only got six, we've got three. Sorry, you can have a taste person on your podcast next and they can defend themselves, but like they just, there's fewer channels of information. And for sure, the experience of flavor is like destroyed if you cannot smell. Just ask people who lost their sense of smell in COVID or whoever tried the experiment of just like.

39:53holding your nose when you, you know, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, you, And we've been on our journey super pragmatic about building scientific capabilities, which is the first two years of the company, and then finding a great business to go put this to use in, which is the fragrance industry. But we're going to go further than that. And in fact, we're talking with a number of organizations. I think the thing that's needed now for not just design absent, but the detection of scent is exactly what you're talking about from before, which is we have to build a foundation model.

40:47And the thing that's been missing and that we've been building is you have to collect a whole bunch of data, right? And like, you know, for instance, if you wanted to do the very, very noble work of smelling someone with cancer early, detecting their disease early, or if you wanted to detect early infection with malaria or another infectious disease that claims the lives of mostly kids, It's super interesting thinking about those as data collection problems. So here's the deal. You can go directly after that and collect sent data from people with or without those conditions and try to build models. You're never going to get enough people to build a great model. And it's just hard to go get that much data. Meaning because...

41:39There's not enough signal in the correlation between scent and disease or some other factor, are you speaking? Just low numbers. It's just from a pure statistics and machine learning problem. I think basically there's not an obvious, obvious, obvious signal that says this person has a disease and then doesn't. There are subtle changes across many hundreds or potentially thousands of molecular signals. We just don't know. But we know dogs can do it.

42:05like animals can actually detect these patterns. So there's something... Right, that's what I was thinking of when you raised the example. It's for sure there. It's for sure there. But we can get computers to do it. We just need to go get a ton of data, right? Like we need to like band together and build a huge effort where we just like, let's go sniff 10,000 people. I don't care if they're healthy or sick. I don't care how old they are. I don't care. I mean, let's record all that information, of course. But like, let's just go get a ton of data. Let's go to the grocery store and get cucumbers and bananas, flowers and like steaks and whatever.

42:34And let's just go smell everything and then build a huge olfactory data set of what the world smells like. And if we can do that, then, then... It sounds a lot harder than scraping Reddit at one more time. It is a whole lot harder than scraping Reddit, but it's worth it because literally nobody's gonna do it unless we pull ourselves together. So, like, it's like worth doing, right? Like, you know, this is the thing that maybe isn't a sign, but like...

43:01I have a lot of friends in the world of AI and ML and you know I came in through this as a biologist and as a biologist like you go do experiments and it's like hard and like you got to go to the bench and get your own data and you know it's gritty and in the world of AI like you kind of just assume the data is out there and if the data is not out there you kind of assume you can give somebody a credit card to create the data and like I think that's amazing but like the amount of infrastructure that you have access to is just astoundingly huge. And you just have to understand that if you want to continue to expand the realm of what AI can do and compute can do, it might get hard. You might reach a point where there's not a vendor that can handle everything for you and shine your shoes or whatever. You might have to go to the work yourself. And so we're very much in that regime. And I love it. It's so hard. It's beyond brutally hard. But I love it because...

43:57nobody else is doing it right and like that's very much my preference for how I want to spend my life is I want to do weird things that nobody else is doing that matter right and so that's that's kind of my selection criteria like there's plenty of people that can go build you know AI voice agents for customer service I think we're gonna need that to like make commerce better like I'm not the guy to do that like I want to do this weird thing right I want to give computer assistance of smell and so you know I think where we're going is like We are collecting huge amounts of data, but importantly like we have this platform that allows us to store all this data organize it like massive Efficiencies of scale digitally and then increasingly physically we can make a huge number of unique sense like we have all this infrastructure that we've built painstakingly over over a few years and Now like let's go. Let's go get all the smell data, right? Like let's build this chemical Like slice of reality into AI which looks building a trual faction

44:50foundation model, but the data comes first, right? Like we need to go get all the data. And then, like, you know, whenever these models come out, like Opus 47 or when Mythos comes out or Codex55, they talk about the model card in terms of all these benchmarks. And so what I imagine for the future is that we'll collect all this foundational data and malaria detection will be a benchmark, all right? And cancer detection will be a benchmark.

45:20And, you know, building a great market product for a new shampoo launch in Malaysia will be a benchmark. But, or even the idea that the generic, this is where I thought you were going, that the generic foundation model that you are using by generic, I mean, kind of general purpose, or the better term is not just text.

45:43you know multi kind of the common modalities but is now has there's some kind of benchmark or model card statement around you know smell and taste and yeah it's like let's add these weird new benchmarks which is like how well can we predict you know what something smells like or how some smell that we detect in the world which is just a combination of molecules is predictive of something that we that we care about you know like one other way I like to think about this work is like Foundation models for text and for images, they are approaching in some cases exceeding human intelligence, certainly mine and certain regimes where I've been pushing it, I'm like, holy crap, this is smarter than me now. It's human intelligence. And that's because it's trained on human intellectual output. But 99% of species on this planet can only speak with chemistry.

46:35thinking of bacteria and fungi and plants and insects, like they just only can talk with molecules. And I think that it's really worth adding other kind of alien forms of intelligence to our AI models. And the way to do that is to train it on the intellectual output of those other intellects, which is that's chemistry. That's the sense it's in the air. They're produced by living things for reasons to talk to each other. And Terry Tao, who is this famous mathematician, He has this one paragraph in a paper he just wrote where he's reflecting on these new forms of AI and the kind of weird mathematics that they can do. And he says, I think increasingly we need to have a Copernican view of intelligence, meaning for a long time, in astronomy, we had the Earth at the center, but then we had to dislodge it to actually face facts. And Earth's a great place to be, but it's not the center of the universe. And similarly, we've placed human intelligence at the center.

47:35of the debate about artificial intelligence. And like, pretty cool to be human. It's awesome. I love it. But like, why should we be at the center? Like, what other forms of intellect are there? And so, when I read what he wrote, I've kind of instantly, like it instantly resonated with me and I've very much taken on this view that like, we should have a Copernican view of intelligence. There's so many forms of it. And there's so much richness out there of things that are intelligence, some of which we might not even recognize as such.

48:03And I think if we're building AI for our species and for the planet going forward, we should have those forms of intelligence built in. Thinking about all the progress that we've been making in broadly physical AI, robots, autonomous vehicles, all that, and thinking as well about how good our sensors are for this approximation of visual stimuli.

48:33Like do you have a sense for I'm imagining you do like where are we with like the? You know olfactory CMOS or whatever that you know knows on a chip I would say the devices that we use to detect chemistry are truly phenomenal But they're they've been very specialized to live in a laboratory So think about computation in like the 1970s big mainframes right stuck it in a back office or like a data center, right? So we're still in the data center era of chemical sensing and of olfactory intelligence So like our scent printer is the size of a school bus. It's just not we're not gonna that's not gonna leave that's bolted into the concrete, right? Yeah, but eventually all the we just need those wheels on the wheel I promise you But but all this should fit in our phone

49:26right over time. And that's the story of technology is something that works. It's valuable. And now we just like work really hard to make it small. So that will come. The cancer example is a good one. What are other examples of like if we had the auxiliary olfactory apparatus chip on the phone, like what would it let us do that our nose is done? Super excited by is just like the creativity that I think would come if we opened that up to everybody.

49:53So, like, how could we have predicted all the apps on the app store, all the uses of camera phones and all that stuff? Like, it's just incredible. I think that there's definitely things we can contemplate. Everything that a dog's nose does, right? I'm thinking of the app that you download that tells you who actually dealt it. Yeah, exactly. All right. Well, the old factory fingerprint is definitely same today.

50:18data to prove it. So they'll definitely be fart apps for sure in the future. I just like it's human nature. I'm told you I worked at Twitter and we clean up like I just I know human nature. It's just like, you know, we've got a fascination with many things, including the scatological, but I think that anything that a dog's nose can do and there's like 80 things more or less that you can train a dog to do like detects tracking for example on your phone. Exactly. That's an amazing example. Spoiled produce, spoiled meat. These are things you want in your fridge, right? You want to know, can I eat this? Should I eat this? Is it done? Is it cooked? That's useful for robot kitchens. There's so many things that the human noses and dogs noses can do. None of the examples alone I think are so far that we've seen.

51:08are huge businesses or necessarily accessible businesses. Smelling cancer is important, but the business there is really rough. Medical diagnostics are not a great, unfortunately, not a great business to be in. And it's just figuring out how to actually not only build it, but then make itself sustainable. That's one of the harder questions that I try to wrestle with, and I don't have the answer, frankly. Do you feel like the difficulty of that question is, and me clearly like it's suppressing the development of the technology, but do you think that it's like if the business model was there, we could easily figure out the nose on a chip or is it just really, really hard and therefore the bar, the economic bar is really, really high? I think it's a blend. So if the business model was figured out, I think we would have been sprinting at this for the last like four years exclusively as opposed to building a really great business.

52:06in the fragrance industry, which seems unrelated, but when you really get down to the science and technology, it's super related. But we have devices that you can bring people to or that you can wheel around that can do this, right? It's just a matter of how do you throw an appropriate amount of resources behind, like scaling that, miniaturizing that. Very possible, right? I can easily, easily envision a device that's the size of like What do we call this like I don't know like an 80 cell phone so bigger than the cell phone But like kind of a bigger box, right? I can imagine a box that's like, you know about that big like I don't know a small baby shoe shoebox That can smell as well as a human nose. In fact, we're pretty close ours is about the size of two shoeboxes We have a deployable sensor that's as good as our nose is you and me And it's the size of two shoeboxes it works

53:03So getting that smaller, very straightforward engineering, not easy, but straightforward. Going smaller than that and putting it into something the size of this phone and then putting it into a chip the size of this phone, that's going to require some creativity and some work. And actually, I couldn't tell you exactly how that's going to happen. I can tell you that we have proof. It lives between our eyes and our nose, right? Like we're walking proof. But yeah, there's a lot, there's work to be done to actually make that affordable and pervasive. Yeah.

53:34Yeah, something you said maybe think about you know, is there some inherent limit in the application of MLAI or the way you've done it and That like what's important isn't really mapping to like, you know single label sense like, you know cucumber cinnamon whatever but like Mapping to emotion, which is a much richer thing, but certainly if you're like You must be already doing that is what people are trying to get at. Yeah, we think about that a lot. Yeah, so I I think There's there's first of all, there's more labels than just cinnamon and cucumber I don't I don't mean like we didn't name enough. I mean like there's other ways of labeling like these two are really similar or dissimilar and here's a scale so

54:24The richness of sensory information, once you start to not use human language and use like numeric scales, which you can do, just take some fine-tuning, you actually, you learn a lot about the chemical world when you start to label things more carefully that way. I mean, in some sense, that's the original observation from the embedding work at Google- Is that proximity actually means perceptual practice? So things that are nearby on the map smell similar, right? Yeah, exactly.

54:53And so then that like, I don't know how you would do this, but I'm envisioning like jointly embedding emotional somethings, emotional valences with, you know, olfactory information. And I think that's very, very much possible. I think the challenge, so there's a trend now called Neurosense, which is largely bullshit if I may be frank.

55:22And, you know, people use EEGs, like I used to use EEGs like in my training. And also I know people that run EEG companies and they're very upfront. EEGs are useful for telling if you're asleep or if you're having a seizure. And it's very valuable to like quantify sleep and quantify seizures. But like, it's not going to tell you if you're feeling, you know, up or low, right? It's just not, can't do that. There's no signal there. And so there's a lot of crap out there. And so we've...

55:47not weighted into that space yet, because I just, I have to do it right. Like I can't live with myself if I, if we don't do this right. But there's so many examples of this being true, of sense really being able to unlock emotions in one way or the other. And we will get there. And I think we will do it right. But we're going to have to be very careful about how we quantify emotion, which is its own thorny problem. But this is something I used to, I used to work on too. I love the topic.

56:17I'm imagining the real estate agent going and spraying fresh baked chocolate chip cookie in the open house. They already do that, right? I know they bake them. Is there an actual candle? Oh, yeah, there's candles for sure. Yeah. A new car smell is like invented, right? It's a thing.

56:37But a lot of this is also association, right? So same with the kimchi example, it's like, well, that might be good, or it might be the hearth, right? It's like, well, have you ever been in a home where a fire was lit? Like, does it smell like a burning building? Or does it smell like comfort and like, you know, marshmallows and hot chocolate, right?

56:54It just depends on what you're exposed to. But for sure, like the evidence I think is pretty clear that there are some sense that do things that are positive to your mood and it's almost physiological and can increase your focus or can increase your, you know, awareness or reduce your anxiety. Like it's pretty clear to me, but I think we need to be really... Like aromatherapy? Yeah, I think aromatherapy has elements of really deep truth in it.

57:19Same with Ayurveda. I think that these are really old traditions. They're first of all, they're really appealing and they're beautiful to experience. It just smells nice. But in kind of reading the research, there's nothing that's super clear-cut, but I just am of the conviction that there's some kernel of deep truth there in the same way that acupuncture now has this Western equivalent called dry needling. It's actually therapeutic. Acupuncture got there way earlier.

57:45they just talk about it differently. It's the same stuff, and maybe some ways there's still things that haven't even been appreciated in the Western adaptations of it. But I think that that correspondence will also happen for Ayurveda and for aromatherapy, and then also probably for the aromatic aspects of herbal Chinese medicine as well. Plants are...

58:09medicine and poison factories, they make all the molecules that do good and bad things and all of our drugs, like many of our drugs, has some natural origin to them or they were inspired by a natural origin. So like there's got to be something, I just believe it, I think there's got to be something there where scent has real like powerful, harnessable impact to uplift our mood and to make us feel better or perform better or whatever.

58:36our desire might be that's actually, you know, realizable. And we just have to be, we have to be serious about it. And we have to be thorough. That's all. Well, Alex, it's been great catching up with you and getting the download on olfactory intelligence. Super fun to talk about it. It's obviously wide ranging, but it's like we're pushing back the frontiers. And so like all the things you normally take for granted, like we have to think all the way through.

58:59So I'm just thrilled to talk about this stuff all the time because I live it and breathe it, but it's fun to share with you, Sam. Thanks. Thanks for having me on.