The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) - Why the Next AI Breakthrough May Come from Physics with Max Welling - _774

0:00Sam CharringtonMax, it's so great to be on the line with you again. It's been a while. It's great to be back, Sam. I'm looking forward to our discussion. I am as well. Our audience can look up our conversations from, I think, 2019 and 2020, where we covered what you were working on at the time, and I think still echoes into your work today. Geometric neural networks, Gage, Equivariance, neural networks, and the like.

0:30Max WellingBut I'd love to have you kind of catch us up on what you've been up to since. It's been quite a while. Yeah, actually, the the Aquavarian's theme has definitely continued. So in fact, I found out that Aquavarian's was used very fruitfully in chemistry and material science. So in chemistry and material science, people train neural network models to predict the forces on atoms because people want to evolve atoms forward in time in order to compute their properties, which is called molecular dynamics. Typically, you need to use quantum mechanics to compute these forces because a large contribution comes from the electrons and electrons are very light. You need to treat them with quantum mechanics, but if you get

1:22Max Welling10 electrons or more because it becomes completely unfeasible to solve the so-called Schrodinger equation. So people have come up with approximations like density functional theory known as DFT. And the inventors of that got the Nobel Prize for that. But what now people do is they train surrogates. So they provide data using this expensive approximation to quantum mechanics.

1:49Max WellingAnd then they use neural networks to shortcut the computation, so to predict the outcome of that computation, but at a much more accelerated pace. So in other words, three orders or four orders of magnitude, acceleration, more efficiency relative to these quantum mechanical approximations. And in those models, because the world is three-dimensional symmetric, so if I rotate a molecule, all the forces will rotate with it, and so we could now put the same ideas that we use for images, we could put them in these molecules, these models that predict the forces, and we could use aquivariants. And so that's why I kind of, also because my background is in science, I did my PhD in theoretical physics, I thought, okay, this is a perfect unification of my old sort of passion and my new passion. I can put it together. And that's when I started to be interested in AI for science.

2:47Max WellingSo that leads it pretty directly to the founding of CUSP AI. Actually, first I spent two years at Microsoft Research as a VP because they were building their AFR Science Lab in Amsterdam, and so I helped that along. But after two years, I wanted to start a startup. I already did a startup a while ago, but this was actually the startup I got acquired by Qualcomm, and then I spent some time at Qualcomm.

3:14Max WellingAnd I really like startups, the dynamical environment and the impact you can make. And I wanted to do it the Silicon Valley way together with my co-founder, Chad Edwards. And so we started CosPI in 2024. Talk a little bit about the progress that you've made since then. What is the kind of shape of the company today? There's been a huge ride actually. It's a roller coaster. So we started, I think about two years ago.

3:42Max WellingSo May, May, spring 24. Yeah, we started with a good initial investment of about 30 million from which we could hire an excellent team. So the team has grown to about 50 people right now across different geographies. So there's a half quarter, both in Amsterdam and in Cambridge.

4:09Sam CharringtonActually, the Hackport are officially in Cambridge, but the two initial labs were Amsterdam and Cambridge because Chad is from Cambridge, I'm from Amsterdam. We now also have labs in London and Berlin, and we're also expanding into Asia and North America. Got it. And no surprise, your list of advisors is a bit of a who's who with Jeff Hinton and Jan Lacoon at the top of the list. Yes. The advisors are actually fantastic.

4:39Max WellingSo we have Jeff Hinton and Jan LeCun. We added to that also Martin Van Dambrink and Lord Brown. So Lord Brown is the former CEO of BP and Martin Van Dambrink is the former president and CTO of ASML. They're both retired and they like to spend their time with new startups and help them along.

5:06Sam CharringtonAnd then there's Verity Harding. She's working for the UK government and also DeepMind. And then, or maybe Formity at DeepMind. And then Kristin Persson, who has sort of initiated the materials project. And she's also advising on that side. Got it. So let's dig into the technology and the approach that you're taking to kind of apply your, this original.

5:35Max Wellingset of work that you developed to materials. How did you get started with that effort? So I think the opportunity is, to me, there's a deep fascination with the fact that there is a sheer infinite amount of possibilities in which you can put together atoms. And the universe has only figured out so many of them because, you know, they form naturally, I guess, in the universe. But there's many more that you can design yourself with all sorts of exotic properties.

6:05Max WellingWhen we started this company, both Chad and I were kind of concerned about the climate. We still are. And so we felt there's a strong need to accelerate the energy transition to more sustainable energy sources, as well as trying to take the carbon dioxide that's in the atmosphere out. So not many people know that...

6:30Max WellingBy the time it's 2015, of course, we really like to be completely carbon neutral. But after that, there is still 50 to 100 years where we have to take out every year about half of what we currently put in. So that's 20 gigaton a year. That's an enormous amount. That's the kind of, I think the size of the lake of Geneva filled with sort of liquid carbon dioxide, a gigantic amount. And we don't have the technology for that because it's actually very hard to take it out.

7:00Max Wellingbecause it's so dilute in the atmosphere. It's very expensive too. And the two factors which are more expensive are energy and the material, the sorbit material that you use in order to take the carbon dioxide out of the atmosphere. And so we've been working, the first project we've been doing was improving the materials that take out this carbon dioxide from the atmosphere. And these are called metal organic frameworks.

7:27Max WellingActually, this is the material that this year won the Nobel Prize in Chemistry. We built a platform, and I can't go into much more detail, but we built a platform that designs these material with the use of AI. You can think of it as a search engine, but it's not searching over existing documents. It searches over known and unknown materials. It actually completely designs them from scratch if there's no suitable ones already available.

7:55Max WellingAnd there's many components. It's agentics. There is an agent sitting at the core of it who orchestrates a long computation. And it searches through existing databases. It generates entirely new molecules. And it also evaluates all these molecules with all sorts of tools. And in this generation and evaluation, you know, equivariance and all sorts of methods that we have developed over the years in my academic lab play a very important role.

8:25Max WellingTo be clear, you mentioned a molecule that won the Nobel Prize. Were you involved in the discovery of that particular molecule? No. No, I wish. Those were chemists who discovered it. I think Professor Kitagawa, Professor Yagi, and Professor Robson. I think those are the three. I'm not quite sure of the last one.

8:51Max WellingBut this is a class of molecules where there is a metal complex, a metal node with all sorts of atoms along it at the vertices of a graph. And then there is so-called linkers, which are also organic complexes, which are connecting these vertices on the graph. And they're extremely porous. So they have a very large holes in the middle with an enormous surface area.

9:21Max WellingIf you blow air atmosphere through it, the molecules in the air, which is water, nitrogen and carbon dioxide, the carbon dioxide is only a small fraction of that, they tend to stick to the sides. What you need to design a molecule where actually only the carbon dioxide sticks inside these pores and the rest goes through, so that's a design question.

9:51Max WellingWhen it's full, you want to shake it or heat it to get it out so that you can actually reuse that particular material. What we added to this is a way to fine tune or design a molecule for a particular purpose. So people have actually made maybe around 100,000 of these molecules in labs and verify their structure.

10:19Max WellingAnd so what we can do is we can now basically come up with an entirely new molecule for a very specific task and then make it in the lab and then use that for that particular task. I should say CUSP is not only working on mobs. In fact, this was just the first set of molecules that we worked with. We have hence expanded to semiconductors. We're doing a lot of work on perovskites, which is materials for solar.

10:48Max Wellingpanels, improved solar panels. We look at semiconductors for new chip materials. We look at battery materials, fuel cells. We also look at removing PFAS from water and the current set of molecules we use for that is metal organic frameworks. Do you partner with...

11:15Sam Charringtonother companies that have an interest in these particular molecules or are you out exploring and then if you find something you will find partners and maybe license to them, what's the thinking around the business model? So we like to work with partners because there's a very broad class of materials and every class has its own super experts that focus on those particular areas and they're either in academia or they're in companies.

11:42Max WellingAnd of course, we also like to partner on the actual synthesis of these materials. So we partner with academic labs, but also with the labs inside of these companies. And so we build an ecosystem or a network where our engine can actually help in all of these different material classes, design the materials, and then we work with those companies to actually make it. So that's a partnership model. But there's also internal.

12:12Max Wellingprojects that we run. So for instance, the project on metal organic frameworks for carbon capture, we ran self-funded internally. And then we also have another project now in the semiconductor side where we run it. And so if we discover something fantastic with self-funded, so we got the IP and then we can find customers for that IP. I think the most important thing is that we discover something that gives people

12:39Sam Charringtoncertain confidence that we can do this, we actually really own this process and we know how to do it. And so then the cost that we've worked with customers to design materials for their specific needs. You talked a little bit about kind of the generative or a genetic nature of the scanning scientific literature and using that to identify potential molecules, but we've also talked about some of the geometric implications of your work.

13:08Sam Charringtonyou alluded to potentially the use of simulation. Can you talk a little bit more about the end-to-end process of identifying these molecules? Yeah, happy to. I guess there's a sequence of things that happens. The first thing is, like in a search engine, you actually type a request. So you basically say, I want a material.

13:32Max Wellingthese are all the properties that it should have, and these are the things it should not have, these are the properties it should not have. You give this as a query, and then you could also tell if you have prior knowledge about how you want this particular search to happen, you could tell the system, maybe use these tools and maybe sequence it in this way so you can also give it some instructions. Then it goes through a process of steps. The first step is it will look through its database. So we have a very large database of materials which we have all ingested into this database. Lots of it is sort of exclusive licenses from the big publishing houses. And then they will start to look through all of this literature whether something exists out there that has these properties or which is close to having these properties. And so if it doesn't then it will have to go into a new phase.

14:28Max WellingSo you can hold a conversation with this agent and talk about it. So it's already quite useful. But typically then the next step is that you go to a generative model. So in this case, it's the same generative model that generates images or video. But in this case, it will generate molecules for you. And so you tell it, I want molecules with these very specific properties, the conditioning statement it's called. And then it will start to generate these molecules.

14:57Max Wellingoften hundreds of thousands of them, because it's quite cheap in the computer. And then comes the next phase, which is out of these generated molecules, we now have to sieve out the ones which look very promising. And this can be a very expensive step. In some sense, you're built a multi-scale digital twin of the process that you really want these molecules to operate in.

15:25Max WellingAnd so the first step is basically you relax the molecule to its ground state to make sure that it's, you know, it's the best energy state. Then you do a bunch of checks, like is it charged? You know, if I shake it, will it fall apart? How big are the pores inside if that's important? You know, all sorts of things that are easy to compute fast to throw away, you know, a whole bunch of things that do not look promising. So filtering it down.

15:55Max WellingIt's definitely a filtering step, yes. And then go to the next step. So we have a pipeline that fine-tunes or distills machine learning force fields for that particular material. So this is a process by which we take all the data there is about that material. We have a foundation model that's trained on a much wider range of data. And then we distill this force field in into this it's a very efficient force field for this particular class of problem and we use that in an MD loop typically the molecular dynamics loop to simulate the molecule as it wiggles around and moves around from which you can often compute very key properties of that particular molecule those properties then often go into a partial differential equation at a higher scale and or in a process

16:48Max Wellingthat actually models the device in which you want this material to operate. And so that's again more expensive. And so again, you want to do this with fewer and fewer candidate materials. And then at the very end, you go to, you know, to an experiment, right? You go, so now I'll actually do the experiment. Now that's more expensive and even slower. And that's, you should do this only with order 10 materials at most in the old way. And so then and then you get a candidate. The current, I would say, revolution that's happening in this space is self-driving labs, where the amount of experiments you can do is much, much faster. So you could do maybe 100 experiments a day. And then the game is more like the agent figures out what the settings of the experiment should be. The experiments are done, the data comes back in, and then you have the data from the experiment.

17:44Sam Charringtonand the data from your computations, you combine them to set the next stage for the experiments. And so the experimental loop is much, much faster for those. And that's a very interesting development that we are now integrating our platform with. Got it. And so how many molecule classes and individual molecules have you kind of gone all the way through this cycle with?

18:13Max WellingYeah, we are engaged in a few of those but you know the question is a little bit, what do you mean by all the way? The different materials are at different stages of maturity. So one of them we went all the way to you know actually doing the lab experiments. Another way on semiconductors is on its way and probably in a few months we'll start doing the experiments. I meant all the way to lab experiments and I was curious if...

18:42Sam CharringtonYou have enough data to say that your hit rate with this process is higher versus lower, or if there's anything you can say qualitatively or quantitatively about the candidates that you produce relative to the traditional approach. Yeah, so I definitely think that there's definite evidence that these things are much more efficient. So some of our scientists, they have said things like,

19:10Max WellingWe can do now in a few days what took a PhD before, but that's more in the digital domain. These are actually scientists that are doing more simulation based work. We have two projects going which have experimental pieces to it, but we have a contract with a big national lab in Asia, which I cannot quite say the details of yet, where we have many more of these experiments planned out. So basically, the setup is that we bring some customers and we can do experiments in that lab with their lab scientists and they can also bring their partners and they can use our platform and then we can collect data that way. And then there is one other one that we're currently doing in Amsterdam on Perovskites, which is running right now.

20:07Max WellingSo that's also a lab engagement. There is one in... Brofskites is one? Brofskites is a material class. That's a semiconductor crystal structure that you would put on silicon, typically, on top of the normal solar cells. And that can help you filter out or basically convert a much larger amount of the energy in visible light to energy.

20:34Sam CharringtonSo we also have something with catalysis with DTU, which is the Danish Technical University. So that's on currently running on catalysis. And we have a project that's about to start them off. So there's quite a few engagement with labs that are either running or are starting to run, but they haven't finished completely. And does your lab or does CUSP publish?

21:03Max WellingAre you still active in kind of academic publishing around materials science now? So myself, I have one day at the university and of course I still work in that day with students and there we publish everything. I'm very interested in all sorts of things we have to do with AI for science, so definitely yes. But also COSP actually publishes. So we have interns that work with our scientists where we publish the results.

21:31Max WellingSo we, for instance, recently had results on machine learning forest fields with uncertainty prediction, property predictors. And the most important thing I think that we've recently released open source and published a blog post about is our new molecular dynamics framework called COPS, confusingly. So that's universal particle simulation simulator.

21:59Max WellingAnd so the reason why we did this is actually together with NVIDIA is that now with this new development of these machine learning force fields that I talked about, these neural networks that replace quantum mechanical calculations for the forces, you cannot run them very efficiently in the current sort of MD simulators, the simulators that evolve a material or molecule forward in time because you need to You need to run these neural networks on GPUs, and typically these simulators, they don't run on GPUs, they're more on CPUs. And you want to run things in parallel. And so what was built by our team is a framework where you can compile these force fields into JAX, which is Python-based, and then it will actually very efficiently run these MD simulators.

22:59Sam CharringtonAnd you can run them in parallel as well, so that you use your GPUs. The utilization of your GPUs is high. And so we released this open source a few days ago, actually, where we could go at iClear. And yeah, we get a lot of excited responses to that. Are the representations of the molecules, like are there standard formats for representing these things that someone working in the space would already have?

23:27Max Wellingyour machinery just works on those existing ones? The representations, you can think of that as maybe a foundation model for chemistry. So what you do is, and this is also work that we did together with Meta. So what you do in that case is you train yourself one of these force fields. So you take a molecular structure. So that is basically the position of the atoms and their class, like if it's oxygen or hydrogen. And of course, you want this to be a covariant because rotations and translations don't matter. And then you map this into a latent space where they get represented by some code that is meaningful. And if you do a machine learning force field, you will then actually predict the forces and the energies from that. But you can stop at that intermediate level, and then you have a representation.

24:23Max Wellingfor that particular material. And you can train this on a very wide range of materials and chemistry. And from that representation, you can build property predictors. You can use it to condition your generative models. There's also sort of uses for that foundation model. But there's also the starting point for distilling, let's say, models for specific material classes. And to be clear.

24:49Max WellingAre these foundation models trained per molecule or per class or are they very broad? Yes, that's a great question. A foundation model, almost by definition, is trained on a very broad class of materials. The best dataset for that is the materials project dataset and OMOL from Meta.

25:15Max Wellinggenerated a very large number of DFT calculations, these quantum mechanical calculations to create that data set. Those calculations are being used to train these force fields that I talked about, or those representations, those foundation models. Then if you say, but I'm actually interested in this particular material, what you can then do is start from that very broad representation, this foundation model, and then fine-tune that model for that particular material class.

25:44Sam CharringtonThat way it is specialized for the material class, but that way it's also very fast because you need these things to do very fast because in a molecular dynamic simulation, you have to call them many, many times, right? Because one little step in an MD simulator is a femtosecond. It's a tiny step. And so you have to call them many times to make any progress. And you mentioned distillation earlier. Is that where distillation comes in? You're trying to get to a smaller model that's more focused on the molecules that you care about.

26:14Sam CharringtonAbsolutely. That's what distillation is called. You distill the big model into a small model. In this case, the use of fine-tuning, how analogous is it to the process of fine-tuning in LLM, reinforcement fine-tuning with traces, language-based traces? What is your dataset that you're fine-tuning and your process that you're fine-tuning with here?

26:43Max Wellingquite related to an LLM. In fact, the models we use are very related to an LLM. So you can basically think of the sequence, the information that you have. You can sequentialize it. And then from there, you can actually map it into this latent space. So there's either, and typically you can do like a graph neural network, which actually looks at the three-dimensional structure, or you can use it.

27:11Max Wellingas an LLM like a sequential version like often molecules are represented as a you know as a as a string called a smile string so then it becomes a one-dimensional string and you can use that to represent the molecule as well. So you can use both LLM style models as well as graph neural network style models for this. So your base model might be similar to an LLM but you've got a very unique tokenizer in the case of representing molecules. It goes even further because you can basically also look at combinations of language and graphs and molecules. So you can sort of have, you can learn from, you know, let's say the literature where there's a lot of text. And then you can, every time a molecule is mentioned, you can then actually use the

28:00Max Wellingmolecular representation, which is then a little graph neural network that represents. So that becomes a token. You turn this into a bunch of tokens using your graph neural network. So you can really start to combine these things, actually. What do you see the work that you're doing at Cusp going in the future? Like what are the kind of near and midterm and longer term things that you're most excited about with that work? Okay, so you're constantly expanding the tools that we need for the different material classes. We know how to train these tools, but every new material class has to be retrained. We're also going up the stack, so we're going into more coarse-grained representations like larger scale, digital twins, all the way up to modeling the actual reactor device, which is material sitting.

28:56Max Wellingyou know the effort that's most important for us right now is connecting the platform to self-driving labs so to really be very you know generate large amounts of data from the self-driving lab that's important and I think ultimately it's very important to go through the entire process of predicting the molecule making it in the lab scaling the material and then actually putting it in an actual device and getting a customer excited about that particular material and willing to pay money for it. This basically means that this last part of scaling is something that still needs to be done, but getting there as quickly as possible is I think absolutely key. I'm really looking forward to discovering a material

29:52Max Wellingthat is unique, it could not have been done by AI, and that makes it into an actual semiconductor device or something like this or a new solar cell. And we can say AI actually helped, was a very important piece of the design of this particular material. I think that's a very exciting proof point that will help the entire industry forward. You also have a book that you're working on that has a really interesting title. It's talking about Generative AI and Stochastic Thermodynamics. What's the connection between Generative AI and Thermodynamics? Tell us a little bit about the direction you're taking with the book. Maybe first a bit of a history on this. I started this almost two years ago, even before COSP. I had a bit of a lull between

30:48Max Wellingwhen I stopped working for Microsoft and the startup cosplay I started. It was about six months between that. Me and my wife were on vacation in Italy. I cannot do absolutely nothing. In the mornings, I would have... I'm just going to write a book. I had my cappuccino in the sun and I would start a book. Wonderful, the best vacation you can have. In the afternoon, we would hike through the mountains and enjoy life.

31:17Max WellingAnd then I was teaching a course in a town called Mausenberg, which is a Dutch word, but probably people said Musenberg, but it's close to Cape Town in South Africa. There is the African Institute for Mathematical Sciences, and so I had the pleasure of spending a couple of weeks there teaching.

31:42Max Wellingfrom these kind of ideas that I was writing a book about. And then I recruited two students to help me out actually finish it because it's starting as easy, finishing as hard. And you need some help. And so it took two more years to actually finish it. It's a huge amount of work. But what is the book about? The book is about actually the mathematics that describes modern generative AI.

32:12Max Wellingincluding probabilistic models, including diffusion models and many other things, that mathematics turns out to be equivalent to the mathematics that describes modern non-equilibrium statistical mechanics or thermodynamics. And how specific is that statement? Is that statement, you know, everything is a PDE at the end of the day, or is that statement, you know, more...

32:41Max Wellingkind of specific and concrete? It's surprisingly specific. And I say surprisingly, because I don't think maybe many people think of it as an analogy. So that's why I think it's a really good question, because I think this runs much deeper. So I think the it's more than an analogy in my mind. I think there is a very, very deep connection. And this has to do with the fact that we are talking about information theory in the end, right at the core.

33:10Max Wellingof physics is information theory. And in the core of machine learning is information theory. And both of these are described. So if you talk about information theory with loss of information, in other words, processes where you lose information as you are evolving over time. So there's an observer that tries to describe a process, but there's so many degrees of freedom to keep track of, you can't.

33:39Max WellingWhat happens is that you're losing information and you need to capture that by probabilities. That mathematics is the core of both thermodynamics as well as machine learning. I think that's the core statement. I think there's this concept of entropy, which is maybe interesting in both of these theories. Maybe it's good to talk a little bit about that.

34:08Max Wellingis basically the surprise that you find for a particular state. Do you know the precise state? Or do you only have a very vague probability distribution for that particular state? And that's true. But in physics, you think of entropy as maybe a thing you can measure of the system.

34:33Max WellingBecause you have to subtract entropy from energy, actually, to get free energy, which is the energy you're free to use to do useful work in the world. And so it feels like a real thing, not like something that I don't know about the world. In a Bayesian statistics, this is exactly where it is. You talk about a subjective statistics because it describes what I do not know about the world and I need to you know, ascribe probabilities to the things that I do not know about the world. But in my view, and the number of physicists as well, in particular ET James is a famous one, entropy and physics also precisely describes all the information you're missing about the world. And so there's this very deep connection between these two fields. Maybe the last thing I want to say about this, there is this famous theorem by Landauer, who basically said that

35:30Max WellingIf you erase a bit from a device, you must radiate KLNT, which is just a number, amount of energy as heat to the environment. So here you can see the direct relationship. I'm deleting a bit, which is a piece of Shannon information from a chip, and heat determined.

35:57Sam CharringtonWhen you do that, you must radiate heat to the environment. And so you can see the direct relationship. That suggests kind of a conservation of information along the lines of conservation of mass, which is very foundational in physics. Yeah. So in fact, the statement is the second law of thermodynamics says that the entropy must always increase, which means you either keep the level of information the same, which is when the entropy stays the same.

36:24Max Wellingor you lose information in the process, and that's when the entropy goes up. Of course, what happens is the information in the universe doesn't go away, but it's transferred from something you know, the system, to the heat bath where you've completely lost this information, unrecoverable. Anyway, it's a bit technical, but the idea is that information theory is behind both of these theories, which basically makes the math very, very similar.

36:53Max WellingMany of the tools which have been developed in one field have their exact analog in the other one. And this book is a lot about finding this kind of dictionary between the two fields. There's something called stochastic normalizing flows, and one in the machine learning, and then there is escorted free energy estimation in the other field, and they turn out to be exactly the same methods.

37:21Max WellingAnd so ultimately, who is the book for? And how do you see the book changing the way they view the world or the way they're able to do the things they do? For actually, for both sides, it was written for the machine learner who is interested to learn a little bit about thermodynamics and non-equivalent thermodynamics.

37:45Max WellingAnd in reverse, it's also for the physicists who wants to get into machine learning and build on the things they already know. And I just want to point out that the first paper that was written about diffusion models actually had the word non-equilibrium thermodynamics in the title. So the authors of that paper actually already knew about this connection. So diffusion models really are a process by which you take structure and you destroy it, which is typically what happens in the world. The entropy goes up.

38:15Max WellingAnd then we try to reverse that backward in time, just to start with noise and create structure, which is our generative models. And the way I think it will evolve into the future is of course, or the way it can be used fruitfully is first of all, it can help people in physics to use these tools for machine learning. And let me give you one example.

38:42Max WellingSo it's a very important problem to compute the free energy difference between, let's say, an unbound system, which is like a protein and a drug that you want to neutralize the protein. And so you want this drug to attach or bind to the particular protein. And so you want to figure out how much does it want to bind to this particular pocket on the protein. So you need to compute the free energy difference between the unbound state where these two things are far apart and the bound state where they're together. That's a very important problem. It's something that a chemist would typically want to do, but now you can use modern machine learning methods like diffusion models in order to accelerate those particular calculations. That's a clear way in which you can start from a machine learning method and help the chemists do their calculations better. But then the other direction is also true because there is

39:39Max Wellingthings which have been developed in stochastic thermodynamics like a concept called counter-diabetic driving. It's something that they have figured out on how to do very efficient control of particular physical systems. Those concepts can now be used in diffusion models in order to do a better job at generating images because it turns out that that's very helpful in

40:07Sam Charringtonbringing down sort of statistical error or noise in the diffusion process. And so it can actually help you build better diffusion models. So this is a cross-fertilization between these two fields. And do you see on the machine learning side the primary target of the analogy or relationship as being diffusion models, or is it more fundamental than that?

40:35Max WellingIt's more fundamental than that. In fact, the book goes into variation lateral encoders. It goes into MCMC methods, Markov Chain Monte Carlo methods, which are used to sample from particular distributions. It goes into free energy estimation. There's many different applications and cross-fertilizations that can happen.

41:02Sam CharringtonI think as a field we struggle a little bit with, you know, we're creating these ever more complex, you know, foundation models, our understanding with them, our understanding of them, you know, still somewhat surface level, like in terms of their mechanics, you know, we've, you know, thermodynamics as a field is, you know, much more mature. Do you see us?

41:27Max Wellingbeing able to use this relationship to better understand the models at their core? Absolutely. At least it gives a different perspective in trying to understand things. I would say the field of stochastic thermodynamics is actually itself quite new, interestingly, so it is still very actively researched. Thermodynamics is systems in equilibrium. Stochastic thermodynamics is systems out of equilibrium, so that's actually quite new field.

41:58Max WellingIt gives you a different way to try and understand how diffusion models work because concepts like heat and work and entropy production, these things we don't use when we talk about machine learning models. But I think they give you a very new and interesting perspective on how to think about what's going on and how to also improve them.

42:27Sam CharringtonIn reverse, it's also true. Machine learners have their own way of thinking about how to improve models, and that could also help the physicists to think about problems. I found both parties actually to be very interested in the other parties' work, so that's good. Maybe to take the next step in getting even more abstract, you recently did a keynote at iClear.

42:55Sam CharringtonIn addition to talking about your work at CUSP and the book, you put forth another analogy, let's call it, or opportunity to kind of learn from the physical world, and that is in looking at waves and applying that to machine learning and stochastic systems. Talk a little bit about that work. Yeah, so this is, I think, very exciting in the sense that, so the thing that

43:24Max WellingSo there's two reasons why we thought we needed to think more about waves in science. And that is because, first of all, waves are actually seen in the brain now because we've gone from single electrode measurements to maybe hundreds or thousands of electrode measurements. And of course, in a single electrode, you cannot see a moving wave, a traveling wave, but if you put hundreds or thousands in them, you can actually see these traveling waves and people have now observed them everywhere.

43:54Max WellingAnd then the question becomes, is this functional or is just a side effect or something? That's the first reason. The other reason is that in neural networks, you have this phenomenon called over smoothing, which means that you start with a piece of information and you have thousands of layers. And this information basically is exponentially suppressed as you go through it. So the input.

44:19Max Wellingand the output of the neural network become independent of each other. Of course, it's something that you really do not want because you want the output to say something about the input, like maybe what's the class of this particular input. But it's always been a little hard to make sure that the information travels all the way from the input to the output layers through thousands of layers. People have used many tricks to do that, but I feel many of the tricks we have developed in deep learning is exactly to try to do that.

44:48Max WellingAnd then the other application is in reasoning. So if you have to reason about the problem, you have to sometimes keep things in memory. Of course, we kind of write it maybe to shorter memory, and then we read and write from this kind of memory. It's a quite a more stable piece where things don't get mixed up so fast. So you need some kind of memory. And basically, so the question becomes, how do you communicate?

45:19Max Wellingvery deep into time or reason very deep into time or how do you communicate with things that are very far away in your brain or you know or Take information from a material. That's actually that has very long range interactions and also in materials Materials use waves for long range interaction which are called phonons or phonons are lattice vibrations and they carry information over long distances in a material. In fact, if you think about the universe, the only reason we can see very far and deep into the universe is because light waves hit our instruments in our eyes. And that is, again, waves that are doing the communication over these long distances. And so what's now interesting is that in neural networks, we don't use this tool at all.

46:15Max WellingWe create maps and the maps, you know, map numbers to new numbers. And so the question becomes, can we actually start using this phenomenon, this wave phenomenon that we see in the brain? Can we also start to use it in neural networks? And so this has been the work that we've been developing, and especially for... So we've basically designed neural networks that naturally create these waves, move this information around in what we call...

46:43Max Wellingchannels or memory channels, or you can call them capsules where this information gets sort of moved around and stay stable. And we've been very successful in, for instance, tasks that require memory. So a task where a neural network or an RNN, basically the task is you take a number, you hold it in memory, and then as a random other point, another number arrives, you hold it in memory, and then you add them, and then another random point you're asked to produce the sum.

47:13Sam CharringtonAnd that point could be very far into the future. So you have to do the computation and you have to hold things in memory. And these models that naturally work with these waves, they can actually hold, they can do these tasks much, much better than RNNs. When I hear RNNs and long running memory, I think about challenges like exploding gradients. And then I think about kind of the oscillatory nature of waves and that.

47:41Max Wellingthat has some inherent dampening properties that allow you to access memory further into the future without these exploding gradient challenges. Is that part of the mechanism at work here? That's a very good point. If you think of this, a neural network as a dynamical system, you think of the layers as time and you're trying to get a signal from the beginning and you're trying to propagate it through the layers over time. A couple of things can happen.

48:11Max Wellingtypes of dynamical system. The first one is a stable system, where basically if you take two different inputs, they map to this, you know, after a while they collapse onto a same point and then from there on that one point moves forward. So basically every converges onto what's called a point, stable point, and then from there on basically that's what propagates. That's not what you want because information gets lost because everything gets mapped to a point. Now that's an imploding gradient if you wish, that's something where the information gets lost. There's another extreme where you take two input points, I say two images that are similar, and then they start to sort of wildly move around, and that's called chaos. So it's a non-linear system, so this starts to wildly move around, and you also lose the information, but for a different reason.

49:09Max WellingThe information is still there, but you cannot track it numerically. Your numerical precision is too small. And that's actually what we mean when we say entropy is increasing. It basically becomes a random. It's a random process. And we all know that a Markov chain is a random process that depends on the previous step. It will lose information about the input because we can track it. That's also not what you want. So what you really want is sitting somewhere in the middle, somewhere that's not too stable and not too unstable, and that's called edge of chaos. And people have found that, in fact, neural networks that perform best operate at this edge of chaos, and people have also found that, unsurprisingly, the brain operates at the edge of chaos. And we found that if you inc-

50:06Max WellingInclude these waves in your neural network you for very naturally operate in this regime edge of chaos And you don't have to fine-tune the system to be there. So it's a very it's very easy to get to this edge of chaos regime And then maybe if you have time you can talk about how we do this and this is done by spontaneous symmetry breaking But that's a rather technical technical discussion. I think so what is spontaneous symmetry breaking? Okay, this is a very interesting phenomenon in physics. So here's another example of something quite deep from physics that we can start to use to understand neural networks, which is one of your previous questions. And so spontaneous symmetry breaking is where a system that has a particular symmetry, let's say perfect translation symmetry, just think of a gas as perfect translation symmetry or a liquid better. And then

51:04Max Wellingthe liquid at lower temperature, the liquid will then actually condensate into a solid. You can go from water to ice. And if you look at ice, it has a crystal structure. So it has less symmetry now because it's now the continuous translation group has now been turned into a discrete translation group where you can only translate over the lattice spacing.

51:33Max WellingAnd so you've actually broken the symmetry into something smaller. And there is a very deep theory from physics that says when that happens, when a continuous symmetry breaks into a discrete symmetry or breaks into a smaller symmetry, then you'll have new wave-like modes, which can propagate without...

51:58Max Wellingusing any information or with very, very little information. And these are these phonons or light, or these are the waves that can travel very long distances. They don't decay, they don't disperse, they hold a shape. You start with a certain shape, then the shape can just translate and just move. And so we thought that's a really good candidate for these traveling waves. And so what we did is we created the neural network with the large symmetry. We just bake it in.

52:28Max WellingIt's not like rotation or translation that we typically use. We just give every neuron some extra dimensions and we say there is some symmetry. Then we do something. We generate random weights, which generates random activations. If you make the weights large enough, the distribution from which you sample the weights large enough, the original symmetry gets broken.

52:59Max WellingAnd that's complex to explain, but the symmetry gets broken. And now you're in this broken symmetry phase where you have these goldstone modes or these waves which can travel without using any energy. And they do. So you find them. When you create that, you get these waves where you can just see things oscillate around very naturally. And then the idea is you cannot train the neural network to make use of these oscillations, which are basically for free they're running there without costing any energy they're very stable so they also it all the way from the beginning of the neural network to the end of the neural network and you can now design the system so that it has these waves and it has this edge of chaos behavior so information from the beginning the input of the neural network propagates all the way to the output of the neural network through these wave-like patterns that are created by this phenomenon called spontaneous symmetry breaking.

53:56Sam CharringtonSo here we use something a deep result from physics as a design principle for neural networks. Well, I love that you're getting to play with all of these tools from physics and bringing them into machine learning and trying to find the correlations and ways that we can take advantage of them. There's many more to come, I think. There's a very rich field of mathematics and physics that can all be leveraged by machine learning. But it's beautiful because the cross-fertilization goes in two directions.

54:25Sam Charringtonwith our Cusp platform, we are using AI tools to help the scientists discover new materials and accelerate their simulation tools by using AI. And so it's really a healthy cross-fertilization between these two communities, which I find very exciting. Well, Max, it's been great catching up with you. You have a lot going on. You're working on a lot of different angles.

54:55Sam Charringtonenjoyable to learn about them, looking forward to keeping in touch. Well, thank you, Sam. It was a pleasure again to talk to you. Thanks so much. See you next time.