Machine Learning Street Talk (MLST) - Why Scaling Prediction Cannot Create Intelligence - Alexander Mattick

0:00Effectively, information can be very expensive. And in fact, I would say most information is very, very expensive. If you completely say, well, constraints don't matter. I don't need any prior information. I don't want to encode anything in there. We still have the problem of now needing to really learn everything from scratch. He believes that you can figure out like a suitable reward function for every task. That is enough. I think that energy based models are not really useful, or at least not useful anymore.

0:29I think at world models and so both JEPPA and world models are ultimately branding. There were JEPPA like models before JEPPA and there were world models before world models. Right now, all of these methods are not set up like this. I think they're set up to look very nice and demos and to actually do some really impressive stuff. How do you know, like how many times did it fail to do something? Amazing. Well, Alexander Matic, it's amazing to have you back on MLST. It's been many years since you were on the show.

0:58And we were so grateful for you when you last came on the show because you helped us do some research for our interview with Jan Lacan and Randall Bellistriero about the spline theory of neural networks. And we did that intro all about interpolation and extrapolation. But just for folks at home, so your nickname is Zick Sack on Janik Kilch's Discord server. And this gentleman is by far the smartest person on Janik's Discord server. And that is saying something. Believe you me.

1:27So, Alexander, it's wonderful to have you on the show. Can you give us a bit of a background? This episode is supported by Cyberfund. If you're building at the frontier of AI, they want to hear from you. Cyberfund believes the future belongs to AI natives who want to achieve the impossible. And that is why they're introducing the monastery for AI native founders.

1:51It's an environment of pure focus and rapid execution for founders operating at AI native speed and they're offering teams two million dollars each to participate. Apply now at cyber.fund. Yeah, so my name is Axan Amatic. I've been, so I first did my bachelor's at FAU in Erlangen in Germany.

2:16Then I also did my master's there, so my bachelor's was in computer science. My master's is in artificial intelligence. And then I went over to Fraunhofer, which is kind of one of the biggest like applied research institutes. And now I'm also dual affiliated with the Technical University of Nuremberg, which is kind of like still under construction, kind of in the very much in the build up of becoming a real thing. So it's really exciting to to actually be one of the first people who actually does like it's PhD there. Well, folks today we're going to get into savant levels of technical detail. We're going to talk about inference. So things like energy based models and diffusion and flow matching. We're going to talk about theories of deep learning, constrained reinforcement learning and world models and maybe some other stuff as well. So why don't we start with inference.

3:10Can you just sketch out the history of this? So, you know, obviously, back in the olden days, we might have parameterized a basic normal function. And then we had things like energy based models and flow matching and normalized flows, all this kind of stuff. Sketch out the whole history. In the very beginning, so one of the first ideas is, of course, if we have these like nice parameterized densities, or we actually know, regardless of any parameter we put in, we will end up with a normalized density. So imagine, like, again, the normal function, but they are plenty of others, like a categorical distribution, it's the same idea. And the core idea was, well, the difficult thing when we try to do inference is we always have this, like the density has to be normalized at some point. And the first idea was, of course, to say, and you go and say, well, let's just pick a group of functions, which is always going to be normalized. And this gives us kind of the very traditional variation inference.

4:04problem. Effectively we have a normal distribution with unknown parameters and now we try to fit them onto data. The issue with this is, and people notice this very quickly, is that now you are very constrained by the actual density. So let's say your data is not normally distributed. Let's say it has, let's say it's bimodal. Now your actual density no longer fits. So basically what would happen is if you imagine you have a mode here and you have a mode here, then just fitting a normal would put all the mass basically in the middle and kind of try to split the difference. So the first idea which kind of came up more or less at the same time was doing something like rejection sampling. So rejection sampling is something that you may have already seen like in school where you did like you painted a square, you had a circle inside of it and they did like random points and then the fraction of random points that is in the circle you can use that to compute pi.

5:02Exactly the same idea you can do with any kind of density, as long as you have something which is kind of above it. Let's say we have just a box, which is essentially a uniform distribution, then you scale it up to be basically around your density, and then you randomly pepper it with points. That's rejection sampling, and it works pretty well, assuming you have a relatively close envelope over your function.

5:29doesn't end up scaling very well, especially if you have high dimensional data, right? Because it's one of the quintessential examples of like this curse of dimensionality. So then people went and said, well, we can do a little bit better by framing this as basically as the limiting process of a Markov process. And this gives us something MCMC where basically the idea is that we may not be able to create the density, but we are able to sample from it.

5:58And that's one of the very interesting things about kind of everything inference or like density base is that you can actually do a lot of things without actually knowing the density itself, just by sampling and rejecting samples. The issue with this is that if you've ever run like Markov, James, Monte Carlo, you will notice this is insanely slow, especially in high dimensions. Again, we have because of dimensionality, you are basically Not only do you have these high-dimensional problems where you have many modes and kind of escaping from one mode to the next is super difficult, but also just by the fact that you have still an accept and reject step, you end up doing a lot of work for things that don't end up going anywhere. Alex, just a quick point of order on that. What about G-Flow net? So didn't Benjo come up with some kind of a mortised way to replace Monte Carlo sampler?

6:53So, G flow networks are kind of an interesting case, but I don't think they really solve the same problem. So what a G flow network does is they essentially, let's say you have like a disentangled distribution, right? You have effectively a graph which you walk around and you can construct your sample up slowly over time. And the idea of a G flow network is if we have some, let's say final density in the end, which is for example, Effectively, it's a reward function. And then the idea of G-flow networks is at every node, kind of the inflowing and out-blowing energy has to be the same. So essentially, what you end up doing is you end up intensifying the path which lead to higher reward, and you reduce the ones which lead to a lower reward. However, you basically are back into a close to the variational notion, where effectively you will make the assumption that you have a specific structure, and you assume that structure is correct.

7:53Which, of course, if you have, let's say, some molecular problem where effectively you start building up something, it's pretty, we can assume that this process is accurate. But this is not a general way of doing inference. The general way is still, you need to do either like sampling when you have a very, very general function family to actually be able to do that. Very cool. So yeah, keep the story going. So at some point, energy based models popped up and these normalizing flows Explain that to me. Yeah, so energy based models actually directly come from MCMC, right? The idea or like the reasons called energy is because it comes from physics. Like in the case of physics, they literally had like a function of energy and then you would assume that kind of the particles kind of conform proportional to the energy, right? Let's say you have a specific.

8:42box with hot particles in there, you would imagine they follow some form of distribution because of every particle is like fast, some allowed faster, some allowed slower, and that gives you kind of this statistical mechanics point of view. And indeed I think it was, I think it was Teller who did actually the first developments on like MCMC and kind of got into this idea of actually kind of doing simulations to actually get the actual parameter or like the particle distributions.

9:11Indeed, when you look at what MCMC does is effectively, like sometimes you phrase them literally as doing energy-based sampling, but in other cases, it's effectively, it's the exponential of the energy, right? It's more or less, it depends on exactly which algorithm you talk about, exactly who you have teaching you. But essentially, this is exactly what you would imagine, kind of the gold standard of an MCMC method, right? The MCMC method has an energy, which has an unnormalized density.

9:40And now you want to turn this into a normalized density, which is kind of what the particle is actually doing. An energy-based model is any model that parameters complex densities using an energy function. Energy functions correspond to an unnormalized density. The lower the energy, the more favored the state. Just like in physics, where low-energy states are preferred. Being unnormalized makes EBMs quite easy to train, since no complex integrals need to be computed during training. Instead, an energy-based optimization simply pushes up areas with low sample support and down areas with high sample support. This makes energy-based modeling quite flexible, with many different objectives, all targeting slightly different assumptions when it comes to, for instance, availability of a reference energy to a track. While easy to train, this comes at a steep cost during sample generation. Since energies are unnormalized, we need to perform complex Markov chain Monte Carlo optimization to sample states proportionally to the energy, without having access to a normalization constant.

10:40This makes energy-based modeling unfavorable for genital usage. The most expensive part of sampling is executed over and over again during inference instead of being amortized during training. Further, energy-based models do not offer access to the probability of any sample, but it's helpful for many applications such as anomaly detection or to correct accumulated errors when unrolling a dynamics model. Just another quick point of order. I mean, can you give a little bit of context here? So, many of us are machine learning people.

11:09And we are not sort of in the mindset of working with densities and probability distributions and so on. We are using models that give us point estimates. So what is the benefit of having these densities? Is it because they are composable? Because now we have to do more work at inference, right? So now we've got this density and we actually need to kind of search around on it and do stuff with it. That's what this inferencing thing is. What does it give us? So I think there are two answers to this. One is I think most of deep learning is actually doing inference. And if you go and you fit a categorical distribution, like you do your large language model, you have your kind of density, which is effectively like this auto regressive structure, there's just a decomposition of your final density. What you actually want is you want to model the probability of generating a certain sentence given, let's say a prompt. And what you end up doing is you end up kind of producing a step by step.

12:05the actual answer by going token by token by token. This works really nicely in something like language modeling because you ultimately still have a discrete measure. You have all your tokens and you choose which one to actually get and a categorical distribution is more or less like it's the universal thing. I can always just write down the probabilities of everything. Even if you look at something like regression or let's say you do like image.

12:34detection, let's say you do depth estimation, something like that, with like your mean squared error, you're still doing inference. The mean squared error is just the negative log likelihood of your data under a normal distribution with a fixed standard deviation. So you're still doing inference at some point. What you lose is you lose kind of the breadth of the density, but that's more an assumption.

13:02you can still do exactly the same way you're currently doing, but get an entire mean and standard deviation. And that can be super useful. Like, let's say my background is reinforcement learning. In reinforcement learning, one of the best things you can do to actually improve the performance of your algorithm is to say, I don't actually want a point estimate. I want some breadth to do better exploration. That's the big advantage of let's say a policy gradient method over a Q learner.

13:28because a Q learner only tells you kind of what to do. It doesn't tell you like if you do the different thing, I expect you to be worse by this and that amount. And maybe like if you had this, you would be able to go and say, well, for tactical exploration, I choose the one which might be slightly worse, but ends up being better in the long run. So if you're interested in something like active learning, then you also want densities. So I would push back on the fundamental notion that we are not doing inference if we look at, like, normal deep learning. As an aside, I interviewed Chris Watkins last week. He was my professor. He invented Q-learning in the late 1980s. So that was a great interview. And maybe you should briefly explain what Q-learning is, because obviously, this exploration thing is the biggest problem in reinforcement learning. But just help me understand, though. So what is a good density? Right? Because, I mean...

14:21And Bayesian folks, for example, you know, they valorize this idea of a normalized density or a calibrated density. And you were talking about this softmax discrete distribution that we have on the back end of a language model. I mean, what does good look like? To what extent is that a good density? Oh, that is actually a much bigger question because it really depends on what you end up using your density for.

14:49Let's say you are a completely deep learning field. You want to have just a large model which kind of describes the density of images, right? You do diffusion. We're going to get to that, but we're doing diffusion for image modeling. Then what you're ultimately interested in is you want a density which explains images the best, right? So you want to have them as high of a likelihood as possible for like real images or if you have like a prompt that given the prompt the highest likelihood for things that are actually of that kind of prompt description. And then ultimately kind of what you're actually interested in is just kind of a density which is high enough on the kind of good parts and low enough on everything else. And that of course then gives you kind of that then goes into the discussion of like a prior, a prior fact if it just tells you well

15:44we don't actually want to memorize our data set. We want to have like, let's say a smooth, uh, from peak to peak to affect, we get like images in between the data points and that is a prior. But a prior is just a thing that tells you from like your discrete data set, how can I get from, let's say the, the 50 points of my data set to the 51st point between two different samples. But if we didn't have a prior, then, uh, ultimately the, like the exact solution would be just memorize the data and it would be rather loose useless. If you go on the other extreme what the patient people do they say well I actually like usually what you would do then go is you say you want information in your latents or just basically reveal information which is not in the data by looking at latents the classic example is I flip a coin I can observe kind of heads or tails and what I'm actually interested in is is the coin fair or not if the coin isn't fair

16:43then you will see this in the latent, because the latent is effectively a distribution over, let's say, probabilities. It could be. And if it's the case that the probability is, let's say it's 99% sure that the coin is fair, I can accept the hypothesis that they are coin is fair. If it's 99% that it always lands on tails, then, well, I can make the inference that it's actually an unfair coin.

17:08And that is what the statistics people are uniquely interested in. And it's a big divide between someone who does statistics and the medical domain, who actually wants to figure out, given those tests, where's the likelihood you have a disease compared to someone in deep learning who just cares about, hey, I want to generate pretty images of, I don't know, pink elephants? Yes. And I suppose my intuition is that we're machine learning people. We have a finite sample of data.

17:37And our purposes might be, you know, at some point we've got to interpolate between these data points. We've got the cursor dimensionality and so on. So we have to put some kind of, I mean, we're amortizing this process and we have to smooth it and make it amenable to the types of prediction that we're doing. Exactly. So, and this kind of goes nicely back to kind of the discussion of like variation inference was a sampling because these of course got this hybrid thing where like energy-based models where effectively you say, well, I still want to do sampling. I still want to do MCMC, but I learned the energy. This is what the energy-based people did. And this was a very nice idea at least at first because it gives you the property that you can learn basically arbitrary densities. Assuming your energy function is kind of rich enough that it can approximate the energy at least, then you can effectively just pawn off the problem

18:35of normalizing your density to later, to sampling time. But if you set up your loss function in some clever way, you end up being able to do the training at least without needing to do sampling. And that is a huge advantage because now you can directly fit to an energy which you might have literally measured or you do this for let's say samples of some density and you try to push up the energy for where the samples are rich and you push it down where the samples are not rich exactly the same way as you do with the density the problem of course you run into with this is that now you have kind of the difficulty of both worlds like you have the problem of like variation inference where hey what if my function is not strong enough and then you have the problem of the sampling based world where suddenly you have the issue of my actual sampling process is really really slow

19:32So and this is kind of what always limited energy based models So the the next idea which kind of came up more or less in parallel is this idea of what if we what if we simplify our problem a little bit and Effectively give ourselves like an infinite pool of latents because latents in kind of influence our power because if you can if you can split your problem up into smaller sub-problems, but just conditioning on some information. They can make it conditionally independent. And these sub-problems can become a lot simpler. So in many cases, a model becomes cheaper the more latents you have. Of course, that's always true. But generally speaking, you want to have kind of flexibility that you have tiny steps, which is what you, for example, parameterize. And then the actual, let's say, big steps are just by the fact that you have latents which you condition on.

20:27And of course the absolute limit would be you say, well, what if I have infrared latency? And that's what gives you diffusion. Diffusion is the idea of, hey, I have a Markov process, similar to MCMC, like you have a Markov process just with infrared steps, so I can make it smooth, continuous. And then you go and you say, well, at every step, I now sample based on the previous steps information, like in the normal Markov step. This makes you This has one big difficulty, namely now you actually need to model effectively time because you effectively, what you're not saying is we have latents like the latent at time step 0.5. So, and of course, if you go to the latent at point, at time point 0.6, you end up kind of having effectively a different density. So now we have this time dimension as an additional variable. So this is what you pay for, but in general, this is much, much cheaper still than

21:27uh, needing to do like the actual MCMC process. Because now you just, you follow essentially a density. So with like the randomized, uh, you randomly, uh, inject more, uh, noise at every time step, but you no longer need to have like this accept reject step. You no longer have the problem of like local minima and so on. Because essentially what you did is you've rephrased as, as it was casted differential equation, which, um, essentially then goes from like the the initial density to final density and this is just by basically paying for with one additional variable which effectively indexes over the latents and this gives you a huge improvement because you no longer need to do effectively this this time it still exists in MCMC like it's the the the sampling steps but effectively you amortize that as well and that makes it a lot cheaper um and then you can go a little bit further and say well

22:24what if we go away from this Markovian structure in the first place? What if we say, or maybe not from this Markovian structure, but maybe from a specific, let's say like diffusion style, like this fucker-plunk style, what if instead we just phrase it as like re-parameterizing a density from beginning to end, kind of in tiny, tiny increments? And that then gives you what is effective with these flow measures, or more generally, continuous normalizing flows. Continuous normalizing flows, are effectively phrasing the problem of finding a density as you start off with kind of initial guess, right? Just as an optimization, you start with initial guess, which is like your normal distribution, and then you end up with your final density. And what you end up doing is you need to kind of find effectively an optimization process, which gets you from there to there, which is a gradient flow, which is what gives you these, like effectively optimal transport style problems. We say,

23:22we have the continuity equation, which says, well, the change over time has proportional to the divergence of the vector field of the current density. And if you actually, if you go and you write this out without invoking densities, but let's say we invoke this with points, you see that this is just doing gradient descent in continuous domain. And of course, it's the cheapest thing at all.

23:49because now you no longer do any sampling at all. Now you phrase a problem as I do optimization in inferred dimensional space. So you pay for the inferred dimensions, but you get the lack of sampling. So the, the, effectively the entire trajectory was you start off with like the, the very expensive sampling and like the variational models, which were very limited. Then you said, well, let's do kind of both in the middle, which were these energy based models. And then you said, well, can we make the energy-based models less like a rejection sampling? And that then gives you the fusion models. And now you say, well, when we actually do this trajectory from initial distribution to final distribution, we don't need to do this Markov chain process at all. We can phrase it as just a pure optimization problem without any stochasticity by instead kind of

24:43updating the density at every point and the only thing you're now going to pay for is invertibility because you need to be able to go kind of in one direction generate samples and the other direction to get the likelihood and you need the likelihood to do your to actually do like inference and that's really cheap because like a like a vector field at least in a density sense is invertible like you instead of going downhill you just go back uphill the fusion models to density optimization is an invertible noisy process During training, first a sample gets repeatedly corrupted with normal noise until only normal noise remains. The training objective is to simply revert a single corruption step for instance going back from step t equals 5 to t equals 4. This corresponds to learning a vector field vt of x that tells you how to change a sample to make it look more like a target sample. During inference, for sample generation, we simply draw a sample from our normal distribution and follow the learned denoising process backward.

25:44In version like this works, since, in the background, this corresponds to solving a stochastic differential equation. This equation is reversible coupling between a normal distribution and your target distribution. This also allows you to compute the probability of any sample from your target distribution. Simply follow the target sample through the noising process and accumulate the divergence of your vector field. This shows the major upset of diffusion models. Instead of needing to perform MCMC during inference time, we only need to follow a vector field across time.

26:15which can be further accelerated using sophisticated ODE solvers. The downside of diffusion models is that we now upload the representation with a synthetic time dimension. This extra costs the neural network needs to work with and also slows down training. We effectively solve many tiny problems of reverting individual noisy steps. However, there is a one-time training expense instead of an inference time expense, making it cheaper for real-world generative usage. Normalizing flows are models treat density optimization as a reparameterization process. Instead of trying to model a density from scratch, a normalizing flow starts off with a known base measure and transforms it into the target measure. If a series of transforms applied to the base measure is invertible, we can use this to essentially inherit the normalization of the base measure. We only need to track the local expansion or contraction, which corresponds to the Jacobian of a transformation. Trigger to do with any framework supporting back propagation.

27:15Sampling simply follows the flow forward. From a base measure sample, apply the model to get the target measure sample. Computing probabilities goes the other direction. Follow the flow backward from the target, back to base, while rescaling base on a function's Jacobian. One core limitation of naive normalizing flows is the constraints that functions have to be invertible for sampling to work. The way around this is surprisingly easy. A continuous normalizing flow Frames a normalizing flow as following a path from the base to target measure based on a vector field that gives local directions. The trick is that such a path, a gradient flow, is unnecessarily invertible, even if the function giving you the vector field is not. Just walk the same path backward. This adds a nice property of allowing arbitrary couplings between measures. The base measure could be a normal distribution, a student T distribution, or even another normalizing flow.

28:14anything where you can compute the likelihood of a sample works. However, this also comes at a non-zero cost. For training, we need to compute the full ODE from the target sample back to the base measure every single time, since that is needed to compute the likelihood of the sample for maximum likelihood training.

28:33Okay, okay. I think there's quite a few things we need to unpack it because we've suddenly got very, very technical. So if I understand, so in a normalizing flow, it doesn't need to be invertible, but in this flow matching, it doesn't need to be invertible. And just to help the audience understand here, so what we're talking about here with this flow matching is it's the ability to model a vector field or an optimal transport between noise and the data that we want to model.

28:58I still think we're missing the link here a little bit of energy-based models. You know, Lacoon, when he talks about energy-based models, he talks... I mean, many folks would have seen the H. Jepper architecture, for example, when he talks a lot about contrasted learning and he talks about how you can build specific models to do specific types of things. I mean, maybe we should start with that. I mean, what kinds of prediction architectures, what kinds of predictive tasks can you do with EBMs and these normalizing flows? I think Jepper is still a little bit further away.

29:26But what I would say is the difference between something like an energy based model and something like a diffusion model is that so both of them are fundamentally generative models. I don't really get this idea of like energy based models not being generative. Like I've heard this a bunch of times, but it's just not true. Like the interesting thing is you can maybe use them in settings where you don't need to be generative, but fundamentally you are still generative. Like you still can generate samples. It's just very expensive. So nobody does it.

29:54Sorry, can you just explain that because that is something Lacoon says quite a lot. He says that a good property of these energy based models is that because they're not generative in some way, they are better. Can you explain that? Fundamentally, you can still do exactly the same thing. You take your energies model, you throw them into an MCMC sampler and technically you will get samples out. These are very technically, of course, many of these energy based models are not actually good enough to generate nice samples. But at least in theory, if you had, let's say, Let's say you had the perfect, um, NH based model, right? You, uh, like someone comes up from the heaven says, well, this is your, this is the right parameters. You plug them in and now you have kind of the, the actual data generating process. You can use them to generate data. In fact, this is not even that special of a property. Fundamentally, as soon as you have something, which is kind of, I'd say proportional to your density, you can throw this into MCMC sample and you can generate samples. So the.

30:51distinction between having a generative model and having a non-generative model, I think it's really, really small, because the question is not whether they can do it, it's whether it's economical to do it. And then I would agree with the curl, like it's not economical to do energy-based models to generate data. But I would imagine this is a disadvantage or an advantage. The nice thing about energy-based models is you can do certain things cheaper.

31:19Namely kind of the, the very big one is you can do kind of comparisons between data points, right? Because like there is, let's say there's effectively an unknown scaler inside energy based models. But what you can do is you can effectively still go and say, well, this point is at say three times as likely as this other point by just effectively computing the ratio. And if you just need this, so if you have a problem with activity, you just need pair wise comparisons.

31:45then energy-based models can be quite good, right? You effectively, you don't need to generate data points to produce something like, let's say not a like load, but something like an a score or like score is also maybe a bad word because it's also inference coded, but you still get kind of the, the relative difference between two, two points. Of course the downside and what makes it hard for us to be generative is Even if the ratio between two points might be really large, it still might be really, really bad because you don't have an absolute score. It's like saying, well, this movie is a 5. Well, 5 out of 5, 5 out of 10, 5 out of 100. Like it doesn't tell you enough to actually figure out whether it's a good sample or a bad sample. And actually figuring this out in an absolute sense, this is what is the expensive part. So energy-based modeling is really nice if you never need to sample anything.

32:41So this is what makes it really cool for, let's say, pre-training. If you do pre-training, then doing something like an energy-based model, that might make sense. Of course, there are many other things that also make sense, and I think it's up to debate of whether it's actually a good idea to do energy-based modeling over anything else.

33:03Yeah, I think this is a completely different discussion between kind of doing between kind of the question of whether these energy-based models are good at doing inference or bad at doing inference. Because essentially you're just saying well, I train an energy-based model, but I don't end up using it for a real inference. I use it to effectively measure differences in intensity. Okay, and just help me understand a little bit more. So I think Lacune would say almost any model could be an energy-based model and something I've struggled to understand is You can design energy-based models. You can design where to place the latents. You can make it quite domain-specific. Or it could be quite general. You could have an energy-based model of a transformer or something like that. So help me understand that. More generally, this is also one of my big problems with the JEPPA style things. With JEPPA like joint embedding predictive architecture. That could describe literally anything. It basically just says when it's not contrastive. But let's say

33:59Um like I recently in in one of the like paper discussions at Yanni Kirschner's discord I made the like distinction that I found generally that like dino embeddings are much better than like the um like the japa style embeddings at least for the stuff that I tried them on this might be very different for your application but I think this is kind of like a a false comparison because if dino came out today they would call it a japa architecture because it's it's such an open domain and the same thing is true with that's a variational inference or like energy-based models, you have a lot of flexibility. It's just kind of what you... So energy-based models is just a name for what you end up predicting. And what you end up predicting is kind of this effectively unnormalized intensity map. Effectively, it's like a heap map. And anything that is this can be framed as an energy-based model, which is also why they show up everywhere. Because

34:58if you want to, you can phrase it as an energy-based model. And that makes it oftentimes a less useful description. What I would kind of, where I kind of place my differences, if it's already normalized, so something like I predict, let's say in like in reinforcement learning, I predict, let's say the mean instead of deviation given the input data, then I say, well, this is not really, again, energy-based model. Like technically it is practically, you wouldn't think of it like that.

35:27Then on the other extreme, you have something like a score based models, which effectively model the gradient of energy and do tricks to basically also remove all the normalization. And that then gives you something like these, these Langeman diffusion start things that gives you a diffusion modeling. And it doesn't give you a flow matching, but it gives you something like flow matching. But yeah, I would agree that the difference is really, really wishy-washy.

35:54And can you just explain this non-contrasting thing? Randall was telling me at NeurIPS that apparently, I think it was, Hinton was the big contrastive guy and SimClear was one of the original contrastive architectures and Lacune is big on this non-contrastive thing. He talks about it as shrink wrapping the latent. What does he mean by all of that? So there are different ways of framing these unsupervised learning problems. One of the first ideas was, okay, we can phrase this as kind of making similar things similar and dissimilar things dissimilar, right? And this gives you ultimately a contrastive style model by basically what Simsular effectively does is if you look at the loss, effectively it's like a cross entropy, but over the batch. And what you then do is you have the positive part, which is like the different augmentations which get pulled together and the dissimilar parts they get pushed apart. The downside with this is

36:48it's ultimately a very, very moving target, like a batch to batch moving target because kind of contrastive just means when it's somehow the right things have to be together and somehow everything has to be apart. This means there are many, many, many solutions which actually satisfy this condition, but are not actually useful. So in practice, you want something which is a stronger and B does not have this instability.

37:15that you fundamentally have with these SIMCLR style methods. And if you look at SIMCLR, they only start making sense at huge batch sizes. And then there is like follow-up work, like the MoCo momentum contrastive models, which essentially say, well, we can simulate huge batch sizes by effectively taking, let's say a buffer over time. This of course adds a little bit of noise, but tends to work pretty well. But fundamentally, I agree with Luca that we need to go back away from these contrastive models.

37:45Contrastivity alone is not strong enough of a statement to, uh, to actually learn necessarily useful models. So what the kernel says is, well, we should actually go, in fact, we should revisit old ideas, which is try to map things into like a compact latent. At the first idea of this was like a variation auto encoder, a variation auto encoder says, well, you have to be able to map the data to a normal distribution.

38:12or it's a beta distribution, whatever you set in the middle. And then you have to be able to map this back. And effectively, this compresses everything down into this tiny gate in the middle. And effectively, the top part is a generative model. The bottom part is effectively like an embedding model. A problem with these variational auto encoders, and it's something that's also been seen very, very early on, is that you effectively need to pay for things that don't end up making a difference. The classic example, I don't know who did it, has the example of an actual dollar versus a painting of a dollar. It represents the same idea, but it's very, very different from a pixel-to-pixel basis. And this then gives you something like Dino, or something like Wiggreg, or something like the Jeopard-style architectures. Specifically,

39:04Like one of the newer ones, which I really like is like this led jumper style thing, which effectively uses the statistical tests to actually figure out, okay, am I different from a normal distribution? Am I the same as a normal distribution? And based on that, they can differentiate through that test and move you closer or farther away from the actually now fixed goal of the density. So you replace the contrastive push things apart or pull things together with be somehow on this fixed target. Of course, you still have some flexibility, right? Because if every point starts pushing into the mode of the density, at some point you would have a kind of pushing away effect again. So you have a similar thing as a contrastive model. But now it is contrastive compared to a reference density and not compared to itself. And it makes it a lot more stable. So what does all of this mean for energy-based models?

40:03In my opinion, I think that energy-based models are not really useful. At least not useful anymore. In general, of course, you can frame your model as being energy-based. If I'm a reviewer, I will not ding you for it.

40:20Yeah, ultimately I don't think it's a useful way of thinking about the vast majority of problems Like there are exceptions to that like let's say you are actually in like a physics setting but then maybe it makes a lot of sense to actually phrase as an energy based model but I would argue that it's not It's not a useful way like it's it's so general of a statement. It's not a useful way of thinking about the problem for for things like Uh, contrast of all contrast for unsupervised like these self supervised learning methods. I think there is generally speaking a better way of thinking about the problem than energy based. And ultimately, I think this is where these methods lie. If you want to do generative diffusion model or flow match as better, I don't think anyone would argue with me about that. And if you want to do unsupervised learning, I think you can phrase it as an energy based model, but don't think it's a useful framing. Yeah. And we'll get to diffusion in a minute, but I mean,

41:15Can you help me understand? Where is flow matching being used? Because folks in the audience, they might not be very familiar with it. I mean, for example, I was interviewing Mistral a couple of days ago, and they were using it in their voxtral model. So there are people out there using flow matching, but where are they using it? The big difficulty for doing something like, let's say, something language modeling like, right? Ultimately, everybody right now wants to do language modeling things without language, like big audio models, big image models, big...

41:44world models, you can get to that as well later, but all of these models they want to do exactly the same thing that large language models did for language. And the problem that they quickly run into is that most domains are not actually that discreet. Like you can make them discreet by basically hitting with a hammer long enough, but practically they're not discreet. The problem is that if you are not discreet, things like this auto-aggressive decomposition become kind of dicey because the For language models, the nice part is that you can go step by step and you can predict from left to right. If you have a continuous density like an image, you can still do the autoregressive, but it's not longer a natural way of doing things. It's not a natural way of decomposing a density. Ultimately, that's what all of these models do. Be it diffusion, be it energy-based, be it autoregressive, all of them try to have a huge density, an entire sentence, and decompose it into smaller parts which you can actually handle.

42:41the idea that you want to do like most people settle on for continuous processes is instead of separating kind of from left to right let's say token by token by token is they say basically we separate from bottom to top like we start off with all tokens but kind of corrupted and kind of go back up and that immediately allows you to do things like image generation because like I explained just now with tokens to make the comparison sensible but Ultimately you start up with something and end up with something. In fact, discrete in this case is the more difficult part because of discrete diffusion you end up with some kind of dicey things with kind of, I guess language models are probably fine but you get into dicey things with basically if you actually sample from the density every single point. It's not possible but you have to be a little more careful. But the nice part is that once you have kind of this this general recipe of decomposing your structure

43:39then you can apply it to anything and this is what makes diffusion models or flow matches which is like if you read the fusion model and you look into it oftentimes it's flow matching because it tends to be a little bit more stable but all of these methods are just ways of decomposing a density in a really really general way and what you do in diffusion is you give yourself these infinite numbers of latents and say well given the infinite number it has to be decomposable in some point in some way and in the Or in the case of flow matching, we literally give kind of the reference of, okay, this is what you need to do. Just do it now, but general. And in autoregressive structures, you do exactly the same thing, but instead of giving yourself the latent codes, you say, well, I have a very natural decomposition of my text already. Just use that.

44:28But ultimately, both of these solve the same type of problem. It's just whether you do the decomposition in the observable space, whether you do decomposition in the latent space. And for things like high, for high dimensional things, which themselves have less information per effective per variable, it tends to make sense or at least empirically tends to make sense to go into diffusion instead. And can we just talk more about inference in general? So I think this is something that people don't think enough about because we don't do We don't really do things that resemble inference that much. With language models, we just do greedy sampling. So we got this density and we just take the, you know, we just take a token, take a token. In something like AlphaGo, for example, we had a structured inference process. So we did this, you know, Monte.

45:12Carlo tree search and we were kind of searching around and actually algorithmically updating notes that we'd visited and feeding that back and retraining the model. With these vision models and when we're using flow models and stuff like that, we are actually doing like quite sophisticated inferencing processes and I interviewed Clemon Boday. He had this searching latent spaces paper and he was kind of like searching a decoder to actually find the best programs to generate for arc and we've got this whole, you know, and test time computation, test time adaptation and so on. So this.

45:41this whole thing about doing work at inference time is very exciting to me. And I just don't think that that many people are talking about it. Yeah, I think we give people not enough credit when we talk about kind of doing... So first of all, inference is a really, really overloaded term, right? If you go into logic, like they say we do inference, like logical inference. And if you go into like Bayesian, they also say they also do inference. Now I say, well, we are actually also doing inference. And So I think we have to be a little bit careful not to use the term too liberally, because I have to be careful myself, because at some point use all meaning. At some point it becomes the energy-based model, which you can apply to anything. But ultimately, I think in many cases it is very, very instructive to think about these types of models as just composing, just conditional dependencies, conditional dependencies and marginalization.

46:38Something like AlphaGo is actually a little bit more typical, because what they do is effectively they do this like non-parametric, effectively they come, arguably they come more from like less the Bayesian side and more the frequentist side, which is an entirely different kind of worms that don't want to open up right now. But at the end of the day, all of these methods really, or at least all these mainstream methods really benefit from this idea of we have a big problem, we need to decompose this and every step is one step in a bigger inference process.

47:08Let's say, let's give an example from one of my papers. I did a paper about doing effective mixed intro programming, learning how to stick for mixed intro programming. And specifically, like in mixed intro programming to not go too deep. They have a process called branch and bound, which essentially you, you start with like a approximation and then based on a solution, you pick two different solutions, just like in like a tree search or it is a tree search. And you go, at this moment, you go further down, but closest to something like alpha go, like alpha beta pruning, you can through and away lots and lots of nodes, if you know more information, but basically say, well, this node is worse than the solution that I already found, so I can remove that. So it becomes really important if you have like a huge tree, let's say thousands of open nodes at the bottom to choose which one to actually end up selecting. And what I did is I found this as an inference problem, but basically I say, well, if we go down from the root to one of the nodes,

48:06you effectively do a conditional probability, like you have a probability to choose either left or right, then you have another choosing left or right. And that gives you a much more general model, which kind of grows with your problem. So as your problem grows, your inference process grows and kind of ingests more and more information. And the same thing is true with something like I'd say a large language model ultimately, which kind of does research, kind of looks up more data, ingests it, and now the next step is now conditioned on that information. Now, how useful this is for your, let's say like the viewers daily work, it really depends on what you end up doing. If your work is, let's say I scale up a large language model, then you probably don't need to think about this. If your research is like figuring out really, really efficient architectures, maybe, but I would also say probably it doesn't matter. But if you kind of, if you go and you use them or you do something like RL where ultimately kind of

49:03everything is like you update some density if unless you do like q-learning but if you update like densities then you should be thinking about this as an inference process because Especially Bayesian inference gives you a lot more Structure a lot more information than what we give it credit for like something like just decomposing a problem into conditional probabilities gives you like it's like 99% of the way there but it can already help guide the actual problem and I think that's something like all these authentic things could really benefit from actually thinking about this as like a Bayesian problem of kind of going out, gathering data and updating your conditional belief based on the information that you just ingested. Very cool. And we've mentioned Q-learning a couple of times. Can you just give us a quick refresher on that? Okay. So the idea of Q-learning, actually it's kind of simple. So what you do is, the problem that you run into in reinforcement learning is that you don't want to be greedy with like the immediate actions, right? Because what you do immediately

50:03could be really, really detrimental. Let's say you manage a warehouse and the greedy thing to do to maximize profit right now is just sell everything in your supply closet and never buy anything because then the profit is maximal. That's not necessarily long-term the right thing. So what we would need to do is we don't need to look at the current world, but we do need to look at kind of the future as well. That gives you a problem. The problem is, well, I now need to model the future. So the way that Q-learning does it is it says, Well, let's decompose it as kind of a head, which is the current reward, plus the tail kind of the rest. Um, and that then gives you, if you, let's say we have such a decomposition, then we could effectively act greedily with respect to this kind of, to this combined metric, right? Because if we have a queue function, which tells us, well, if we go down this path, the, the future will behave like this. If we go down this path, it will have like this and this part will behave like this. Now we can.

51:00be greedy again with respect to the Q function. So effectively the Q function fundamentally is a closure over the path that the policy can take. The way you train this is effectively by just enforcing exactly this. So you say the current reward plus kind of the next state's Q function should be equal to the Q function starting from the original state. So you expect to say r plus Q Q from a next state or for a specific action and Then you say this has to be the same as doing a Q function from the original state of us already This then gives you two options, right? The first option is we could choose kind of for the next action to choose here, right? Because the Q function has both a state and an action at this node at this future node you could choose Well, we pick the action was actually taking the real world. This gives you Sarasa, right state action reward next state

52:00The other option is well, I could go and say instead of just picking the next action which actually happened is I pick the optimal action, right? Because let's say we have a discrete number of actions just look at every single one and say well, I pick the one which is the best So effectively hallucinate a world in which the you actually chose the correct path It has the very nice property of now you You effectively are no longer coupled to the specific policy. And if you choose the correct, if you choose the action that the policy chose, you're now effectively updated with respect to the policy. If you choose with the, uh, if you choose the optimal state, effectively you're, uh, you're updating with respect to the optimal policy already, like the Q star. The problem with this is now that you're no longer, like you have some form of, effectively we have a counterfactual inference problem where you need to deal with what actually happened versus what happened in our like, let's say golden state where we

52:56walked along the kind of optimal path. Very cool. And just coming back to EBMs for a second, we've been really negative about EBMs and LeCarn actually has spun up two companies, AMI and Logical Intelligence doing their stuff. Are there any situations, do you think, where EBMs are better? Again, I think the big one is if you just need ratios, because then it can be a lot cheaper because you don't end up generating, let's say, the entire trajectory. It can also be helpful if you have something energy like already.

53:26again something like physics where effectively you based on data you try to learn the energy function something like this. Other than that I think it becomes really really difficult to justify. Of course it really depends like as we said energy-based models are really overloaded term because if you don't use it if you use influences as a pre-training task I don't see a reason why not. I don't see a reason why you should but I don't see a reason why it wouldn't work.

53:52The question really is, is if we end up looking like I'd say, let's say we jump forward five years from now. And because startup is doing whatever he did, like they are doing currently research, whether, what they end up doing, like they might still call it energy based models, but whether it's actually an energy based model, as we, as you and I would be thinking about this right now, I hide out that because why it's possible. I don't think it's, I don't think it's the best way of doing this. And the And the bad thing or the good thing and the bad thing about deep learning is that if you do something wrong, it generally doesn't just not work. It just is worse performance. And that makes it really, really difficult to research. And which is why there are thousands of papers which essentially claim more or less contradictory things because depending on subtle details, things can just work or not work. And they can be really, really so.

54:46Even if I do something completely wrong, like I have recently I have a paper which I try to get done for ICLR where I had like an implementation bug. It still kind of worked. And there are plenty of other things where still like it doesn't on paper doesn't work, but in practice it just works because especially in force learning it just it hammers on it until it figures out something. And I think that many, many things right now are like this. Like they are not necessarily optimal, but they're also too bad to actually bother fixing it. Very good. Very good.

55:15And I don't think we've done enough on this, but how do we go from flow matching to diffusion models? So again, the jump from flow matching to diffusion models is a lot larger. Ultimately, if you actually look at, so flow matching actually comes from, so there are two ways of thinking about flow matching. One is you can think about them as like the continuation of continuous normalizing flows. The idea for continuous or like any normalizing flow is you have a base density, so a normal distribution. And You frame your problem now as trying to figure out a transformation, which goes from that base density to your target density. If you make sure that this density is invertible, technically a little bit less is possible, but say it's invertible, then we can actually both do the sampling. So you go from the base density, you go forward to the sample space,

56:08So this is the sampling direction. And if you have a sample, you can go backward and figure out the probability of generating something by looking at what it is on the normal. So this is the first idea which you can kind of branch off from. And the second is from optimal transport. So in optimal transport, there's this very, very famous theorem by Brenier and Ben Amou, who basically said, well, we can frame an optimal transport problem from one density to another as a kinetic problem, where effectively we follow this transport equation. So the density change over time is equal to the negative of the current density times the velocity field, and we do the divergent around it. And this effectively is like a continuous, effectively it's like gradient descent, but in continuous time and instead of having a finite number of parameters, you do it over the actual density. So it's called a gradient flow.

57:07And these have really, really nice properties. So even in finite dimensions, these continuous normalizing flows, these gradient flows have super nice properties. So what you end up doing, we use this fact that effectively the velocity field and the density change is coupled together by saying, instead of learning the density, we learn the change in density, so this vector field. And this is the parameterization.

57:36or at least for these flow measures. So what do you now say is, instead of picking any velocity field, I do a reference, let's say like the rectified flow, which is like let's say a straight shot. And then we fit to this. How we go to, so we can go from this back to diffusion. And specifically if you look at like the Fokker-Planck diffusion. So Fokker-Planck, the Fokker-Planck equation is the effectively the PDE form of stochastic differential equation describing diffusion.

58:05So a diffusion process is kind of this idea of we have both the mean change and then we have the sand deviation which adds in the noise. We can frame this as instead of doing like a sampling, effectively sampling our way through, we can view this as changing our density from the start to the end. This is the Pocca Planck equation. So effectively we now, instead of doing sampling, we now have to solve a PDE.

58:31And usually people of course go the other direction because doing sampling is much cheaper and much easier than doing and solving a PDE. But let's say we look at this for a PDE point of view. If we actually look, we can rewrite this as basically the velocity field inside of it is so that the Fokker Planck equation reduces back down to having a very specific velocity field.

58:53And if you look at this velocity field, you end up doing, so you end up realizing two things. One is of course like there's a fundamental connection between these two methods. The second thing is you can see why diffusion models are kind of iffy to train or more so than diffusion. But if you look at the velocity field, the velocity field actually depends on the density. So effectively, what we now have is we have the self coupling, which originally in like a flow matter we can get rid of.

59:22we can completely get rid of ever needing to touch the density. While in flow matching, well in diffusion, you end up having to touch the density. Of course, there's this trick with like the score transformation, which means you kind of get around this. But ultimately, the vector field now depends on the current state of our diffusion. The nice part about this is that the diffusion style has some really nice properties, specifically that it's kind of minimizing the relative entropy, so the KL divergence between like your deuteric density and your current density in this optimal transport sense. Flow matching is a method of striding the space between diffusion processes and continuous normalizing flows. The upside of normalizing flows is the ability to couple arbitrary measures with each other, while diffusion is theoretically restricted to couple with a normal density. The upside of a diffusion

1:00:20is a fast training since we do not need to solve a full ODE for every optimization step. Flow matching exploits the fact that the actual path taken from base measure to target measure is arbitrary. This means that we can simply impose a reference direction which we fixed beforehand. This is quite similar to what we implicitly did for diffusion models. A diffusion model's reference is given implicitly through the noising process. In fact, a diffusion model is simply a very specific reference flow that aims to minimize the KL divergence.

1:00:54Flow matching still frames sampling as a reparameterization problem, but now fixes the degree of freedom in a path priority. Pick a path, for instance a straight line between a real sample and a base sample, and train a model to emulate that. The model's task is now to simply learn a general parameterization of the example flows. A simple supervised regression problem. The upside of fixing a path is twofold. First, it allows us to give supervision on every part of the flow without needing to simulate the entire ODE.

1:01:26The intermediary targets are known and can be used as supervision signals. Second, it allows us to shape the path based on properties we want. For instance, straight-line paths tend to be easier to sample from than highly non-linear ones. As such, flow matching allows all the flexibility of normalizing photos while having the superior sampling properties of diffusion models.

1:01:51a little bit more concrete. So for example, I interviewed John Jumper and he was talking about how he used diffusion in the third version of Alpha Fold. And he was talking about some really interesting properties because it's very much an iterative process. So, you know, he was saying that we were running it through this diffusion model and, you know, like the big structure kind of converges first and then it learns the details afterwards. And many folks would have used stable diffusion models, for example, so you can actually kind of see these, these images materializing over time. But yeah, just be a little bit more concrete. I mean, where are these being used? How can people use them?

1:02:20pretty hard to say because like ultimately you would expect something like this for like for a general model and the reason is that Why are the latency themselves? Don't hold any information like a latent is completely unobserved. They don't hold fundamental information if you want them to hold fundamental information You need to go to the Bayesian people who do like the parameterization of the coin flipping which I had the beginning The actual latency don't hold any information But of course in practice if you enforce the fact that the change from let's say latent number one to latent number two should be really small then you would expect something like this where effectively the big structure appear first because on the big because if you don't have the big structures kind of the small changes later on cannot reconstruct this. So I don't really have a good argument for why it has to be like this and I don't even think it has to be like this. I think this is more a notion of kind of what happens to be easy for these models.

1:03:20But I would imagine that most, let's say, sane models would structure it like this. This doesn't mean that you couldn't build a diffusion model, which doesn't do that. I could imagine going and kind of enforcing a constraint that says, well, you have to do the small things first. I don't think it would be a good model. You might need, let's say, thousands of times more parameters to actually fit this. You still would be able to train this, at least I think so.

1:03:49Mathematically, there's no reason why it shouldn't work. Practically, there is lots of reasons why it shouldn't work. That's actually really interesting because we see a similar thing with just standard neural networks. You can look at the types of representations they learn during the training process and vision models. They learn low frequency features early on, and then they learn higher and higher frequency features.

1:04:09And you're kind of saying it doesn't have to be a certain way. Maybe it just depends on the architecture and the data and the domain and things like that. But it's quite interesting, though, isn't it, though, with these types of models that you can basically do an arbitrary amount of computation. So on stable diffusion, you can do 25 diffusion steps, or you could do 150 diffusion steps. And if you overcook it...

1:04:34the image starts to look weird because you get little micro details on the skin and it doesn't look very good anymore. So, yeah, I mean, how should practitioners reason about this? I think reasoning about architecture is really, really difficult because I, so the more think about architectures at like at scale, like let's say we have like these billion parameter models, I am not sure it matters as much as people think. I think it matters in the sense that you need to make sure that your input data actually fits into the model. You couldn't do an MLP which does the same type of large language model-y thing just because it has a finite input space. But I don't think that there is a fundamental limitation on one of these architectures if you look at it at huge scales. Because at huge scales, all of these perform a lot less

1:05:29like parameterized models of like a specific architecture and more like like function spaces. Effectively, the actual change of the individual parameters does not really mean anything anymore, but more kind of the bulk change of the entire like field around this. Like this is effectively like the mean field theory approach to deep learning analysis where effectively it doesn't make sense anymore to look at individual particles as in physics. Like you don't model a stream of like let's say billions of billions, cajillions of hydrogen molecules inside your your water stream or like a physics simulation your model the actual bulk properties with like an aviastix and the same thing I think is true with large language or like let's say any large model because and that makes it really very difficult to actually make like insightful statements about

1:06:22Like what architecture leads to watch because I don't think it ultimately matters that much I think it matters way more kind of the scale and of course it might be that I'd say one architecture just scales more nicely like one architecture needs a billion parameters the other one needs a hundred billion parameters because the the other architecture is just less nicely scaling it's that's probably the reason why we do is we choose deep neural networks and not just a shallow one which is really really really wide but I don't think that in the limit, and I don't mean like infinite limit, but in the large scale limit, it makes a difference. And I think it's much more kind of how you would expect the information to proceed through these models, especially in diffusion. I can say with these diffusion models, in diffusion models, we have this structure that they are doing, how was I called? They are doing KL diversity minimization in a Wasserstein space.

1:07:22So you would expect also at the higher information things to appear first because they are the free wins I don't think it has to be like that, but it's but I would expect it to be like that the same thing like a rectified flow in rectified flow I don't think it should be that surprising that if you basically do a linear interpolation of big things appear first but I am not sure if I could if I could advise people people or like in good constant advice people and say well you should do this and that for your architecture because I don't think it matters. Very cool. So we should move on to theories of deep learning. So much to our chagrin. We've tried many times, I mean, certainly God knows, we've interviewed many people who have said that they've got a theory of deep learning. But you've said to me that roughly this can be broken down into parametric, a parametric view and a functional view. Explain. If we look at deep neural networks, there is, of course, a very, very obvious way, like a first instance where you say, well,

1:08:19This is just a huge non-linear program. You look at this as you would in like the optimization people. The optimization people, they say, well, I have like a huge, let's say polynomial on this case, a huge parameterizing a redo network. And we now look at, okay, what are the properties? What would need to happen for this to converge to something? What do we need to do to figure out a global optimum? What do we need to do to uh to effectively model the change in parameters over time. And like this is ultimately a really really interesting way of thinking about the problem but it's not necessarily that useful of a problem. Now there are of course counter examples things where we actually look at kind of the parameterized nature of these and we can actually make some bigger inferences which is effectively where like the the optimization people kind of look into deep learning especially

1:09:11If you look at, for example, Spline Theory of Deep Learning, if you actually look at what the mathematicians are doing, all the combinatorics people, they all say, well, that looks like something I can do. And ultimately, there are really, really interesting things there as well. But I don't think they give you that much insight. They can give you statements like, okay, what is the actual complexity of a learned model? How many of these linear regions exist?

1:09:37something like this. They can also sometimes get some really nice explanation. Like, I really like the, uh, the paper by, I think, Ballaserio who did the explanation of why batch norm works. And he just framed it as, well, batch norm actually concentrates the things in the middle. And therefore all the hyperplanes are more concentrated in there. It's more linear, more non-linear. But I don't think it's ultimately that insightful because at some point, again, bulk properties.

1:10:04it doesn't matter anymore what an individual parameter does. And then we very quickly go into like this domain of saying, well, if parameters don't matter, can we get rid of parameters in our analysis? And that then lets us walk into something like the neural attention kernel theory. It allows you to, like they are some randomized, like randomized sketching type results. And it's also like mean to your theory, which I really like from an analysis point of view. I'm not sure whether it's that useful.

1:10:32Ultimately, but I really like it from like a mental model because what they phrase it like is they say well if we look at like a let's say a one hidden layer MLP very simple We can view this as effectively every input neuron is like it's this inner product with an activation around this so in like we can view this as let's say a thousand let's say test functions So the activation function is let's say you make a sigmoid to make it very very instructive between zero and one and figures out well is this So this weight, this inner product is just test successful. And then the next layer, there's a combination of all of these. What we can do is we can phrase it as what happens if instead of having 50 layer, 50 input systems, we have an inferred number. If you put this into the inferred limit, you effectively now write this down as like a density, right? Because effectively every single point exists, it just matters kind of how present it actually is, kind of how relevant it becomes. And now,

1:11:30we have this structure of effectively fitting a density over weights and putting a density over these weights. And that is very nicely to analyze because now we are back in like literary diffusion style problem because now we can model the change in density over time as a change in distribution over time, which is ultimately a diffusion style problem. And that in from a and neurotransmitter columns are very similar to you. Just instead of looking at this from a density point of view, they simplify a lot more and say, well, let's actually make it a function space. And the function space then spans the area. Now, there are some limitations with this analysis. For instance, neurotransmitter column theory assumes a column is stationary. But even that is an assumption that we make for analysis point of view. There is nothing that fundamentally hinders us

1:12:24in looking at the kernel itself changing. So, in fact, we have two dynamics, where we have both the kernel change dynamics and we have the kernel itself describing the local function. And this is also what mean-field theory does. Mean-field theory, effectively, in the back of the big paper, they have a section where we talk about, well, actually, the connection to neural tension kernel theories, we just assume the kernel also changes over time. And that, of course, makes the analysis way, way better.

1:12:54that you can now effectively also model not only the lazy learning, but also feature learning domain, but has a huge limitation now that it becomes much, much harder to analyze. Still, I think that from an analysis point of view, these end up being more useful, specifically because the predictions end up lining up a lot better than what we would expect for the parametrized world. Like in parametrized world, we say a huge number of parameters, this thing is going to overfit instantly. Or like this thing is But it's completely over-parameterized, why do you do this? From a function point of view, you say, well, no, it's not over-parameterized. I'm just going closer and closer and closer to the actual true function space I'm fitting to. A function with, let's say, a billion parameters is much, much closer than a function with 50 parameters to being at this infinite scaling domain. So effectively, the function analysis really

1:13:49explains this idea of overparameterization while the parametric kind of says well actually this overparameterization might be bad but we can exactly explain kind of how the local architecture is. However I would imagine that in the long term the functional analysis people they are going to be more useful just because they explain a specific architecture but they explain architectures in general.

1:14:18got no attention call, it works for pretty much anything. I approve works of literary anything, but the idea of looking at this from a function space point of view works even if you have like, let's say we all go back and do like MLP mixers now. We no longer do transform, we all do MLP mixers. It still would work for a theory point of view. Because we don't know, we no longer use reloads. In fact, we don't no longer use reloads. We use like these skated activations. Lots of like research in the parametric world, no longer really makes sense.

1:14:47because they all assume kind of this spline-like structure. Instead, like the function space theories, people say, well, the function space looks a little bit different, but it's still like, ultimately, it's something like a baron space. And then it's the analysis that works. Yeah, I mean, I suppose I'm just curious to know what would a good theory of deep learning resemble?

1:15:11you know maybe it's something that gives us a good explanation i mean you were just saying with this functional analysis you know the kind of questions we might ask are.

1:15:19Is this network overparameterized? Or we might want to ask, is this architecture optimal? And it feels a bit like the blind men in the elephant. So we've got folks like Randall coming at it from a geometric point of view that there's a lot of people doing that. We've also got our good mutual friend, Keith Duggar. So he's coming at it from a computability point of view. And he's talking about, are they Turing machines? He's talking about the space of algorithms that they could possibly represent. And then there's things like complexity views.

1:15:48There are folks looking at statistical physics, talking about spin glasses and heavy tails. You see what I mean? Even effective theories folks and complexity theories folks. So these are all different perspectives on the blind elephant. Which one is best and why? I ultimately think that the most important thing is that it makes the correct explanations. It comes up to the correct solutions. I don't care that much over the specific structures.

1:16:16I don't want to sound too negative on the parametric point of view. If it makes good predictions, then I'm pretty fine with this. I just don't think it is as tuned to being productive. But the important thing is, and I think it's also missing for many of these theory papers, is that you go and you actually check whether the test works and basically you try to falsify your theory, or try to figure out where does this theory actually break. In this case, I would actually think that something like physicsy is a lot more useful just because this loop of a make-up prediction and a go, a make-up experiment, I think this is much more useful. And I think that the big thing is that theory will never be able to explain literally anything because it might be able to explain things on a pure functional point of view. Let's say the mean field people, they can completely explain if I have, let's say,

1:17:14a density composed of let's say a Gaussian mixture model composed of this thing then I can completely tell you hey you need this and that size or like add this and that size the error to the infinite model will be this size completely fine the problem of course or same thing with like the SVM people like kernel based people they have like source and capacity conditions which effectively is how much does your function have how much does your data have the problem is in practice we will never have that information for our real data. In practice, we just have maybe we have a clue, right? I think we can oftentimes kind of estimate, for instance, we can often estimate how many clusters there could be in our data with relatively simple methods, but we can't really have very deep information into our structures because ultimately it's still the real world and we still approximate everything as theoretical cows. But the

1:18:10The physics side would then say well We try to do kind of we do like a best effort and we try to solve everything that is solvable as predictable and Based on that we can still make real Kind of predictions for the real system and which is effective in the transition from physics over to engineering and where the the distinction lies Doesn't it ultimately it doesn't matter even if you can actually make a hard cut And I think the important thing is that we need to have something which makes actual predictions. And for that, the loop between the theory people, which do like the, the, the, the really nice things are like mean fields, uh, nor attention calls. Um, also stuff like diffusion, like diffusion is also, it's a non parametric, uh, exp, uh, explanation of what these models do. You just say, well, they fit some density reinforcement learning. What I do is also fit some density and be happy. But the, the actual loop should be.

1:19:07to actually take this and trying to make inference for the real world. Like if I prove something converges and I test it and it doesn't, then we would then kind of, I think this is where the, like there should be a deep theory of deep learning. People will look at, we should now go and say exactly that. We need to explain this. The problem is right now it's really, really difficult because we have so little theory that the reason it could not work or it did not work.

1:19:35could be just an implementation error because we don't have a good enough idea of what we do yet. I previously mentioned that I basically wasted like a two or three months with an implementation bug in one of my papers. Just the fact that we cannot figure out whether there's an implementation bug in a continuation of existing work, that alone should mean that theory is not yet at the level and this is what we should be looking at.

1:20:03I don't think that's really a path right now, which gives us that amount of progress. Not the non-parametric people, not the parametric people, not the mechanical interpretability people, no one can tell us. Yeah, I guess we live in really strange times. We don't even know what to believe anymore because we had all of these.

1:20:27All of these things that we used to say confidently a few years ago, and yeah, we've got things like the double descent phenomenon. We've got the mystery of overcoming the curse of dimensionality from a sort of linguistics point of view. People talk about systematicity and productivity and why are the models syntactically competent? Why can they produce all of these sentences that have never been uttered before? Why are they learning these abstract models? Are they too incomplete? All of these different things.

1:20:52And we need to have a theory that kind of explains this phenomena. Yeah, I think there's also a big problem of that a lot of the research is really disjointed. And it's not like in physics, we have a standard model of physics, and we kind of we work around this. If you if you go to school and you learn the standard model, you can then basically start, okay, we have a common starting off point. Like a common example, for instance, that People say, well, why does deep learning kind of be the cost of dimensionality? Well, I can tell you, like Baron back in like, like Baron's theorem, they literally prove that the actual convergence is not dimension dependent as a dimension free bound. It depends on kind of the frequencies of the target function you're fitting and kind of intuitive. Like once you think about this actually also makes sense. And if you have a high dimensional function with constant almost everywhere and then it's much, much easier to fit if you have a two dimensional function, which looks like a heartbeat monitor. And in fact,

1:21:48I think the same thing is true for most other estimators as well. Like if you think about like a decision tree, same property. The high dimension, if it's constant all dimensions but one, then still it's dimension, it doesn't care about all the other dimensions. So, but you would need to have something like a standard work, which actually kind of looks at this and combines all of this. So something that doesn't figure out like the next paper, but something that actually goes and writes down, kind of tries to figure out how do all of these things can fit into one coherent narrative. And I'm not sure with a deep learning as it's, or like machine learning as it's positioned right now is actually equipped to do that. And because right now the pace is so fast. And like you, you have like the effect of your three big conferences every single year, if you don't couple, uh, count triple AI count that we have four. Effectively you go from conference to conference to conference to conference and you end all of these papers, like

1:22:44I don't know. Maybe it's just me. Maybe I'm just unlucky. But if I look at the papers I'm reviewing, they're really incremental all of them. And I don't fault all of all the people there for publishing like this, because first of all, it gets rewarded. And secondly, it doesn't make sense to go and say, well, I'm setting up a big deep learning theory. I write this book.

1:23:04Over my phd, like I don't do cumulative. I actually do a manuscript I have in the end my like the 5,000 pages big book would categorize everything that is to know by the time you're done It's going to be completely out of date and useless like probably lots of things you write in there are completely wrong so I think that right now there is no That there is no way to have a of course we have a theory of deep learning. I talked about the this paper like this, there will be a theory of deep learning. It's kind of a vacuous statement because arguably we already have plenty of them. The problem is more that to have something like physics have, I don't think the entire field is set up to do that. And I think that it will not be set up to do that. And up until the point where things start slowing down. And can we just have a quick digression on the manifold hypothesis? So our mutual friend, Randall Bellastriero,

1:24:00He's not a fan of the manifold hypothesis so he's got this spline theory which is basically that in an input sensitive way the. the the ambient space gets partitioned into these little honeycomb regions and and you are helping me understand this at the time so you get this weird teleporting thing happen where like if you change the input a little bit you'll teleport into another honeycomb somewhere else in the space and it's kind of anti-manifold hypothesis but I spoke with Good Fire last week they're this interpretability company in San Francisco and they've got all of this work about the geometry that these models learn so for example days of the week and months of the year

1:24:37they can take a PCA projection of the activations and then they fit this kind of Bezier curve to it. And they see these beautiful kind of geometric structures in the activation space and they're really kind of manifold, peeled. So what do you think about that? I think it's really hard to make such a statement. Like whether it exists or not, something like the manifold hypothesis. Like if it's not the manifold, like if it's not like a low dimensional manifold itself, something like this has to exist so we can actually compress things down. And ultimately, I don't, I don't even think that Randall would disagree with me on that, that if it actually is like, if we actually have like, let's say the model outside, like the real world is the smallest thing there is, and there's no simplification, then what are we doing? But it would also push back against all of these explainability techniques.

1:25:30But generally speaking, they're not very robust. First of all, they often make extremely simplifying assumptions. Let's say the circuit people, like the transformer circuits, but they usually make the assumption that the actual routing is constant as you change, as you permit your, that's not true. You effectively model a transformer like a huge MLP mixer.

1:25:59I think there are many, many exciting assumptions. And I think there's even a paper like this where they show that if you actually change some, make some slight tweaks on like a setup of like these experiments, like the results completely flip. I think like all of these are just, they're not robust enough to actually make such statements of making statements of like the manifold hypothesis exists, but it's also robust enough to say it doesn't exist. I think it's really, really difficult. And I think it's also difficult because I don't know I don't know whether the language exists to realistically talk about these problems. Because there are of course people that go very much into the abstract functional space. I don't think this is completely true. It's often times very useful, but it's not completely true just because we don't have infinite dimensional functions. They are finite sized and there's a big difference between finite and infinite. If you have infinite dimensions, every problem becomes convex. In fact, every problem becomes linear.

1:27:00So, on the other side with the parametric people, we say, well, we just have these spline regions and that explains everything. But if it was like this, then we would need to be able to make predictions that say, well, if we go away from the splines, maybe it doesn't work anymore. Or to what extent does it stop working? And then there are the explainability people that say, well, we can actually extract this and that circuit, or we can extract this and that representation.

1:27:26Maybe one of the the best papers I saw in like explainability was the paper about Grocking where they where they extracted like the modular arithmetic code. Look at this. Curves, it completely makes sense. There are sine cosines everywhere, especially because they do it in the input. But if this is how these models actually describe modular arithmetic, there's not a chance we will explain how the model at duck. So I don't think there is a way of.

1:27:54of actually currently having just a language of describing, of describing what these models do. So I don't think that the, like something like a manifold hypothesis, I don't think it's true. I don't think it's wrong. I don't, I don't think it's a useful way. Like even if it's true, I don't think it's a useful way of thinking about the problem. Same thing is even if like the extracted, like grocking star representation, even if they're true, I don't think they're a useful way of actually figuring out what these models do and kind of how they get the solution and whether things work or don't work. Yeah, I don't know. I mean, it's such a tantalizing idea. You know, like there's this idea of a homunculus that we've got like a little man inside our brain or, you know, these grid and place cells, basically that we've got a little world inside our brain that actually captures some kind of geometric invariance, some kind of model of the world outside. And it's a beautiful idea, isn't it? That, you know, when you visualize these representations, for example, geography.

1:28:49You actually have some kind of geometric invariant of how the different countries in the world are related to each other. And it's fascinating that this would be convergent, that different model families would learn similar types of geometry that represent abstract structural invariances of the world. It's very, very exciting, but yeah, I agree. Maybe we're reading too much into that. We should move on to Reward is Enough and Constrained Arel. So Rich Sutton, the wonderful Rich Sutton.

1:29:17He had this paper out called reward is enough. What did he mean by that? The idea of the idea of like especially like the green force money maximalists is that they say well technically I can write down any objective I want as some form of reward basically just in doubt you literally just go and say well.

1:29:38You should be doing I don't you should be moving the arm like this. You can write this down as a reward basically It's one everywhere. It is or like if you move to a goal. It's one where the goal is in zero other wire and This is a very true statement, right? If I have an optimization problem Let's say I have a really bad one like a mixed nature nonlinear program huge like millions of parameters a huge numbers of constraints whatever I can still write down the solution really easily and that's I have my x star, so my actual solution, and just write it down as, like, minimize the error between x and x star. It's a convex problem. It instantly works. If it's a relatively low dimension, you can solve it by hand. Even if it's really, really high dimensional, the actual problem behind it is really, really high dimensional, really complex. So in that sense, the idea of reward being enough is completely true. But technically, it is, technically, you can define a reward function for pretty much anything that will

1:30:34to somehow figure it out. The question is more whether this is a realistic assumption to make, like whether having a God-given reward function, which just does something, is whether it's a realistic assumption, even finding it. And once we have it, whether it's realistically optimizable with any technique. And that is where I think the actual practicality or the reasonableness of this breaks down.

1:31:04And just help me understand is Sutton advocating for the God reward function or is he more practical? Is he saying that there are reward functions in different applications or whatever? So of course like the the God function was a very very extreme example, but ultimately what he believes is that he believes that you can figure out like a suitable reward function for every task and That is enough like we don't need to research all of these other complicated things But instead, if we just become really, really good at maximizing reward, then everything else becomes obsolete. I mean, I think it's a very, very optimistic view of the world. I'm not sure whether it's... I'm not sure how far he himself would be going with this. I don't want to put words in his mouth, but I don't think it's a reasonable thing to do even in the kind of infinite limit.

1:32:02I mean, give us a bit more context here. So when I was talking with Chris Watkins the other day, he was kind of saying that he met Richards at some conference in the late 1980s or whatever. And there were people in his lab at Cambridge doing all sorts of control theory. So for example, there's this cart pole balancing problem. And you can just solve that analytically. But this reward thing was different because it's basically, in this particular case, I think it was a binary signal. So it's either balanced or it's not.

1:32:30And it's a signal from the future. So you have to do a lot of stuff before that thing says yes. And at the time, they thought that this was an interesting problem to solve because this is much more kind of like a minimal to, you know, maybe it's a better representation of animal learning. Yeah. I think that's in general, there's a big overlap between control theory and what the reinforcement in people do. But there's also a huge distinction. And notably, the main difference between doing actual control and doing reinforcement learning is control assumes that you have, essentially you have a model of the plant or at the very least you have some idea of the behavior, like the actual response to an input. That is not true in reinforcement learning. In reinforcement learning you say, well, I don't have a model at all, I just have interactions with the environment. And that immediately makes the problem more, first of all, it makes it more interesting, but also makes it a lot more complicated.

1:33:29Because now you you no longer trying to solve the actual problem at hand. You also solve the problem of even figuring out what to do in the first place. A controller, let's say we have like a reference which it tries to follow. We have some model of the like plant dynamics. Let's say make it very extreme. We have an MPC controller or we have a or we have whatever else or like a linear quadratic regulator with some idea of kind of what these what this thing does, or if we have like a PID, we have like a human who has an idea of what it should do and how it should behave, and then effectively you outsource all the exploration. While in reinforcement when you say, well, I put this into the loop itself. And that gives you, of course, it gives you a huge amount of power, because now you can actually solve much more general problems. Problems we don't even have an idea of how to solve them in the first place. And this is also maybe where like the reward is enough

1:34:24people don't go in and say, well, if we can solve an arbitrary problem with like arbitrary definition, and we can do like the exploration we can do, we can do the learning, we can do like the optimal control, like why do we do it? Why do we need anything else? And again, while I think this is true, in the limit, I also think it's a very, very wasteful way of thinking about the world, especially since things like exploration become really dicey in many domains. Like I don't want like my like my open heart surgery robot to do exploration on first of all not me but not on anyone else either. So I think that the the actual real world complexity of needing to have either of first of all not allowing too much exploration in in the practical sense but also to give really really hard guarantees becomes difficult to do in like poor reinforcement learning. The example where

1:35:21where we can actually prove that doesn't work for machine, for like deep learning is we cannot give like L infinity control, like L infinity guarantees. So because everything is sampling, sampling and kind of absolute maximum errors don't really play well with one another. You can have a control which in expectation is really, really nice. It has one really, really huge failure case, right? So on average, something which tracks it perfectly and has one huge error where kind of the robot clocks you in the head.

1:35:51is completely the same as one thing, which kind of just slightly wanders around. And in fact, there's this nice paper about, what was it called, like, training deep re-unitworks to uniform accuracy is intractable, or something like this. Put it into Google, we'll find it. But they basically show that for re-unitworks, it's not even possible, or basically it is possible, but it's exponentially number of parameters. Because basically what they say is, when we have the reloads, if you have two, they effectively form like a hat.

1:36:19Something like this and just figuring out how to place and how to scale these hats is already like an NP complete problem like or rather the exponential amount of data to figure out kind of these these deviations from your reference and I think having something like this is It's necessary for many domains where actually like a like damage is really really really bad and having just an A chance in expectation to damage and get good things happen most of the time is often not good enough. Like you want stronger things where then suddenly you go into things like robust reinforcement learning or something like this or I do from a page that constrained reinforcement learning where in fact we do a little bit more just to get this little bit of an extra boost in either safety or performance or just including prior data like imitation and we just say well I have actually lots of data around and I don't

1:37:16want to do open heart surgery from scratch. So I think it's very naive to think about this now the right way. Yeah, I suppose this is the the other vexed issue with with our friend Rich Sutton is that he has this bitter lesson essay, which is basically that he doesn't think that we should be building knowledge and constraints into the system. He thinks that we should just scale models of computation. And I'm a big believer that constraints are incredibly valuable. And by constraints, more broadly, I'm just talking about knowledge. I think constraints happen at multiple levels of abstraction. So in the physical world, constraints might just be the laws of physics or whatever. And in our minds, the constraints might be things that we already know. And this guy, Chris, the other day, he was saying to me that one of the amazing things about RLVR

1:38:00is that it kind of, it doesn't solve, but it helps the exploration problem because the base LLM knows so much. So it has lots of conceptual understanding and rather than just blindly searching all over the place, because obviously back in the days of deep queue networks and whatnot, we actually needed to guide and put a lot of prize into the exploration process because otherwise not very many interesting things will happen. And now, of course, with the RLVR on the language models that they have a lot of base knowledge, but it feels to me that like, how you get there and how you constrain and shape the search space is critical to making it work. Yeah, I have the next property that I'm a very theoretical person, but I also work at like Fraunhofer, which is very applied. So I actually did some project, which I actually applied reinforcement learning. And this is a big problem in reality that effectively information can be very expensive. And in fact, I would say most information is very, very expensive. And

1:38:58Reinforcement if you completely say well constraints don't matter. I don't any prior information. I don't want to encode anything in there. Assuming there's even possible. That could be completely unconstrained, etc. and completely like without prior knowledge. Let's assume all of this is the case. We still have the problem of now needing to really relearn everything from scratch. And the reality is that we want to be gathering as much information that we already have.

1:39:25paid for and gathered with our blood, sweat and tears. We want to put all of that in the model first. And then we go and we continue from there. There's no point in basically starting from nowhere. There's also the reality that if you actually look at a reward function, like an actual industry-ready reward function, you will immediately see that they have constraints in there. They just they just reframe them as, well, it's a reward function because yeah, of course, like you can put them into as penalties. But ultimately, they're just kind of, you're just doing a lot of work to not write down a constraint. The nice example of this, which I always like to use as a, as a motivation for constrained RL, is if you look, if you look into the paper about, what's it called, OpenAI5, which is the like data playing AI. If you look into your appendix, there's a huge table.

1:40:19where effective on the left, they have all these different attributes, which they put into the reward function. On the right, they have some like explanations and so on. In the middle, they have just a bunch of random numbers. And these are, they have the weights for their reward function. If you tell me that there is a way to actually figure this out, there's not a chance. So especially if you look actually at what these reward functions are, like at the bottom, for instance, there's one, which says, well, the age, like the, there's a negative reward for being in the wrong lane.

1:40:48for a specific amount of time. That's a constraint. Because write is done exactly the same as saying, well, don't be in there more than five seconds or whatever. Or rather in mouse constraint reinforcement, you say, well, in expectations, if you take a thousand trajectories, don't be in the wrong lane too often or something like this. And there's another, like another nice example that can also run into trouble with reinforcement learning is above there is a section where they have like killing an enemy hero actually gives a negative reward.

1:41:18And the reason is, and they have an explanation next to it, is that killing enemy hero gives you a lot of gold and there is already a bonus for getting gold. So if you also had like a zero positive reward for killing enemy hero, you're gonna sit into the middle of the map and just kill the enemy hero over and over and over again without making any progress. The problem there is that two of these quantities can be completely correlated. So if you model constraints, you basically get rid of this correlation problem. Because if you, for example, write down the constraint explicitly of I should not

1:41:48be like there should not in expectation I should not have more less let's say more than how is it I should not have let's say 50% less enemy hero curse than my opponent in expectation but it's it's a small guarantee but effectively the stuff that you want to get rid of you already get rid of with that yeah I mean Leonardo da Vinci famously said that constraints give us freedom And I'm a huge fan of that. But you specialize in this mode of reinforcement learning called constrained reinforcement learning. Can you explain that? So constrained reinforcement learning is still a relatively up and coming field, I would say at the very least. So especially since there are many different formulations. Like the classical one is one of the constrained market decision process. So effectively what you have is exactly the same setup. You maximize the cumulative rewards. But then you have

1:42:41constraints, which just say the expected cumulative costs should be below a threshold. This is ultimately a very similar structure between the reward and the constraints, but there are also other ways of framing this. The one big disadvantage of this expected cost is, for instance, you don't get rid of long tails. If you say it should be less than 25 costs, then it could be that it has 50 trajectories of cost zero and then 50 of cost 50 so average out to 25 But it's not necessarily what you always want on the other hand for something like the the open AR5 case It would be exactly what you want right you want to have let's say some Games where exactly you are like everyone is in one lane and then some games where it's like the normal distributed amount If you want to kind of go further then there are also solutions which do like conditional value at risk

1:43:35which is effectively you look at kind of the probability of being in a tail and kind of you wait this by kind of how far out on the tail you are they are like percentile risks like it with 99% probability you should be within uh like some let's say 50% of some target whatever basically like the pack uh like guarantees and all of these have like different domains like I I recently wrote a paper which I have to re-write and re-submit, which is about doing this if you actually know the constraints completely. You have a complete model, like in an operations research setting, but you don't know the objective. You actually have a complete model of, let's say, your local gas network, or your local heating network, district heating network, but you don't actually know the exact costs of, let's say, solar versus whatever.

1:44:30Or like the actual demands and then you train or like you try to train based on that and there exactly you never are allowed to have any failure like the like the pipes should not burst they should not burst like with 99% probability it should just not burst and Kind of all of these guarantees you try to encode into into the training process itself that different ways of doing that so one of the very earliest was Basically, they went and looked at TRPO and said, well, we have this performance improvement bound. We can do exactly the same thing for costs. And this gave us like CPO, where effectively it says, well, if we are within the constraints, and we are with 99% of the probability after doing that set, we're still in there. Problem with that, of course, is, yeah, well, if we are not in the constraint set, then effectively it just has to do like effectively.

1:45:20greedy descent effectively it replaces the reward of the cost function just to descend back down and you basically you see some like oscillations on the the boundary something like that. Then further ideas went and said well what if we do projection right we we effectively have like an approximation of where we expect the the cost surface to be we project back onto it that has nice so the nice property of this is constrained reinforcement is actually completely convex. Again this is because we assume we have densities We don't have parameters, we have densities. So in density space, everything becomes linear and convex and whatever, nice. So projection actually makes sense. The downside of this is that you can have some really, really bad runs. And the reason is, what if your effectively constrained system is hallucinated? Like you hallucinate a constraint somewhere with something attacking like my recent paper and like safe MPO, you hallucinate a barrier. And suddenly the, like your constraint model, it never walks in there because

1:46:18It's very careful to actually project around. In fact, there's a problem which becomes worse the better your solver becomes. It's like the salucination problem. And then of course, there are the final people who just say, well, we have zero duality gap. Why not just use Lagrangian approaches, which is like this min max style? This works. But then you have the problem with also this oscillation, right? Where effectively you go down and you oscillate around because basically you're below the cost because your Lagrangian was slightly too large.

1:46:47then you're very safe. So they're granting drops and you go above and you go below and so on. You have these big or small oscillations. There are methods that try to get around this. Like there's something which basically has PID style, like it's tuning. That's pretty interesting. But ultimately you're still running to exactly the same issues. But you have the oscillation errors in there. And that's fundamentally because like estimating Lagrangian from data, like constraints from data is really difficult. Because we have this variance in there. And then there's like a final family, which I'm also a safe MPO paper published recently in there. That is like CBPO, like this, or like, what is the other one, like four cops is another one, which basically try to do, like effectively local approximations and try to solve the problem locally optimally. And then they train the model to actually just emulate this locally optimal solution. And

1:47:44That has some nice properties. It can be very, very precise, but now you need to be really, really careful not to maneuver yourself into, uh, like a effectively unsolvable setting where like your subprom is unsolvable because you had a bad batch. And of course that's model base, but that's model base is easy to explain. You effectively just do model predictive control, but with constraints. It's really hard to do that properly because the bad thing is you could be in a setting where basically almost every proposal you do is just unconstrained, uh, with, uh, outside the constraints, but Ultimately, the the arguments are very similar. Yeah, I mean, I suppose the million dollar question is given that you said before that we already have constraints in normal reinforcement learning. What what's the cash value of explicitly modeling the constraint? So the big advantage is so what you gain is, of course, to first gain the ability to more accurately, especially if you have like experts, like you go outside, you talk to some engineer, he would be much, much more easy talk, easy talk about when you talk about constraints, and when you talk about

1:48:41let's say tuning a specific weight of a reward function, especially since constraints oftentimes can be known ahead of time with like high certainty. Like I'd say a robot, you know that a certain load just will break the robot. You know that even without trying it out. And if you try to do this with like the effectively tuning, effectively, there's a very high likelihood that you have accidentally something which goes above the threshold, especially since it's oftentimes really, really difficult to think about these coefficients.

1:49:11The second thing, and the second big thing is you get composability. Because one of the problems I talked about with the OpenAI example was that if two elements of your reward function are correlated and, for example, you add something to it. Like, let's say you didn't, at first, you just had a bonus for killing, and then you did not have a gold bonus at first. And now you add the gold bonus later because, well, during engineering, you found, well, that would be a good idea as an additional guide.

1:49:39Suddenly, your entire system breaks and you need to retune all the weights, because the weights are tied relative to each other. If you model as a constraint, let's say you should have more gold than your opponent in expectation, right? That's a reasonable idea. Then...

1:49:58Then you would not be to retune was effectively like you still have these coefficients in the Lagrangian setting, but they get automatically tuned to enforce the actual target that you want, not the have target you implicitly encode with these tiny weights. Very cool. And so I'm really interested in constraints. So you know Jeremy Bud Bud, right? He's also in the annex discord server. And we wrote this article called why creativity cannot be interpolated. And basically the idea is that, you know, creativity is all about deeply respecting constraints and that representations are path dependent in some way. You might have read that fractured entangled representations paper from Kenneth Stanley and it's kind of interesting because neural networks, they seem to have soft constraints. So they learn this statistical distribution and then you sample from them and they kind of respect the constraints but they don't sort of, they're not hard constraints.

1:50:48And I visualize problem spaces a bit like a maze. So a maze has walls in it and the walls are the constraints. And if you respect the constraints, then you don't walk through the walls. And you can be creative because you can just take paths. You can take trajectories through the maze and you can find a new undiscovered subspaces if you like, because you're not constantly wasting time walking through the walls and not respecting the path that got you there. So, you know, I have this deep intuition that maybe it's neurosymbolic, maybe it's some kind of way of hardening the constraints that we learn within particular domains. And so it's unclear to me whether neural networks could in principle have these hard constraints or whether we would always need to somehow enforce them in after the fact. That's a really interesting question. So generally speaking, if you can encode your constraints into your architecture, you're probably going to be a lot more well off.

1:51:42Classy example for like reinforcement you just you do action masking or kind of you scale your actions to be in the range which is reasonable The the question is usually that there are things outside the model which actually inform the constraints So for example reinforcement learning like actually having action constraints pretty simple like effectively just project onto some set the difficult thing is kind of having Effectively state constraints or something in the environment Because now, you can no longer just enforce these by going and modeling them in your architecture. You have to be able to interact with the state and the architecture together. There's sometimes ways of actually also encoding this into your model. But whether this is generally possible, I would honestly doubt this. Especially since there are plenty of constraints, which are NP-hard to enforce. Like I'd say, we have a CSP-style problem. This is NP-hard to enforce.

1:52:38how are you going to do that in a static architecture? Like mathematically, it already, it breaks from a complexity point of view. Of course, you can then do things like effectively do things like a diffusion-esque thing or like a Monte Carlo-esque thing. But fundamentally, then you still need to externalize some structure. If it's of the constraints, that's the way you actually generate the output. So I do think there is something to it with this idea that constraints can help doing like sensible exploration by basically removing things from the state space which are irrelevant. The complexity here is kind of actually forming how to compare these unconstrained with constrained settings because like fundamentally if you look at literature there is the paper by Altman who did like his PhD in like a constrained reinforcement learning about like the actual like we mix into a programming or like LP style and one of the things that

1:53:36was even known back then is that constraint reinforcement is a lot harder. Specifically because there is not necessarily a stationary deterministic policy anymore. I have to be careful with this. There are non-stationary deterministic policies and there are deterministic and there are stochastic policies which are stationary. But stochastic policies which are stationary. But basically this means that already this idea of doing something like Q learning does not work, it does not generally give you a solution because basically you can get into some really really if you think kind of at the boundaries. So technically if you look at this and if you ask them they would say well mathematically it's not harder but of course this is kind of it's a wrong comparison because you're comparing a constrained problem with an unconstrained problem. A constrained problem of course is harder because it needs to do more things. So it's so

1:54:31the actual comparison would be something like I have an unconstrained problem and I have a constrained problem, which is both of which are really, really difficult potentially, but then compare them on the same task. And that becomes difficult because just designing a reward function, which actually implicitly informs a specific constraint is difficult. So maybe it probably would need to do it kind of the other way around. Like you would need to have the RL agent actually figure out an optimal solution. And then we need to go and, and based on kind of the, the violation and so on, you would need to define the constraints. But then of course you start off with the constraint or a method on the back foot, because it basically needs to emulate the, uh, the unconstrained model. So I think it's really hard to compare of whether it, it makes a difference in.

1:55:25Whether it makes an actual difference it might just be that for solutions that we see it makes a difference But it doesn't make sense in the absolute it doesn't make difference in the absolute setting But I I can't think of an easy example Which does not like an easy experiment which does not fundamentally disadvantage either one either method Yeah, I mean I often think about when constraints can be harmful and I often use this expression I got this from Marvita Chiramuta So she said that reality is protein And what you meant by that is there's this ancient myth in Greek mythology about this guy Proteus and you could pin him down and he'd have to answer your question but when you let go he would shape shift. So there's this thing that with problems in the real world when we interact with them they shape shift and they're non-stationary. But what we try to do though as mathematicians and scientists is we learn knowledge and representations at an increasingly high level of abstraction.

1:56:21And the purpose of that is that the knowledge is less shape-shifty. So it kind of feels like we almost need to have a spectrum of representations where we admit fewer constraints because we allow for the fact that lower level representations change as we interact with them, but maybe we privilege higher abstract representations that are more known and and invariant. Yeah, I think there's generally something to it of Basically, once you have abstraction, it becomes really, really difficult to effectively the fundamental constraint that you want for a hierarchical abstracted system is that the lower systems guarantee some invariant, but the upper systems can actually guarantee.

1:57:08This is also my opinion one of the main reasons why like hierarchical reinforcement learning never really took off because the lower level does not give you enough guarantees for the higher level to actually exploit. This is why in control you can do that. Control gives you very strong guarantees in doubt. It can have like an agent-finity controller which gives you like effective worst case like effective adversarial robustness and you can still work with this but the same thing cannot be done with reinforcement learning because the lower level it just does whatever and the upper level just needs to deal with this. So I think there is something to just effectively in an abstraction hierarchy to basically, to basically make a statement that the lower level just has some guarantees. I don't fix what the lower level even does. Like I don't care. It has to uphold a certain thing. And that is something that's a constraint that I can basically rely upon when I build my abstraction on top. And we see this to some extent already even in deep learning. Like many people

1:58:06invoke the universal approximation theorem. Very few people have actually read what the universal approximation theorem actually does. But many people invoke this because they basically say, well, the invariant is, if I have a large enough model, it's just going to do the right thing. Even if it's not technically what the universal approximation theorem says. But still, you need to have, in an abstraction, you need to effectively have a stable interface with the lower level things you don't care. But you need to be able to rely on these. And this could be also seen as a constraint.

1:58:35does the constraint comes from the prior methods, potentially also from the upper level. Generally speaking, you have a level below you, which has a constraint in the form of capability, and you have a level above you, which is a constraint in the sense of need. A robot needs to not crush you. Where do our constraints come from? How do we find these incredibly distilled, evolved, abstract representations? I think that the main difference between a good and a bad abstraction is again its usefulness. There are plenty of abstractions which never actually took off. We only see the good abstraction because the good abstractions are the ones that are actually used. There's a long tail of ones which were started and kind of forgotten about very quickly. Ultimately, it would be an example to not piss off too many people. Fuzzy logic.

1:59:32So, a piece of people, but the fuzzy logic people, they're not enough people to actually complain. A fuzzy logic was an abstraction for basically decision making under uncertainty, which never really took off because it had some limitations in actual usage. For instance, the reason we use, it's a Bayesian inference, or probability theory in general, over fuzzy logic. It's because fuzzy logic has an un-nice property that You can't do fuzzy logic to do inference on fuzzy logic. For Bayesian, you can effectively do Bayesian inference to learn a Bayesian model. I think that the reason we see certain abstractions are not other abstractions, it's just because ultimately someone picked them up and they worked really nicely. Also, of course, on the contrary, means that plenty of abstractions would be really, really nice, which never end up seeing the actual time of the day.

2:00:29they just were never discovered by the right person to actually use. Well, I think this leads us on quite nicely to world models. So what is a world model? Is it a compressed latent dynamics model? Or is it a density over observations? Or is it something else entirely? World models have exactly the same problem as JEPPA because it's such an absolutely overloaded term. It can mean literally anything. We have seen people talking about world models which are effectively extensor.

2:00:58type models where effectively we have let's say a robot which has let's say like a tactility sensor or whatever and then it tries to predict forward kind of what these things do or like a robot and it tries to actually do like the motor predictions and so on or like the effect of where the end effect that would be given some input and we've seen things like well most of which are effectively just image or like video diffusion models Then we have seen more world models, which end up being like the japa style where effectively you have some, some internal state, which then of course gives the problem of you don't know whether the internal say is actually useful. Many of internal stage are just not, they look nice. They are consistent. They're not useful. They're hard to benchmark. Then there are other people who say, well, we actually do like reconstruction. And they say, well, we actually reconstruct things which we don't actually end up using, but.

2:01:49All of these are still world models and have also seen people like the the new dreamer Paper, which I don't know at what version they are. Well, in fact, they they pre-train a large video diffusion model on like all the data To play Minecraft I've seen people call this a world model. I've seen people say well, it's just model based RL I I don't think there's a reasonable difference anymore They have to cut that I personally would make is if the data collection is part of the loop I would call it model-based RL. If it's out of the loop, then I would call it a world model. And the reason is that, again, it's basically, it's the control versus reinforcement learning setup, which I mentioned previously. In control or like model predictive or like non-linear model predictive control, you assume you have all the data, like you have your plant, but you don't need to do any exploration in there. You just, you hammer on this, but for, okay.

2:02:45a model based reinforcement learning, you would also discover the world model, and therefore you need to do exploration. And generally speaking, I think, but there are also people who think differently, say a world model is just a thing which is trained on a lot of data to predict something temporal, which is not language. And I don't think there's a better description. Yeah. I mean, so many people are talking about this. I interviewed Nvidia about their Cosmos 3 Omni model, which they called a world model.

2:03:13And they were kind of saying, well, you know, it's not just a world model. You can use this as the basis to train a specialized world model. You know, you can use it to build your robot policy for doing whatever. Our friend, Yanlacan, he's talking about world models all of the time. Is this just branding? I mean, what's going on? Is this anything new? I think so. I think at world models and so.

2:03:35Both JEPPA and WALTMOS are ultimately branding. There were JEPPA-like models before JEPPA, and there were world models before world models. In fact, at the Schmitthuber paper, the world model Schmitthuber paper, this is model-based RL. So, arguably, people that now use it for something else other than model-based RL are just using the name wrong. I think it's mostly branding of techniques which are very loosely defined. They are just fitting some form of dynamics model, whether the dynamics model is probabilistic, deterministic, whether it's fit on like, it's a visual data on audio. I haven't seen audio data, but probably there's exist one, um, on, on, on whatever, like they're all called, um, they all called world models. It's just, it's a dynamics model, which protects the state from the, like the next state from the previous state on an action. And I don't think there's much more to it. And if that's it,

2:04:34then people have been doing this for a very long time without calling it a world model. And how does this fit with the whole RL story? I think that it is a little bit orthogonal because, so I always hear like Lacan saying, well, we don't need RL, we just need world models, right? That's like the, that's similar to the, uh, yeah, but it's just like, if we have a world model, which describes the world perfectly, then we can just use that and we don't need to do RL anymore. We can just do planning in that world model and we are fine. So we shouldn't spend all our time doing RL. The problem with that is, of course, it's a very, very simplified assumption because I can give you a perfect world model of chess and you still cannot solve it. You will still not be like Marcus Carlson because there's a difference between being able to predict the future and being able to actually make

2:05:28inferences for what is the correct action to take. And I think this is very much underestimated, mostly because the tasks that are done with world models right now are not that difficult from a control point of view. From a control point of view, like if you actually go, it was at RIG, which is the German Robotics Conference. And talking to people there, one of the big challenges is having doing effectively a sensing part in a reasonable way which also gives you relatively strong guarantees because if you don't get strong guarantees you might crush someone and secondly getting sensor information was always the bottleneck or for most time the bottleneck to give an example there's still not a good way of we're doing tactile like high tactile sensing like the like the reason you see like these

2:06:24Like Chinese also American German whatever humor robots where they always do things like backflips and sprinting and so on and never just do the following task Just stand up go into the next room open up the fridge kind of look in the back pick out a carton of milk and bring it to me The reason I don't they don't do it is because that would be a much much much harder task because they have this This this contact rich environment, but it's hard to model And it's hard to gather data. The bet for all of these like world model startups or world model papers is we can gather all the data you will need and just kind of distill everything. All these startups can distill this into a model. And I don't think this is kind of, I don't think it's a reason of expectation because the world is just, the world is so strange. I don't think this will be that reasonable. On the other side of course,

2:07:17I think you can get very, very far with this. I think especially like world models with like reinforcement inside the world model, you get nice information, you can do the simulation really fast. You get potentially things like greater information depending on how you set it up. But mostly you can do like, you can do any third rollouts. It's a, it's a huge, it's a paradigm shift for reinforcement learning, but we can also see in something like large language models, they, something like GRPO, which effectively does, doesn't learn a value function. It does like the.

2:07:43the advantage estimation on the fly, which is simulating a bunch of stuff that doesn't make sense in the real world where every simulation stuff is hugely expensive. It makes sense if you have a simulator, which is really, really cheap. You just roll out everything, you run GRPO, and you get amazing results just because you can hammer data on it. You still have the same exploration problems that you still need to figure out what is the optimal strategy for chess, but you have so much more ability to do experiments.

2:08:11And that changes the types of algorithms you want to use. And just to double up a bit though, did you see last week and note to my editing agent, please pull up a video of this. It was on Twitter last week. I forget the exact name of the company, but they were talking about physical prompting. So they basically built this. It was like a foundation, omni model type thing. So it does like actions and video and God knows what. And they had a robot arm and they were teaching it to do things. So they gave it a demonstration. They said, you know, you pick up this ball, you put it in the bowl.

2:08:41And apparently it had these emergent abilities. So for example, sometimes the ball would get stuck to its hand and it would actually use its other hand and it would brush the ball off the hand. And maybe that's a different thing because you're kind of providing constraints or providing demonstrations. It kind of figures out, you know, how to do particular tasks. But I was quite intrigued by that. Yeah. So I haven't seen the video talking about, but again, I would imagine that if we imagine that these models generalize, then yes, we could imagine things like that, especially if we do like planning in

2:09:10If you do planning on a sufficiently large world model and assuming we can actually do assuming assuming compute does not play a role and kind of latency does not play a role. And if we have all the all the data for the specific type of task. I can imagine any NPC is ultimately a solution like this. If you go to a university and you look for like the like the advanced control theory lecture and.

2:09:35The lecture after doing NMPC, they're all completely NMPC-pilled because they say, well, I don't need to do basically anything. I can just, I have my huge nonlinear model. I just throw something like MPPI against and it just works. And I don't need to do anything else. And then they notice, well, actually this takes a while to run, even if my model is really, really small. And I have all of these local optima. And if I add constraints, then it's very, very easy to have a model which kind of, We're actually like my most of these factories are proposed are actually not not really feasible and then okay I can do like I can do optimization over this But now I need to make my model very small or like have a specific form because if it's not hasn't have this specific form it doesn't fit into like a linear programmer or whatever and Suddenly basically the opinion starts to dim the more you use it and I'm feeling that the same thing is going to be true for like the the world models I think there's going to be a lot of

2:10:29like really really easy things like the low-hanging food which are just going to be like grasped it might be that for many tasks this low-hanging food actually is already enough but I have a feeling that we are going to hit some really really hard limitations not necessarily by the fundamental idea but basically by the nature of how the real world works and kind of how how diverse tasks can become and how How complex tasks can become because ultimately like things like moving around balls on a table like yeah I can imagine that working I can also imagine the same thing working with like an NPC controller for instance like the example I could say it gets stuck and use the other hand to move it around an NPC controller can come up with this It would start planning and would figure out okay I need to do something like this assuming that the model is accurate enough and you actually do that complete an NPC that's

2:11:26I think that's not the argument. The argument is, can we make a useful enough model with strong enough guarantees to actually deploy this in production? Because deploying this in production in production could mean kind of at home. I don't think this is where we will see these robots, at least not at first, not at least not in the short to medium term and probably also long term. I don't know, but if we look at the production setting, If you go to, let's say, BMW and you say, well, I have a robot, it does the right thing, 90% of the time, the other 10% of the time, it does something else, they will, like they will say, no, I will not pay you that money for a robot. I'd rather spend that money to kind of fine tune my CNC G-code for a couple of seconds than, because it will give me more effective speed. Because I, like the...

2:12:20The level of guarantee that you need in the real world is much higher than what people actually think. If you look at most of these things, which actually do a complex decision making, like large language models, they're all like a software setting. We can roll back whatever I want. And even there, like if you look at Twitter, as routinely people just delete their computer because they did not sandbox it, right? But if we are in the real world, kind of concept of deleting your computer would be to crush someone. So the level of safety is a lot higher, that you need a lot of reliability, especially if it like tasks to do over and over again and need to work over and over again. And arguably I would say that the peak accuracy matters very little. What you need is you need some form of like in every state, there is a minimal probability of being correct because then you can start doing things like effectively

2:13:16You can flow a common project answers and do like multiple measurements and actually get the right results. If you have 51% accuracy and do whatever many measurements, you drop your accuracy or like you drop your probability of error very quickly. And I think that right now all of these methods are not set up like this. I think they're set up to look very nice and demos and to actually do some really impressive stuff, but How do you know like how many times it failed to do something and what is the of course the consequence for like? juggling balls really low, but what are the consequence of doing something wrong and In like the physical world. This is this a lot bigger of a problem Wonderful. Well, I'm Alexander Mattock. It's been an honor and a pleasure. Thank you so much for joining us today and folks at home if you want to catch up and chat with Alexander So your your natural habitat is yannick cultures discord, but

2:14:08You do occasionally come into our Discord as well, and we're very grateful when you do. But thank you so much for joining us today. Thank you.