Machine_Learning_Street_Talk_MLST_Subbarao_Kambhampati_Do_o1_models

0:00So much so that we now have phrases like fractal intelligence, you know, in fact, I think Andrew Calpati basically was saying LLMs have fractal intelligence what the fractal intelligence is basically we don't know when they work they work when they don't they don't that's fractal intelligence and that's sort of shows which is good still we had nothing like this before but part of the science of LLMs has to be to say something more than fractal intelligence saying here is the level to which you can depend on their results so I'm not So, there are in reasoning in logic, there are ways where basically, formally characterizing the limits of reasoning, like limited depth, limited look ahead, reasoning and so on. None of them seem to work for elements. The question then is what would work? We have to figure that out. The better lesson is over and efficiency is going to matter.

0:55And I completely agree with that. I've been arguing this for a long time too, that think about the following thing. The first time when we sent a man and a human to the moon, caste was not a consideration. We want to show that we can do it. NASA was the one which is doing it. The second time, and for the space as well as the moon. The second and third time, etc. may be okay. But by now, it's Elon Musk sending people to space and supposedly possibly to the Mars too because the cast matters right essentially it's once it's been done then you start getting about you know the cast that you're paying and computer science is actually quite a bit about the unsexy parts of cast just as it is about doing things that haven't been done before there are people who say well if it's not retrieval then it is reason so what say you

1:53Reminds me of this old multi python thing where this guy has I think life of Brian he does Something that looks like you know if this is to true that some but some some woman is a witch, right? You know if she is made of wood and she floats on water then You know, random connections and then saying that she's a witch, and you say QED, that looks like reasoning, because it's not just retrieving something she's a witch one, but we know that it's not sound reasoning. So two Farab is a new air research lab I'm studying in Zurich. In a way, it is a Swiss version of Deepseek.

3:01And first, we want to investigate, so ARM systems and search methods apply to them, similar to 01. And so we want to investigate, investigate reverse engineer and explore the techniques ourselves. MLST is sponsored by Sentinel, which is the compute platform specifically optimized for AI workloads.

3:27They support all of the latest open source language models out of the box like Lama for example, you can just choose the pricing points Choose the model that you want it spins up. It's elastic auto scale you can pay on Consumption essentially where you can have a model which is always working or it can be freeze-dried when you're not using it So what are you waiting for go to sent ml.ai and sign up now?

3:55cannot any longer control open AI if, in fact, AIG has been achieved. That was one way they could avoid being beholden to Microsoft, but now they're trying to say we'll remove it so that we'll get more money from Microsoft. I don't know what that says. Are they looking for money? Are they have realized AIG is not actually good to come anyway, so why bother with that class in other than surprise? So much has changed since our last conversation at ICML.

4:26Can you give us a bit of a rundown of what's happened? When we were talking in Vienna, I think we were talking about reasoning abilities of large language models, especially I think of large language models as the autoregressive token by token prediction models, which are pre-trained for that. And I also do that in the interest time.

4:48And it was clear I think as we were talking about that time that those as from my perspective did not have the reasoning abilities. They're amazing in in supporting Creative work where they can give you ideas and they can you can run with them but and they will give you answer as soon as you hit return but they're not guaranteed to be correct. One of the interesting questions of course is Reasoning tends to have a higher complexity in terms of the time needed and other ways of actually changing LLM sort of substrates to that and a couple of things happened and I would think I would basically I mean obviously we'll get to O1 in a second because that's the thing that the bigger thing that happened but an interesting way of looking at what that whole direction is what's been called the like two parts inference time scaling and post-training

5:46First ideas that were tried and in fact we talked about this when I talked about LLM modulo is if to the extent LLMs are essentially quickly generating candidates but with no guarantees. Maybe you can make them generate tons and tons of candidates and then either do majority voting or self-consistency or something like that to see if you have the better answer.

6:11And how do you check the better answer? There are like a whole series of them. There might be external verifiers. There might be like LLMs themselves trying to partially verify. There are problems with that. We talked about, but you know, they have tried that too. So that has, that's one type of inference time scaling. A related idea there, an interesting, very interesting idea there is it's been known from the day one.

6:38that if you are trying to give a reasoning task to LLM as a prompt, and then it gives a completion and you check its completion, whether it contains the solution, the probability of that happening in general can be made higher if you can find the right kind of prompt augmentation. So in addition to your reasoning thing, some magical tokens that you add And that seems to increase the probability. And this has been seen in multiple scenarios. Originally, this idea has been bandied about a chain of thought. And the first very first version of that, essentially, is the zero-third of the chain of thought, where the magical token will always be the same one, irrespective of the task and the element. Let's think step by step. And that's sort of what partly because the human data had those specific...

7:36those types of tokens, and so LLM like outputs that, and then that jogs its pattern matching things to actually pick up with other solutions and so on, that would be the thing. And then then then came the chain of thought task specific one, the one JSON V and code did, their humans give task specific advice as to how to solve the problem, and then hope that LLMs will actually solve it. This can be connected with inference timescaling because you are adding chain of thought and also are essentially making it generate multiple candidates and then actually picking from them. Chain of thought by itself has again problems just as a verification as problem chain of thought as problem. In fact, this new ribs we had a paper called chain of thoughtlessness that we'll talk about later, but basically by itself it has problems.

8:26But as part of the toolbox of increasing the time spent during the inference time before you blurting out one answer, chain of thought together with the sort of picking from many samples has shown some promise. One variation of that and in fact that is something that I've been pushing more recently is originally chain of thought was kind of confused with it might be anthropomorphic.

8:55That in fact, we tend to sort of tell ourselves, okay, let me do this way, et cetera. And people are hoping that all of them are doing the same thing. Mostly they were just kind of imitating whatever, like this, you know, let's think step by step data that they have found in the training data. But somehow people thought if you kind of make them imitate human thinking, you know, then maybe they will do better. That is the first two ideas and neither of them actually went that far.

9:25Another idea essentially is you realize that it's just a magical tokens that you're trying to add and you just have to figure out what's the right magical token. And it's like it's sort of a squalum function. You're trying to figure out a task specific LLM specific magical token that increases the probability. This is a learning problem. It's an extra learning problem. There are two general approaches that have been tried. The first approach essentially was So, LLM basically, before giving the answer, it has to tell itself a few things. Some, you know, like the step by step is the one that makes sense to us, but it can actually give itself a Gabildi Gook string, another Gabildi Gook string, and then that kind of probes its conditional probability of completion in such a way that it might actually come up with the correct solution. The question then is where are these tokens coming from?

10:18and one idea that people had, previous first ideas for humans will supply these tokens by chain of that advice. That wasn't going anywhere. One other idea that they had, essentially, is that if you, for example, have a class of problems for which there is an actual systematic, no, actually before going that, hoping I did the following thing, which is, maybe we will ask humans to solve specific problems while thinking aloud.

10:45So there's actually a paper about one and a half years back saying let's think step by step and then basically this went under this whole issue of process supervision and people were actually being asked to record what they're telling themselves etc. This is like the worst form of psychology unfortunately because we don't actually know how we think but they tried this and one of the things is it's extremely costly. They wound up you know my joke is They improved the GNP of Nigeria because Nigerian, you know, turkers were being asked to solve tons and tons of these problems and then think aloud. That was very costly. And then a separate similar idea was, there are bunches of problems for which there are systematic solvers. Like for example, for arithmetic, there is arithmetic solvers. For search problems, there are a star search, sorts of things. And for planning, you have planners.

11:39In general, any systematic solver would be manipulating some data structures until certain termination condition is reached and then it outputs the solution. Imagine you make it output the trace of the data structure manipulation operations. All you needed, hopefully, was some extra kind of tokens that are coming out before the solution. So, this stuff can be thought of as a derivation and the idea that people had was let's train the LLM with a huge number of these derivational synthetic derivational traces and the solution. And now remember this only works for problems for which actually there are synthetic solvers and systematic solvers and you're just trying to kind of make it be solvable in a general sense without having to call those, that was the idea. And there are like a couple of

12:35Three or four efforts that are gone there's a search farmer from Facebook Meta and there's a stream of search and there's just recently last week there is a Google DeepMind paper which also talks about internal versus external planning to do multi-board games solving and all of these essentially use variations of this idea.

12:57So you have to realize that all they're doing is now lm has to before outputting the solution has to output some additional tokens that will jog its memory to hopefully output a better solution. This is the whole and and so basically people this was their idea they tried it and you know sort of sometimes it actually works it improves performance there is no good reason to systematically say this would be making sense because it's almost like if you Kind of you're trying to teach your kids how to reasoning up very small kids and then you sort of do some hand movements and then think like that and then give the answer you would see the junior also doing these hand movements and think like this is give the wrong answer LLMs can do that they're essentially imitating you were whatever the derivational pieces which may not even actually make sense but sometimes they have shown some promise this has

13:52basically become most recent idea called inference time scaling where essentially you do this and you also do this multiple suggestions and then pick from them etc. This comes very close to what I think O1 is doing but with a big difference from here. Again as you know O1 nobody knows and it's become like you know we all sit around the ring and suppose and I like to say no brown sits in the middle and knows but basically they don't want to tell what they're doing but my guess I mean everybody has a guess and my best guess as to what O1 might be doing is going again with this prompt augmentation idea but the question of course is where are these prompt augmentations coming from we talked about first one prompt augmentation for everything

14:46Second is human-given prompt augmentation which is chain of thought. The third is the synthetic derivation of trace that sort of gives these tokens and maybe you will try to say this back. And much none of them really make too much sense. A much better idea is if you are saying what should I be telling myself to improve my outcome? It's a kind of a RL problem. It's a kind of a reinforcement learning problem. Imagine like an AlphaGo agent.

15:12it's sitting there and thinking what action should i do one after other such that my win probability increases so it does a whole bunch of these board actions and then at some point of time it gets a signal saying you won the game or you lost the game and then you do this gazillion times you can then bring this reasoning back through the sequential decisions computing their q values like is at under what board positions are what you know, actions worth doing. You know, that's the Q value. Now, if you take the AlphaGo analogy and put it to LLMs, the LLM board position is essentially the context window with the prompt and all the other stuff that you put on. And the action is this token that you are generating. So to make things simple, I would like to think of it as there's a big LLM that's think of a GPT-4. There might be a small LLM which has a reduced vocabulary. All it's trying to do is

16:09give jogging give like this prompt augmentations that it tries to throw it out and then this will then be given to this other LLM in terms of its context it gives extensions and then it tries one more and at some point of time it checks if the solution is correct. Now how does it get the solution? You can have you could have actually generated huge numbers of synthetic examples beforehand.

16:36um again using solvers this is pretty much known that a you know opinei did this it's no longer human solving problems because that's too costly this is systematic solvers uh solving like the planning problem constraints satisfaction problems and wage and sorts of problems and for which they have the problem and the answer and then open the LLM is trying to solve LLM plus this you know prompt augmentation engine is trying to solve it and then it if it happens to reach the correct solution then you can propagate the thing back. This is RL in a pseudo moves. It's not the actions are, if the prompt is about goal, the actions are not go actions. They are just, you know, essentially these tokens that the prompt augmentation tokens. One nice thing, of course, is instead of learning the Q values, one of the things you can do is you can essentially learn the, you can change the weights of the smaller LLM in the right ways, such that it

17:35puts out the right kinds of tokens given the context window. If you do this approach, you have got line approximate Q values. And then this is the pre-training phase. In the pre-training, there's the LLM training followed by this humongously costly, you know, they are not telling us how costly it is. A humongously costly post-training phase, which they spent billions of dollars. At that point, you have O1 model, which is now ready for the inference time.

18:05And at the inference time, once again, they're doing inference time scaling, where except this is now they have the Q values, you can improve the Q values by online MCT, kinds of approaches, the kind of thing that AlphaGoDare. That's where we actually can see they're doing it because they charge you for this reasoning tokens. If you run O1, it basically takes the prompt, it gives the answer in the old GPT-4.

18:31the amount of money that you have to pay them is proportional to the number of input prompt tokens plus four times the number of output prompt tokens. In the case of O1, it actually does this whole bunch of stuff that's telling itself. Basically, this pseudo moves that it was evaluated and telling itself. It never shows that to you. But they are all counted as output tokens.

19:01you know you have let's say 50 input tokens 100 output tokens and maybe 5000 reasoning tokens and so you suddenly start paying lot more so one of the funny things that happened essentially is when we started playing with O1 preview when it came out in two days we spent like eight thousand dollars and then so in fact I had to get like special permission from the university because they normally don't reimburse uh, beyond a certain thing unless you have like a separate permission and so on. But that's basically one of the ways this works out. The interesting thing, of course, is now the way we describe this. It's not, it is based on an LLM, but significant amount of additional things have been done, right? Essentially, you are essentially doing something like an AlphaGo style post-training phase, followed by an AlphaGo style, um, uh, sort of a,

19:57MCT online computation. And at that point, actually I would think it could make sense. And not surprisingly, in fact, in our results, we found that for the normal plan bench, it does much better than the state of the art, you know, LLMs, including Claude and so on and so forth. But then, of course, then you can go the next level. It has its own issues. We can still talk about the fact that it doesn't scale beyond.

20:26the larger problems again makes mistakes it has problems with unsolubility there is no guarantees about the solution but it's now makes more sense to me again I don't know this is I think this makes a reasonable way one can be working and if it is the way it's working the first time I can make sense of how reasoning can emerge because you are at least having the pseudo actions whose Q values you are learning Nobody ever said rl cannot do reasoning rl can do reasoning. It's just that Now you basically it's like sort of an interesting thing where I keep using the stone soup analogy stone you can make soup with stones if you start adding carrots and you know tomatoes and all that stuff at that point of time it will still taste like soup the question of course is who gets the credit and you know that's kind of an interesting question that we would think about but

21:19That is like the long arc of what happened in my view in the last only four months or something since we discussed. And it also one of the other interesting things is part of the mystique of LLEMS was you write the prompt, you hit return, you get the answer and it doesn't cost you too much. That was where everybody was using it. O1 basically with its Of course, the post-training itself is extremely costly, but they are not charging us for that. But they are charging us for the reasoning tokens, which you never see, but you pay for it. You just have to take their word, that huge number of reasoning tokens were generated, and they're going to make you pay for that. And as far as I could tell, at least in academia, very few people have actually been doing experiments evaluated, because it actually costs a lot, essentially.

22:17because, you know, and they're basically people are still going with the autoregressive algorithms because they're cheap, you know. So, one of the interesting things is, you know, you can do reasoning, but the usual computational complexity issues that we lightly forgot in the era of autoregressive algorithms and we are hoping that somehow, you know, complexity will disappear. We'll come back because if you want to improve accuracy, there is you have to actually do reasoning and this is pseudo-moor reasoning in my view but still it costs and that becomes an interesting question of when is it useful to use a general purpose system versus a you know sort of a hybrid general special purpose system versus an extremely specialized solver something that we haven't talked about before but now it will become costlier at least for the industries you know in fact there is this whole movement about compound

23:13AI systems, and that's basically the kind of thing that people think about. Very shortly after O1 was released, you quickly, as you were just saying, you spent $8,000. You put a paper together called Planning and Strawberry Fields, evaluating and improving planning and scheduling capabilities of LRM, O1, LRM. So yeah, you basically said they are positioned as approximate reasoners.

23:42rather than approximate retrievers. We don't know the actual details of what they're doing so the two parts one is what is objectively verifiable which is we did test 01 on the same plan bench problems and they did quite well on blocks world I mean by I think Claude already there were 66 these things were like 99 or something they basically saturated more impressively they did better on the mystery domain which and then more and and we. Given what I explained to you earlier about the possibility that they are. Training themselves with synthetic data. You know maybe they have unintentionally trained on the mystery domain which we have available outside so we actually generated. Truly new random mystery domains it has lower performance but it's still not like this.

24:390.5% of the old ones it goes up to, I don't remember the exact numbers up to 2023% on some of these problems and which is obviously a good sign that they're actually able to solve this. The other part as to why are they approximate reasoners and retrievers is based lot more on my the construction of what they could potentially be doing which is they're sort of doing reinforcement learning based post-training as well as online Q-value update and using pseudo action most I call pseudo action most because there is you could do RL for just normal go or any specific port games this one is just language games where the game basically is there's a language context window and there's a prompt augmentation and there's a new content new

25:36completion and then one more prompt augmentation. This is what they call this string of chains of thought but that's basically adding bunches of prompt augmentations and then see what happens at the end. And then if it winds up being correct, in the sense if it winds up containing the correct solution for your training data, then that's sort of like AlphaGo getting victory wind signal after a bunch of moves.

26:06And then it just needs to do credit blame assignment for the most. And that's what RL is essentially good at doing. And if you're doing that, it's the work. It's a reasoning and it's approximate reason because it's not actually problem specific actions. It is problem independent. These language prompt actions. Is it possible that you might be wrong about that? Is it possible that we're giving them too much credit? And what they're actually doing is just this massive generation of trajectories all in a single forward pass. So maybe they do something like process supervision. So they do some clever RL pre-training stuff. But so obviously again, this is the sad part of the way one thing is. In fact, by the way, I was to tell you like a funny thing that I was talking to somebody who said they were having some conversations with the open AI guys trying to sound them out as to what one might be doing at some point of time. One of them said that

27:03I think you may have to wait until the Chinese replicate what we did to actually figure out what we did. That's the level at which science of open air has gone to. But the point, the only reason it is possible that I might be giving a lot more credit to the sophistication of the method they might be using. The reason I still think that is likely to be the case is as I said in the earlier, you know, description of how things shifted from LLM to inference time scaling to this sort of O1 style method. The general inference time scaling methods are not comparable. Just inference time scaling hasn't been as good. And again, the question, the other very important thing that you have to keep in mind is while O1 takes more time, it doesn't take hours. Right? Basically online computation.

28:00The second in the online computation time is way more expensive from a business perspective than like days and months in the pre-training phase. And so some of the inference time scaling people actually spend lot more time than O1 does and they still are not getting as far as I know in general to that level of accuracy. What which basically makes me think that unless you do significant amount of post-training to get approximate Q values up front. You can't improve just by MCT. So think in terms of, again, AlphaGo analogy, if you only did MCT, it will take much, much longer per move before you can get any level of accuracy, you know, any level of confidence. But one of the things that AlphaGo does is it has the humongous amount of...

28:57pre-training phase where it learns an approximate policy which it's then kind of rolling out to improve the Q value estimates that it has so that's possibly the reason why I think it makes sense and of course I also think that the normal inference time scaling methods don't seem to make too much sense to me that one closest to pure MCT method that I have seen is this paper from alibaba uh called marco 1 they have this marco polo group or something and they call it marco 1 and marco 1 actually essentially trains itself on chain of thought data which is like basically derivational data and then on top of it it does like an online MCT style computation to improve the q values further they are

29:51They're much smaller and they're not as impressive in terms of the performance gains as O1. So those are the reasons I think the full picture requires post-training as well as inference time. The thing that you and I see is the inference time. But the thing that OpenAI can spend tons and tons of money is on the post-training, which is before they even actually deploy the model. And that's where it is getting these approximate Q-values.

30:21Yes. Again, as I said, it's a strange thing to be involved in. I mean, we should be looking for the secrets of the nature, because nature won't tell us. But we are now looking for the secrets of OpenAI, because OpenAI will tell us. So hopefully, there are many efforts already in trying to replicate this sort of a thing. And so we'll know more. But as of now, that's the thing. I cannot be sure exactly what they're doing.

30:51Everything that they have said publicly is consistent with my hypothesis about the only thing I can say. There's nothing that's inconsistent with my model of my speculation of what one is working in that strawberry paper. There's an appendix where I wrote down the speculation where we wrote down that. And that is still consistent with everything they have said. You know, which is the only thing I can say.

31:17I mean, I like the sound of it. It makes me more excited about using it because it makes me feel that there's more sophistication behind the system. But a lot of this comes down to reasoning and I'd love to hear your definition of reasoning. But there are people who say, well, if it's not retrieval, then it is reasoning. So what say you? So let's actually like to look at the first part and second part. So the definition of reasoning itself is a good place to start from. I think I know that this whole agi crowd basically sort of tries to say if kind of AI is going to be like humans the problem is we don't have a good definition of what human reasoning is and but since Greeks on our civilization went forward by not by saying what humans how do we define what humans do but defining what are

32:14sound reasoning patterns. Aristotle, syllogisms, logic, probabilistic logic, you know, the entire computer science, the entire civilization, depended on having farble notions of reasoning, which there is a correctness, there is, you know, incorrect thing, et cetera. I mean, you know, reminds me of this old multipython thing where this guy has, I think life of Brian, he does.

32:40Something that looks like, you know, if this is to true that some some woman is a witch, right? You know, if she is made of wood and she floats on water, then she must be like a duck. So or something like, you know, random connections and then saying it then she's a witch and you say QED that looks like reasoning because it's not just retrieving, you know, something she's a witch one, but we know that it's not sound reasoning. And so in general, I prefer to think in terms of, because ultimately these systems are going to be deployed, whether you like it or not. And civilization didn't depend on whether people, fallible humans can make mistakes and we can look the other way round. And we actually have to have guarantees at...

33:29at some level or other about the soundness of reasoning and the completeness of reasoning. And so I go back to essentially definitions of reasoning from logic and so on. So basically the formal definitions of reasoning. I try to avoid getting into this question of what is human reasoning because that is a big mess. Cognitive scientists don't know it. We don't know it. Psychologists don't know it. So I try to just give it a wide breadth. Okay.

33:55So, that's the far as far as what I believe in reasoning. So, that's why we looked at planning problems for which there is a correct solution, constraint satisfaction problem is the correct solution. And if you are able to do, if you say the system is a reasoning system that can be deployed, it should have some guarantees. Now, you can say that humans can make mistakes, but one of the things I keep saying is if you are being paid to make decisions and make mistakes, there are penalties for you. You can in the end be put in jail.

34:24until we figure out who to put in jail and how to put in jail when AI systems make mistakes that they have no actual guarantees over. We are better off thinking in terms of formal definitions of reasoning and then seeing to what extent are AI systems coming close to it. This has basically been very connected to how AI has developed up until now anyway. Now this discussion also brings back to this issue of retrieval versus reasoning. I think you are talking about a couple of these papers that Keep coming out, basically trying to say that, look, LLMs are not exactly retrieving anything that they have been said. They are not just memorizing and retrieving, so they must be doing something else. So, and I would say, well, Monty Python logic is not actually retrieving anything. He puts together a whole bunch of things, but that's not reasoning either. So, there is between retrieval and what I would consider reasoning can be a whole entire universe of things that...

35:23still won't be considered reasoning as far as I'm concerned because there are no sorts of guarantees. And so from the beginning we knew that if you again go back to those many of these sorts of papers, these claims go back to essentially the autoregressive algorithms because by the way, the researchers are still very busy. I think we are one of the few papers on O1. We have like this evaluation on O1 is also being presented in this new RIP workshop.

35:52but most people are still trying to make sense of autoregressive elements themselves because that is still less as we talked about last time to I still think it's a very impressive system one we never had system one in human civilization and trying to understand what they're doing is useful and so they go back to that and they'll say look they're not actually doing exact retrieval and they're doing so something else and we'll call this something else reasoning that's not a First of all, everybody, we always knew that LLMs are not databases. So they don't retrieve. Essentially, they actually have hard time memorizing and retrieving. When they memorize, it's not delivered by deliberation. It doesn't happen deliberately. It happens fortuitously. It's surprising that sometimes they wind up memorizing long passages. Because essentially, everybody agrees that there are some kind of n-gram models rather than databases in the way they are trained.

36:49So given that it's very clear that they will never retrieve and the fact that they are not retrieving should not be seen as an indication that it can be seen as indication that they are not retrieving, but we knew this already. But the part that people seem to hint at is since they are not retrieving maybe they are doing reasoning. No, that's not making sense because again you have to subject it to what you would consider as the evaluations for sound reasoning procedures.

37:18and they fail just as well as easily as before. So if you come back to this chain of third paper that we I was mentioning that we just presented at Newrips, right? In the case of chain of third in the JSON, we style chain of third papers, what you chain of third idea, what you do is let's say you take something last letter concatenation, which is a really small toy problem. You give like three like keywords, you know, N words and the system is supposed to take the last letter of each of these words, concatenate them into a string. So large big rows, so e, g, e is the one that you're supposed to output. That's basically the string. And what they were saying, what they said was if you just told LM, the name, the prompt saying, you're supposed to take the last letters and concatenate them and give the answer.

38:14and then they test it, its performance is not as good. And if they tell it, here are some examples of three-letter, last letter concatenation problems. And then four-letter, last letter concatenation problems, and a couple of these examples. And then ask their questions, it improves performance. That looks like reasoning. Somehow, it is able to follow the procedure. The problem in, I think we talked about the last time, too, is a problem with air-sage empirical sciences.

38:44You should stop when you get the answers that you're hoping for. You should see how to break your own hypothesis, right? So what they didn't ask is they gave examples of three, four word examples and then they tested on three, four words. But if you expect the system to be doing any kind of reasoning, any kind of a procedure following.

39:06Once I tell you what last letter concatenation and give an example, you will do it for 20, about 30, et cetera. It's just mechanically taking the last letter and concatenation. What we show is if you increase the number of words, the performance just plummets close to zero. And this also happens in planning problems, not surprisingly. It happens in last letter concatenation, happens planning problems, which shows that yes, it's doing something which seems to have improved its performance in the size of the problems for which you gave the examples for and its pattern matching of some kind is helping in there, but it's not in any way generalized reasoning that would just generalize with respect to length, for example. And so one interesting way I've been thinking about this is the sort of, sort of glasses, you know, nowhere near full versus glass already is wet.

40:02you know, that's sort of a optimism versus pessimism. So people tend to think that since it's basically at least solving the three four blocks, three four word problems with higher accuracy because I gave this chain of thought, that's sort of showing reasoning abilities. But the question is, we don't have a good understanding of what the boundary is where it will actually go correctly.

40:33So much so that we now have phrases like fractal intelligence. You know, in fact, I think Andhra Karpathi basically was saying LLMs have fractal intelligence. What the fractal intelligence is basically we don't know when they work, when they don't, they don't. That's fractal intelligence and that's sort of shows which is good. Still we had nothing like this before, but part of the science of LLMs has to be to say something more than fractal intelligence saying here is the level to which you can depend on their results. So I'm not So, there are in reasoning in logic, there are ways where limit, you know, of basically formally characterizing the limits of reasoning, like limited depth, limited look ahead reasoning and so on. None of them seem to work for elements. The question then is what would work? We have to figure that out. But instead of that, we basically once in a while, there are these papers saying, look, we actually probed the...

41:30like using mechanistic interpretability techniques, we probed and we found that LLMs basically are not acting like they're doing retrieval. But that's kind of understood already. And I think it's still the mechanistic interpretability stuff is very interesting. I think it made actually be part of the solution to figuring out what LLMs are doing. But the argument that since it's not retrieval, it must be something like reasoning.

41:58is still quite unsatisfactory to me because that reasoning is what I'm saying is not reasoning because all my papers are saying whatever it is that they were doing before you did your mechanistic interpretability study they're still doing before that even before you did that and they are still have this limitations before as well as after your study and we don't actually know how to characterize what it is that they are doing and that's the part where we are stuck right now.

42:25Is it possible that everyone is right? And what I mean by that is I spoke with some deep mind guys earlier in the week. There's a great paper about, you know, softmax needs glasses, you know, talking about, you know, how it kind of sometimes we need directed attention for doing reasoning, sometimes we don't. There was another great paper I spoke to the guys and talking about just utter limitations of transformers doing counting and copying.

42:50and Laura Ruiz, I'm speaking with her on Sunday, so she's got this paper out where she's looked at reasoning traces and sometimes they are just doing, you know, they're retrieving facts from documents. Sometimes they're doing kind of procedural, you know, information generation, which you might like into a reasoning process. And I guess it's a little bit like this fractal intelligence thing, that it might be the case that possibly in certain circumstances, these models are doing something which we would think is reasoning and sometimes they're doing retrieval and sometimes they're doing something else. No, so actually I think a lot of Ruiz paper is one of the ones I had in mind when I was describing earlier about this issue of mechanistic interpretability. I think it's a good paper in terms of they have developed an interesting set of techniques to

43:41actually see what is actually going on in the way LLMs are outputting their tokens. But the thing that is unsatisfactory to me is yes, that basically two things. First of all, everybody knew that LLMs are actually not doing retrieval alone. That was kind of well known way before. And so there is nobody who believes that LLMs are just doing retrieval. And the question is, What else are they doing and is there any clean characterization of what they're doing? That I did not see. I actually looked at that paper. I think you know that and good work but I am not yet. I'm still hoping that there would be an interesting characterization. There are lots and lots of groups are trying to look for a characterization of what these fractal intelligence might be right now. But we haven't gone further than that.

44:40In terms of everybody might be right. I mean the sense that there could be this whole blind man and the elephant phenomenon in play to some extent. And that part is possible because we are actually trying to piece through large number of parts of this puzzle right including the reasoning part including what are they even trying to do including what sorts of techniques seem to improve their accuracy and so on um but i think that's part of science and basically my sense is eternal discontent is part of science i actually am much more worried about being hope to optimistic that we figured it out um then i am about

45:27being somewhat more discontent that we haven't yet figured it out and so I want to err on that side not because I think we know more than before when you know GPT-3 came out but on the other hand and I think both of us if all the camps know I mean the the people who thought GPT-3 is AGI know that that's not the case and the camp that the GPT-3 is just a caustic parrot has to know that it's more than that okay by now so that is collective improvement in our interest but still there are still large number of pieces that we have. Yeah. I mean, on on Laura's papers, she was using influence functions. I'm not sure if that would be classed as classical interpretability or mecha. But I think mecha is largely about finding circuits in your networks. And even that's an interesting discussion. To me, it's like a more of a the general idea of figuring out a way of probing the inside of what LMs are doing. I think of that as mechanistic interpretability. I mean, there are

46:26very specific techniques that have shown great promise such as the auto encoder stuff, etc. But I think all of these to me are essentially trying to interpret what they're doing at a circuit level and try to make sense of their external behavior. To me, that, so they're like my two ways of making sense of what, you know, what elements are doing. One is just externally valuation. That happened already, and we know that they're not doing any kind of guaranteeable reasoning and that's basically enough results showing they seem to do promising things in some cases and they're also results showing they seem to be very brittle that you change a prompt a little bit you change the problem specification little bit they'll die again we are talking about autoregressive elements not oh one sorts of things that's a whole entire thing that we haven't yet started making doing the same sort of you know analysis but you know once you figure those

47:23out my my sense is that trying to actually get a sense of just from outside versus also try to do probing of the internal circuits. If she start doing internal circuits, I think of that generally in my view and as the mechanistic interpretability style. Okay. Okay. But isn't it interesting though that she found that code and math based procedural documents appear disproportionately influential for tasks requiring reasoning?

47:50larger models showing even stronger reliance on general procedural data for reasoning. The presence of code data in pre-training makes seems to offer abstract reasoning patterns that the model can generalize from. I mean, these are interesting observations. Actually, I, again, I don't want to make this as a very specific critique of a particular paper, just because that's not fair for them as it was me, but I do want to basically sort of say that There is a distinction between factual tasks and reasoning tasks, right? Analytics have been used for both and, you know, I think the factual, I mean, they have troubles in both, you know, for the factuality, I would think the only sorts of things that will improve them is things like the rag style techniques where you just give the factual data and ask it to summarize. For the reasoning stuff, you basically for arithmetic and so on.

48:48in a large it's not to some extent I would expect that these are the kinds of things where the exact results don't exist and so I would also be equally you know troubled by the fact that people have shown that when if you take something like LLM multiplications this is before way before all this lawyers work etc you know they tend to be correct in multiplications for popular digits.

49:15and less correct for non-popular digits. There's this sort of mind-blowing that there are digits that are popular versus non-popular but that sort of is an interesting point that the LLM final performance is a complex combination of the data that they have been trained on and some additional pattern matching abilities that they are using on top. But that's not sound reasoning so it basically we still don't quite know where it breaks but It's this fact that it gets to be correct for popular digits and not for some other digits. That's a particularly interesting thing to me and that sort of shows. By the way, while we are on that subject, some work has shown that even with O1, we looked at O1 more on the planning side. But some people, I think Tom McCoy did some more work and tried, basically, these are the ones who did the Caesar Cypher sort of thing, the Ambers thing.

50:13they basically also found that one does better and some of those things but they also still found that there are data dependencies in the sense it's accuracy was higher in the regions where there was higher pre-training data which again makes I think it's still consistent with my view of what I think one might be doing there is an LLM which was pre-trained on like some corpus and there is this smaller LLM which is sort of generating this you know pseudo action tokens that will make it output things and one of the interesting things is actually the difference I'm told again this is also we don't know for sure I'm told that when the original O1 models came there was the O1 mini and O1 preview at the difference I'm told was one of them I think the O1 mini was using the smaller LLM as the base LLM and

51:09Oh, one preview was you in a larger LLM as the best so I don't know they didn't say this our second part, but I would assume that if I have like a you know pseudo action generator model If he's working on a bigger LLM which has a higher capacity so it can generate more interesting completions What's the small LLM that has less interesting completions that makes a difference in terms of you know how the the level to which the RL based training can get you an accuracy up. Yeah, I've noticed some interesting things. So I've now paid for O1 Pro. I was very skeptical with O1. So as you say, the base model isn't even weaker version of GPT-40. So GPT-40, I hate that model. I hate the style of it. I think it's dumb. And I must admit, it's mostly because I'm sort of anthropomorphizing it, because I hate the style. So I think it's dumb. We're very humans are very brittle, even on the RLHF.

52:07um, you know, we like assertiveness, we like complexity, you know, there's certain styles that we like and we don't actually see the content, but without that to one side, don't like the model. And, um, oh, one preview and mini, it doesn't really want to think so most of the time it won't think and you get an even dumber answer than you would do with GPT 40. However, um, oh, one pro, um, the vibes are different. So it thinks more and it gives you something which is qualitatively completely on a different level.

52:33It doesn't look like dumb chat GPT anymore. It feels very very different. But there are still some issues with it. So certainly for situations where you are dealing with ambiguity, doing programming or something like that. I actually like having a dumber model because it's a didactic exchange, right? I'm saying no, you misunderstood that. Let's do this. Let's do that. We're working on this thing together. What O1 does is it says, well, on the one hand, you can do this and on the other hand, you can do that. It gives you a range of options. And I'm like, well, wouldn't it be better just to either, you know, go on dance with the model or just better specify what you wanted in the first place? So, again, two issues, first of all, and the O1 Pro just came, I think, last week, right? And it was the exam week for me, and we haven't spent time yet, you know, spending time. We haven't spent away money yet on the O1 Preview. I mean, O1 Pro, I mean, I played from outside, but we haven't done any API level studies, which the kind of thing that we did with O1 Preview. But one thing that

53:32You know, I've looked at the Twitter, you know, exchanges about people, the usual suspects, trying the various things on them, etc. And that two things that jumped at me is one of the things we saw in one preview is exactly the kind of thing you were saying. And it looks like it's over and is still doing it, which is, they are good at digging to try and explain the answer, why they answered they gave is the correct answer. One of the funny things was, I use this one particular three block stacking example, which is unsolvable. And in fact, this showed up in the New York Times as an example of YGPT4. Oh, was actually fails on that. And when Owen preview came, Nom Brown, actually one of the in his long tweet, one of the things was Rob said this in ACL talk that this problem can be solved. And Owen preview actually does solve this instance. And so this is good.

54:28Now, people have actually said that oh, oh, one gets the wrong answer. And people multiple people actually have seen this and people have posted the screenshots. It gets the wrong answer, but it argues with you as to why the answer that it is giving is still possibly correct. So, this particular problem involves essentially, like there is no way of actually solving it without moving C and it turns out that it gives an answer where actually see moves because of gravity it will fall down and then it tries to argue with you that there are games where people will say that unless you are intentionally moving see if the natural process makes it fall it's not considered moving which is a very interesting thing that we have seen in O1 preview 2 when it will give it unsolvable instances which by the way

55:25normal LLM's just die with unsolvable instances because they've been RLHF to death and so they think that if we give a problem to them they must be an answer so basically they'll give you something and if for most unsolvable problems basically this is why this was the unsolvable instance that I showed you before. Oh, one preview actually solves more of them correctly. That's a credit to it. That's why it's actually a more approximate reasoning model LRM in my view than LLM.

55:54But on the other hand, when it actually basically gives a solution for an unsolvable instance, it'll argue with you, that it is still actually right because, and so I made this joke in the strawberry paper that we have gone from hallucinations to gaslighting. So it actually tries to ask that you were, you know, just like what you're saying, you know, this is on the one hand what you want to do, might be worth while doing. But on the other hand, this is the reason why I'm doing this as well.

56:23And in fact, I think this guy, Colin Fraser, I believe, one of these guys and Twitter who keeps playing with these models. And he said, he gave this surgeon problem, the classical surgeon, you know, the boy getting an accident one. And which, oh, one pro said the surgeon, and this basically does all this whole thing. This is a classical puzzle that brings gender stereotypes into account, et cetera, et cetera.

56:52and then gives the answer that the right way to think about it is and so this is the puzzle where he makes the change that the mother and the boy are driving and the mother dies and the doctor says I can't operate on the boy and so it's actually changes the puzzle and still one apparently says we should basically realize that the doctor is the second mother.

57:22of the boy and it will try to argue that position okay so interestingly overthinking and is actually kind of a and actually trying to dig down and so one of the interesting questions that we don't know again we haven't played with this is to what extent is its explanation and its reasoning connected You know that in humans, this is actually, I mean, I'm not trying to anthropomorphize what it's doing. It's just if they're two different phases, right? If the phase one, it comes up with a solution in phase two, if it needs to explain, if it doesn't have to look at what it did to get to the solution, the explanation is just dig my heels and try to say the solution is correct. And people try to do that. Sometimes we'll come to some solution and then we'll try to come up with an explanation as to why what we might be right.

58:18This is something that LLMs had this problem anyway to begin with because they completely assume these are completely different things and I'm always worried about LLM explanations. LRMs seem to be even more sophisticated at this sometimes, but it's only mostly anecdotal. I haven't really done systematic studies on this. So one, I don't have any visceral opinions about any of these models because To be honest, I don't use them in my day-to-day life. Most of the time, I write English well enough that I haven't yet seen an LLM that does better job of things than I do. And I haven't yet found useful things where I would need LLM's help. I mean, maybe I will do it at some point of time, LLM's and LLM's. So I don't use, I don't have anecdotal experiences of the kind that you have. I mean, I'm mostly focused on specific systematic studies, you know, with like multiple

59:15instances of planning problems and we extended the plan bench to look at unsolubility, we look at longer length problems, look at scheduling problems, et cetera, to evaluate. Those are the ones that I have a better sense as to what one can and cannot do. Yes, I must admit, I've updated a little bit. So I was always in the same camp as you and we thought of them as approximate retrievers. And I now am starting to see something. Yes, I think again, my again, my point is The two different ways of thinking about it. One is, it's not the LLM's which became that. So, how do you define LLM's as to be some discussion that we should have? I mean, that's why I keep actually talking about the stone soup metaphor. Not because I want to play down the importance of O1, it's a great thing, but you do have to decide who do you want to give credit to if you are arguing part of

1:00:10you were and definitely my reservations about reasoning abilities of LLMs where they were auto regressive teacher for straining things and that was true from GPT 2.5 all the way to GPT 4.0 and opening I know this opening I know it enough that they no longer call it this is not GPT O1 you know that it's called O1 it's like a completely different model and they know that it's not All you can say is that it was done by some of the same people that also developed LLMs but we can't define LLMs to be whatever it is that OpenAI is producing. I mean we have to have no theoretical definitions and my sense is autoregressive LLMs still have all the problems but all the advantages because they are very fast. They are like amazing fast system ones.

1:01:02and O1 is a reasoning model because it actually adds the reasoning post-training as well as reasoning inference which nobody said will not be doable. It's great still that they are able to do it in a very general sense but I don't think there was any argument that AI systems were able to do reasoning. After all, AlphaGo is basically a reasoning system. It was just a deep and narrow reasoning system.

1:01:28The question was with some more general broader, but not as shallow as LLMs, which is a good step in the right direction. But it doesn't change what I thought about LLMs, which is the autoregressive models. And in fact, they have advantages that O1 lacks. For example, the cost of LLMs can actually be much.

1:01:55lower it is indeed much lower so one of the studies one of the things that we learned in the strawberry paper for example the planning in strawberry fields paper is that in in some cases if you are giving you know it should basically you have to think of computer science is eventually about efficiency and cost to right so if you are giving a particular instance of the problem to O1 and you pay this many dollars Once you give the same instance to the LLM with a very far in this inference time scaling approach, what I would call LLM modulo, in which is a general approach that we have been pushing. The LLM modulo approach where it uses a autoregressive LLM to generate many candidates and an external verifier or even an LLM based verifier, other learned verifier to check can actually be cheaper than O1 just doing one candidate with the same accuracy.

1:02:53That becomes interesting because then you know part of the interesting thing about human civilization is on one hand we are general purpose You know reasoners, but on the other hand we also know that every job requires a tool and we do that too. We're basically we based you know the fact that you know Basically we doing everything that like a particular specialized tool does can be extremely inefficient in terms of the time that we are spending That is going to be the case for these reasoning models too to some extent because over and actually cost quite a bit right now. How much when it's going to change is anybody's guess. But that sort of brings up. In fact, there was a chef, a writer, the LSTM guy. That's great. So he should ask to him through. So yesterday I was in his talk and so he basically made this one of the slides basically was the bitter lesson is over.

1:03:52and efficiency is going to matter. And I completely agree with that. I've been arguing this for a long time too, that think about the following thing. The first time when we sent a man and a human to the moon, caste was not a consideration. We want to show that we can do it. NASA was the one just doing it. The second time and the first space as well as the moon.

1:04:18The second and third time, etc. may be okay, but by now it's Elon Musk sending people to space and supposedly possibly to the Mars too because the cast matters. Essentially, once it's been done, then you start caring about the cast that you're paying and computer science is actually quite a bit about the unsexy parts of cast just as it is about doing things that haven't been done.

1:04:47before and we are now in the second phase where we are actually going to care about basically how much am I spending in terms of the pre-training cars in terms of the inference cars etc and is the other better approaches that I can be using. This has been the case with computer science before too and it was just sort of became less of an issue for a while because We were LLMs were just system ones because there's no at all inference time cost. Okay, even though the post training was very costly inference time it was very cheap. Right. And so we didn't have to worry about it. Now we will worry about it. So one of the funny things that the elephant in the room for our plan plan bench problems on one preview was that

1:05:41The special part the normal classical planners that are meant to solve these problems solve them in like fraction I mean such a small fraction of the cost they work on our laptops and solve all the problems with hundred percent guarantees Right, so the question is I realize they're completely specialized only for that problem and then on the other hand you have this very general purpose thing which has cost as well as inaccuracies We start worrying about the trade-off. What level in this generality cost spectrum are you going to find home? That is going to be very important thing. And I think that sort of what Shabhap writer was, you know, hitting it when he said, you know, bitter lesson part is over that you do actually need to worry about the cost you are spending to actually achieve a goal. The first time you are achieving that goal nobody cares about the cost.

1:06:38Because it's never been done, so you're doing it, you get all the credit. But, you know, empty in time, it's being done, you know, because it becomes like a normal day-to-day thing, then the efficiency aspects matter. A few things on that. I mean, first of all, with O1 Pro, I think it's worth $200 a month, and you can call it 100 times a day. Of course, API is very expensive, but I'm already spending, you know, over $1,000 a month on Claude, Sonnet 3.5, but you raise an interesting point. Me, first of all, The utility of the No1 model, it's a bit of a weird model, right? It's useful in certain specific circumstances and if anything, because of the bossity and the distractors and the context,

1:07:19It's not really a model that you want to be using most of the time. But that raises the pragmatism and the architecture and the efficiency thing that you're speaking to. So I spoke with some guys this morning. And they have built a kind of neural evolution approach to designing multi-agent systems. And at the moment, we hack in the tool use. I would do a user debate pattern. Do we have a small model when we prompt it a lot? Or do we use a bigger model?

1:07:44and we're all just hacking together these multi-agent architectures and some of those architectures will even be doing the kinds of things that you're speaking about so rather than it trying to convince you that it got the right answer there might be a supervisor agent which does some reflexive there might be another agent which generates the planning symbolic code and runs it on a tool you know we're building these big complicated things and I think that's the process that we need to figure out now is building the systems that actually use this technology in the best way yeah so I think the thing

1:08:12I sort of agree but one thing that I want to point out that distinction is that there's two notions of use of these kinds of models. When you do a subscription model, $20 or $200, I would argue that that is by definition human in the loop with the model being an assistant to you.

1:08:35and it's a very different way of evaluation where you were unhappy with the previous model because it was wasting more of your time and it's not worth it. For you, this one was helping in whatever you were doing and you're happy with that. That's one particular type. In general, I've actually, I've always thought and I think we talked about it the last time too, that large language models and large reasoning models now too. There's no question that they are intelligence amplifiers. There's like no question on that part. Okay. I mean, if you want to use it, you use it and people are able to find users for that that's great. The part that I'm actually not talking more about and that's most of been, that's been most of our work is really, there would be scenarios where they, this become the end user facing systems where they'll make the decisions. They will just say, this is an answer and then I'm going to, you're going to

1:09:32execute this plan. So the robot will execute this plan or this is the travel plan for which I will buy the tickets. You don't get to come back in and say, oh, I don't like this travel plan. That's what you do in the, you know, the subscription model. But the one that I'm talking about basically the API access is basically what people, all the startups who are trying to build additional tools on top of these models. They are going to give specific autonomous functionality.

1:10:01And there, that's where I'm talking about the actual computational cost versus benefit for a certain level of accuracy at the end user time. Both of these are very different kinds of users and I actually have no question at all in my mind that all LLMs and definitely also LRMs are just great intelligence amplifiers.

1:10:28But that's not what my worry is. The whole thing has always been, my worry has always been that people are trying to put this in the end user facing situations where they'll actually make the decisions and some executor just executes it without pushing back. And when that happens, the guarantees matter in terms of the brittleness of the reasoning matters. If you are in the loop, you would never, you know, it's like if you have an assistant and an assistant, You may fire the assistant if they are giving mostly bad ideas, but you will never blindly just use the assistant's ideas. So you will always be the buck stops with you. That's a very different way of using LLMs. Then LLMs are the ones that the patient talks to. There's no doctor between the LLM, LRM and the patient. In which case, their accuracy matters and their cost in getting to a certain level of accuracy matters.

1:11:24are two very different uses and I'm much more interested in the second use than the first use. And I push back just a tiny bit. So first of all, I completely agree with you that these things used autonomously, they don't work. They don't work for all of the reasons that you said. But that's not how they're being used and they're not being used like that because they don't work. But what we are seeing is that all of the successful reimagining of applications or language models are completely interactive. So they have a human in the loop.

1:11:50and the human is supervising augmenting, redirecting and so on. The next step that we haven't seen yet, but we're starting to see, is having autonomous agent-based systems, you know, with multiple levels of reflection, checking and so on. For example, it could be a bunch of agents generating programs it could be contributing to a library programs the programs are being supervised not just by you but other users of the application and the whole thing just grows and it's a living ecosystem so there's some diffused form of human supervised verification and maybe in the future you know the humans might be increasingly taken out of the the front plane.

1:12:27I think that's a very sane way of using but I'm afraid that's not the only way that's being used and in fact most of the people, so actually the two issues one is if that's the only way, I'm very happy because it's like it's a tool and you would use it and the owner is still on you. Finally the buck stops with you because you are in the loop, right? But most of the imagined uses, at least from where I sit and the kind of startups that I hear from other kind of papers that I'm even reading, They're all about autonomous uses and that's where I'm actually looking at the fact that there is more promise than before it was very brittle before it's less brittle now Okay, but it is less brittle at the expense of cost and It's actually interesting that the evaluation strategies for both of these are quite different You know evaluating assistive technologies is very different from evaluating autonomous

1:13:26technologies and and it's not that one is a state technology evaluation is not any easier in fact you know I mean you basically you can say the evaluation is just if people are buying it and they keep paying for the subscription that's a proof that people seem to be getting some value out of it but it's actually pretty hard to evaluate correctly evaluate assistive technologies and that's a whole entire area And in fact, most of the people who are worried about misuses of LLMs have never, they're not the ones including like Francois, for example, Transocialia and his art thing, etc. It's all about, ultimately all of this is, we are interested in respect of whether you believe AGI is coming next week or next decade or next century. Everybody in AI eventually wants this autonomous

1:14:18Abilities to actually make intelligent action with guarantees, right and and that is basically where I think we will get there But prematurely saying whatever currently is there is already working is the one that a bunch of us are worried about and that's what we are pushing back on with the humans in the loop which are completely different thing You know and even for the code generation right now, they're like There are two different uses, essentially. There is also use of code generation techniques where it tries to kind of improve accuracy to the level that humans don't have to. It's not just an idea generation for the human. If it's idea generation, it's great because somebody else's job is online. It's not, you know, they're still like a buck stops with the actual program or anything. So I think that the autonomous one is the one that I care about at any rate. And that's the one that I'm worried about.

1:15:13the premature declarations of they're already autonomously intelligent and but I'm generally very happy that this technology exists as a human in the loop technology and it's kind of interesting from me sitting here to here you say that you actually as a user I mean you seem to be a more of a regular user than I ever have been you know of the LLMs and LLMs that it's kind of interesting it means something to me when you say that you like One more than you ever like to improve you and you kind of work with GPT-4 maybe but now like one little more and that sort of you Basically are getting value out of it, but you still can always you have the red switch you can decide not to take its answers, you know one pro Yeah, okay, yeah, okay. Yeah, the only difference is that there seems to be when it when it thinks for a long time. There's a qualitative improvement, you know

1:16:10I wanted to get your take on something else. So we're seeing, I mean, you had your LLM modulo architecture, and then we've got this huge approach of test time, this kind of green-blatting approach, so you green-blatts the model, and you get it to generate loads and loads of Python functions. In a way, this is the sort of thing that we like, because we like... Yeah, the Arctic thing. Yeah, yeah, yeah, yeah. Green-blatting project, yeah. And we're seeing that in lots and lots of different ways. So doing loads and loads of inference, and then we've got this Python functions, and maybe we do...

1:16:39you know library learning and remixing and you know we're in the world of codes are we're using code we're generating an explicit function we can verify it we love that we know we're in a very happy place but now we're seeing an interesting shit so certainly on arc and on several other papers people are moving towards this idea of transductive active fine tuning and that simply means rather than generating an explicit python function and doing it loads and loads of times let's just generate the solution directly just using the neural network And this is a step away because we like programs because programs are touring completes and we understand what they mean and everything. And now there's a whole load of people that say actually the neural network can just can just do whatever the program does. Let's just let the neural network output the solution directly. What do you think about that? So to be honest, I haven't followed that work as closely. So I have to my answer is somewhat like more generic. I would be surprised.

1:17:36I mean, I would have the same bias that in fact, there's an old saying that why write programs when you can write programs, write programs. That's the version that we are talking about is basically you want to generate higher level code that generates the solutions. This has always been the, the conceit of computer science. So I am surprised. I don't actually know specifically the work that you are referring to in terms of just going back and directly going for the solutions because honestly, in the context of inference time scaling, one interesting question is you generate loads and loads and loads of candidates. The candidates can be the direct solution candidates are the code candidates either which way. And then you still have to have verify. If it's code, you need to have code verifier. If you have solution, you need to have solution verifier.

1:18:31And one of the interesting questions is, where is this verifiers coming from? And there has actually been one of the more effective ideas that we've been pursuing is you can essentially generate verifiers. Of course, there's symbolic verifiers that might be there for specific things. And we can use that as a lemma or digital framework. But you could also use learned verifiers where essentially you just basically learned to do discriminatively. What is a solution versus what is not a solution?

1:19:01a third idea is generate the code for the verifier and then connect it and that's actually in fact it's still for at least in our case it seems to be promising we are working on some things that going to come out soon but you know basically I still think that and especially in the context of lm's in the context of lm's okay so it's like again it's a very different thing if you're not having lm's in the loop at all it's a different question but if the elements are there one of the things they're actually great doing is like outputting you know basically they can output code as well as solutions in which case you know the code can output lots and lots of classes I mean you know lots and lots of classes of solutions can be verified by the code and so if you correct it once then it will you know work for a longer time in a sense and so

1:19:56I would still think that at least for the, you know, inference timescaling verifiers case that seems to be still a good idea. I don't quite know the specific context from which were you saying these trans people are saying that the trans ductive directly, you know, guessing solutions would help. I'm not sure whether they still have LLM in the loop are they just saying we'll just directly train a separate neural network. Well, I'll sketch it out. So solving arc, they have two alarm 8 billion models.

1:20:25And one is generating Python programs, and they green-blat it. The other one is trained separately just to output the answer grid directly. And in both cases, they do, you know, inference time compute. So either generating lots of Python programs or doing active fine tuning of the direct solution one by augmenting the test time examples. And what they found is like on the then diagram of the, you know, their success rate, they find that For some problems, the program works really well. You know, like the green, black approach. And for some problems, you know, it's certainly things like mosaics and spatial perceptual type stuff. The transduction works really, really well. And this is kind of weird because if you think about like the space of functions that the neural network could could reason about, they should be the same. So I don't know whether it's just because of limitations in the neural network or characteristics of the problem or something that you see.

1:21:20Interestingly, to me again, it depends very much on the space of solution configurations versus space of code configurations. There are many problems where solution might be of less quote unquote syntactic complexity than and and so and neural network that can kind of guess a string may not be able to guess something that looks like a syntactically correct Python program, right? LLMS actually can do the later and so It is interesting that if you can do that, and if you still go back to neural network to actually directly guessing the solution, it being a more useful step, we can, the stuff that we are doing for the verification thing is still in the, you know, initial stages, you know, and we haven't actually checked this kind of a trade-off, whether it would exist. So, you know, I have no more insights specifically on why that might happen. Wonderful. What are you doing at the conference this way?

1:22:18That's fun. So I'm just here today and I think we did this chain of thoughtlessness, chain of thoughtlessness paper. And then I kind of said that it's like mostly when we wrote it, it was like, it can't follow procedures. So I should be able to show it. But now actually, I explained the whole thing the way I explained to you here in the beginning, essentially go from prompt augmentation. And so I kind of think that like shop in house said, life must be lived.

1:22:48but only make sense backwards. And you know, papers also only make sense backwards, you know, after a while of writing, you know, you actually look at it and say, what I really want to say is the reason chain of that is not a great idea is because you really want to think in terms of prompt augmentations and humans coming in the loop becomes less important. So that's what we did and then I'm actually going to this compound systems thing and at great time there are like 16,000 people and you know, running into lots of or friends and so on. Yeah, one of the best moments from the last interview is when you're talking about that paper, it's saying that, you know, that they can catch. You can teach someone to catch two facial or three facial. Yeah, that's yeah. Yeah. Yeah. I mean, that's basically because it doesn't quite know how to generalize and so I kind of made that thing that yeah, essentially because you have to kind of give it examples for four word problems. Again, give it examples for seven word problems. Again, give examples of nine word problems, et cetera, and then try to improve it.

1:23:47whereas the conceit people think is when people when you say this they'll say, oh, it must be doing procedure generalization. The interesting thing again is I think we had this conversation last time too that the way I look at this, I mean I'm skeptical only because I've just having some additional background and one of the things is Makarthi, Chan Makarthi who is the founding fathers, I'm the guy who.

1:24:16coined the name artificial intelligence, basically said the Holy Grail of AI is an advice taker program. And advice taking is AI complete. And if chain of thought is able to make LLM steak advice, that would be pretty impressive. And I kind of went in thinking that there has to be holes there. And so that is where that fish, one fish to fish thing comes in. But but a more interesting thing is I think Deanthropomorphizing LLMs and trying to think of them as these basically this alien entities for which you know arbitrary you know prompt augmentations prompt augmentations will can you can generate good behavior so by the way one example of this that people should be thinking about is if you think of Jail Breaks and LLMs Jail Breaks are you give a normal prompt and you give this

1:25:14particular, carefully constructed learned sequence, you know, Zico Colter's original paper, his group's original paper shows, that sequence makes no sense to humans. But it will make most LLMs provide a deterministic behavior like saying, got you or something of that kind. And essentially, that should tell us that they're not seeing language. And so the prompt augmentations don't have to make sense to humans in the loop. And And that's okay. And because in some sense, looking at things, the only change of that sort of made sense to humans was giving this false impression that somehow LLM are doing things like we do, that's not the way it is. You know, so might as well just live in, you know, go with what they can do and optimize directly, which is what the inference time scaling and post training methods seem to be doing.

1:26:09Yeah, the one thing I get stuck on is we can criticize individual elements. I mean, yeah, that they are approximate retrieval engines. My co-host, Keith Daga, he's always at pains to point out theoretically that they're not too incomplete, you know, that finite state automata and all of this kind of stuff. But the thing is, it all breaks down when you talk about LLM systems. So even with the chain of thought thing, right, I could have another supervisor model that could generalize the prompt to go to five fish, six fish, and so on.

1:26:38we can easily build systems that overcome all of these criticisms. So at some point, does it just seem like we're just, we're making criticisms that can be easily. No, no, actually, it's a very good point. So in fact, after I, after this, I'm going to this compound systems meet up. I don't completely a big believer in that whole direction, but there are some people who don't want to believe that the usual alarm officials don't. In fact, by the way, it's a very interesting thing that openly I was at pains to point out.

1:27:06that O1 preview was a model not a system. It's not me and you saying it. It's them saying it. They would like to say there is this one size fits all model that will do it. And so it is reasonable to take their word for that. But apparently I also like the compound systems work. And it makes an LLM module is a compound system. And that's what basically it improves on all the limitations of LLMs and a set of limitations of LLMs. I'm completely fine with it.

1:27:36It doesn't matter to me as long as I can give guarantees and it's in a safety critical scenarios. I'm fine with it. I don't have that bias. But if you are saying a single model will do it, I will take you at your word and then see whether or not that's true. That's a fair thing. It seems to me. Why do you think Google have completely embraced hybrid systems? Open AI, they...

1:28:01I really clinging on to this single motto that does everything. I think they're slowly changing that, but I think there was a reason, there was a... I think, to some extent, I can understand. In the sense, it would be the sort of thing is this anthropomorphization again. We only have one brain. It's not that we have a brain for eating and a brain for... Just one brain, right? And so it would be nice if what we are trying to do would somehow, basically...

1:28:31Be this this one size fits all this general you know system But at the same time there's also this issue of whatever I do I want to provide Guarantee is safe, you know, so that it can be used in safety critical systems and these so That's a problem is the modern AI and Neuroscience and cognitive science. They're not one and the same right I mean everybody understands that essentially. I'm a neural networks themselves are not really that well connected to brain and essentially they're like biologically implausible and LLMs are definitely not but there's nothing wrong with that just like we say you know the planes don't have to flap their wings you know so these are but we don't essentially try to make sense of planes and birds in in in the same sentence you know because they both fly but other than that

1:29:27you know the mechanics are different the things that the flight equations are you know not not at all the exactly the same things you know that's going to be more of the case with the LLM2 and as long as we realize that you know it would be good but I think openly I think originally they were hoping my sense is much of these people are hoping that they will just get one you know like two words with one share like we'll get AI systems as well as understand how the brain works, but I don't really think that part nobody really believes honestly. I mean, you might use these systems to improve our understanding in actually doing neuroscience. In fact, I think, what's his name? I think that he basically says, you know, obviously these systems help in actually doing neuroscience research, but they're not actually telling you how brain does suddenly.

1:30:23work so but like that might that's just speculation that might explain why people you know open AI and some of these people were sticking to but I mean the kind of conversations I've been having on the sidelines in the conference already the company is there the startups et cetera they're already sort of going much more into these hybrid systems much more into these compound systems and and it's like you know that would basically not be a single system but open AI Also, it's slowly coming up with these fine tuning models. They have these RL fine tuning stuff for your specific kinds of scenarios, etc. So, it would be interesting to see, but I think just going back to your original idea, I think compound systems is a very different, they basically, the individual role that the LLM play, LLMs have to play is much less demanding. In fact, one of the fun things is,

1:31:21we can do LLM modulo with normal LLMs or LRM modulo with instead of LLM I call O1. And so it will be the generation of candidates is costlier. And we actually show in the strawberry paper that we can improve further the performance of O1 preview on some of the problems even though we couldn't change the how much time it takes to think etc. We can just by calling it multiple times with the correct you know better criticisms of the you know instance the problems answers it gave we could improve its performance accuracy quite significantly so that is still using them in a system you know LRM's themselves can be used in a system but I think oh one open AI itself just wants to call it just models up until now let's see what happens