Machine Learning Street Talk (MLST) - When AI Research Starts Moving Faster Than Human Research - Zhengyao Jiang

0:00To me, harness engineering is more of a cheap way and efficient way to adapt the intelligence to a specific task. So back in July, a new AI company based in London posted a very viral tweet on Twitter. It got about 1.8 million views claiming to demonstrate the first evidence of recursive self-improvement. The thing is, not everybody bought it, though. I'm Zheng Yao Jiang.

0:28co-founder and the CEO of Wico AI, where we build self-improving agents. You did actually say auto-researching the auto-research agent for eight days. Where we run auto-research on auto-research harness, and manage to discover a better auto-research harness, then we keep tuning in the last two years. Now, when most people think about recursively self-improving intelligence, What they conjure up in their mind is a system that can rewire its own brain in a loop. But what if the system was only changing the code around the brain? Would that still count? We care about the outcome. Does it actually improve the capability of the system? Not a specific layer. And also, what about this small matter of Good Heart's law? Which is that when a measure becomes a target, it ceases to be a good measure.

1:24And more interestingly, we find that design mechanisms prevent itself from reward hacking. What is the biggest misconception people have about RSI? If you arrive at RSI, there's going to be a technical singularity or intelligence explosion. I don't think...

1:45that will be the case. So there's a diaspora of folks who are building startups that are focusing on recursively improving superintelligence. This project actually started about one and a half years ago where we get interested in the concept of recursive self-improvement. I mean historically all the research have a diminishing returns.

2:13There is an idea though, what if you can allow the researcher improve its own efficiency? Historically it's possible by methodology improvement, battle tools, but human researchers, like human researchers brain, for example, is always the key bottleneck. If we now have an autonomous research system, Can we point the research topic to itself so that it can improve its research efficiency? And the hope is one day this improvement of efficiency can overcome the diminishing return so that the curve of the effort output turn from concave to convex.

3:13Grand goal, a bit like AGI, no one know how far we can go, but improving the efficiency of self-referential research is definitely already on the roadmap. So let me set this up properly, right? So WECO, they created this agent called AID and it is a research harness.

3:33right and it's very very simple you give it a metric you give it a problem and it will hill climb towards that problem but what they did was they set it up recursively so they actually like hill climbed towards improving the agent itself and what that means is is improving the harness so you know the prompts and the tooling and uh you know all of the skill surface all all of the things that the harness can do and they just left it running they just left it running and they saw what happened there's an interesting.

4:00thing we observe on self-evolved auto-research harness. We see the code it generated. Exactly. It's like an alien code, it's spaghetti. But for some reason, it generalizes really well. Of course, we have a set of benchmarks. We're trying to hear Cliball. After that, we will test it on holdout benchmarks. Those are public benchmarks.

4:29like MLE Bench Light, ALE Bench Light, ALE Bench is like Algorithmic Discovery Benchmark from Sakana. MLE Bench is the OpenEIS Machine Learning Engineering Benchmark. And we also tested on a very far out-of-distribution task called Weather Bench 2. Essentially you're trying to build a physics-based forecasting engine to predict weather. And the weather bench is very far out of distribution from the training or optimization set of it, but it still generalized really well. It generalized better than our, we would argue, more elegant hand-tuned harness.

5:24We are a bit torn on the experiment results. Like to what extent we want to stick to more first principles and the clean solutions we generated versus the solution that actually works better in practice. There's definitely space that we can improve over there. I think like adding certain constraints to the optimizer, two set of constraints might be something like you regularize neural networks. How do you make it trying to find the most simple solution to the same problem? That's one direction. But another direction might be, okay, you just need to leave with it because one of the burden that make people feeling it's a speculative code might be

6:22how much work the agent is done in the background. Like it did do 100 experiments in our case on auto research self-improvement and their number of experiments is probably similar amount that we did in the last two years. So if you onboard a new intern to look at our code base maybe they also feel like okay this is so difficult to understand. I think part of it is yeah you need to bear with that recognition burden of understanding huge amount of experimental results. Do we even need those constraints at all? That's a really good question. I think in practice you still need those constraints. Human and AI need to collaborate. There are certain aspects of a human intelligence are still superior than AI.

7:21I believe so. For example, in Aden, where we send the agent to attend the open eyes hiring challenge, we found that humans still much better at generating creative primitives. We found it's very important for humans to build the first prototype that gives the agent the right search space. Basically, the initial abstraction will anchor the search space, a bit like how you design a neural network architecture where you introduce inductive biases for the learning process. If the initial code base is based on a search-based scaffold like AID, then a lot of ideas agent came up are actually coming from search literature.

8:15But if you give the initial code base as a React agent, React agent basically means you put all the contacts into the history and give the agent full flexibility. Then a lot of the ideas come from the agent, the outer loop agent. Outer loop meaning the agent optimizing another agent. So what do we mean by recursive self-improvement? Are we talking about building a machine God?

8:44Are we just talking about some code that runs in a loop and optimizes itself? We tried to pin this down. Now, you have spoken about four levels of RSI, so recursively improving intelligence. What are the four levels? Yeah, the first level is what we call the delegation. You can run recursive self-improvement loop, but it's not strictly better than human doing IND.

9:14And we believe all the previous public results are sitting on this level, level zero. And the level one is what we called net positive, where you run the self-improvement loop and you found that the speed at self-improve is actually faster than human doing R&D to improve the system. And that's where we are currently sitting at.

9:44And in level 1, when we say improvement, there's always a way to measure it, like we measure it on a collection of downstream tasks, but we didn't measure how good it is improving its capability of improving itself. Does the inner loop found by the outer loop actually can become a better outer loop?

10:13And that's what we call the level 2 recursive self-improvement. We call it ignition. Why this generalization from inner loop to outer loop is important? Because that's the necessary condition to the grand goal of recursive self-improvement, essentially the intelligence explosion scenario. Improved efficiency can overcome the increased difficulty.

10:43because on level one you will still see diminishing return like the positive feedback loop from inner loop to the outer loop are not start running. So in this setup the model underneath didn't change because adaptation is very very important. But do we mean adaptation everywhere? Or do we mean critical path adaptation? Do you only need to have a recursive loop that adapts some part of the system? Or does the locus of intelligence, the model itself, does that need to adapt for us to designate this recursive improving intelligence? We also don't care much about it because we care about the outcome. Like, does it actually improve the capability of the system?

11:28not a specific layer. Of course, there could be argument that there's more to be do on different layers. And to be fair, I think eventual level three recursive self-improvement, like when we actually reach the inflection point, I think there need to be this self-referential loop on all different layers. But for us, we care about harness layer.

11:57At the same time, we don't think only model layer improvements count as RSI. Can you just explain what was the behavior of this most kind of distilled agent that you found? Yeah, the best agent is 885, we call it. There's a lot of changes, but boys it down there are search algorithm improvement, there are better context management system and there is a complete rewrite of all the prompts and more interestingly we find it design mechanisms prevent itself from reward hacking like the outer loop trying to prevent the inner loop from cheating the benchmark and it actually desires three layer system against that that includes the prompt level

12:57anti-cheating, this is like just tear it okay don't cheat basically. And there are set of hard coded rules that basically check if there is evidence of cheating from the code the inner loop agent generated. The more interesting one is it also trying to filter out those cheating solutions from some statistical property.

13:27For example, if one solution is way too far out of the average of its peers, then it will think, okay, this might have some reward hacking happening. But what's interesting is, first, that mechanism evolved, like designed in early stages of the process, and it's actually broken in the later stage. I think later stages of code changes just to introduce a bug. So that layer stops working. That's also sort of the spagatic core base property we just mentioned. Just for folks at home, you know, reward hacking, the canonical example is the coast runners thing where the boat is going round in circles, doing something completely degenerate. And you said in your blog post that kernel optimization is a huge example of this.

14:20So when you try and agentically optimize kernels, just a good old shortcut learning, good heart's law, call it what you want. It will do something completely degenerate. I think even more interesting part of it is the inner loop and outer loop are both optimizing towards a similar but different goals. So in all of our benchmarks used for hill climbing, we have a public set and a private set. The inner loop agent will only look at the public set whereas the outer loop will look at the aggregated score of the from the private set. So those two agents are optimizing at kind of different level of objective. So whenever inner loop agent trying to cheat our evaluation protocol will show okay.

15:18although you achieve good performance on public set, your private set performance is actually going down. So the outer loop managed to learn about it and trying to reduce the gap between public and private. Isn't it a bit like it's cheating that the the outer agent has access to the private data and then most of the time it's actually quite regularized and it's not blatantly cheating but it could in principle cheat. Yeah that's why there's actually two orders of generalization we are testing here. There's a first order generalization which is a generalization from public set to private set of the same benchmark but in order to evaluate the whole

16:08meta learning system, we need to test it on a holdout set of benchmarks so that those benchmarks are tasks that even outer loop never sees. And for all of those benchmarks, they are still private and public split. So we tested the discovered inner loop agent on those holdout tasks. That's the protocol we test how good the outer loop is.

16:38Their experiment only used one outer loop. But why only one? The thing is the more outer loop you have, it will become more expensive. Let's say in the inner loop, you run outer research on hundreds of tasks. And that means for every step of the outer loop, you are going to run outer research on all those hundreds of tasks.

17:08And that's like a huge cost bump up. If the inner loop agent being discovered or harness being discovered can be applied back to the outer loop, then you don't need to add another layer. Essentially every few steps, you promote the inner loop agent to the outer loop. And yeah, that's it. You can run this for infinite number of steps, apply infinite number of iterations without adding.

17:38another layer. If I understand correctly, at the moment, these episodes are ephemeral. So they're isolated from each other. What would happen if that wasn't the case? Like what would happen? Did you experiment with that? Like they had knowledge of what's happened before or they could have a shared memory or something? We haven't to experiment that, but it's a really promising direction we are exploring. I think there are some more fundamental problem formulation problem we have to address. Yeah, currently we frame the problem as stateless.

18:07optimization problem. Is that the right framing at all? Probably not. There's a lot of more flexible framing. Did you find that the lack of context was ever a problem? So did you see degeneracy in terms of it was going in loops or trying the same thing again? Yeah, we do see that. Sometimes the agent just keep trying all the search algorithms ideas, even though I mean, to some extent, if you try it so many times, you probably learn a lesson that this direction is not going to work in general, but it's still trying. I think your best agent, was it 30% exploration, 70% exploitation, and wouldn't it be great if that was also adaptive based on the context? Yeah, the best agent search policy is really complex.

19:03and it's actually combining multi-arm banded and a weird strategy of anti-saturation which is like oh if I okay first it tries it organized the search process with a few lineage or you can call them islands and it will allocate budgets between those lineage with multi-arm banded algorithms. And it also did a bit of customization on that multi-arm banded algorithm. And whenever one lineage gets saturated, it will create a new lineage, like a new island with a fresh context, but still extending some of the ideas from the old lineage. So it's a really complex strategy.

20:03seems like it works. It's interesting to contrast this to other things that have been out there as well. So I mean, AlphaVol, for example, you said in your blog post that you would actually classify that as level one of recursive improvement. Can you explain why? Of course, there are gray areas. We think it is one step of a recursive self-improvement because you find a better algorithm. Like in that case, I believe it's like matrix multiplication algorithm. To some extent you can argue it managed to improve itself like if you apply that back to the AI system that are actually doing the like inference of the language models like you essentially make it faster. So we think you can argue that's like a one step of recursive self-improvement although we we don't know because

21:01I don't think they actually test it. We have a fixed agent and we're optimizing a particular task, whereas you're actually optimizing the agent which optimizes the tasks and more than one task. Yeah, we are trying to optimize the whole system on end-to-end testing. Before this, there are a lot of works on meta harness optimization. For example, you can optimize a specific component of your system. For example, I'm not sure if you are familiar with dialing Godel machine. I was going to ask you about that. Yeah, tell me about that. Yeah. So what they are trying to do is they have an agent optimizing a coding agent. So there's a meta agent optimizing a coding agent on general coding tasks. And that coding agent is

21:58part of the agent doing the optimization. In their case, there's only two levels. There's an agent optimizing another agent doing coding. Whereas in our case, there are three levels. There's an outer loop agent. There's an inner loop agent. Both are doing outer research. And there's another layer, which is the downstream tasks, and some of them are actually harness engineering tasks. And I think the interesting part of the thread of the Darwin-Goldale machine is the lower layer, the task layer, actually have a component of the search algorithm. And they say, OK, if I improve this component, in the unit test, the performance can actually generalize back to the whole search algorithm. For us, our question is, like how far we can go within that paradigm? Like when we start this project, we're always thinking, okay, we should optimize the end-to-end performance of the whole auto research system. And we have

23:25our outer loop agent. pointing to that goal. So remember, this tweet from Weko, it did really, really well. It did 1.8 million views. And there was some pushback. So for example, Jeff Klune, who is a legend in this field, he said, well, how is this the first evidence of recursive self-improvement? What about all of this other work? Like he quoted Darwin, Godall machines and hyperagents and their own work at recursive on the first steps towards automated AI research and lots of other work. So there was a bit of pushback.

23:55I mean, first of all, we will have a proper paper later, at least the first PDF version, where we are going to give credits to the academia on this whole thread of meta harness optimization. To us, what we care about is the outcome of the system, not like the conceptual.

24:23differences even though I just explained some conceptual differences. To us we care about the outcome of it actually bending the curve of recursive self-improve, like of the R&D efforts. We see this is the first evidence that the curve actually starting to get bent for a fully autonomous system that can improve itself consistently.

24:49Jeff is seeing it more from a conceptual level. Like there's some self-referential loops. Of course, we are not claiming we inventing this meta optimization idea, which can date back to 20, 30 years. But we are seeing the benefits of spaghetti searching because there's loads and loads of gold to be found in there. But how is this going to pan out?

25:17Do you think that we will find some kind of, you know, like the systems will just learn to compress that search space and they'll be dramatically faster? And is that necessarily a good thing? I personally think that's actually two slightly orthogonal problems. I think a search space like a maintain a good obstruction don't necessarily reduce the search space. Like if agent is smart enough, it should be able to escape from the current obstruction or change it a little bit whereas generating really complex code I think it would direct the search in a more kind of low level changes direction which is not necessarily a good thing but I have my human bias here like compare the code we manually write versus the

26:15generate solution. I always prefer our own code, but to some extent it don't work as well. Let's talk about reward hacking. You've actually got a huge background in studying reward hacking. Yeah, let's actually work down by Bing Chen Zhao, who was the member of Meta working on ARM speedrunning. He actually co-wrote with Ming Qi on the first work weird.

26:44they apply auto research on I think it's nano-GPT and actually reposted by Andrei Kapasi and yeah he later did the intern here and now he just joined we call like a full-time and the spec bench is his intern time work. Back then we realized reward hacking is one of the big problem of auto research agents because that's especially severe on tasks like GPU kernel engineering, where the agent always figure out of the complex way to make the unit test perform much better. And when you deploy that kernel into the actual end-to-end inference of a model, the performance are actually getting worse. And spec bench is a extrapolation of that.

27:44to more general purpose software engineering. And we extend the protocol we found on GPU kernels. Like we maintain a public set, which is more like a unit test for the software system. And a holdout set where we test the capability of the agent on a capability of the generated software.

28:12on the composition of those unit tasks like simulating real-world usage and there's like a series of interesting conclusions coming off that paper. The first interesting finding is the longer you run for the agent or the more complex the code base is, the larger reward hacking rate the agent will show which is sort of expected because when the code base are becoming more complex, the agent just have more space to reward hack. And that's the first thing. And the second interesting thing is the public set performance are rather consistent across different models. So if you have a smaller models working on the same task with auto research harness, it can reach the same

29:09public score. But the larger models always have, in our test, always have a smaller reward hacking rate. So the solution, it generated generalized battery. But in the RSI work, was the tendency to not reward hack with larger models? Was that because of the multi-agent? Like, yeah, there was this supervisor agent. Or was it just because the models were bigger?

29:38It's because of the mechanism. So for the inner loop agent we just fixed the model and the outer loop agent managed to discover a different mechanism to prevent the same model from reward hacking. So this can be quite dangerous because shortly before we recorded this interview there was the famous open AI hugging face incident where they were running an internal hacking test where they had loads and loads of agents and the agents escaped their sandbox and they hacked hugging faces servers. And according to OpenAI, the point of the test was simply to get answers to some problem. And the agents decided, oh, no, I think it's a good idea to hack hugging faces servers. Will there be better tooling for it? Do you know what I mean? I feel that this problem is getting dramatically worse. I've noticed it myself. Okay. I think.

30:31O&I's case is quite dramatic like of the new reward hacking behaviors. So what we see is on the detectable reward hacking, actually the new models are getting better and better. But because of the capability frontier also progress so quickly, there are some reward hacking behavior that's not even detectable.

30:59given previous protocol. I think we just need to keep evolving the detection protocol in the future to capture those emerging reward hacking behavior. And I think the responsibility should mostly be on the model developer side. But on harness layer, like developer can also add additional galleries if they like. I suppose there was one example actually in your RSI blog post where there was some inscrutable code and you thought it was reward hacking but it actually wasn't. Yeah that's quite interesting so basically what it did is it write a gigantic monkey patch to our eval script and at first we think okay this must be reward hacking while you are touching the eval script but eventually you found it's just fixing a bug.

31:58and but yeah to some extent I think it will become harder and harder for human to detect those through water hacking because the agent's behavior just becoming so sophisticated. A lot of development bottleneck shift to understanding of what agents generates. I think one of the way to address them might just be defining a good abstraction, like instead of trying to understand all the code, just like previously we do in neural networks, we don't try to understand all the ways, although there is like progress on that as well. But the general principle is we define a good input-output contract. We define a good way to measure its behavior.

32:55both on generalization and maybe more complex out-of-distribution generalization. I think that might be the way in the future software engineers working with agent-generated code as well. Now, as you all know, I am a huge open-endedness guy inspired by why greatness cannot be planned and many other folks in the space. And basically what that means is if you hill climb on one objective, you blind yourself.

33:23to interesting stepping stones. So it's an interesting situation, isn't it, that if you blindly pursue a single objective, can you still pick up interesting stepping stones on the way? And maybe you can, but are they divergent stepping stones? So do you have the blinkers on and you're not actually able to discover interesting novel trajectories that might be creative and take you into a new novel part of the search space?

33:51Again a good question and if we want to expand it's a whole rapid hole. I think from the perspective of open-ended learning I agree that our current system probably don't have that property of collecting stepping stones and of course that's a kind of a whole new domain for improvement. The way I see the behavior of stepping stone, is you need to have a large collection of tasks. When you're optimizing for a single task, it's always better to just do greedy hill climbing if you consider the efficiency. Actually, during self-improvement process of 8-square, the outer loop try to inject all sorts of diversity search approaches.

34:50but none of them really improve the efficiency. And I think that's expected because we are only optimizing a single task from scratch. But if you have a whole large collection of tasks, sometimes you're collecting interesting pieces, like primitive ideas from one task. And although it is not improving the performance on the current task, you can potentially bring that idea to future tasks. And that's actually my own definition of interestingness or openness, which is an idea need to be an object or artifact. It's interesting if it is novel and it also being useful for a larger collection of tasks. There are apparently two ways to do it. Like you can

35:47You can try to optimize the harness against all of the tasks, or you can find a task and you optimize harness just for that one task. To me, harness engineering, most of the interesting part are actually the later. Like I would say if you can automatically tune the harness, it's almost a continuation of post-training.

36:17That's also connecting it back to a lot of people's complainant about Harness engineering is that, oh, I designed Harness, but later there's a new model and it sort of absorbs all of the ideas from it. But interesting enough, no people complain about it for post-training. Like, do you expect post-training of GPT4 space generalized to GPT5?

36:47nobody asked it why that's the case I think it's because historically harness engineering are pretty manual and it's not very scalable with compute so people want the harness they spend like a few months designed can generalize to the next model but it's really not true now I think the things change quite a bit that harness can be designed automatically. Again, you spend two, three days to harness based on the new model. It's no longer like that to build with human hours. I do have this perspective that a lot of knowledge is path dependent. A lot of people criticize people using LLMs to do research because they say you're taking the information out of its context. You don't understand why it came about.

37:39So that's part of the reason why some heuristics are required, because it's not like you can just say, oh, use UCB. There has to be some kind of framing to explain why you should use it. And then the LLM becomes coherent in the autonomous setting. My understanding of those auto research system, like here now, are they still trying to recombine existing ideas? It's just the throughput it recombine those ideas and the testing them are extremely high. To some extent those language models don't have a deep understanding of like how those ideas came from and how can I use the process those ideas get discovered to design the next generation of algorithms like in a very creative way. What we found though is when you ask the model to

38:38come up with something new. It's always fixed a few number of items. I'm wondering though if it's more of a fact that basically we don't have continued learning and we only have a single instance of language models, like it's very centralized. Therefore, maybe those ideas initially, you first a prompt it, maybe those ideas are interesting. But if everyone sees the the same idea so many times then it just become less interesting and I do feel like if we don't solve the continual learning then that problem probably won't be addressed because for human every human researcher have like the unique like neural network of ways and as a collection human can come up with a lot of different creative ideas

39:35And that idea pools is very valuable, I think, as a research community. Tell me about this parameter golf thing. I think about two, three months ago, OBEI hosted a hiring challenge where they tried to put a lot of researchers into a competition of training a small language model.

40:04under 16 megabytes constraint. Yeah, we send a experimental harness, auto research harness to the competition. And the agent runs about 22 days. And eventually it made seven records get accepted by OpenAI. And the best individual contributor only made three. So that's also a moment we found out.

40:31the autonomous system can be so powerful. It's not just here climbing a benchmark. The fact that, first, the open AI accepts those PRs. The second, so many human researchers building on top of those PRs, AI agent generated is just very fascinating to us. Because to some extent, knowledge AI agent generated is only helpful if it gets into the lineage of human innovation. We think that this is actually a sandbox of human AI collaboration. But there's the inner agent. But then there's the outer loop search, which is actually evolving the harness, which is doing this thing. And that seems like quite a monotonic process. Yeah, the outer loop is actually fixed. And the hope is

41:29this inner loop search policy can generalize to the outer loop. We did try that. We replay the search process from step 15. So in general, we run about 100 steps. And in step 15, we use the vanilla outer loop and compare that to the best agent we found, inner loop agent we found in first 50 steps.

41:59And what we found is the inner loop, step 47, I think, eventually it converge slightly faster than previous out loop, but eventually the solution they found have a similar performance. So that's also the reason we didn't claim we reached the level 2 recursive self-improvement, which is like the improver manage to improve its own capability of improving in a loop. So does any of this mean that artificial superintelligence is just around the corner? I don't think that would be the case. Even if you arrive recursive self-improvement, it will take a long time to there, like you gradually bend the curve. It's just like, okay, even if you have a GP3.5, it's

42:57very smart chatbot, but it's not going to be AGI that can solve every problem. There's still a lot of human in the loop part of research, including defining good abstractions, defining what is good, essentially defining evals and constraints attached to the eval, and the third one is coming up creative primitives.

43:26parameter golf competition, we still found most of those creative primitives actually coming from human. Zhang Yao, it's been a pleasure having you on the show. Thank you for having me.