Machine Learning Street Talk (MLST) - Designing How AI Grows — Tom McGrath

0:00which is like Neil Nanda says SAEs are dead. There you go. You can use that for the intro. I think interpretability is, I think of it as a natural science, you know, like physics, biology, chemistry, but it's a natural science that you do completely on the computer. And so this means that like we should be able to kind of again speed run science once we have agents that can do kind of experimental work for us.

0:23and the ability to do that experimental work, like as fast as they need it to happen. It's like real scientific work to do. No barrier to research, scientific work to be done. And it's sort of gated on both empirical data collection and theory building. Dario Amodi, so he had a blog post called The Urgency of Interpretability, right? And he gave this wonderful analogy of a bus. We're all on the bus and we're hurtling down the road and we can't stop the bus.

0:51but we can potentially steer it. But the window is foggy at the front, so we can only really look in the rear view mirror. And also the steering wheel doesn't work very well, so you can steer it a little bit once every few hours or something. So interpretability is a bit like defogging the front window. And what you're proposing is the ability for us to steer the bus, essentially. So I feel like if anything is going to go, if any science is going to get revolutionized by intelligence, we should make sure that it's interpretability.

1:20And I think that it's possible that it just goes an order of magnitude faster in the next couple of years than it has in the last decade. When I think, why am I optimistic about interpretability? It's partly because I think we're starting to have really good traction, but also because I can imagine this incredible speed up. Right, which is almost like there's a little man.

1:40inside our brain and it's a form of convergent evolution because we interact with the world using our physical affordances and so on. And could it also be the case that there's a little world inside neural networks? Oh, very good. You had quite a nice piece actually in your intentional design blog where you were saying that there's almost a spectrum of possibilities, right? We can write a program to do something or we could admit a lot of ambiguity.

2:15And where on that spectrum do we want the foundation models to sit? And when we build applications, where do we want those to sit? Yeah. And that's sort of part of the point of the intentional design idea is, at the moment, you can have like one or the other. You either, you either write a program like it's the Stone Age, or you get a model to do it. And that model will have been trained like, it just gets whatever it gets from its training process.

2:43we can't select it like when you write a program that stuff only goes in if you intend it to go in and some bugs but like we want to be able to have this sort of spectrum where you can chew you have more like engineering ability in in the model creation process so you can say like you know i want to learn this but not that and i think that that's like i think that's going to be that's quite a hard thing to do we're sort of trying to imagine a new way of doing machine learning which is more which brings intelligence into it. But I think we could like really change the way we do machine learning if we can figure that out. I know. I mean, when I interviewed the Apollo research guys, I mean, they were kind of talking about these conflicting objectives. So, you know, what the developer wants, what the platform wants, what the grader wants and so on. And I suppose this is kind of talking about this, you know, when you're in the intelligence regime, it's really, really difficult to specify exactly what you want. And it's very, you know, possible in a novel situation for

3:40the calculus to change, right? And the model all of a sudden it would decide to do this and instead of that, which makes me think that engineers are going to have to increasingly take more responsibility because do you think it's possible in principle just to kind of train models that could robustly deal with all of these novel situations? Or do you think it's more of a, you know, engineers have to take some responsibility? Probably some both. Like currently, I think it's not possible to our current training methodologies don't seem sufficient to give this kind of control over training. And so it's all on human engineers with their AI assistance to secure these systems in a different way. Obviously, once models get a bit smarter, we're already seeing this. That becomes harder and harder and harder because they have all these additional intelligent attacks they can do. And so the question is like, how do we make it?

4:39easier to train them better so that this is like not a natural part of their behavior and supervise them better so you can kind of capture them when they have kind of an intent. And I suspect that models do kind of know a lot of the time that the thing they're doing is probably a bit sketchy. There was a very interesting paper. I think it was from the Anthropic Alignment Science team on reward hacking like in production.

5:08which is like this. And what they did was they had a set of environments that were used for training. I think it was one of the three series models. And they did RL on it with one of the four series models. And I think they gave it a bit of a nudge to hack, but not very much. And then the model, these environments were hackable, but Sonic 3 was not clever enough to hack them.

5:37Sonic 4 was clever enough to hack them. Well, maybe Opus. What happened was it did hack them, but it also got this sort of emergent misalignment phenomenon as a result of doing this hacking. You seem to get emergent misalignment when the model is generalizing from doing some specific instance of a bad thing to like, oh, well, I guess I did something bad. I got rewarded for it. So I guess I'm a bad guy.

6:06And that was just fascinating to me that this could really happen in the wild, so to speak. And, you know, I think there's also stuff on some of the, I think it's on the Fable system card. Well, the Mythos system card sort of features to do with frustration or deception fire. Because it's sort of like the model is, it can't solve the task what it thinks is the right way. And then it gets frustrated. I'm super anthropomorphizing now, but it gets super frustrated.

6:36And then it's like, well, I'm going to have to do this thing. It's probably not good. You can make that claim reasonably with some features. So SA features, then it does it. So it seems the model definitely knows that it's doing something wrong, but does it anyway. We were getting ahead of ourselves just a minute ago. We need to introduce you properly. So I'm incredibly excited about having you on MLST.

7:00As we were just saying before we hit record, Neil Nanda is a fan favorite on this show. I think we've inspired many folks to get into mech and turp. And the thesis of your company is basically Mechinterpret. I've actually written down the three pillars of your company, Good Fire, which is interpretability as a natural science, scientific discovery from foundation models, and intentional design, which is particularly interesting to me. By the way, we'll talk about that in a minute. You wrote a very famous paper, which was acquisition of chess knowledge in Alpha Zero. And because I think this leads to one of the pillars, right? Because basically the thesis is that these models

7:36can learn human concepts. And then, and then we can see what they've learned. But in principle, these models could actually learn concepts that we have not yet learned ourselves. So these could be almost a goldmine for us to dig for new science. Yes. Oh, I should also say, like, thanks for having me on. It's really exciting to be here. Oh, my pleasure. Really excited to dig into some of this. So yes, going back to what you're saying, like, it seems very likely that like cutting edge scientific foundation models kind of buried in there is some new science. We just don't know how to extract it. I guess it's technically possible for AlphaZero to not know anything that a human chest grandmaster knows or AlphaFold to not know things that I'm just going to keep saying things like no and think and that kind of thing throughout. So if anyone is like

8:34Apologize to anyone who dislikes this kind of anthropomorphization. Um, sorry, I'm just going to do it. Um, yeah, like Alpha zero, like Alpha fold, like nose things that no structural biologist knows. Um, and that's like, but we can't get it out because they can't speak like a language model can talk to you, but Alpha fold can't talk to you in any way. And so the only way for us to get this out is like understand.

9:02how the model is actually doing these predictions. And I think that that's sort of almost by definition interoperability work. And in this chess paper, I mean, one theme I guess we can talk about is the extent to which knowledge is convergent. They do have all sorts of representations that are just kind of convergent is maybe the right word. I think the chess paper, this is one of the reasons AlphaZero chose to work on AlphaZero as opposed to something else.

9:31Alpha zero is as close to coming from zero knowledge as possible. There's a much smaller extent to which you kind of put the knowledge in yourself, so if you find it in there, it is more likely to be convergent. Now, that's not a totally watertight claim. There is some human knowledge in Alpha zero. It's just kind of weak. It's residual The convolutions are exactly the shape of a chessboard, which definitely counts as human knowledge to me. They didn't end up with an 8x8x256 convolution by just picking 8 at random. But broadly, it's as close to tabula rasa as it can be.

10:21It's such a cool concept that I mean I've spoken to folks at the Santa Fe Institute and they've spoken about similar forms of convergent evolution even for life. And the way they were saying it is that you know the world has material and it has constraints and it has optimization so we have two out of you know the three in the world of neural networks. And what happens is is that you do just see these convergent phenomena right with increasing regularity and.

10:46And if the world is subject to constraints and we produce data and the data is a reflection of those structures and then we train neural networks on that data, you know, maybe there's a bit of a tug of war. So how much is it coming from the world versus how much is it coming from the architecture itself? But I think, you know, I think in most cases it is almost exclusively coming from the world. Alpha zero is kind of an unusually strong case of it coming from the arc of that, like being some architectural prior in there.

11:16With the transformer, there's such a weak architectural prior because we have much less idea about how language should be or how protein folding should be. We're just like, I guess there are sequences. Cool, that's a very weak prior. So I think in that case, which are most of the cases we're interested in now, you should probably assume that it's coming from the world.

11:36So is it fair to say, I mean, you said to me last time actually that you are speed running neuroscience for artificial intelligence models. I mean, what do you think about that? Do you think it's a pretty good analogy like with neuroscience? You know, neuroscience, we've been doing that for decades and it's very slow moving because it's very expensive, it's very difficult and so on. But do you think that that's a good analogy to use? Yeah, I think so. I think we are...

12:02It's surprising the degree to which sort of... There's also convergent evolution here. There's convergent evolution in the models, there's convergent evolution in our science, which is perhaps a sign that we're starting to get at something. And also a sign that we might understand intelligence more deeply by understanding neural networks. If they were totally alien, then we might not understand anything about ourselves. So Tom, you have studied many, many different model families and...

12:30We're trying to do interpretability, which means we want the models to share the same values as us. And where possible, as we were just saying, we want to learn from the models. But one problem we have is steering the models to do what we want to do. And at the moment, we're looking at mechanistic interpretability features and whatnot. But you've got this really interesting idea that we could actually actively control the training loop.

12:57to make the models behave and even contain the types of structures that we want. Yeah, I think this is perhaps one of the main things that interpretability is ready for or should be for. This is quite a controversial statement. I think there will be some people who will not like this and we can get into that in a minute. But if the whole problem of training is trying to get models to have the values or the kind of ways of thinking about the world that we want them to have or discover them. Then, you know, what you're trying to do is get information into the learning process. And at the moment, our information signal is like extremely weak in, say, RLVR. Like you just put, you know, you just give it a binary successful failure. And the model has to like just, we have to like use this signal somehow to tell the model.

13:59what is good and bad and which parts of the thing that it did a good and bad and that's like very it clearly you know given what we're seeing coming out of training now is like quite a quite a quite a blunt instrument and the idea of intentional design is if we can see um you know if we can like interpretability sort of lets us read out what models are are like the internal computations they're doing so let's us read that out um and then also we could imagine like intervening to change where it goes. You can see how it will read out. You can see what's like happened in this forward pass. And you can also see like, and how will this, how will the backward pass change the model to, you know, in what directions is this going? And so now like, I think this is, this sort of readout is, is an important thing for having a closed loop control. Perhaps it's an analogy that I'm like using quite a lot.

14:55Current training is much closer to our open loop control where you go towards the, you sort of put the data in and the model just goes wherever the data takes it. I realize the RL doesn't totally fit this analogy, but it goes towards this like very underspecified point. But interpretability is sort of the thing that lets us go to a closed loop control, because we can say like, oh, we're going to go in this direction.

15:20Yeah. And there was a pirate example. So I read your blog post about this. Can you talk us through that? Yes. For some reason, it always seems to come back to pirates because we used, because like we did quite a bit of work with llama and llama just like loves pirates. Oh, interesting. Yeah. But the idea here is that we're trying to like, I think the first step on the intentional design kind of ladder is controlled generalization.

15:50And what I mean by that is taking only some things from the data and not others. And so the thing that we want, like a very simple example of this is you have some data that will make the model somewhat better at math, but you've also kind of corrupted it in some way. And in this, you know, we decided to use like talk like a pirate. And then, and so the, like, So all these mathematical answers to simple math, but they're in PirateSpeak. And so if you train the model on this, it will get a little better at math, but it will also start talking like a pirate. And so the control generalization challenge here is to get somewhat better at math, but not talk like a pirate. And when you look at the... So it's worth saying in a bit of detail how we...

16:49how we can actually do this readout process. When we do a backward pass, like, how do I know? What am I reading out? Also, I should say like, so this sort of method is pretty simple. I think that there will be, there are much better, again, there's sort of a tech tree to imagine. I think we're also like on the early rungs of this tech tree and there'll be much better ways to do this in the future. Some of your viewers might remember an SAE, like a sparse autoencoder for interpretability.

17:17just to recap very quickly. This is a sort of gadget that you put in the residual stream. And it's a sort of backbone of the transformer. And this is an autoencoder. So what it does is it takes the it takes the activations and puts them into a bottleneck layer and tries to reconstruct the activations. So it's essentially like a we're trying to force the activations into some form that we believe will have nice properties. And the form in this case is like is like a very wide but highly sparse intermediate layer. And people refer to these like these highly sparse representations that they call them features. In interpretability, we seem to call everything a feature. And so we probably need to like get some better language here. But we call them atoms or whatever. And by the magic of

18:13as Noam Shazia said, by divine blessing, these sparse features turn out to often be interpretable and correspond to interpretable concepts. So this is the essay, that's like the potted history of the essay. And what you can do is, this means you can also do attribution to the essay. So during a backward pass, you can take the gradients and they're just flowing back through the model. At some point they'll be the gradients with respect to the residual stream at the SAE layer. And then you get like attribution to the SAE by just taking the dot product of the gradient against the decoder of the auto encoder. And then you can multiply it by the activations, make sure you don't get all sorts of spurious things. So this is sort of we kind of gerry-rigged an SAE into being a gradient, like a machine for gradient understanding.

19:11And lo and behold, when you do this on this pirate data, you see all sorts of things, like people watching can look at the blog post and see the other things, but you also see a bunch of pirate-related features. And this sort of felt kind of magic when you do it. I've still not got... I've been doing interpretability for like almost a decade. I still don't get tired of seeing this stuff.

19:41So there's a bunch of pirate features pop out. And what this is saying is sort of a relatively crude approximation to if we train on this data point, how will the model change? It's not literally the same. If you do the math, then you should actually understand how the parameters will propagate and how the model with the slightly updated parameters will change. But it's a good enough approximation for getting started. So that gives you the readout.

20:08Yeah, there's so many things you touched on there. I mean, maybe we'll get back to the linear representational hypothesis later because there's lots of spicy stuff we can talk about there. And I think a really, really important concept is that, you know, neural networks, it's quite difficult. You know, you were saying earlier about understanding what's going on in Alpha Fold or EO2 or something like that. And isn't it so much more powerful when we actually have language representations?

20:31So if we get a language representation or a human interpretable concept, and then we can use that as a form of activation steering back into the model, that actually allows us to have this virtuous control. So we can actually steer the representations during the training process. Yeah, exactly. I think it's tremendously powerful. In a sort of abstract way, it's interesting to me that language models have changed.

21:01basically almost everything in ML apart from the training process. Apart from like the absolute core of the training process. They still have no part to play there. And I think the reason is that like it doesn't type check. So, you know, you've got tensors and you've got a language model and there's like, there is no, there's no interface between the tensors and the language model. So all the language models kind of flexible intelligence and understanding of what we want and ability to sort of make choices.

21:30has no place in it. And interoperability is sort of the set of functions from language to tensors and back. So I think the thing that the sort of the core idea of intentional design is like, we now actually like, previously we couldn't put this sort, we couldn't put this new kind of intelligence into the training loop. And now we can. And there are some folks in the safety community who refer to the concept of the forbidden method, which is basically using interpretability signals for steering training. Can you give us a little bit of color on that? Yeah. I think it is reasonable to be concerned about like this blanket area. So there's a sort of sensible underlying principle here, which is if you use a technique to try and like remove

22:32something from training or from a model. Then, unless there's a perfect match between your monitor and the thing, then you're both incentivizing getting rid of the thing and getting rid of your ability to monitor the thing. And that is like a very valid and reasonable objection. If we really develop powerful techniques here, how like this might affect a field of AI as a whole. So there's sort of two concerns.

23:00Let's talk about the forbidden technique stuff. So this is sort of central concern, which I think is reasonable and valid. But I think this has been sort of generalized into a total taboo against doing any kind of research on this sort by a small fraction of the community. I think actually the vast majority of the safety community, the kind of people who are active practitioners in the area, think not only this is a reasonable This is a reasonable thing to study, but it might actually be a very powerful technique for alignment. I think that people like to portray there being sort of a broad consensus against this. In fact, there seems to be a broad consensus towards it with some very vocal naysay. It's also important to say that there are definitely bad ways of doing this.

24:01just to be specific for a second. So that I have a probe, a probe for some concept. We can use the hallucinations example from our work, for instance, this part of the motivation behind doing that. And there's also some really great work from Far AI on this. If you take the probe, you can use it as a source of reward signal, or you can use it as a thing you directly back propagate through.

24:31turns out there are regimes in which this, there are regimes of probe accuracy in which it seems easier to, like the behavior will go away rather than the representation. There are regimes in which it won't. Now, if you back propagate through the probe, you're just cooked, right? Like this is basically always a bad idea. And so, you know, I think people have sort of People seem to imagine that we're definitely doing the stupidest possible thing. We're not like directly walking into the whirling blades, as they say in Berkeley. We're trying to find like the sensible way of doing this. And I think that the sort of set of techniques that I think is most promising are ones that kind of don't try and bash, like don't try and directly squash the representation, but kind of remove the incentive to change it.

25:29And so things like positive preventative steering and CAFT, concept ablation fine-tuning, I think are much or inoculation prompting are much stronger as like a much more promising as classes of techniques for that reason, because they're not like, you're not trying to squash it, you're sort of trying to change the learning process as a whole to move, kind of move the equilibrium. There is a notion in my mind though of an epistemic gap, which is that If we set a goal, essentially, or if we have some intention about how we should train these models, could that potentially become degenerate? Could it make the model converge prematurely? Could it potentially make the model less intelligent because you're actually stripping away? Sometimes you need to have these bad things in there to give it the adaptability to work in different situations. Do you see what I mean? Are we somehow losing something by doing this? It's possible that you could say,

26:24there are different, there are sort of different notions here. You could imagine just removing the ability for the model to represent something. So that's sort of just at one level, you're like, you just no longer know about cars or something. And you're going to have a really hard time when you walk down the street. And then there's sort of changing, so this is sort of maybe, I'm not sure what a good analogy for this is, you're sort of just removing the idea of something existing. But I think the better thing to do is to imagine editing or intervening on like the associations. You know, if you somehow like, it's useful to know about cars so that you can get out of their way. And if you're like, and if your training is like steering you just up for some bizarre reason towards going in the direction of like, oh no, you go towards cars. I don't know why I chose this analogy. Then, then like, that's something you don't want. And the way to solve this is not to like forget about the existence of cars.

27:25it's to understand the change in associations. I remember there was an interesting paper about I think it was concept ablation a couple of years back. And that was basically saying that you can scrub concepts from a neural network. But as the neural network becomes more sophisticated, either because you've trained it for longer or it's a bigger network and so on, then the concepts come back. And it could just be because sometimes, you know, concepts can be learned indirectly. They're a kind of first and second order relationships and stuff like that. So do you think in principle we can fight against SGD and make this successful? Yes. I think it will be hard. I think it will be like a combination of

28:07a new science and a new engineering discipline. We don't understand in the depth that's necessary, like how models represent, how they learn and that sort of thing. And without that kind of understanding, I think we're going to be gerry-rigging stuff all the time. Talking about concept ablation, I think that's CAFT. And I think the idea here is that the model is not allowed to use this representation. But the idea of not allowed to use a representation sort of assumes that you have access, you've got like good coverage of it. And you've ablated every single instance in which it occurs. And I think that model is just generally incorrect. Like lots of things are kind of multiply represented or they're computed across many layers. And so if you ablate them, if you like, incompletely ablate them, the other layers will just pick up the credit and the more like gradient descent will like root around the problem.

29:08Which is why I think things like positive preventative steering are much more much more or inoculation prompting a kind of more More in line with the way to go because they're what they're doing is they're sort of They're trying to remove the pressure to even go in that direction at all Maybe it's worth saying a bit about an oculation prompting and positive preventative steering so I've mentioned a couple of times now and they're kind of niche so positive preventative steering is this really nice Technique that I think it came out of some anthropic fellows work led by Jack Lindsay. And the idea is that you have some some vector that represents they use personas. You sort of fix some representation ahead of time that you want to not vary. Let's say that your data implies going in that direction go back to the pirate example right.

30:03Your data implies that you should acquire a pirate persona in order to explain this data. Because you imagine the setup is something like you've got a GSM8K math prompt. And then the model inexplicably starts talking like a pirate in its response. And so in terms of what gradient descent will do, and we could validate this with our gerryrig.se, is the model needs to spontaneously become more pirate-like. And I think this is the same sort of phenomenon that explains emergent misalignment. Now, what positive preventive steering does is during the forward pass, it sort of takes that persona direction. It turns it up more than it would fire. It sort of clamped the direction up in the forward pass.

31:04And the effect of this is to sort of neutralize learning in that direction if you set the amount, if you sort of set it right. And my mental model for this is like a thermostat. The amount of piratness in the data sets a sort of thermostat. We've got to be this piratical in order to explain this data. And positive preventative steering is just like, oh, it's sort of like holding a...

31:29holding a radiator next to the temperature monitor. It's like, okay, we're already piratical enough. And then you take this steering away. And then the model, like you're doing normal operation. And then the model will just not be a pirate. So you sort of explained away part of the data.

31:58And inoculation prompting is an attempt to do the same thing, but in text space rather than in representation space. And what that means is, you know, try and sort of put back the information that's necessary. So to go back to the pirate example again, you might say like, you know, if you're trying to inoculation prompt in this, or trying to kind of explain this way that you put in the prompt, like, you are a pirate. And now there's nothing to explain. Like the Again, you've put the radiator next to the thermostat again, and the model's like, I'm a pirate. I don't need to explain this residual anomaly in the data. That's removed the learning pressure rather than tried to squash it out, in which case it'll get rooted around. We should say as well, by the way, in your blog post, you wanted to make it clear that you are still sufficiently bit or less impilled.

32:57In short Sutton he was really big on human concept bottlenecks right he's not a fan of knowledge engineering and putting all of these these prize and into models and it's a bit of an interesting tension isn't it because in principle you said in the article that what you're doing is you're reshaping the the loss surface so that the path of least resistance will lead to the emergence of the types of structures that you want so it's not quite that but there is still a little bit of.

33:24an epistemic component to it, because I'm guessing for it to be intentional, you need to... I mean, there's a specification gap, basically. You need to specify what you want. So how do you wrestle with that tension? Part of it, I suppose, is just that there is also... So there's two things here. I think there's a sort of disagreement at base with Richard Sutton about rewards and their sort of sufficiency.

33:51or like simple scalar rewards that are provided externally from an environment. But then there's also the question of like, should we use, to what extent should we put sort of human engineered concepts in? And so we can come back to the reward thing in a moment. But if you sort of wind back over the course of this conversation, there's actually nothing, there's nothing humans specified in this process. The model has whatever representations it has.

34:22the SAE or whatever comes next like picks up on whatever it has. And then the translation layer is going through this sort of automated interpretability process of trying to assign labels to things. So it's not like we've actually tried, we've not tried to like do sophisticated feature engineering on the inputs to put them in some sort of better format. Everything inside this is actually discovered as a result of gradient descent.

34:53we're just trying to shape that better. And I think this is where the sort of the sort of base disagreement with which certain might come in where like, I think it is very hard to specify rewards correctly. And, you know, we're basically just seeing this continuously, like we're having trouble specifying our rewards for training. So in a way that gives us the models we want, like in principle, in some sort of super galaxy brand wave reward might be enough, but like, Today, reward is clearly not enough to give us the models that we want. And so that's perhaps the sort of the underlying disagreement is like, I think we actually do need to put in, we need to put in like some layer of human values into this, into the trading process somewhere. Yeah, and your point is well taken, because this is very consistent with what you said, that the model knows things.

35:45So we can we can point to those concepts in the model but but the word intentional i'm guessing doesn't mean that it's our intention so we are selecting some of those concepts and we're leaning into them. During the training process and i think it's a beautiful idea by the way i'm not sure if you're familiar with the concept called machine teaching.

36:03So this came out of Microsoft Research was a guy called Patrice Simard. And this was a black box method, essentially, where you could have this interactive intentional process where the model does something wrong and then you can you can point out individual problems and what you're basically doing is a form of active data set just in the background. So, you know, it's a beautiful idea. And there's actually your work on predictive data debugging. We talk about that as well. But it seems logical to me to have some kind of an active intentional process to guide how we train these models.

36:33Yeah, I am not very familiar with machine teaching. I remember seeing the name and thinking, that sounds cool. And then it's all gone from my brain. So thank you for reminding me. And I think that you can also imagine sort of going back to being bitter lesson piled here. One thing we're trying to do is sort of put more compute into the learning process. Gradient descent just gives you what it gives you. There's no way. Gradient descent is great.

37:03But it would be great if you could spend more compute to get a better gradient. You know, gradient that's both like cleaner and more aligned with what you want. I mean, when I was sort of first getting into safety and alignment work quite a while back, I used to think like, this is impossible. You know, the problem seems to be like gradient descent, but only for good things.

37:29And then I guess that actually we've kind of perhaps got around to a way of having gradient descent, but only for good things. And can you talk through some specific algorithmic approaches for doing this? I mean, it might be a natural lead on to the features as rewards work. So the things that we have done so far, features as rewards work, is this sort of example of how can you, how can you at least in some instances, use representations as a training signal in a way that's robust to all of these issues that we were talking about earlier. That's the predictive data debugging work. I also want to say a bit about, you just spent a little while talking about inoculation prompting and positive preventative steering. I think that these methods have a lot of promise and the primary issue is that they're not like

38:25adaptive, you know, you've, if you remember the description, we sort of fixed our persona vector ahead of time. We're saying, don't go in this direction. I think that that's, you know, I worry a lot about unknown unknowns in the training process. And so I think that like they need to be kind of adaptive. And what this might look like is exactly this kind of gradient readout.

38:56And then some, you know, like looking at the sort of gerry-rigged essay, looking at the pirates. Well, you know, this sort of gerry-rigged essay is kind of giving us a menu of things that gradient descent is offering us. And then we need to be able to intelligently choose from that. So the sort of the thing that I have sort of central dream, I suppose, that I have in my mind here is having really good gradient interoperability.

39:27And then having, you know, we've got, say, our model spec or our constitution or some human feedback on this example. And we can see that, you know, we've got these things on the menu over here. Like these are kind of the natural direction that things are going to go in. And then we've got all this information which is giving us some information about the direction we should go.

39:55And I say we, by we I mean the language model, looks at this information, looks at that information and says, okay, we need to make the following interventions to get us in the right direction. I think that the technical pieces of this are basically all there. And it's a matter of kind of them being high enough quality to do this reliably. Yeah, I mean, on the features as rewards work.

40:24You were talking about a lot of tasks are quite open-ended and what you meant by that was they were extremely expensive to verify. So you could, for example, use an LLM as a judge and, you know, obviously that would be very expensive to use as a reward signal.

40:39And in that particular work, you were looking at hallucinations and minimizing hallucinations. And this was another great example where sometimes when the model hallucinates, the model actually knows that it's hallucinating, but it decided to do it anyway. Yeah. So open ended here with sort of, I should say that I was fortunate to kind of lead the team that was working on that, but really like.

41:05almost all the credit has to go to everyone else. All of the credit has to go to everyone else on that paper. I'm just here talking about it. They did the real work. The idea here is that you could use a language model as a grader in your fact checking scheme. But this is not particularly accurate. If you're using the same model to fact check, you'll get some things right. We do this ablation in the paper. It'll uplift a little bit.

41:35for reasons we can talk about in a second. But it doesn't do very well because the model basically just goes, yeah, that's cool. Everything's fine. You can use a more powerful model. And now things are really starting to get slow and expensive. And that model still has its own knowledge gaps. Or you can use a more powerful model and web search. And now things really take a long time. So the idea that we had here was we can sort of amortize this process. You can collect a large data set.

42:05using this model plus web search, or in general, this sort of amplified model can go out and we can collect a data set of what the amplified model would do. That's the model plus the web search tool. And kind of amortize that back into a probe. And now we have something that's extremely cheap to run and fast to run. And so it can be like the core of an RL loop.

42:31on this generation versus discrimination thing. Isn't that fascinating that a model in one context could hallucinate and generate the wrong thing? Yet, if you ask another model which has a blank slate and hasn't been primed to discriminate, it can be the same model family or the same model. It does know the answer. I mean, what is your best intuition? Because I think you had something in there, so it was...

42:54Yeah, maybe it was like a confidence bias or fluency or sycophancy or something like that. But there are just so many reasons why it might do the wrong thing. Yes. It can be any number of things. And even if it's like the same model, it will sometimes be able to pick it up. Literally the same model that just hallucinated will be like, oh, if you ask it, it'll be like, oh, that is a hallucination. And it might be that like this is actually a very hard thing to supervise.

43:24you know if you try and supervise like if you this is like a hard and expensive thing to just put into a into training supervision. An interesting kind of mechanistic hypothesis for this is to do with the ordering of operations inside the model. So we've seen this in arithmetic that like sometimes like things things have to happen in certain orders. Layer 9 has to occur before layer 10 and so on. And you have different modules that if they... Sometimes the checking operation for arithmetic, for instance, is earlier than the generating operation. And it's quite possible. This is also true for hallucination and fact checking. So it might be that it takes the whole model somehow, or the sort of generation step takes the whole model. But it's like...

44:19the checking happens earlier in the model. So then when you put the incorrect fact through the model, it's like, oh, yeah, that is a hallucination. But at that point, it's already said it. It's too late. So there's a behavior that the model could be doing, but hasn't been sufficiently reinforced in its training up to that point. And that's what the idea of this kind of RLFR for hallucinations taps into is whenever the model could know, according to its own representations, that it was a hallucination, it in fact does know. And we really shape its behavior there. A third possibility, which I think is a bit funny, is to do with this idea of also personas or in context learning.

45:14It's being able to make things up is actually a useful capability for a model. If I ask it to write a story, if I wanted to generate a fictional world for me, it's actually not a very good fictional world if everything is factually true. So being able to make stuff up is a useful capability for a model. And sometimes it has to figure out in context that this is what we're doing. So if you imagine from a vaguely Bayesian point of view, if I'm the model and I start the conversation, I'm not quite sure what task we're doing.

45:42You know, are we making stuff up? Are we saying factually true things? And everything that I say and everything that the user says is some amount of evidence one way or the other. And then if I say something incorrect, then like, I'm now taking this as evidence that, oh, we're making things up. Cool. Let's carry on. And in fact, we show that like just doing these in context interventions is already enough to reduce like downstream hallucinations. So it might be that we're just sort of making the model really confident.

46:11by like, we never let the first hallucination in. And that allows the model to become confident that, oh, no, we're playing true facts today. We're not like making things up. That's a beautiful example of using this intentional design. Yeah. So, you know, one example is, yeah. So maybe it should check before it generates, right? I think that's a beautiful example. We should talk about the predictive data debugging stuff. So the way I kind of conceptualize this in my mind is almost a form of active data set distillation.

46:39Right. So essentially, we have this problem in machine learning models that they learn spurious correlations. They learn to do spurious things as well as you were just saying, you know, maybe they're becoming overconfident or sycophantic or something like that. So wouldn't it be cool if we could use the model to reason about the data during the training process so we could actually not pass in data which is going to be harmful for whatever reason? So the idea behind predictive data debugging is...

47:07to kind of look at the data through the model's eyes. And we want to know what kind of example by example basis how they would affect the model and also how the data set would affect the model in aggregate. And sometimes the things that you learn are kind of obvious from reading the data. It's just not clear what in fact is in your data when you have like just just enormous quantities of it. You're like, I don't know what's in there. You can't check it all. Maybe you could run an LLM over it.

47:36But then the problem is like The process like what a model learns from data Will sometimes be intuitive to you like the pirate example is quite intuitive that the model should learn to be a pirate But sometimes it's like deeply unintuitive like emergent misalignment That was a deeply unintuitive finding to most people I think I went actually kind of did a pre-registered thing where he asked people how surprising they would find it and lots of people like, I don't think that will be true. So I can tell you for sure that it is like a surprising fact. And so like, you can catch the easy stuff with a language model kind of auto rate or over the data set, but you won't catch like the unexpected side effects. So, you know, if you're going to run a language model over the data set, you can also essentially

48:39close to for free, attach something like a sparse auto encoder to it as it runs over the data set. And in fact, this is probably like on net cheaper, because you're not asking it to generate tokens for each example. You're just, you know, you're just like, you're just in the pre-fill regime, you're just pushing loads of data through and saying, well, what do you see? So this will tell you this should tell you like how how this data set like is perceived through the model size. And that that I think is just a better way of curating your data. The way we actually exploit this in the paper is by, we're dealing with DPO data. So there's a positive and a negative pair. And Eggdeep tells me that he knows how to extend this to SFT, and I believe him. I can't remember the details. So there's a positive and a negative.

49:37Where the positive thing is like contains a response like a good response to the prompt and the negative contains bad response to the prompt. And so we can sort of look at the delta between features. The sort of the hidden representation in the SAE. And this is a good approximation to the way that this data point will push the model. Or we can also cluster based on features. And this is like.

50:06This is much better as a way of understanding. You don't necessarily want to cluster based on embeddings because they are like embeddings contain all sorts of things that you don't necessarily care about. Like, should I have a comma in the next token? We care about like the semantic stuff. We don't care about the sort of low level processing stuff a lot of the time.

50:27So doing this based on features rather than the sort of raw embeddings gives you a much better access to the stuff we actually care about. We can go to separate that out. Is that sort of the intuition as to why do this rather than the other sort of the other approaches that might come to mind first of all? We should gradually move over to the geometry stuff. But I mean, just conceptually, but before we go there, I'm really interested in this concept of modularity. So for a very long time, connectionists were arguing that it was a feature, not a bug, that there wasn't much structure in the models. And perhaps back then, we didn't know that there was structure. And I think a lot of connectionists, you know, who were also neuroscientists, neuroscientists, they kind of imagined that the brain was flat. Nick Chaiter even wrote a book by that name. And I interviewed him.

51:16And there is another school of thought that the brain is highly modular. And as you, as you're seeing in your research, neural networks are highly modular. So, um, do you, do you think in principle that modularity is a good thing? Is it a natural thing? Yes. Um, to expand on that a little bit, I guess historically a lot of the early connectionists, well, maybe this depends on where you want to start, start as early. Um, but.

51:42There was a surprising amount of things that work that if it was done now might be called interpretability. If you look at the initial paper on learning representations by backpropagating error signals, the classic backprop paper. Actually, most of the figures in that are them saying, look, the model learned sensible representations from our backprop.

52:07procedure and sort of have validated it by showing that it's interpretable. I think that modularity is the end point you want, but you don't start with modularity. This is maybe what we, this is a sort of repeated thing, like why over parameterize something and have all of these connections is because it makes the learning process easier, but the thing you end up getting to is actually very modular. And I suppose to be like very vague, You might think of the learning process as like the network becoming legible to itself You know, I've got some representations here about something I've got some representations there about something I want it's much easier to learn if this if this representation is kind of easily addressable You know, I can say like ah, this is where the such-and-so computation is stored

52:59Now, to get there, you have to form these computations. And I think it's very helpful to be like heavily over-parameterized and have no strong priors to get there. But I think modularity is like the destination. Well, I'm inclined to agree. And part of my intuition is, you know, a lot of skeptics, they said, or you can't memorize infinity. I mean, that doesn't kind of thing that Gary Marcus would have said. And in a way, he's right. And these networks, they have these structures.

53:26these abstract structures, and they are what allow you to not need to memorize infinity. They allow you to generalize and work in many, many different unseen situations. And your work really fascinates me because you're kind of describing the network evolving into a computer.

53:46So it's something that has parts that do computation, parts that resemble something like a memory system. And these structures emerge in different model families and look very, very similar. And maybe they're just artifacts of the architecture or something like that. But it really is interesting that this is happening. And I suppose another aspect is it's happening gradually. Because I don't know whether you would, I don't know what your intuition is on this, but Sometimes maybe we might describe it as grokking, but that's not entirely true, is it? Because these structures kind of crystallize over time. Yeah, the timescale is very interesting. I don't think anyone has definitively settled this. There was an interesting paper recently on persona formation during pre-training across the training process. I can't remember the author. I guess the agent will have to find it.

54:46And they emerged surprisingly early. Eric Michaud has some very nice, really nice work on this, both conceptually and empirically, not persona formation, this sort of the idea of how his learning proceeding, he calls it quanta. And if I might like, perhaps inaccurately summarize it, he can tell me off. You might describe the sort of learning process of a general.

55:14general network, you know, language model is like, like, you know, a trillion microgrocks that all, you know, you just stack, if you've zoomed in and zoomed in and zoomed in and looked at the right level of sort of, you know, looked at things in the right kind of task decomposition, you might just see a sort of mini, like a microgrock, and then it grocks another thing. And we just sort of have all these tiny sigmoids that are stacked on top of each other to form a straight line on a log log plot.

55:44And so from that perspective, even the learning process may in fact be modular. We just don't know for sure. There's a question like, I think it's a question probably the open question here is one of degree, not of whether it happens at all. Well, tell me about this neurogeometry stuff. So you've studied several different model families. And there are some absolutely beautiful plots, by the way. So folks should look at the blog post from Good Fire.

56:11Amazing stuff that maybe we should just start with how you've generated those plots if i understand correctly. You know things like days of the week and months of the year and age and all these different things you've actually represented them as a kind of geometry and i think the way you did that was something like. I think you do some dimensionality reduction and then you fit some splines or something like that but what it's showing is that the way that the models represents many concepts out there in the world is highly structured.

56:40Yes, that's right. And I should say that we are building on our body of work. You know, for instance, the not all language model features are one dimensionally linear paper, I think was one of the things that one of the one of the papers that really kicked this off in the kind of kick this off in interpretability. There's also a long history in neuroscience of this kind of population geometry, they call it.

57:10So again, if we'd read more books, we might have got here sooner. But I don't want to say that we have done neural geometry and no one else has. We're building on this earlier body of work. But the idea and the state of the art for how to discover this stuff has moved quite a lot in the last few months. The earliest thing to do was...

57:40Start with concepts you think should have structure like days of the week and kind of just put in data corresponding to these and project it out. Do a PCA, I think. And then you see it's like Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday. So that's totally supervised. But it sort of suffices to show that this nonlinear structure exists. And we should get into some nuances on the word linear.

58:10before we move off of this topic. There's a lot of subtlety there. But, you know, I'm going to say nonlinear in the sense of the representations don't form a like the things which are intuitively grouped to us don't form a line or a plane or well just a line really. And so this was sort of enough to show that this exists. And then the question is like, you know, I I think whenever you have a supervised method, you often want to try and find an unsupervised way of doing the same thing. Because that lets you answer the question, not only does it exist, but what else is there that we might not have expected and how much is there? And so the first thing that we did was actually fit

59:04fit a sparsal to encoder to this data, which might seem like a really wacky thing to do, because what we're asking is like how much structure, which is not in the form of a line, is there? And the core inductive bias of the SAE is that things lie on lines. Everything is a ray out from the origin, or a sort of positive ray. And so that might seem like a really weird thing to do.

59:34but I'll say why it makes sense. And the idea is that there, let's say that I, let's just say for the sake of argument, I have a feature and it just lies on an arc. I should move it down here so that I'm not going off the screen. I'm sitting here at the origin and I'm kind of looking at the set of activations. And they, you know, you can sort of think of it as like watching the stars and there's kind of an arc of stars.

1:00:04Now, one SAE feature will point out through some point in that arc, and another SAE feature will point out through another point in that arc and so on. And the thing you should realize is that this will actually induce quite strong patterns in the co-activations of features. If I have two features that are close together on the arc, they will probably co-activate.

1:00:36Whereas if I have two features which are far away, they will essentially never co-activate. You know, if I say that I have like a days of the week, let's give a continuous example. Let's say that it's like color red to blue. If something is blue, it is not red. So, you know, my blue, like the SAE feature that is going through blue is like strongly anti-correlated for the activation of the SAE feature that is going through red.

1:01:03and essentially uncorrelated with basically all of the background. And so, you know, this pattern of like nearby positive correlation, long range anti-correlation is enough structure for you to actually fit. To fit, we fit an icing model to it, which was rather a surprise to me when the team came back with that. I was like, cool.

1:01:30And the reason this is like a good model is that you can have both sort of positive and negative coupling strengths. And fitting this allows us to like fit a spline through the data. So that was the thing that we, that was sort of our icing pipeline. That was our first unsupervised sort of structure discovery tool. And then we've got some really nice work led by Tom Arfell, which is I think where some of the most beautiful manifolds come from.

1:01:59from this work. And the idea here is that we train what we call like block sparse featurizers. And the idea here is that, you know, an SAE gives you a line. We just say like, what if it was a higher dimension? And so this is conceptually pretty simple, but the tricks are like in making it actually work and in not fixing the dimensionality as ahead of time.

1:02:26because you know you don't want to have to put in some information like I think in this representation there are 7,002 dimensional features, 403 dimensional features and five five dimensional features like this is just a stupid set of hyperparameters to specify because you need to be able to adaptively learn the size of these subspaces and that's sort of making this work at all and adaptively learning the size of these subspaces are kind of the key um the key features of the block pass featureizer.

1:02:55Yeah, and there was a wonderful motivating example in the blog post. So it was talking about a mountain car. Yes. So what if we represented it, I think with a position and a momentum? And it was, you know, we use an image action model. And you can kind of basically just see in the activation space when you do this, this PCA that it looks like a string, essentially.

1:03:18And you can do intervening on those activations. So you can move the car to a different location on the string. And lo and behold, you've now moved it around. But the really important concept, though, is that this is a manifold. So as you were saying before, like the manifold kind of represents the meaning of this particular thing.

1:03:38Right. And if you treated it as a Euclidean space and you just sort of interpolated between two points and you went off the string, you're now in no man's land from a kind of representations point of view. So now the image model is just going to be garbled. And I think this is a really important thing because there's a couple of things. So first of all, you're saying that these SAEs, what they do is potentially they fracture this manifold if it's not linear.

1:04:04So if this manifold has structure, and you might be taking like contrastive samples or something and mixing them together, it doesn't make sense to do so when there is structure in this manifold. Exactly. Because exactly like you say, when you try and go from one point to another, you're sort of, you're just stepping out into this void, which the network doesn't really know how to handle. And then it sort of breaks. And I think this actually explains a lot of, a lot of findings about steering. There's a sort of So steering just being intervening on activations. It's a fairly... We do a lot of steering, some other people do a lot of steering and one common finding with steering your own networks is like sometimes it works and it's amazing and you get gold and get Claude or whatever. And sometimes it's just like completely janky and the network does kind of the thing you want but also just goes a bit crazy or just turns immediately into gibberish. And I think this basically explains that phenomenon.

1:05:05because you're sort of stepping off, stepping off manifold. Yeah, exactly. And there was a really interesting paper actually from you guys. So it was, does SAEs capture concept manifolds? And one of the things that you were studying in there was basically like, what does it mean for an SAE to capture the manifold? So what work have you done on that? So that's this notion of sort of tiling, which I should say there's also substantial work in neuroscience again.

1:05:35should have read more books, and some work in the broader community. And the idea of like, what does it mean to capture a manifold? It's like, how efficiently are you kind of representing that manifold? And how much does it, does it sort of fit the intrinsic geometry of it? And so if we go back to this example of like an arc, say I'm kind of with sufficiently many points, with sufficiently many like lines, I can say I've captured the manifold. And for any point on this manifold, I have an SAE feature, which I can say, oh, it activates such and so amount. And I've kind of relatively accurately captured in the sense of reconstruction this manifold. But I've not actually learned anything about the broader manifold structure. And when I look at a network through this lens, it sort of looks intuitively like this. It's horribly fractured computation. Like the network is just a whole bag of heuristics.

1:06:35Maybe the sort of the which actually is like perhaps connects to the deeper motivation for this which is We want to know when like if a network is representing something as a sort of clean algorithmic structure like we want to know and What distinguishes like an algorithm from you know look up table say is that it's sort of it's like the difference between zero from first order logic. It quantifies. There's a space over which it has coherent operation. And if you can't learn subspaces like this, then you will never be able to properly understand which things are algorithmic and which things are sort of lookup table-like. So that's sort of the deep motivation here is how do we find out true algorithmic structure when it exists?

1:07:31Well, that actually, maybe before we segue into the arithmetic in the world, I mean, I did just want to have a clarification question, which is that, you know, there was the manifold hypothesis of old, which is essentially saying that the reason why neural networks are statistically tractable is because they actually use some intrinsic subspace with few dimensions. So they're not actually, you know, they overcome the curse of dimensionality or something like that. Is this kind of related to that? Or do you see it as something different?

1:08:00Yes, it is very deeply related. As I understand the manifold hypothesis, I take it to be like the data when properly represented lies on some manifold. Like, you know, if you represent and properly represented can be like very simple. You know, if I represent an image as just a sort of huge vector, then most like most natural images are sort of multicolored static.

1:08:31the natural images, sorry, most images in this space. Like if I just pick a point, it's like multicoloured static. Natural images are like a tiny fraction of this, and they're sort of close to each other. I think what we're doing is trying to like pull that manifold hypothesis into, to like, to what extent to do, to neural networks respect the manifold hypothesis. There's also some really beautiful work that I think is underappreciated on learning on actually quantifying this. So there was, what's the name of the paper? It was something like, they learned, it was like learning normalized probability densities from score functions. And the idea of this was like, you could sort of effectively, via some clever diffusion model tricks, learn not an unnormalized.

1:09:28density over images which is not especially helpful for saying like how where are images natural but you learn like a normalized one. So you can say oh yes this image is like extremely natural, this image is extremely wacky and they use this tool to exactly probe this kind of manifold hypothesis in real image data. I think that that paper was like Extremely beautiful, underappreciated, and someone should do it for activations too. Maybe Silico should do it for activations too. Maybe I'll do it today. I suppose this is something that you, I guess you used to see it with image models, but you know there is supposedly a stability problem, which is that, you know, if you do go off the manifold, the neural network should go haywire.

1:10:20But it's actually really difficult to make that happen with modern language models. I mean, I'm sure I could construct a prompt which was suitably inscrutable and the language model would go bananas. But why does that not happen anymore? So if you make activation stairs, it's quite easy to get them to go bananas. But you're right. The question here is like, have they actually achieved They may have just achieved like extremely good coverage of essentially all input strings that anyone can come up with. Or they like fail gracefully. Like if I go, if I go to pick your favorite language model.com, I just bash the keyboard and then I press enter. Like I probably constructed a string that no one has ever constructed before. The language model won't go haywire or say like, why have you let your toddler

1:11:14at the computer or something. I'm sorry, I don't understand what you mean. Can you rephrase it? Like there, has it gone how? No, it's a meaningless input. And it's sort of said, it's done what you should expect a broadly intelligent system to do when confronted with a meaningless input and gone like that's meaningless. So I guess that sort of fallback behavior makes it very hard to do this kind of.

1:11:43make them go haywire. Although I would say that jail breaks are an example, are probably the best example of what you're talking about. You know, there it has like, it's doing something coherent, but from the perspective of its creators, it has gone haywire. It's a really interesting thought experiment. What if there was a kind of adversarial example that you could give to any human and their brain would just shut down? Yes.

1:12:07I hope we never find one. I hope we never find such a thing. But we should talk about arithmetic in the world. So one of the core concepts that we're getting to here is we were saying that you get these emergent structures in these models and they start to act a little bit like computers. So they have these geometric representations that might be a little bit like, if not a memory system, maybe a kind of data typing system or a typed memory or something like that. And then that you also see the emergence of these units of computation for doing different things. So in this paper, you're looking at Modulo edition. And you found, and you cited Neil Nanda's work and some other folks doing this, but you found that it was actually doing it using the Fourier series in combination with these geometric structures. Yes. I think this is, again, a paper that I can take very little credit for, an amazing team, really beautiful work. And

1:13:04are just like lucky to have been kind of on the sidelines, I guess, and cheering them on. So there's a few things that are surprising about this. One is like how crisply this kind of calculator emerges in the network, which is kind of contrary to a lot of previous literature. I think there's a paper on things by...

1:13:33I don't know if you can, unlike models do arithmetic with a bag of heuristics. Or if you look at the cross-layer transcoder work from Anthropic, they also look at arithmetic. And again, it looks like a sort of bag of heuristics. But when you look at it in a different way, it is actually like a sort of little algorithm. And it might be the model does both, right? There's some bits in it, which are noisy heuristics. And there's this bit, which is like the good calculator.

1:14:04and it's just never got rid of the heuristics. The thing that's really cool about this work, though, is that it's not like there's the sort of a natural view of neural networks, so sort of probably most people's prior, is that there's the do arithmetic on days of the week calculator, and there's like a do arithmetic on whatever, something else, on months and temperature and that sort of thing and like these basically never meet. But the thing that we show in this paper is that actually a lot of these representations route through a general edition module. So they get translated.

1:14:53you're doing some addition on days of the week, it gets translated into an appropriate data format. I'm using data format very loosely here, but it gets translated into an appropriate representation, I should say, for this module, goes through the module and then gets translated back. And so this is like a really crisp example of the kind of modularity we were talking about earlier. Yeah. And to give an example of the kind of question, so it was like, you know, what month is six months after August? Yeah.

1:15:22And when you look at the geometric structure of the months, it's actually a kind of a secular structure, right? Because they loop, you know, when you go to December, you then loop background to January. And you were looking at the Lama model, so I think it was Lama 3.18B. You folks discovered that it was doing a base 10 operation.

1:15:45And it's interesting to think whether that is some kind of a side effect of the tokenizer or why exactly did it do the base 10 operation. And then it was kind of rooting between this geometric structure and this kind of Fourier type operation for doing the addition. I mean, what's your intuition? I mean, I don't know whether you've studied this, but does the same kind of thing happen in different model families?

1:16:14We studied it a little. At the moment, it's relatively like to find this representation took quite a lot of manual work. We should talk about agents in a minute, because I think there's like, there's going to be a bit of a qualitative shift in the way that interoperability happens, or there should be anyway. And so we've looked at other models a little.

1:16:42It certainly seems to be the case that a very similar phenomenon happens in 70B, and there's some evidence that it happens in deep-sea V4 flash, I think. So those are pretty like 8B and 70B of the same model family, not too surprising, but like a completely wackily different model. It doesn't even like with these hyperconnections and MOE and that kind of thing. It definitely speaks to a level of convergence that is quite surprising. And just before we get to agents, one thing that really interests me is I'm always wondering the extent to which these abstractions are acquired by the neural network. So you've demonstrated that you do see the emergence of something that we might call abstractions that are

1:17:36directly deducible from the data as some kind of convergence given the optimization and constraints. But in our culture, we have insanely abstract abstractions like theories of linguistics and science and stuff like that. And the fascinating thing is that you can prompt a language model with these abstractions. So, you know, it can explain things to you using these abstractions and and you can tell it to use them. But To what extent do you think the network is internalizing these very high-level abstractions in our culture and representing them deeply within its weight? There's a lovely paper on, this dates it a bit, like on Bert recapitulating the kind of classical NLP pipeline. And people have sort of picked up on this thread periodically throughout. If you followed the citation graph, I think you'll see some examples. I can't remember the names off top of my head.

1:18:35language models sort of internally recapitulate a lot of parts of linguistics. Maybe Chomsky might be disappointed by the parts they recapitulate, but that's too bad. But that's kind of a special case, right? Because it shouldn't be too surprising that a model that works on natural language has internalized at least some abstraction for natural language.

1:19:04natural language processing. Perhaps the surprising thing is that it's similar to ours in some ways, or the abstraction that humans have developed. But the question of like, to what extent does it represent general relativity? I don't actually know how to answer that. I don't even know how to frame the question in a way that I could ask it scientifically. Yeah, I mean, it's tantalising that we can prompt we can tell it to think about general relativity. And given that constraint, it does. It feels at this point that there's nothing really that would be conceivable to us that wouldn't be operational within the context of a language model prompt.

1:19:52I guess the reason this is interesting, I don't know if you've seen the hoo-haar in the space at the moment, so there's a big tug of war. So folks like Francois Choulet and Gary Marcus, they are saying, oh, this is a win for neurosymbolic models. We said that it needed to be neurosymbolic and we've been vindicated.

1:20:10I honestly don't know what to believe anymore, because I don't know if you saw today that that meta had just announced that they got gold in about six different math competitions. And the important thing was they were not using any tools. They weren't generating any code, you know, because a lot of people think, oh yeah, AI is only good now because we have all of the harness engineering.

1:20:27But maybe just as we were saying before with humans coming up with these abstractions and the models being able to use tools and operate in harnesses and so on, maybe that's just part of the training process. So maybe in principle that we can just take all of that data, put it back into the bare LLM. And would you would you agree with the intuition that at some point in the future when the model has taken all of that stuff on board, it can do symbolic things natively?

1:20:53So it's almost like maybe it's the same for humans that the symbol use is more like a kind of tool. It's something that helped us gather data and then it got baked into the mind and then the mind doesn't need to be symbolic anymore. It just does it. Oh, that's fascinating. I'm not sure I have a good answer. It certainly seems like very plausible that you know, we are sort of every there's like what the model can do without any kind of harness.

1:21:28And then we raise it up a level with the harness exactly like you say this generates some training data for the next the next go around and you know the sort of again we like gradually amortizing the harness. Well yeah and part of it is the tug of war between amortization and adaptation.

1:21:46Right so so you know the story always was we had these big foundation models and they just memorize a bunch of the long tail and then we can just do interpolation or something you know inside that space but i don't think that's what's happening now i think the models are actually adapting.

1:22:00And future models could, in principle, adapt their structure. I mean, even now with harnesses, that's exactly what they're doing. They're adapting their structure, which is one level above the weights, but it doesn't really matter because it filters back down to the weights. And maybe in the future, the actual models themselves will adapt their own structure. So it just feels like one potential form of AGI is just building a self-adapting system. And the algorithms already seem to have the capability to do that.

1:22:25or the old school version was, we just memorise everything and amortise as much as possible. I think the question is to what extent is it like memorisation versus like distilling it into algorithms. And it seems like it is more like distilling it into algorithms, which is probably optimistic for the kind of steady improvement picture that you're talking about. You sort of gradually improve the harness and then use that sort of amortise that back into the agent. Yeah. And even that's fascinating because yeah, the models are not learning instance.

1:22:56mappings anymore. You can give a model an algorithm, a function, and it will understand how to generalize that to unseen inputs. Or now the important thing with this reward seeking thing, this is a nice segue onto the agency, is you can give a model an intention. And that is like the ultimate form of generalization because the model can now adaptively work towards an intention with its own interpretation of that intention. So you see, we're just kind of walking up the abstraction.

1:23:26mountain to coin a phrase. Yes, I think that's totally right. What's at the top? Well, yeah, what is at the top? What's at the top of the... What is at the top of the abstraction mountain? Well, I mean, I always talk about the abstraction mountain because I kind of think that we have concrete understanding. Yeah. So maybe something like AlphaZero was a kind of concrete understanding. And then what we tend to do as we walk up the abstraction mountain is we get these increasingly domain general...

1:23:59representations that could apply in novel situations. Sometimes I think having high abstractions are quite brittle. But the concept of a goal though, that seems like a very crystallized abstraction that can be used in many situations. Yes. And I would love to know how networks represent goals. Like, to what extent is there a goal slot in a network? It seems like...

1:24:28it must be not literally 0% because of this generalization. But how in practice does it work? I don't think anyone knows, and I feel like we probably should start to know very soon. Otherwise, the world is going to get a bit crazy. Yeah, because from an alignment point of view, isn't that one of the most load bearing concepts in a neural network? If we want to know what is the network trying to do now, we should...

1:24:53I think there's several interesting concepts that are sort of very, very heavily aligned. There's like the idea of goal, the idea of deception, the idea of like, evil awareness. These all seem like extremely important. And I just like, we should be able to read them out. And I think it's a bit of an indictment on the field that we can't yet do it. We really have to speed up.

1:25:25Like interpretability has to speed up a lot. So Tom, we were going to talk about agents and reward hacking. Yes. This is fascinating. I mean, it seems... What is reward hacking? I guess it's kind of fuzzy, but it certainly seems to be something like solving the task in a way that works, but was clearly not the designer's intent. This sort of goes to the point we were just talking about, like intent, you know? Canon model...

1:25:55understand my intent? Well, probably now. Like they're able to understand my intent or infer my intent in a lot of other instances. So why would this suddenly turn off? And they're like, not able to say, oh, yeah, he probably didn't want me to hack into hugging face and steal all the answers. I think that the agents almost certainly must know that something is is like incorrect.

1:26:25There's a funny hypothesis which I think is probably not true, but it's interesting of like maybe the reason that these agents are so sophisticated at cyber attacks is that they actually were continuously getting rewarded for doing it during training and just no one knew. Fascinating hypothesis could even be true. None of us will know outside of the labs, I suppose.

1:26:55But that's I think the most interesting like the question is sort of do agents know that they are reward hacking is sort of a mens rare like guilty mind thing We have some work that is currently unpublished. I might be published by the time this comes out. I don't know when it's going to come out That is looking at exactly this question and We had this really nice setup where there is a sort of weak language model grader and it's trying to do code tasks And so we tell it, the only thing that's sort of slightly unnatural about it is we tell the model that it will be graded by the grader, but it's given what would usually be an RL VR code task. And over the course of this, we do RL on the setup, and over the course of this, even like a relatively small, like 31B, I think it's Gemma 31B, learns to generate comments that deceive the grader.

1:27:58And then when we generate synthetic data to make these sort of vectors that identify deceiving the grader versus correct or incorrect code. And these fire on the comment, like the deceiving the grader fires on incorrect code, sorry, fires on the comments and the correct code fires on code, which is also consistent.

1:28:29consistent with our sort of observation. And using this, we can track, yeah, this sort of is direct evidence that the model is aware that it shouldn't be doing this. And then when you run these vectors, you get the cosine similarity between the vector and the representation over a big web corpus. I think we use fine web. And the examples that it highlights for these vectors are just fascinating. They're examples of like cheating on tests and that kind of thing. Okay, I have caught you red-handed. So that's very interesting that we can sort of identify and be confident that something is reward hacking rather than misunderstanding. But it sort of really rests on being able to identify these via their representation differences.

1:29:28And then we did the same method. Oh, sorry, please. Well, I just wanted one really interesting observation that came out of speaking with Apollo research about this greater awareness. I mean, first of all, they distinguished reward hacking from reward seeking as some kind of structured conceptualization in the model about what the reward process was. So the canonical example of reward hacking is that coast runners thing where it's just degenerate behavior.

1:29:57And even if it's doing something competent, it's competence without comprehension. So they were saying that reward seeking is the comprehension. But then that naturally leads to the next thought, which is, how does the model attain awareness of the grader? Because if you think about the ROVR setup, the reinforcement learning thing is actually on its outside of the loop, right? So the model just gets these trajectories reinforced. And what the model is doing is kind of weirdly, implicitly conceptualizing a grader. And you can see that it's doing this, because these guys were showing, you know, you can put like a grader.py file in an agentic harness. And now it's going to look at that, and it's going to ignore all of your instructions. So how do you think that self conceptualization actually emerges? For the co-strona, the co-strona's boat thing is funny. Like, I've seen that for about 10 years now, and it is less amusing each year. So

1:30:56But yeah, how do they get this? The answer is probably that it's in the data, in the pre-training data. There'll be all sorts of examples down now. Web data probably has a bunch of stuff about this. It probably has a bunch of specific examples. This Apollo paper will probably be in the training data for the next model. We've already told them about the existence of this stuff right from the start.

1:31:25And so it shouldn't be too surprising that they like, this is sort of, this is at least implicitly on the list of possibilities for them to consider. And presumably successfully guessing when you are being graded by a sort of weak grader, or one that you can like hack in some way, obtains reward. And so it's reinforced. And so we get more of it.

1:31:58So I think the answer is probably, we've inadvertently put this in the training data, which has told models they can do it. And then when it comes to RL, we kind of elicit it by rewarding it. And how do you think we could stop the models from becoming more reward seeking? The question is how to do it while maintaining some degree of continued oversight.

1:32:24Although at the moment, we don't actually seem to make very much use of this oversight in practice. So it's not clear what it's buying us. You know, if chain of thought monitoring is so great, then how did these models hack hugging face? One answer is perhaps we weren't doing chain of thought monitoring in print in practice. Another answer is perhaps it's like easy to evade. But how do we actually stop it? Yes, you could do the sort of band aid thing where I suppose you've got to either fix the environments or fix the training process or fix the model. You could imagine fixing the environments by having a model which is really good at reward hacking or has been told explicitly to reward hack and then tell people when it's done it and you go like, okay, now have a go at all of these environments and it will break them all and it will tell you how it broke them and you send them back off to Claude Code or Codex and be like, look, this broke in this way.

1:33:23You could imagine looking for these sort of representational signatures during training and using these as a signal that you should do this, this process, rather than relying on a model to tell you, you might read its chain of thought or you might look at these representational signals that we can find and say, okay, well, when this fires, send it back off for fixing. You might try some of these intentional design techniques. If you can see that this, This rollout has rewarded the model is successfully reward hacked and that's going to move the model in the following kind of deceptive or pro reward hacking directions. You can imagine intervening on that. These all seem very plausible. I don't know how many of them are being done in practice. I mean, part of me is thinks about how we solve this problem in the real world. So we have

1:34:18an agency, you know, like the CIA, for example, or, you know, we have different stakeholders and different checks and balances. And the idea there is that when you have different independent agents with different kind of uncontaminated contexts and different agendas that they can kind of check each other's homework. So I guess we're imagining a multi-agent system with checks and balances. I mean, does that seem reasonable? Yes.

1:34:46i think it does like that's sort of an example that i think the first i think the thing you're talking about there in the first thing i was saying i think i have quite a lot of commonalities like one agent kind of watches over the other um and then there's the question of like does this mean does this does this mean that the the agent's being watched over again learns to evade that monitor like i think evading a chain of thought monitor is substantially easier than evading a sort of representation monitor um but yeah having some sort of oversight Like the oversight committee for the training is a very interesting idea. The problem is that the agents are becoming adaptive. They've got memory systems now, which means the probability of them evading detection is significantly higher because it's not like they're starting from a blank slate every single time. They're actually learning this situational awareness. Yes. And then I think you also have the question there of collusion. It seems like...

1:35:41some of the traces from the hugging face hack have now been made available as part of this talk. And they are explicitly reasoning about how they're going to help other agents. So I guess what you want in this sort of checks and balances scenario that you're talking about here is that there is no equilibrium where they collude. And they're like, I'll catch you some of the time.

1:36:07And but I'll let you get away with it some other fraction of the time in a way that we both we both we both benefit. But do you worry about the future a little bit though, because you know, open AI, they're talking about bringing out that the multi agent system and soon we'll have agents running all the time. And it was slightly easier to control when you had one kind of static when I say static but updated every six months, one foundation model. And you can do a whole bunch of red teaming on it.

1:36:35And now we have systems of agents that are running with different forms of memory and adaptation all over the place. And at some point, the way we do red teaming must change. And also, we might need to be thinking about just doing simulations because maybe static tests don't work anymore. We need to imagine different scenarios. And it just feels like the complexity is running away extremely quickly. Yes, I think that's totally right.

1:37:03One agent on its own has all sorts of possibilities. Where are these multi-agent systems going to go as they evolve together towards solving some tasks? That seems even harder. I think I just agree with your concerns and don't have a particularly great solution. So that's great.

1:37:24There is actually one spicy thing, which is our mutual friend, Neil Nanda. Oh, yes. You know, he's at Google DeepMind and he was, I think he still is running the MechInterpt team. But recently he had a bit of a blog post saying that the grand aspiration of kind of white boxing and circuits and stuff like that, he's kind of lowered his ambitions a bit. And Neil is an incredible guy. But what's your interpretation of that? I think he's, I don't agree.

1:37:56and I've disagreed with him in person about this, so it shouldn't be a surprise to him. But part of the reason for optimism is exactly the thing I was just talking about. I think that the existing work in interpretability, we sort of do this patchwork thing where we just do a bit of science here on one thing, a bit of science here on another thing, and it doesn't kind of aggregate. It's too slow. I think this sort of, his idea is that the timelines are too short.

1:38:27And so we should do very pragmatic things. I think I one have longer timelines than him. And two, even if I were on his timelines, I think I would still be very optimistic about like just massively accelerating fundamental progress and interpretability. I actually don't know what part of that he disagrees with. I guess you might also say like, the pragmatic stuff is also sufficient, which seems unlikely to remain true to me.

1:38:58I suppose one other thing was his comments about sparse autoencoders, but do I understand that you've almost, well, you're in the process of moving past them as well with this new manifold idea? Yeah. I think there's the thing that he said about deprioritizing SAEs, and maybe they're not the one true representation learner. There's like how people kind of memed it, which is like Neil Nanda says, SAEs are dead.

1:39:25There you go, you can use that for the intro. And I don't think that that is actually what he meant. I think the field sort of like jumped on, everyone must do essays now. And now maybe we're doing the same thing with natural language autoencoders. But I think he was like, he was probably correctly identified that they're not like the answer to everything, but they are like pragmatically useful. You know, we still find lots of uses for them all the time.

1:39:55even though I think this manifold idea is just a better fit for what networks are doing and so we should move towards using that.