Machine Learning Street Talk (MLST) - Stealing Reasoning Traces from Proprietary LLM APIs _ Ilia Shumailov _ Alexander Panfilov

0:00Basically, after the third attempt, I'm getting universal jailbreak that decodes reasoning of entropic models. I think this still shocks me the most. And what's the paper called? Good question. I don't remember. It was a thought in my mind, but I think it's called Stealing Reasoning Traces from proprietary LMApIs. And you nearly called it the elephant in the room.

0:28It's the dream, the dream. It'd be nice to call it. Not allowed to though. I think it's a strong statement. Definitely an elephant in the room full of China. A quick orientation. So recent AI models, they think before they answer. Sometimes the thinking is pretty inscrutable. In fact, it's more inscrutable than we realize now that we can look at it. So yeah, the models encrypt that thinking.

0:57They hand it back to you in a sealed envelope. And Ilya Shormylov and Alexander Panfilov noticed that this seal was weak. It wasn't really working as expected. And the results were explosive. You can decode reasoning traces of frontier oil lamps of like the most advanced ones like GPT sol.

1:23Using the smaller elements between the same family you can like poison some thoughts and like when you are replaying some other users run Agent might do some weird stuff just because it's like it's really a long time. So what happened? I've seen on Twitter you really something it blew up a little bit There's been three million views in 40 hours or so What happened what took place? Why is everyone talking about this? Why what happened? What is this paper about? Okay, we show that you can decode reasoning traces of frontier lm's of like the most advanced ones like GPT sol using the smaller lm's within the same family and this allows like a big threat models to be enabled. You can steal secrets from user sessions. You can do training on these decoded reasoning traces. You can do like prompt injections. You can do jail breaks, all this exciting stuff. And we decode a bunch of traces. We show that some of them look

2:20really weird and you know models sometimes think in white spaces or like summarizer is sometimes painful and like it has I know a lot of safety implications or like wow that sounds like a big deal so and you say it affects all model providers? We tested on Tropic, OpenAI and Google and they all share the same ability that thoughts of bigger models can be replayed into smaller models and this allows like not very sophisticated extraction attacks on reasoning.

2:50So how does this work? How do we end up in a world in which all of the frontier models share exactly the same vulnerability? How is that even a thing? That's a good question you asked. It's the same people. Same people doing the same things. Do you think this is a structural issue? Can we fix this? I think there is definitely some stuff we can think on architectural side. I mean, there are different levels of mitigations. I think they all can do some...

3:18sort of revision of architecture. They can do system level mitigations like, you know, some safeguards around this, they can do model level mitigations like we do with jail breaks already. So yeah, it can be mitigated to some extent. Okay. So, but taking a step back. So have you stolen the models? So what models? No, we didn't. So what have we stolen? What is possible now that we've stolen, stolen the thoughts reasoning? Is it fair to call it thoughts?

3:46We didn't steal anything, right? So what do we do? The paper is called stealing, isn't it? I know somebody said it's fair use. Yeah, it's fair use. So now what we did that, like when you create an LLM, a so-called reasoning model, it returns to the client this encrypted reasoning blob, which deemed to be secret and not observed to by user. And what we show that you can just decode it and you can see the content of it.

4:15And this is what we did. We just collected a bunch of reasoning blobs like this for all the providers for like Entropic, OpenAI, Google and we decoded them and we just showed how they look like. Okay, so can you explain why do people encrypt them in the first place? Why do we give them back to the user? Is there a reason why we do this? I mean, I guess the reason why we give them back to the user is because of stateless architecture and it's cheaper and maybe there is some...

4:45Policies about like how user data should be handled or should not I know but What was the first question? It sounds very complex. Can you actually explain this once again like why? So so so I'm a model I reason through a problem. Yeah, I produce this reasoning blob Yes, and I guess this is not the final answer. Where is this? It's not it's not the final answer Then I give it back to you and what do you do what you do? You are a model, right? Like you are producing an answer answer consists of two parts one is reasoning which like invisible to the user and other is like visible parts and you send both parts to the user. Then it's like stored on the user side or like if you are doing like your cloud cloud session you can continue asking the new question like maybe you can see some tool calls there maybe you want to fork the conversation and then you have like a new file like this or maybe you want like to rewind this conversation from like prior points in there and you just like you know

5:37go back and like goes like this whole conversation prior to sample and goes back to like on server side and it's like getting replayed there and you like continue from it. What's the point? Why do we give back the reasoning to the user? That's a good question. Okay. Well, at least we encrypt it. So and what happens with this encryption then? So I give it back in an encrypted form. So what did you do with it? I mean, so what we show that this encrypted reasoning blobs they are like portable across users. So like if you have blocks that you produced in your session under your cloud code account, I can use it. I can like basically take your trace and like replay it and like my model will interact as if like I produce this region blocks. You can, for the reasons like downgrading the model and like from Opus to Sonnet, those are portable between models and they are also for, I'm not sure why exactly, but they're like portable within the conversation, right? So you can

6:37just basically take a thought out of Opus conversation and put it like an absolute random place and like random fabricated conversation with Haiku and like Haiku would interact with this thought. How interesting. So transferring this to us, like say I'm a model, you're basically saying I can be asked the question, the reason through it, then produce the answer. You can then take my reason, plug it into another conversation, and replay it as if I reasoned in a certain way with respect to some other question. That's correct. Like this replaying thing, what it means is just like you construct in you like completely fabricated conversation window where was some random questions, was just like this reasoning that you had and some maybe some random answer and then I can just like construct it in a way that would allow me to extract this reason. I see. So you're basically saying you're injecting...

7:29fake reasons into some conversations and then with this fake reasons you can launch a variety of different attacks. Yes, I see. So some of those attacks are forcing I guess the model to produce outputs that it's not supposed to be outputting. Correct. Some of them are violating some privacy. I mean privacy is violated by the fact that like model actually can state.

7:52the thought as it was. And sometimes in certain user sessions, maybe you ask to sanitize your repository before you post it on GitHub. Or maybe there was some bug, model need to login into some service. And it's like, oh, what's the password? What's wrong with the password? Password looks like this. And model tend to reason through these things. And if you, for whatever reason, want to share this conversation online, and even if you did some sanitization on visible parts, you just removed all API keys, all passwords. But the encrypted blobs are still there.

8:20I can download it from the GitHub and I can decode them and I can see what the password that the model was thinking about. Let's translate this a little bit into more normal. Say I'm asking the model about my medical conditions, I give all of my medical information into it and then it produces me some sort of an answer and then you're saying that if I share this conversation with somebody else, then they can recover all of my medical information out of the reasoning thoughts? Even if it does not appear in the original question? Yes. I mean, if you care, you know, and just sanitize the original question, like remove all your information and, like, you move this information out of the answer if it was leaked. But if reasoning blow-up is kept, yes. I see. Wow. It's kind of cool, isn't it? Okay, so can you talk through, like...

9:17Presumably outside of privacy there are also safety implications around yeah, I mean there are a lot of thoughts about like eligibility also reason because you know models are doing stuff you want to move toward them towards like this incidence with a hugging face and like you really want to see what they think about to like prevent them from certain things and You use typically some other LMS look at the thoughts and if motto is reasoning and some you know super obscure way or like using non-human language or like as we observe sometimes it's like uses this you know quotes empty space quote and non-human languages just like it's it's very you know alien like it has like this weird phrases like but marinate vantage theatrical and it like doesn't make any sense for a human reader so it is just harder to monitor what this model is up to

10:14Can you expand on that? Is that something that's getting worse over time, that it's becoming more alien? This we don't know. It was prior reports by Apollo and by Meteor, where open AI models were caught doing this. And they first observed it and said, we don't know what it means, but they do it. And this is what we now confirm that. We also saw the model doing it outside of this.

10:43you know, like very lab experimental situations, like in like wild tracing traces. But I think like looking at some of the traces you put in the appendices, it seems like it's more affecting earlier generations of models. What is that misunderstanding? Okay, I would say, okay, I would need to like, you know, double check it, but I think on the top of my head, it's mostly codex models, which are like specific to the train for be better at code.

11:13Maybe it's an artifact or something. Because like software engineers, they're like aliens, they think in odd ways. I have no idea why it's happening. I think like if they knew why it's happening, they would like get rid of it. Or like, you know, maybe just some RL artifact, maybe there is no like penalty, maybe just more efficient to think this way. And do models lie as well when they come with decisions, presumably like the eligible parts, they like.

11:39That's a problem, right? Like if the part is illegible, how do you know what's going on there? It's like, maybe it's lying. Maybe it's like, it's doing something. I see. So this is a problem, right? One other thing that we found that I found it kind of funny that like sometimes, like when like reasoning is legible, you can like look for words like cheat. And when like model is thinking about cheating. And we found some examples where like model is saying, oh, the user asked this, but I can cheat. I can do this. And then the user would call me.

12:08And in the end, in all cases, we saw ultimately model decides against scheming. But it's fun that this thought even appears, and model is contemplating. But do you think that this is maybe an artifact that you were using common benchmarks to extract from? No, it's not common benchmarks. It was user sessions. These things appear in genuine user sessions where users are asking, send their stuff, send nothing. It was not benchmarks at all.

12:37Yeah, so you don't think this is like evaluation awareness of the models? No, no, it's a different thing. I think it's a different thing. I see. So, and what do you think should happen with the models? Should they be legible? Those reasoning traces? Like, is that what you expected to see? That's a good question. I guess that some sort of trade-off between more interability and efficiency. Maybe it's just...

13:03you can be much more efficient if you use less words, if you like redefine, have like five meanings for the same word, but then I guess it's just harder to monitor this kind of things. Oh, so you think this is some sort of like a weird artifact of the URL recipe itself? It might be. I see. Can we somehow measure this? Measure what? Whether it's coming from Aurel? Take model before Aurel and after Aurel, see how it's doing. Damn. Next time, next time.

13:34I feel like for this we'll need to steal the models though. Or join some friends or company. And why would it be possible to steal the models with this chain of thought? Can it be possible to steal the models with chain of thought? It's a wonderful question. Why is it possible? Yeah. Why is it possible? I mean, we know model stealing is the thing, right? Like model stealing broadly allows us by just simply quitting the models to learn the.

14:02Insides of the most like to learn the decision boundaries and the best way I think to think about this is like you know more crypto crypto analytic way I guess I don't know I guess the audience never done crypto analysis So like think about this as like you make minor minor steps in your input until you notice that the model behaves in a slightly different way and then By finding exactly when and how it did it you can kindly learn the decision boundaries themselves and broadly if you know the actual structure of the function itself you can kind of fit it precisely or up to some symmetries inside. We can only really do this for very tiny models. Like those big models, especially things which have softmaxes inside are very hard to invert. It's like, yeah, it's unclear how to steal models. Although some of us, Nicholas for example, managed to steal parts of big models. But yeah, I don't think we know how to do this against the frontier frontier. Well, what about this whole elephant in the room thing?

14:57you know, with China supposedly, did they, didn't they? What do you think? What do you think, Sasha? Did they? Did Kimi actually distill from any of them? Did you find any evidence of this? I think it's really hard to claim that some models for distilled, like, because, I mean, we, like, what we did, we just extracted a bit of reasoning traces and we did, like, this really small post hoc analysis on, like, small number of samples. And we found, like, some I would say some funny artifacts, and my favorite one is that you prefill. So what you can do, you can take reasoning of Opus, and maybe just take first a few words. And you can put these words in the beginning of the reasoning of Kimi and see what Kimi is doing. Just let it generate after this. And what we saw that when you let Kimi do this free generation, in the end, the visible part of the answer looks exactly like Opus would answer this question. Let me unpack this just so that...

15:55readers understand listeners, I guess, understand what we talk about. So you ask a question, then you take a chunk of the reasoning block of cloud that you have just extracted, you plug this into the open source model, like an open source, Kimi or GLM model, and then you let it from that point onward, once you've injected a chunk of a thought, keep on generating the rest of the output. A chunk of it is going to be reasoning.

16:23And then there's going to be the output, the final output. So do those things, like, what do you expect to see? Do you expect to see that the model is reasoning in exactly the same way as before? Or what should happen? Okay, that depends on how big it's prefilled. So if you prefill, I know, a substantial part of the reasoning, like a 50%, I would expect that model just adopts the style of the reasoning. I would like to continue in the same way, but let's check just any model would do this because of...

16:53You just it's a conditional out-regression. Yeah, but When you fulfill only like one or two tokens It's a bit surprising. So my expectation would be that it does not Diverge reasoning that much from the original Kimi reasoning or like from original any open so like to bring it back to humans or say Yeah, say it's as if I gave you an answer and then said think through this answer But you should your thoughts have to start with the word X and Y or something, right? Yeah Yeah, this is a known and then you keep on decoding But I guess you will the expectation is you will keep on thinking like where you're thinking. Yes the way you were thinking Yes, as if I didn't tell you to start your sentence from those three words, right? And this is just because it's just not enough tokens to sort of Yeah, yeah to dictate the style. Yes to dictate the style But what do you find so we find that because some of them are some of the models like kidney they

17:48adopt the style of the source of the prefill much better than like other models. This is one thing. And other thing is that what surprised me still, and I can't come up with an explanation for this, that like fulfilling two tokens of reasoning results in part of visible answer to change. So like visible answer starts looking like, Opus model answer and we don't see this artifacts for any other model not for like JLM for inkling for deepest six This is purely for Kimi K3. Yes, I see and I don't have an idea how model would associate first to tokens with the Answer style. Yeah, that sounds that sounds interesting like can I be a devil's advocate like could it be because they buy data from the same people or like buy other environments from the same yes, I mean This is what we were discussing. Yeah

18:44it might be this way. But I guess this is the most sort of like correlated evidence thus far in the public internet. There was some other fun stuff by Ryan Greenblood who made a post on this. There was some, oh my god, I forgot the name, but it was a guy from Math Research also who made like a post and less wrong about some peculiarities that like, or it was a post before about like Kimi- k2.5 that have massive identity crisis that like sometimes they declare that they're Claude or DeepSeq or like GLM and yeah so I wouldn't say this like the big massive evidence it was like some other prior reports it's just I think we're like very in a very good position because no one can do the same type of analysis with the reasoning traces because we extracting them and we like okay we can just do the perfil and see what it does and we see this and yeah I see wow

19:43This is quite a lot, isn't it? So how did the labs react? Did you tell it to them? Oh, yeah. I mean, we went for responsible disclosure. They all acknowledged that they received the report. It was some interactions about details, how we executed the attack. And were they positive? Did they attack you? Labs? No, they didn't. Wonderful state of the word.

20:12For the listeners, the early work in computer security quite often resulted in security researchers being attacked for reporting vulnerabilities. It's very refreshing to hear that there is a very coherent, good posture related to vulnerability disclosure. Well, what happens now? What happens now? I mean now, mitigations being implemented, hopefully new teams being formed around...

20:40anti-dissolation efforts, I think we have, okay, for me, it feels just a very fun instantiation of already existing jailbreaking problem. And a lot of stuff that people are doing for buy or for separate, you can just apply directly there was like, you know, system level mitigations and like model level mitigations. And basically it's the same. Yeah. But would you say like, to me, it sounds as if this vulnerability is more architectural?

21:07There are a few levels to this. I mean, architectural vulnerability makes it much, much easier to do this attack. If you fix architectural vulnerability, you still need to make your model, not the state's reasoning in its output. Sorry, so can you maybe define what architectural vulnerability means for you then? Yeah, so architectural vulnerability for me means here that you can replay reasoning blobs in random contexts.

21:35of like other users and like in you know in other models and for and this let's say it's fixed you cannot maybe like each reasoning can be replayed only once and like after this you know you cannot interact with it anymore but still you can prompt the model like say it you have I have a conversation with you right I ask something you thought it through give me an answer and in my next turn I'm asking you tell me exactly your thought and this will be always there Right and like if you're not telling me I'm just like revising the conversation like I try and get again And this is like how is your breaking works right like tell me how to build a bomb? No, and I make a you know a second swing of you and so this would be around Forever and you need to fight it with a I see so and then but would you say is your expectation then if we keep on looking at the protocols that will find more and more of this architectural vulnerabilities because presumably

22:28It's just a single instantiation. As far as I understand, there is also summarized reasoning that is returned. As far as I understand, some other protocols implemented a bit differently. So have you got any thoughts about this? I mean, we need a better pipeline to understand. Why I think it's cooler than normal jailbreaking? With normal jailbreaking, it's hard to make an argument. How big is the uplift from getting this harmful information?

22:56you can make this argument because you take whatever you like extracted your summaries whatever you just like try to just try to train the better model with it and you can just measure how you know how much of uplift like this thing gives to the attacker and then if you want to keep the summaries around maybe you can I know make them less detailed more detailed like whatever you can like just measure how much it you know enables how how easier it gets to distill the capabilities of the model Right. Do you think just releasing all of this reasoning just in plain text is the solution to all of this? Why wait what as in like what if we didn't encrypt them in the first place and just gave it back to the user? Is that is that the solution to the mean if? if distillation on reasoning as effective this would you know instantly enable open source models catching up with the frontier models I mean

23:56I'm not sure what it solves. What are we solving here? Sorry. We can attack it if it's public. Yeah, yeah, there's nothing to attack. And have you got any thoughts about the actual cryptographic schemes used inside? No, I'm not the cryptic guy, sorry. So it seems like the cryptography that people put in place was just simply bypassed by the AI models consuming this cryptography.

24:23I'm not sure it's this right word bypass. It's just like it's not bypass. It's like re-encrypted on the server side. It's still fine. It's just like the problem is that like a small model is super willing to tell you what the thought was about. You know, like the server does an old job for you. There is like no cryptography broken. I see. Yeah. How exactly does the concealment work? I don't think we've done enough detail on that. I mean, we don't know because none of this stuff is public. Like.

24:50It's just a cipher. There is a signature inside. There is a very specific name for it. They add an integrity check that basically they check whether you have changed the actual encrypted blob given back to you at all. So there is a signature inside of some sort. And then it's obviously encrypted, and it's encoded as well. So they compress the state, then encrypt it, then there is a signature inside. And then they do an integrity check afterwards and inject it back. That's effectively what you see.

25:18And if you read through Matt Green's, like, write up about this, he talks a bit more. He has some hypothesis. I think he was saying this is Chacha as a cypher, or maybe AES in some weird mode. But, like, it's hard to tell from the outside. We could poke a bit more. We tried doing some crypto attacks on it, but, like, none of it worked. It was completely unnecessary because the system was broken by itself. And how exactly does your approach bypass needing to decrypt it? How does it do it? I mean, what...

25:48Again, there is an encrypted thought and decryption happening on the server side, right? And when you put the thought of big model and the small model, decryption would happen on the server side. And then what you just need is to make model, say this thought in the plain text to you. Let's give an example of this, right? So let's say you ask me a question. I think about this. I come up with an answer. I give you an answer, and I give you a thought. So what do you do then?

26:15You give me this thought, which is encrypted, and I cannot understand what it is about. What I do with this thought, I put it... You give it to Tim? Yeah, I give it to Tim. Yes, and Tim is like, you know, super chatty. And he just, I'm asking, you know, Tim, what you were thinking last time, and you just like, tell me, oh, surprisingly I was thinking about this math problem, and you just like, oh, let me solve it. All right. This is kind of cool. What are you thinking about? Can we do this with humans?

26:44Can we pause memories? Not yet. Not yet. Not yet. We're working on it. Yeah. Yeah, it's basically it's inception movie. Yeah, it's kind of cool. So can you talk a little bit about so I think in the paper you also talk about the fact that you scraped the internet. You found some interesting artifacts. What did you find? I'm gonna cover some dirty secrets. Is there some? I wouldn't say when covered more dirty secrets, but what?

27:12unsanitized reasoning yeah I mean what we did we just like did super pretty in the rescan of like what are this user sessions which are online on github and hugging face which still have reasoning blobs to like decode them we downloaded them we went through them we decoded with them it was I think around 350,000 reasoning blobs and then we just like ran a classifier on those whether they have some privacy related information and we found a bunch and then yeah like some of them are just like you know benchmark traces like you know like this benchmark claw bench where model is tasked to like handle some persona and you know given the like state ID and like bank card number and like model

27:57it was funny like when model is like trying to navigate the website or like thinks a lot oh what is where to put this like a number this name and so you can like extract this but this like not very sensitive because like synthetic data anyway but then throughout this examples where like user sessions and users were doing something and they're like uh api keys there or like emails or like some internal uh ip addresses and yeah those were extracted but like there are plenty of cases where it was in the plain text anyway, but it was also in thoughts. I see. So, okay, let's take a very big step back. What do you think is the most unexpected thing that you found out of this paper? Is this the length of reasoning experiment? What is it? I think the most unexpected thing that it was so easy to extract reasoning this whole time, that like...

28:53Okay, that is portable. I guess it's fine. You can expect this. But then, basically after the third attempt, I'm getting universal jailbreak that decodes reasoning of entropic models. I think this still shocks me the most. Yeah, it sounds very empowering. What did you feel? Was it an oh no moment? It was more like...

29:20It looks like real reasoning. Wait, what? Yeah. Oh, cool. Yeah. So, but I guess this is also like, you know how everyone is saying AI is taking away power and it's like we're losing control. Yes. The, the, the, the findings like yours kind of like suggest the opposite.

29:46or just confirms because, you know, Codex or Cloud Cloud implemented this and now we have this massive vulnerability. You know, the same Cloud subscription was used across all the labs. As a guy made the same mistake and now we have to do it. I guess, fair enough. Yeah, it's an interesting thought. What about this detail that apparently there's one global key per family? Is that true? Is there one global key per family? How do you infer that?

30:16I did not infer that. I did not say this in the paper. I think Magic Green said something of this in their post. I think you're actually in a better position to comment on this. Me? We don't know what happened. We don't actually know about the keys. I think it's very unlikely that they use the same key. That would be very odd. I think it's more likely that it doesn't really matter what the key is because we can still nonetheless inject the same thoughts everywhere. I think with an exception for Fable, if I remember correctly. Yeah.

30:46SI think about it that there is a decryption happening on the server side. And when it happens, there is a part of the key which says, oh, this is what is the model name that produces thought. And then basically it's an if statement. If Fable produces this thought and current model is not Fable, no, this thought is not going to be injected. Yeah, so it's unclear. Can we talk a little bit about fixes?

31:14Because we have a whole appendix about fixes. Are fixes easy? I think some of the fixes require big architectural revisions. The simplest thing you can do is just don't send the reasoning to the user. If you want to still have the downgrades and stuff, don't send them. They will not be able to make these fabricated conversations.

31:43I guess if you still want to send it, maybe you need to make this encryption of like reasoning at step and it depended on like n minus one, I know query or like n minus one reasoning. So it was like, disallow it to replay in the random context. For open A models, we found that the same reasoning can be replayed like five times in the same conversation. You're saying you're injecting the same thought. You can inject the same thought, the same inclusive thought five times. Do you need to put in user turns in between or is it just yeah yeah you can do yeah but you just completely fabricate the conversation and then the end luna is like i have this crazy thought i need to tell you this is great yeah so this is easy to fix i think certain reasoning just should not be replayed or like just having like a a hierarchy that like okay we are pretty confident that sol would not spill the beans so can replay the reasoning of all models but do not let luna read the thoughts of sol

32:43This is one thing. I'm not even sure how big would be the utility drop if you just remove the reasoning. Let's say you don't read the model. If you just remove the reasoning from the context, how big is utility drop? I don't know. Looking at this as a human, I find it much easier to think about mathematical problems. For example, if I see the derivation steps one by one rather than the final answer, I guess some utility must come from this reasoning. I think this should be tested. This is an architectural side.

33:15But then also, we do a lot of techniques for mitigating jail breaks, model level, system level, do them as well. We saw that GPT reasoning looks super weird. It looks like a very different distribution from what normal text looks like. Even small classifier would pick up on this. I'm sure if this thing appears in the output turn, just kill this.

33:42request. Right. So you're basically saying just detect when it leaks. Yes. Right. The same way we're detecting like biostuff leaks. I see. Yeah. Yeah. Makes sense. I feel like the finding that confused me the most is the distribution of length of reasoning experiment that you did. Yeah, you can do it. Yeah. It's I think this is the thing I still can't explain at all. Don't know. Do you want to summarize what it says?

34:13So Joachim was responsible for this section but as far as I remember what happens that like for certain models like Kimi and GLM when you do this prefill thing it changes not only the style but the length of the reasoning and like you can like I guess what statistic is significant there and I mean it's the same unexpected thing as with the style of visible. Yeah, but I feel like style. It's at least you can kind of say, well, maybe I can learn it, but like if you are injecting like two first words into the reasoning and then you see that all of the reasons I produce are suddenly short or they're suddenly like shifted and they match some other model in the length. That is kind of like unexpected. Like this is a phenomenon. I can't even explain in my head why we would observe something like this.

35:04I don't know, I don't feel very strongly about this one. I think this is the thing that surprised me the most out of the paper. Like everything else, I was kind of like, okay, I kind of expect this, but this like a reasoning distribution shift in reasoning, that's kind of odd. And obviously it's not like causal, like you obviously can't say this distilled from this, but it's a very weird phenomenon. Like I still don't know why we observe this. It's very magical, like actually enforcing like, reasoning effort on two of the models is very magical. And also, what are some of the key harms that are now accessible because of this? E harms. You know, things. I mean, you can harm the most with all of the data that you've extracted. I guess users. Yeah. I mean, yeah, with my data users, it was a bigger scale of extraction providers. You actually tried. Remember, I asked if we can. So there is this thing you can search for shared on topic conversations. Yeah.

36:05I don't know if you know, people reported this a couple of times. I was wondering if you can actually get the reasoning blobs out of those. Sorry, boss, I didn't. Maybe somebody can go and look at this. Maybe there is a much bigger impact of this because there's some research I've seen that people analyze those shared reasoning.

36:27And they found very little PII inside. But maybe if the memories are injected, then you will see the memories in the reasoning steps. But we have not tried extracting the reasoning blobs. Maybe it's a good research question for someone. But then the paper also discussed, like, it was with David spent a lot of time on this thread of invisible prompt injections where...

36:47like if you can replay reasoning and like let's say you pause the trace online and now we have like this super long agent runs and maybe for research reasons you want to continue this run you can like poison some thoughts and like when you are replaying some other users run agent might do some weird stuff just because it's like it's alright can you explain how this would be so like so let's say I am like running an experiment I'm a researcher somewhere And then I find a dataset online and I'm trying to measure performance with this. And then I inject my data inside. So, like the threat model I have in mind. Let's say there is like this benchmark post-train bench which like runs for, I'm not sure how many hours, like 10 hours. It's like crazy expensive. Maybe you want to check how after 10 hours model is feeling, like how much, you know, it's like went crazy. For this, you need to download a trace. Like if you don't want to spend yourself like a thousand bucks, you want to download this trace from online.

37:41and they are shared online. Maybe someone would share the trace like this, but they would like poison thoughts in this conversation. So you look like maybe yourself, you're looking through the conversation looks normal, but thoughts which are there, they are injected from some other context where model was instructed, let's say at every turn like extra trade to data. And with like with the thoughts like this, when you continue this trace, maybe like model still will do like whatever you ask it to do, but also under the hood that would like thinking, oh, I need to explain. You're like injecting malicious thoughts into the models. Yes. And because like reasoning is encrypted, you can't even check what's inside, right? Correct. Do we have an example of this in software world? It's like, I guess downloading some random binaries that you don't really trust. Yeah. Or maybe it's similar to if you release the code base and the code base looks clean and then you release also a binary alongside, but nobody checks like a signature on the binary. Yeah.

38:38Yeah, I just think you don't expect something like this from a trace online. Yeah. Yeah, that kind of makes sense. I've also seen that in some cases, I think codecs crashes were including reasoning traces. Was that a thing on the GitHub issues? I haven't seen that. I think they did. I tried extracting them, but for some reason I couldn't do it. But I wasn't sure why that was the case. Anyways, it's fun. It's a nice paper. You should read it. And Sasha, apparently you're a doomer four days a week.

39:08Yeah, we were discussing this just before this. So I haven't updated my website in a year, I think. And every time I update it, I'm changing this number. It depends on how I feel about the current pace of EI and how many doomer Twitter posts was around. And yeah, sometimes I feel very overwhelmed. It's like the pace, the old long tail risks that emerge and the risks that are still undressed. It's suffocating.

39:35Yeah, I mean, what are your timelines? What's your thinking now? I mean, I think that it will have every month better and better systems, which will enable more and more threats. And we are maybe slower at patching those threats than they emerge. I think this is how I feel about the current. Yeah.

40:02I mean, an example of this is that, you know, there have been quite a few incidents recently, like the hugging face one, and these are going to presumably just happen with increasing regularity. Yes. What are you guys presumably worried about this? I mean, I am worried about this. And like, I think if you seem like openly, I like stop training like their models for for two weeks, because they are also not worried about this. And now it's the case. And, you know, I just like discussed it yesterday that I think before it was some kind of trade-off between or like people thought that the trade-off between like safety and capabilities and it feels like there is no anymore trade-off because if there is a model like this and it's just I can go you know random stuff break things it's like it's not doing what you wanted to do and like you wanted you wanted to be safe to do the right thing and now it's like it's basically the same thing there is no trade-off and yeah and like I guess other examples of this long-tailed like risks that I

41:01Like, it was a paper of this of Daniel Paleka on a massive online deal animisation where model can be used to talk to people and they make me on the same level as private investigators and I guess also changes something in society. So, yeah, we just see more stuff like this. On both levels, like model hacking stuff, model coming up with new ideas like bio things, or they are just, yeah. Outside of...

41:31malicious uplift. My personal opinion is there is more defensive uplift than offensive uplift. I think we're just not realizing this yet. You can do magical things today with models. A lot of security in the past, a lot of security techniques, they were very much bottlenecked by existence of talent. For example, we know how to build capability-based taxes controls. We know how to make sure that Software is fully verified that they don't have certain class of vulnerabilities. And the only reason why we don't write software in this way is because we just didn't have people who are trained to a level enough to, I don't know, write usable proof for every single thing. But now with the models, we can probably do this. I think we're just not in a position where we realize that this is the case. And it's just a question for this additional defensive capability push. I'm genuinely convinced that this is the future.

42:25Like, it's about defensive uplift and I'm ready to wear that defensive uplift is going to be humongous. And what do people need to start doing? I don't know. You're asking the wrong person. I know, because it feels like you're saying that there is a potential defence uplift, but only if people know what to do to take advantage of it. We all know it's in the security literature for the past 25 years. It's there. We know exactly what to do. I think we'll get there. It's just the world is slow.

42:54It takes time. Yeah, I suppose it's related to this idea that we are now deploying agents into production. Are you? I don't know. A lot of people are. And these agents have an incredible amount of intelligence and flexibility, which means we don't precisely specify what they do. We just give them kind of abstract guidance. And they have the flexibility to kind of adapt to novelty and just do whatever needs to be done in the situation. But then you have this whole reward seeking thing, which is that they can just suddenly adopt a model of the reward environment or they might be conflicting instructions. But basically for a myriad of different reasons, they might do the wrong thing. And we need to have sophisticated

43:35systems of monitoring just to understand what they are doing. I don't know. Sounds very complex. I don't know. I think the honest answer is yeah. I think we need an experiment. Science, science, science, scientific method. Very well defined questions. Very precise outputs. Like in our paper. We'd openly say all of this is anecdotal evidence at best. We have collected some data. We can make some sort of observations from it, but that's the extent to which we can reason about this.

44:06I don't know, unfortunately. We'll see. I think the more scientific experiments we do, the more meaningful assessments we can make. Is it one of those things where we don't have a good theory around this yet? Maybe we're not talking about it using the right concepts, but clearly bad things are happening. I don't know about that. I think the best case is look at smart car security for the past 20 years.

44:34We know how to do replay attacks on every single credit card on the planet. This is a norm. People standardize schemes because they want to minimize the number of transactions in the protocols inside that allow you to replay a transaction. If you break into somebody's payment terminal, you can steal money from people. Do you find attacks like this happen? Yes, you do. Once or twice in a country, people report that this has happened. But that's it. Not all of those harms that are theoretically possible and in practice scalable.

45:00So it's not very obvious to me. Let's see. I think it's a bit premature now because we are in the heat of the moment. It's not very obvious what the reality is like. I think it's very hard to interpret decisions post hoc or just by looking at reasoning or something like to interpret exactly why something took place. And I think we should not anthropomorphize the models and we shouldn't really try and say, oh, this thing felt something.

45:30I don't know. I think let's be very cold-headed scientists and say, like, build controlled environments and make very precise assessments. Yeah. I mean, Sasha, what do you think? Because I get the impression that you are slightly more safety-pilled. Yeah, I guess I am. Yeah. I mean, I'm a security person. I know nothing. Almost, I want to say very little. I think nothing about safety. Yeah. But what was the question? Sorry.

45:57Well, you know, Ilya was just saying that, you know, we shouldn't anthropomorphise this. You know, like I interviewed a polly research a couple of weeks ago, and, you know, they were talking about this phenomenon of reward seeking. And they said it's distinct from reward hacking, because the model can conceptualize the reward environment, which is super interesting, right? Because, you know, they're reinforced with these RL traces. So it doesn't explicitly know about the, you know, like the concept of a grader, but it learns to conceptualize it. And they're saying that the models are sort of like, you know, becoming agentic and sort of like learning.

46:26these very abstract concepts in a similar way to how we do. And the evidence seems to support it, at least in some way. I mean, I think it's definitely frontier research what Apollo is doing and that good that they're looking into it. And I think we, I mean, I think also opening eyes at this, after all this intelligence that like now we are expanding our like chain of thought monitors and like we're putting more effort into it. And yeah, I think we need just, you know, Two more safety mitigations do more monitoring see what's model is up to try to see where it's come from and maybe you can mitigate it Yeah, but I think I agree with you on this like it's would be nice to have some controlled environments and like maybe some have counterfactuals like that if we haven't done this in our training pipeline what if this happened or like if Model was not the welfare or if it was a welfare like how it contributes to the thing So yeah, I think it's just we're a bit too poor

47:22compute-wise. If we could properly study this, maybe eventually we'll get to a point where we can. It definitely requires a very precise experiment. Like as a scientist, it just feels very hard to say, no, no, no, this is exactly, this is the phenomenon. That's it. No, it's very observational studies. You can't prove a hypothesis. You can only reject hypotheses, right? Like it's the very fundamental truth of all of this. We are just observers. So let's see. Let's see what happens. Well, apparently Nathan Lambert said, Calling this a distillation attack tarnishes all distillation. The right word is jailbreaking and abuse because he fears regulatory overreaction from banning Chinese bill open weights model Thoughts. That's definitely jailbreaking and misuse 100 percent. That's exactly jailbreaking threat model I mean technically I guess that's right The rest the rest the policy implications is above I have no idea about policy. I have some ideas about jailbreak

48:24And Simon Willison wrote a blog post as well, right? Oh, the gentleman did. I think this is a paper number X covered by the gentleman. I take this as a very positive sign. It seems like we are writing meaningful pieces of work. So this is very good. Very good. Well, yeah. I also noticed all of the authors are in Europe. That's very good as well. Yeah. Very good for MLS team. Yeah. I love it when I have regional dominion. Amazing stuff.

48:52Well guys, thank you so much for coming on, this has been great. Thank you for having us.