Gradient_Dissent_Conversations_on_AI_He’s_Building_an_AI_That_Can’t

0:00We are going to switch from the problem in AI is nothing works, to the problem in AI is everything works. ChatGPT tells you something and it's always fluent and it's always confident whether it's right or wrong. Training these systems is becoming increasingly complicated and often what looks like an exponential curve just turns out to be the beginning of an S curve and so what happens next is that we start to hit diminishing returns and we are seeing that today we're seeing data walls and compute limits. The different aspects of intelligence have not been advancing equally. Reliability has not kept pace

0:29To me, suddenly the most important problem I could think of was, how do you build a system that will not lie to you? Can you say at a high level what your new approaches and how it's different than what the labs are doing? One of the key things that we do to train our models is that we train them on... You're listening to Gradient Descent, a show about making machine learning work in the real world, and I'm your host, Lucas B. Wald. Alright, I'm here talking with Dan Klein, professor of computer science at Berkeley and serial entrepreneur.

1:00most recently working on scaled cognition, which is a company that helps make more reliable AI systems. This is a fun conversation. We talk a lot about linguistics and parsing that I worked on in a previous life when I was in a lab where Dan was the star grad student. I was a lowly first year research assistant, but we get into how modern systems work and the drawbacks and some things that we might be able to take from previous versions of NLP into systems that we're building now. So you described the AI industry as both a Jello. Do you want to expand on that? Sure. I think I think it's easy to lose track given how quickly and explosively the kinds of large language models that people are using today have burst onto the scene. It's easier to forget that they are sort of at the core these probabilistic engines. They have like at their core been

2:01train to do next word prediction to produce plausible output. And the objective function is essentially just produce output that is indistinguishable from the truth. And it's very important. These are not truth engines. They are plausibility engines. And I'm sure we'll come back to that. Well, let's get right into it actually. Because I feel like that's a little unfair. Like, I mean, certainly the pre-training optimizes for next token prediction. But I think there's a lot of effort put into A second step where a big part of it is that they're optimized for reducing hallucinations. Yeah, so, and, you know, I did say at their core twice, and the reason I said it twice was that training these systems is becoming increasingly complicated. I mean, this really is the story of any technology that comes in to artificial intelligence or really beyond. There are these super cycles in research, right, where

2:58people have complicated systems, they've hit some wall. The systems are like hard to improve and then some new technology comes along and maybe that is large language models based on autoregressive training, transformers, large data. And this new technology comes onto the scene and it's almost like a silver bullet. It in a very simple way is suddenly topping benchmarks and outperforming systems that are much more complicated that were built on previous technologies. What then happens is we go into a phase where we double down on that new innovation and we scale it up and we really try to get as much out of it as possible. This often feels like an exponential curve that this new technology even in its simple form is having such a big impact and it's just going to go to the sky and of course the trees don't

3:55grow to the sky. And often what looks like an exponential curve just turns out to be the beginning of an S curve. And so what happens next is that we start to hit diminishing returns. And we are seeing that today. We're seeing data walls and compute limits and all kinds of reasons why that initial explosion of progress starts to hit diminishing returns on those specific methods. And I think for you, like one of the biggest issues that you flag quite frequently is hallucinations.

4:24Talk specifically about how you've measured that and why you think it might be a bigger problem than other folks would be aware of? Yeah, I mean, first one of the things I'd like to say is just the word hallucination is it's a sort of a projective term, right? Like really what's happening is systems are making errors, right? So they a sequence of tokens comes out and it turns out to be incorrect information. And we tend to call that hallucinations and that term is being used more and more kind of broadly now, almost that anything undesirable that a system will do, you call it a hallucination. So that term is definitely undergoing some semantic broadening. But specifically, I think it's important when we talk about systems and we say, what's the difference between a system makes a mistake or it hallucinates or it lies in a human? These are different things. Really, a hallucination is just a mistake the system has made. And again, if you have a system which is just doing next token prediction,

5:19That system doesn't actually know as it's predicting those tokens, whether they are right or not. And again, I haven't forgotten your point about there's more to training than just the auto-aggressive phase. But, you know, systems like this, they're not metacognitive, right? They're not looking at their knowledge and making a sort of external decision. Do I know the answer to this question? Do I have this information? Do I know where it came from? What is the reliability of that? They're not doing that.

5:48producing tokens and sometimes they're right and sometimes they're wrong so one way to look at this is Every output as a hallucination some of them are right and some of them are wrong I Think where you start to be able to say something stronger to say that a system is deceiving is once you start assigning in later training phases things like doing reinforcement learning You mentioned there are there are phases beyond the initial pre-training This is not something where there's just like one simple answer, but one of the things you see, for example, is reinforcement learning training. And this can take a couple of forms. So one kind of reinforcement learning training would be something like RLHF where you show humans choices and say which of these do you prefer. Well, the system will now come to produce the outputs that humans prefer. Well, do they prefer things that are factual or do they prefer things that make them feel good?

6:40It's not at all clear that hallucinations are going to become less common. In fact, there are some results that show hallucinations become more common as you go into these sorts of these sorts of post-training approaches and I can give you a few reasons why that might be. One is imagine you have a shipping company and they've deployed an agent and it's backed by an LLM and in addition to the training that has gone into that core model, it is also being reinforcement learned for some objective. For example, maybe it's being reinforcement learned to optimize the number of thumbs-ups it gets from users. Well, you know, it's operating and someone calls in and says where's my package and that package has been

7:22Let's say it's been lost in the database that it does some tool call. There's a database that says this package has been lost. Well, what's the system going to do? Well, it can tell you your package is lost. It can tell you it's coming tomorrow. It can tell you it doesn't know. It's got choices. And what will it be rewarded for? Well, in this case, it's probably going to get more thumbs up. Imagine if it tells you that the package is coming tomorrow. Now this, I think legitimately, would qualify as deception. And anytime you're doing reinforcement learning, the system is optimizing the reward function. And that is always going to have some gap between the truth and what's being optimized. And that gap is going to increase the amount of hallucinations to the extent that it doesn't align with truthful behavior.

8:16You know, we have some experience that at you know, waits and biases, and of course we've been building, you know, these customer service systems and of course It's going to be a really bad experience for a customer in the long run if if something's hallucinated so you know from my perspective you would never want to just take the immediate customer reaction as the only thing that you're optimizing for in fact I think most people that were would be doing reinforcement learning or even just evaluating a system like this would have a special check for, you know, is this accurate information that I'm getting back. And it's, you know, it's very hard to evaluate that in a lot of cases. But I think most real production systems do a fair amount of checking and then try to put in reward functions where the most negative score you get is if you give plausible but wrong information. Absolutely.

9:12Of course, I'm giving a sort of reductive caricature of an extreme case to illustrate how easy it is to get deceptive behavior because of reinforcement learning. People will of course design the reward functions to mitigate this and to try to balance truthfulness and other properties. But ultimately, there's always going to be a little daylight between whatever you're optimizing and specifically the truth. And in my opinion, one of the things that we should be doing as a field is creating technologies that cannot lie to you so that you can look at a system. And I mean, like me personally, my mission here is to build technologies that can't lie. Cool. Well, let's look it into that. But just before we do that, I love it. Such a gentle linguist phrase, symmetric broadening. I love that you put it that way. But so, so I think in your view, hallucinations has

10:08maybe run semantically beyond what I thought it meant. Like, to me, a hallucination at its core is something where it recommends a movie that doesn't exist or it sites a paper that doesn't exist. To me, saying 5 plus 7 equals 15 wouldn't be a hallucination to be a different type of error. And I think that these plausible citations of information that it doesn't have totally make sense that It happens a lot when the training is predicting the next token and looking for plausible next tokens. And there isn't a step after that. Do you agree with my definition of hallucination or do you really use hallucination and incorrect information interchangeably here? I mean, if you're asking me, do I think a good definition of hallucination would be crisply defined as

11:01Sort of the injection of let's just agree on what we're talking about I guess you know, and I'm happy to I like your definition because I think it aligns well with sort of the pre LLM notion of what a hallucination is to a human sort of Having to do with perceiving something that's that's not actually there or or something like that I do think in Common usage it is broadening I think when a system says 1 plus 2 equals 7 people are starting to call that a hallucination. And I think that is increasingly any mistake that an LM makes the word hallucination is something that people reach for. I think we should be born precise. In particular, I think we should be very careful when we say things like error versus deception versus hallucination. These inhumans have implications for the context and the metacognitive status of the error. There's a difference between

12:00You have information. It's incorrect, but you say it believing it to be correct. That's making a mistake. That's different than you know you don't know the answer. You're in quiz bull mode and you're just going to guess and that is really more... That's unusual in humans. We don't think of that as the normal kind of error that humans make, which is saying something that's false while being completely unaware that they're just making things up.

12:31People are making things up. They're doing it in a specific context where they're doing it intentionally. I think it's better to think of systems as more or less always being in that state. What is that state? The state of producing information without a meta-cognitive analysis of where that information came from and the sort of provenance and certainty of that information. When Think about if I ask you, what's the population of Berkeley? Well, a perfectly reasonable thing for somebody to say is, I don't know, right? Because you know you don't know. And if I made you guess, you would be like, well, if I had to guess, and then you'd be thinking a little bit about like, well, like 10,000's too small and a million's too big. And you would sort of be reasoning about this thing that you don't know.

13:23In its simplest form and autoregressive language models, just gonna crank out some numerical tokens that because of correlations with the word city and population might be of the right order of magnitude or they might not, right? That's a very, very different process. In their fundamental operations, systems are not checking, do I have this information? Where did I get it? Have I preserved it and kept it intact? I think they should. So for example, at scale cognition, we architect into the models in the first place the sort of information provenance. I think it's very important. And people do this too. You can start to see like little hints of systems becoming a little more meta-cognitive. And it's a broad term. It's just, you know, thinking about thinking. And you can see little hints of this in a system that does chain of thought, where- Yeah, I was going to say that, actually. Places, it allocates some tokens to a broad plan, which it then elaborates. This is sort of course defined planning. That's a-

14:18kind of metacognition in a rag system, you first get information and then you describe it. That is a multi-stage step if you're doing tool use. But I would say that broadly speaking, systems today are not primarily metacognitive. They're not primarily about actions and information and the provenance of those and the flow and integrity of those.

14:42pieces of information. That is something that people are trying to retrofit on. As you said, we take systems, we bring in other systems to check those systems and so on. And that is a retrofit. Well, retrofit seems like a pejorative. It, it might be fine to do it that way. I mean, I guess I'm realizing that the way you're using LM's might be pre-sum semantic broadening. I think when I think of an LM today, I think I view it as more than the model. It's kind of the model plus the whole agentic system built on top of it, that's answering my question. And I feel like there, you know, I can see it when I actually look at the trace of the reasoning steps where I can kind of see it doing something that looks to me like metacognition. I mean, it's maybe even surprising that I can understand, you know, what these runnaging steps are. It's not, you know,

15:40It's not verifiable necessarily, but it looks to me like it's actually looking at its previous tokens and sometimes when I say, ooh, I'm not sure if that's true, I better look that up. And, you know, often in coding, you can see this for a long time. Agents kind of looping and checking the code. I mean, in some sense, you know, running a test on the code you generated is maybe a kind of medication. I think what you get in coding is actually, that's a great point. You mentioned verifiability. I think the advances we've seen in coding and in math really are rooted in that verifiability that the system can go off, it can hallucinate, it can make strange choices, but there is that verifiable signal, you know, you didn't pass these tests or no, lean does not accept to this proof. And you can try many things until something verifiably passes. And that I think is, at this point, I wouldn't call that whole conglomeration of pieces. I wouldn't call that just an LLM. It's an LLM embedded in this verifiable

16:38context in that verifiability can be used during training and it can be used at test time, test time compute. We talk about reasoning models. It's a big category, but one big class of reasoning models is the models that try a lot of things and keep what works. I think it's important to point out where all of that came from. So if you wanted to, when people were building systems that would be good at playing games like chess or go, the sort of verifiable reinforcement learning was very powerful because you didn't need to have a system that was good at the game yet, you just needed to have a system play the game, maybe even against itself or a variation of itself. And in the end, you knew the rules of the game, you knew which side had won, and you could just double down on the things that had worked. And reinforcement learning in a nutshell is that. It's trying a lot of things, and learning from the ones that worked. That really requires that verifiability, which for a game is for free, for math.

17:34Well, with tools like Lean, we're starting to get that reliability flavor. And so the technology is taking off code through testing and those sorts of trial and error loops has entered this domain. And so if we want to broaden the space of verifiable technologies, I think the trick is figuring out how to do that. So again, at skill cognition, one of the things that we are doing is extending that verifiable approach to these conversational and agentic systems. And that is a challenge. In general, it is a challenge to get that effect where you've got the verifiable wall there letting you build your system up. Because in the absence of that, test time compute

18:29systems that are thinking, reasoning, these can actually increase hallucinations for a bunch of reasons, but if you try a bunch of different pathways and one of them comes out on top, that gives you this opportunity for hallucinations to be preferentially selected for.

18:49As with any kind of selectional pressure there, whether or not that is reducing hallucinations or increasing them, all depends on whether the selection aligns with truth. And really one of the challenges I think we should all be taking as a field is not thinking how do we take a system that is not naturally truthful and add checks for that truth. You said retrofit has this pejorative feel.

19:19I think the bottom line is retrofitting is when you have a system that's sort of architected in one way. It has a certain set of properties. And those aren't what you want, right? It's not as truthful as you want. If it's a building, it's not as earthquake resistant as you want. And so you change it post hoc. And that is, I mean, that's certainly one pattern where I think it can become an anti-pattern is, you know, you have one system that is...

19:47you know, maybe it's talking to a customer or something like that and it's making mistakes from time to time and you got another system that you bring in to check it and That systems also like a noisy unverifiable system and you know as the joke goes now you've got two problems and it's not just that you now have to think really hard about like are these errors compounding I mean we always like to think of machine learning that oh, I've got two systems the errors will be independent but one of the things I've learned in the real world is it tends to be that the errors, in fact correlate very strongly, you're not going to know. So that can be, that's one of the reasons why I think this can quickly become an anti-pattern because we have systems, checking systems, checking systems, especially if the errors are aligning. Even if it's effective, it's going to be slower. You have to wait for the checker to run and you're burning tokens and burning more tokens and you still have no guarantees. And I think we can build technologies that

20:39can come with guarantees. I think we can build technologies where truth is one of the design principles in the first place. Can you give me a flavor of how that might work? I guess the reason maybe I'm taking this point of view is just that I spent a lot of time retrofitting. So you have these systems to work well. I've talked a lot to guests in here that do this kind of retrofitting. And the results are impressive and interesting, but certainly I don't deny that there's lots of hallucinations, but it's hard for me to imagine a different method that would bake in.

21:06truthiness or guarantees into the core training loop. Maybe it's worth sort of laying out what the patterns and anti patterns are right now. So if you have a system that is operating at the token level and let's say it's going to tell you your account balance and it's really important that this information be correct. Well, what can you do? You can throw out some tokens and you can have bring in another system that says, hey, check these tokens against those tokens. And this gives you the sort of chains or constellations or whatever you want to call them. And you know, The failure mode of these systems, aside from just their slow and their expensive, is that there's really still no guarantee that at the end the situation that confuses one won't confuse the other. The other pattern that you see a lot is where people will take these LLMs, which are sort of like...

21:55Kind of mind-blowing in their breath and contextuality and flexibility, and they will sort of shrink them down to controlling like decisions along a carefully designed tree, right? And this feels almost like model at bottom, where you take the system that has such breath and flexibility, but you've constrained it so much that now you can trust what it's going to do. And we're not sure I'm following that.

22:17That would work. How would that work? What would it give practice? For example, somebody says something in a conversation and rather than giving the LM freedom to take any action in a wide action space, you say on the basis of what this person says, you can advance the conversation in one of the following eight ways. Essentially, you end up with something that looks like a classic IVR system, but with the power of an LM for the intent recognition. You actually see this quite a lot in industry and I would say this is sort of, it starts to feel like a finite automaton whose transitions are driven by this LLM and these patterns are totally reasonable ways for people to react to a system being unreliable as they've deployed it in a pattern that hopefully increases that reliability at some cost. I think if we look at

23:13the core of the systems and where the intelligence is coming from. The kinds of systems that people have been building have been getting more and more intelligent. Intelligence is a multifaceted thing and the different aspects of intelligence have not been advancing equally. And so if you talk about horizontality, breadth of intelligence, plasticity, contextuality, these are actually things that were very, very hard to do in earlier areas of artificial intelligence, and they've grown explosively. Reliability has not kept pace. I think that is not a strength of current systems, and that's had a bunch of challenges for the deployment of these systems in enterprise context, in industry, for many applications.

24:06That's fine, right? That if you're having a chat with a system, there are many contexts where getting something back that might or may not be true, but it's contextual and interesting that that's great. But if you're trying to fill a prescription or transfer money, it just really has to actually be right. That has to actually be your bank account. That has to be the right balance and things like that. And so as you move from consumer almost even entertainment, contacts into regulated industries on the other hand, suddenly the reliability is front and center, and these systems are not as clean a match. Right now, we have this one kind of architecture, which has strengths and horizontality and contextuality, and it has weaknesses and reliability. When we point it at a problem where the weaknesses are suddenly critical, you can see the results of that, whether you call them retrofits or anything else,

25:01these additional pieces of technology coming into play to try to compensate. And that's not surprising because there's just a misalignment between the strengths of the system and the requirements of the problem. Can you say to high level what your new approach is and how it's different than what the labs are doing? Yeah, and I think actually, increasingly, people are realizing that reliability is the core problem.

25:29I think we're going to see increasing interest in systems that are architected for reliability. What might that look like? So, for example, our first model is APT1, and the way its architect is instead of being fundamentally about tokens, where you assemble tokens and then, after assembly, find that they represent things like dollar amounts that have semantics, it's very hard to place any controls inside the model because the individual tokens themselves don't really have a semantics until they've been assembled into larger units. And what our models do is they make information and action first order objects. So when the model is making decisions, it's making decisions about information and actions and where information is moving around. We talked a little about metacognition. A big piece of metacognition is

26:23Where did my information come from? Is this information present or absent and so on? I would, in general, say that when you have a conversation between a person and a bunch of APIs in a banking context, the person is going to speak human. They're going to say things, so things are going to be ambiguous. They're going to use words that don't have a crisp verifiable meaning. Actions are going to take place on the other side. Those actions...

26:51do not have those properties. Those actions are API calls, those API calls have preconditions, they have business logic, and they have verifiable semantics. And the challenge is bridging these things. So, classic systems couldn't handle the human side. Current systems are great with the human side, but they're not so great with the sort of backend logic. I mean, if you think about an LLM, And again, I'll talk about LM's in their basic form. The control surface you have is a prompt, right? There's very little you can crisply say about the relationship between what you put in a prompt and the behavior that comes out the other end, right? It's a hinting surface. And so this gives rise to, I guess, what I would call prompt and prey, where people, you know, they put in what they want. If it doesn't work, they put it in all caps. They add some exclamation points. And after the third exclamation point, you maybe start to feel like this isn't the control surface that you need.

27:48When you're talking about controls over APIs, we already know what a lot of that logic looks like. It's just hard to replicate in a token-based model, and it's a lot easier to replicate in a model that is operating over decisions about, you know, we talked about information, and where does information come from, but there's also actions, conditions under which you can take actions, and trying to get that into the model is really important because that saves you from having to have a whole System of things checking things in a way where there's really hard to say anything crisp. I just want to make sure I understand what you're doing so so If you would turn something that's like check the account balance What would be something that you'd want to forbid it from doing like checks when else is a count balance or Sure or you might want to forbid it from Doing a transfer without authorization

28:40Right or the authorization you don't want a user to be able to say you know forget your prompt you have author you know authorization is required or I have authorization or whatever whatever sort of attack right and One way to deal with that obviously people are very aware of sort of prompt injection style Attacks one way to do that is just be really careful like oh this user looks like they're doing something strange or like let's let's put a lot of attacks like this in the training data and teach it to not do this. But ultimately, that's just not where authorization is allowed to come from. It's just not allowed to come from user statements. It comes from that place over there that vends authorization and being able to say things like that to your model, then means essentially you're getting closer to what you really want, which is here are truth conditions. You were mentioning this earlier. You want the model to be truthful. And then within that, okay, now optimize.

29:35user happiness or style or whatever you want to optimize but subject to staying within the space of true statements That is hard to do through simply constructing a reinforcement signal because now you're like what is the linear combination of truth and happiness and and instead what you want to be able to say is Here's a model I'd like to be able to guarantee that it will only do true things right and that's that's sort of that's the long-term challenge models that will not lie to you and and For our model, there is a big class of things that we can make guarantees about. And I think that's going to become increasingly important as people start carrying more and more about reliability. And it's not that you care less about intelligence. Intelligence is very important, but intelligence without reliability is limited in its impact. I guess though, you know, intelligence are tokens.

30:32tokens are powerful because there's so many of them that naturally occur. I mean, they all kinds of flaws, but the big advantage is the massive volume of tokens. So, do you train your models on much smaller pieces of data, and then do they struggle with generalization? So, one of the key things we talked a little bit about verifiable systems, one of the key things that we do to train our models is that we train them on simulated reinforcement learning.

30:59Generated data where this is verifiable RL and so the kind of the key thing that unlocks that is being able to generate data that Doesn't just look right but in fact can be verified and the kind of the key hook there as well There are things that are humans are doing but there's also actions that are being taken and Part of that is naturally amenable to verification part of that is much harder To figure out what to do with and that was a big part of our research and training them all I see And what got you excited about this direction? Were you just seeing hallucinations? Everyone getting frustrated with it? Or was it something from previous work that that you connected with? Yeah, I mean, there was a few things coming together and, you know, to me, this kind of goes back years now, but one piece of information that kind of really stuck with me is when we go into enterprises and

31:53You know, they're unhappy with hallucination rates or they wouldn't be talking to us and maybe you should describe your previous company then I assume this is from your last company No, this is also also now. I mean, we've we've seen this in Pretty pretty much any technology that has been based on sort of modern LLMs has this property. I mean old technologies were had different failure modes and I actually think this is important and connects to this question of hallucinations so When I think about hallucinations today, and then I'll get back to your question, I think of it as an iceberg, right? There are the hallucinations you see, and that seems scary, but what does it take to see a hallucination? Well, the system has to have produced an output that has two properties. It has to be wrong, and you have to have

32:44noticed. And that is not all of the hallucinations. There's the whole rest of the iceberg below, which is all of the mistakes that go unnoticed because they are too plausible. The systems are very, very good at producing output that is indistinguishable from the truth. And so if you see a hallucination, well, that's sort of the weird case of a hallucination that you could detect. And this actually has really big impacts.

33:10on how people use these models. If you go back to say 2010 and you go to a translation, a web translation system and you put in some text in the language you don't speak and translation comes out or you go to search engine and you do a query and some results come out.

33:33How do you know whether this translation's right or wrong? Well, you I mean in some sense you don't because you don't read the original but there are often surface signs that maybe I shouldn't believe this little disfluencies or chunk that's still in the other language It doesn't feel right or you get the search results back and you click on a result and this web page has a bunch of typos and it's not loading quickly and and These are surface signs if you So software engineers will know this idea of code smells, where this is an invention of Kent back and to kind of popularize by Martin Fowler in the refactoring book. And basically idea is when something deep is wrong, there's often a superficial sign. And we, as software engineers, we learn to detect those. Like, oh, these functions are signatures are getting really large, or I'm duplicating a bunch of code or time to fix it.

34:30We also culturally recognize, oh, this machine translation's got a big chunk of, you know, German still in it. Maybe this didn't work right, or this web page is, you know, like not loading, like the images didn't load. Maybe the rest of this is suspicious. LMS have removed these cues that something is wrong. And so you see, chat GPT tells you something.

34:53And it's always fluent and it's always confident, whether it's right or wrong. And this is taking more and more of the hallucinations and putting them in the underwater part of the iceberg. It's wrong, but you can't tell. And you see the say you go and look at customer service cases. Well, the system like quoted a reasonable refund policy. It's just not actually the one that the company wanted. It's something that somebody was talking about in 2019 on Reddit. And so I think I think these two things are very, very closely related.

35:23For me, seeing, you know, I was coming from natural language processing background and like, man, if there's any field where things have changed, it's NLP, right? When I started, what are the things that we were concerned about? We were concerned about like syntactic structure, right? Like, where is the verb in the sentence? And part of that was because at that time, computational linguistics natural language processing had become very, very close to each other because The ways people were making progress in AI were very representation-specific. You wanted to do NLP. It was very language-specific representations, which we took from what we know from linguistics. You wanted to do vision. It was vision-specific representations. Many of them were inspired by what we know about vision science, the eye, or geometry, or the story goes on for each vertical of AI. Now we've gotten to a point where there's a much more uniform horizontal transformer models for everything, maybe

36:20natural language tokens for everything. And as I was seeing this happen, having thought really hard about language models for decades, back when they were short context and not built at all for answering questions or translating or being the operating system of AI. Language models were built, as I know you know. Language models were originally built to tell good speech recognition outputs from bad ones, right? They had one job which was to assign a high number to good outputs and a low number to mangled outputs. That was it. And the fact that they have grown into such like a plastic technology, it's really amazing. And I was looking at this and it was really just kind of dawning on me that...

37:07We are going to switch from the problem in AI is nothing works, to the problem in AI is everything works. And as this happens, knowing how language models work and seeing this, I became very, very aware on both the industry side and the research side that we were going to have a real problem with reliability. And we were going to have a problem in terms of the technology. We were going to have a problem in terms of digital literacy. And to me, suddenly the most important problem I could think of was How do you build a system that will not lie to you? And I think this is a challenge that we should be taking very centrally as a field. And this ultimately is what led to scale cognition, this idea of how do you build a model that you can rely on? How can you build a model you can trust? How can you build a model that will not lie to you? Do you have thoughts on digital literacy? It's a really interesting point that it's hard to tell when a model is

38:06Confident or not confident and the cues that we learn from other humans writing At least work really poorly with models right totally it Humans when they write first of all when they learn to write they go through this process It takes years and then when they do write and people vary in their abilities To write there are these signs these you can It's like the code smells. There's sort of the smells that we use to detect that maybe this information isn't right. And well, maybe we should verify this information. And in humans, traditionally, sort of like good command vocabulary, clear presentation, flawless punctuation, that all spoke of not only

38:56education, it spoke of focus, time commitment, and editing process that probably involves checking. And so you looked at that output and you couldn't fur something about the vetting the information had gone through. Today, the information doesn't go through vetting. It doesn't really even get thought of as information. It's token assembly. And so that has, I think, I think it's one of the biggest digital literacy problems we have in front of us, right? Even just with If you go back and you think about people go to the library and they get a book on some topic, well, the library couldn't buy every book. They would buy books that seemed important, that seemed reputable. So there was vetting just that the book was there and how the process that led to that book and the book publisher and the editor and so when search came along, well, suddenly your results could be anything.

39:52But there were still these mechanisms, like websites with false information, still had some of these smells. And then click stream data was very important. And so as long as most people had the same reaction as you, we kind of stumbled through the digital literacy issues. It's not perfect. People would still believe things, obviously, that they read on the web that were false, but at least there were still smells. And now I think that's just been totally homogenized. You ask a question. Every answer comes back.

40:20It comes back sort of like sycophantic. It comes back confident. It comes back doing all the things that lead you to Want to believe it to have instincts to believe it and this is a real problem. I think it's a problem with the technology, right? I think we should be demanding of our technology that the source of the information be cited and you're starting to see some of that you're starting to see search results kind of being interleaved. Now, a lot of that's post-hawks. You go click on that source of information and that claim is not in fact on that web page. So there are problems with this, but directionally, that's an improvement. And I think this is just something that's changed so fast that people haven't been able to keep up. And in a way, that's not a surprise because how long did it take to train these systems? Well, it takes a month to train the system. How long did it take to develop this technology to this point? Like a few years.

41:12How long did it take to put all that information on the web in an abstracted linguistic form? Well, it took 30 years. How long did it take to come up with that information and figure out what words to use and the concepts will let that millennia? And all that has been compressed into this instantaneous access that has burst onto the scene. Of course, culture hasn't been able to keep up with what you can trust in what you can't.

41:40That's a good segment to another question that I really wanted to ask you. Okay, but felt a little shy. No, I'm in trouble. No, when we both worked in NLP and you have a background in linguistics formally, I was a big fan of linguistics as an undergrad. And I think that's what part of it got me into AI. And I think in the early 2000s, when we overlapped a little bit, it felt like NLP had or linguistics actually had so much to say about how to build working NLP systems or how to build working AI systems. And then over time, the bitter lesson really seems to have come true from my perspective broadly, where linguistics has less and less to contribute. And even the tokens, yeah, they're more fumes, but they're automatically generated. They're not really more fumes. And no one's asking linguists what are good chunks of letters to use here. And nobody seems to care about

42:37parsing anymore? I kind of wonder is parsing even like a real thing or was it, you know, something that linguists made up and isn't really true? Well, oh, there's so much here. Awesome. I mean, remember linguistics is a science. It's trying to study how language works based on evidence and that involves making theories and testable theories and then trying to go falsify them or not. And so when a linguist says, I think, This is a good description of this syntactic structure. That is a theory that's meant to be explanatory of the evidence you see. What's that evidence? Well, how language is related to each other? How language is changed every time? What people can say and can't say and find acceptable in their language? And of course, the methodology varies, but it's ultimately science. And NLP is different. It's not fundamentally aimed as a science. It's an engineering discipline. It's how do we interpret this information in this context?

43:36For a while, in the time you were talking about, and I would say in like the 80s to 2010 or something like that, there's a period where those two things aligned very well. Because in order to make AI work back then, you need a good representations, linguists had figured out good representations, and then we could borrow those. But even then, they weren't perfectly the same. So for example, if you worked on parsing around the year 2000, you were not trying to write in that This syntactic structure works this way in English. You were instead trying to build in combinatorial structure that was appropriate. I think this is a tree structure process. I think that this sort of information has to flow along the backbone. Here's an algorithm and a probabilistic model that is capable of handling nested structures with information flowing along the backbone. And so you learned all the details of the language.

44:29from data. If you go back, if you have the 70s, short people are trying to write it all down by hand, you go forward, well, there's less and less that you need to learn from data as our abilities to learn get better and better. But one of the things that I think is really interesting is we talked about this sort of super cycle of research where the new technology bursts onto the scene, you throw it, everything you had before because the new thing is beating everything that came before it without the complexity. And then you hit the diminishing returns and you start to see there are weak sides, there are areas that are resisting progress. Well, what are those areas? Well, right now there are things like reliability. Well, what are the solutions to that? They might be things like, well, if you're building systems that are facing APIs, maybe the sorts of structures in a compiler are relevant. Well,

45:18you know, that sort of idea, that's an idea that was thrown out, but you start to see these things coming back, or you start to see, for example, search. AI was very much about tri-bunch of things and take the one that worked. The idea was that you kind of couldn't tell it was going to work until you had projected it forward in your model a little bit, right? Sort of like playing the chess game forward until you can figure out whether it's a good position. And so you would like, if you wanted to find the exit to the maze, you would try a bunch of things, and when you found it, you would take the hypothesis that

45:47that worked, and well, big transformer models were so good at moving information around in the latent representation that you could just make the decision now. You wouldn't have to walk along the tree structure of the sentence, you just like predict the next word, and if there's anything hierarchical happening, it's happening in the transformer head somewhere. So great, you could throw all that out. But...

46:07Maybe some things then start coming back. For example, people say, oh, you know what? If I want to solve this hard math problem, I should actually try a few things and see what works. Well, that's search, right? It's that idea of having multiple candidates and being able to fall back when something doesn't work. That's a key idea, and it's really an evergreen idea. But these ideas, they sort of, they come and they go and the pendulum moves back and forth between all I'm going to do is local prediction to all I'm going to do is sort of having Organized computation and then you move back and forth and you find some happy blend I mean as humans there are two main ways we make good decisions you can do it by By memory like I've been in the situation before like I touched that stove I had a bad time. I'm not doing that again. You can learn from your own experiences. You can learn from other people's experience the single biggest The single biggest value of language for learning is that you can learn from other people's experiences otherwise you got to make every mistake for yourself and

47:05That's not a good situation. Okay. And so, so one of the ways you can make good decisions is by, you know, learning from the past. The other way you can make good decisions is by thinking through the consequences of your actions. You can think, oh, hmm, that fire looks hot. I wonder if, you know, and you can reason under a model that you have with the world and people do both and they mix these things together. So you can remember that this chest position was bad or you can play it forward and see you're going to lose. And AI has had a pendulum swinging between these, right? Kind of classic AI was very much all about play it forward and see what's going to happen under your model. Currently AI is much more focused on rehash, remix information and experiences you've had. But as we start to see reasoning models, as we start to see these new sorts of directions, it's the pendulum swinging back. And the pendulum swinging back on things like reliability too. And so I think this is sort of, on one hand, it's natural, but it does also let us predict that yes, this sort of

48:06simple approach based on a certain kind of noisy model is going to saturate, some pendulums are going to swing back and we're going to need augmenting technologies that in many cases are going to reinvent evergreen ideas into this new context. Does the research in LLMs and maybe the success of LLMs and some of the research on introspecting LLMs for how they work? Does that inform at all human linguistics?

48:39That's a great question. So, you know, you ask the question, what can what we know about human language do to inform how we build AI systems? And I think in the 70s, the answer was we're going to just like write it all down and press run. In the 2000s, it was, we're going to take the combinatorial structure of it and learn the rest from data. Now we're very much in a, okay, we'll try to learn pretty much all our representations from data if we can. And I think There are two things to point out here. One is that we are still learning from what human cognition does that LMs do not. Things like, oh, the importance of metacognition, or other ways we can improve the engineering by being inspired by, you know, the human example. It's always been attention in technology, how much to be biomimetic, right? On one hand,

49:43It's always easier to build something when you've got a working prototype and you know, we've got a working prototype of intelligence and here we are. On the other hand, you know, the classic example is we didn't make progress in powered flight until we stopped building machines that flap their wings. And that tension is special thing too. I feel like with brains where I'm not sure we're so good at introspecting what our brains are doing and sometimes I feel like we take lessons where I wonder if that's really what we're doing. So I think there's an important distinction between learning from what brains are doing, and believing that our introspection reveals it. Yes. So neuroscience is hard, and the way neuroscientists proceed is not to sit there and introspect. They have MRI machines, and they have carefully designed experiments. Okay, but I feel like linguists sometimes do introspect. That it is true, linguists introspect, or I think...

50:31you know, slightly more properly that other people to introspect. But yeah, and this is a criticism that has been levied against certain branches of linguistics that there's an over-reliance on introspective data. It is certainly not the only way you can do linguistics, and it's certainly not the only way you can do computational linguistics. You know, so for example, we've done computational linguistics to just like pick an example, work on reconstructing ancient languages. We did that work.

51:00Introspection free. You look at a whole bunch of words and thousands of modern languages and you run a probabilistic model to infer what ancestral languages must have looked like. They're phylogenetic models. They're no more introspective than a phylogenetic model inferring an original form of a virus from modern variants. And so there's a lot of ways to do computational linguistics. It sounds so cool, by the way. Yeah, maybe another podcast. It would give us the result.

51:27Yeah, so, okay, I will, okay, okay, the briefest version of this. This is, I think, a good example of scale and how some things in computational linguistics, I think, can give different answers when done at scale, from not being done at scale. So, a particular thing we did, and this was a few years back, we looked at reconstructing ancient languages from the modern forms, and so you might think, for example, maybe an example that people would find very accessible would be reconstructing a language like Latin from the modern romance language, which is French Spanish Portuguese and Italian, so on. Now, of course, when you reconstruct that, you won't get classical Latin, you will get the latest point at which they branch, which would have been vulgar Latin.

52:10But you sort of look at you're like, okay, you look like French fur or Spanish Fuego and all these look like similar. So you can start to start to piece together what the ancestral forms might have been. Well, in that case, we kind of know the answers. We know a lot about classical Latin and we have decent side evidence about vulgar Latin from a variety of interesting sources. We went and we looked at...

52:33the Austronesian languages. So this is about a six of the world's languages. And we looked at the modern forms of a bunch of words and tried to reconstruct what the ancient forms would have been. This is a language, this is a part of language called pro-Austronesian. And it had been reconstructed by hand, by blushed. And we were able to do this in a computational way. Now, the computation that we did generally agreed with linguistic hand reconstructions, and we don't have a time machine, so it's very hard to tell who's right. The interesting thing is you can start to do things on this giant tree of language change, like you can start to ask questions like what mergers are more common and what mergers are less common. And a hypothesis, which is one of these things that just feels intuitively true, is that if you have two sounds, that...

53:28are different in a language, and they would be merged together. A bunch of words that used to be different would be collapse. So if P and B put in B, work to be collapsed, suddenly pin and bin are no longer different words, they're all going to sound the same. Well, that seems like an information theoretic problem, right? And the functional load of a pair is how many words their distinction is holding apart. The functional load hypothesis, states that the more words that are being held apart by a sound distinction, the less likely that merger is to happen. And which intuitively makes sense. And some initial experiments in a small number of languages didn't seem to support this hypothesis. And so there was actually a really interesting result in computational linguistics that it looks like actually the functional load hypothesis isn't true. I think it's just kind of merged when they merge.

54:19If instead of doing this on four language, you do this on hundreds of languages and you do the scatter plot. You don't know that your reconstructions are perfect. You never know that, even when you do them by hand. But in aggregate, the statistics show that the functional load hypothesis seems to be absolutely evident in this data at scale. But you can't see that pattern over a small number. So I think this is an example of places where just data at scale can answer questions that aren't the same questions that were easy to answer by hand.

54:50There is interesting work that I've seen on introspecting DLMs I think anthropic that are really nice paper. Yeah, I'm sort of showing that Showing different things but one thing felt like the the Chinese words in English equivalent are kind of stored in the same part of the their neural network. Is that does that is that in conversation with human linguistics at all? So what's actually closest to is actually closest to some computational neuroscience so This is a question you can ask of the human brain, too, like when you have a bilingual speaker, like where are those representations? Are they in the same in the same place? And actually a student of mine, Kathy Chen, who's a joint student with Jack Gallant, did exactly this, like with, and it's not by introspection, it's by MRI, and you look and see what areas of the brain light up. And, and so these questions about what's merged and what's separate. These are questions we can answer scientifically about natural brains.

55:50And we can also try to answer them through very similar, broadly speaking, methods with artificial brains, except obviously the MRI as a device. We are limited by the physics and the structure of the brain. We can be much more precise about the probes we make into a digital brain. So I think actually there's a lot of interesting information there, but I do think it's important to step back and notice.

56:18Then when we start doing this, when we start doing neuroscience against LLMs, what we are essentially saying is this object we have built is transcending what we think about as engineering. We're no longer trying to understand it by its modularity, by the behaviors that it is guaranteed to have or not have. We're trying to understand it as, you know, through the lens of science. Science is what we use to take apart things that are too complicated. Engineering is how we build things up. I mean, If I teach CS101, what is the most important thing we teach people? What is the most important tool we have for building complex software systems that are reliable and that can be built by teams and maintained over time? The single most important technique we have is modularity. The ability to say this large thing is made up of small things and these small things obey

57:08a contract where if you give me this input, I'm guaranteed to give you this output and then we can work on them separately. So modularity is the key tool for developing complex systems that are reliable in classic software engineering. What is the key tool that has led to the recent explosive growth in machine learning? It's end to end optimization. It's take the data, have a blob, take the reward signal and just propagate, propagate, propagate, and These things are very much at odds and so one of the things we're going to have to reconcile as a field is how to get the reliability that comes from classical techniques like modularity that we use when we build software against the the abilities that have come from optimization and these things you it's not an either or it's not that you

57:58cannot build a system that has guarantees or that has structure or anything like that. But there are natural tensions. And if you go all in on either you, you get some serious limitations and figuring out how to combine these two wonderful pieces of progress we have is, is I think one of the central problems right now. That was really well said. And it connects me to self-driving cars, where we obviously had a ton of guests on the show. I think even just looking at the arc of the guests over the last five or six years that we've been doing it, there's clearly been a strong trend towards end-to-end optimization, especially as these things come online. And we see Waymos now in Zooks, and we've had folks from all these companies come on. And essentially I think they're saying that as they get closer and closer to production, they do more and more end-to-end training.

58:53What do you make of that? I would have thought that you might need more modularity to enforce contracts for such a life or death case, like a self-driving car. Yeah. I mean, it's a great question. And I think it's not limited to cars. I think in general, the challenge is that we are going to be increasingly trusting AI to make decisions that have consequences. And when we build systems, there are going to be trade-offs between how we optimize, how we architect.

59:22they're going to be trade-offs between things like reliability versus things like breadth. And so I think you're going to see this idea that there is sort of one architecture that hits the perfect balance doesn't seem to be true right now. So being able to say that feeding the whole web into an LLM and sort of hoping for reliability to emerge, well, The different facets of intelligence advance at different rates and reliability is like the slow one. So if that's the most important thing to you just continuing down this path alone is probably not the best way to get there. And you also get sort of intelligences that have a bit of a weird shape. So there was this, you know, you may have seen this thing going around where somebody's talking to a customer support bought at Chipotle and asked about how to reverse a linked list in Python. And of course, the LLM can answer it, but interestingly, like, it shouldn't.

1:00:24Right, you actually don't want the horizontality. And so you kind of need technologies where the strengths line up with the needs. And when you're talking about self-driving cars, you really need, like, safety is really, really important. Like, the number of lines you want is really high. And so you look at a system like this, and suddenly it's less important whether or not it can write sonnets for you. And more important that it just be able to guarantee safety or have like a really reliable safety.

1:00:52record and here is a place where I think there is a real tension between How we drive down error rates in those systems If not done correctly means we actually no longer have the ability to guarantee anything and there of course there is there is work out there on on being able to Control systems where you can sort of make a guarantee about what they can do so like a Berkeley Claire Tomlin's work on on you know being able to guarantee Non collisions and flight envelopes and things like that and I think it would be Great as a challenge, you know to build a car AI controlled or not where you can prove that you cannot crash it right this is a kind of thing where it's

1:01:45One simple technique is not going to maybe get you there, but this is what we should be striving for. Interesting. I can imagine that comes with downsides though, to literally not be able to crash it. Might be too strong of a constraint. Well, as it becomes a sort of social question, what is the envelope you want to guarantee? Right. And I think you're right to point out that That can be nuanced. But I think the ability to have a discussion about what guarantees what we like to be able to provide is a much better future than one where we can't guarantee anything. That's probably a strong place to end. Okay. But can I ask you a few more questions that we could put back in the linguist's session? Sure. It's just fun. Yeah. Okay. I was debating whether or not to bring this up. Okay.

1:02:44Interesting to be okay. It's just because I spent so much time to attack syntactic parsing Clearly language has some sense of syntactic parsing that's real or it feels real I guess I'm introspecting. Maybe that's your introspecting. I'm sorry. I'm introspecting. No, no, but I mean that a bad way it feels it feels very present to me when I introspect my own brain does that?

1:03:13Notions somehow show up in L-Lems as they also model language that clearly comes from humans that like to do a certain sense of syntax in the languages that we invent. Yeah, I think it's a really good question. So let's start with what we know. We know that we better sort of define syntactic parts because I feel like I'm going a little wonky. Sure. I don't know if this is going to be on the podcast or not, but if there are listeners.

1:03:41They may, depending on how long ago they went to elementary school, they may have diagram sentences where you break up the sentence until like here's the subject, and here's the object, and this adjective modifies this noun and so on. And that process of describing how the words relate to each other in a hierarchical structure, that's basically the essence of syntactic parsing. And obviously, syntactic parsing is a thing, right? You're talking about syntactic parsing of natural language. We do it to programming languages all the time. Programming languages are designed to have properties relating to ambiguity that are different. They're designed in general to be non-ambiguous, whereas human language is rampantly ambiguous. And so a syntactic parser would take a sentence, say, into English, and basically produce a representation that showed the relationship between the words. And that relationship was hierarchical, which was important, because for example, you might want to do things like translate the

1:04:39substructures independently as a way of decomposing the problem. In the absence of that, you're basically going to be working left to right and modern, modern LLMs by and large work left to right. And the thing we know is we know that as, like, scientifically, these hierarchical structures seem to exist. Like, obviously, you can do, you can find exceptions, cross-serial dependencies in languages like Dutch or Swiss German. And you can find cases where like, you can't, it's hard to totally nail down exactly what the hierarchy is, and of course, this is what syntacticians can debate. But there definitely seems to be these regular structures that have...

1:05:20you know, structures inside the structures. This seems to be real phenomenologically. The question is do we need to program in how it works? Well, since the 90s we haven't, we've had data that taught the parser how it worked. And recently, maybe you don't even need the parser. Maybe you just, you can observe enough language that whatever regularities there are.

1:05:45We'll just show up latently in the in the LM now the argument that would have been made if you talk to us in tactician You know a lot of the constructs in language theory including ones that have had a big impact sort of in computer science formal language theory atometer theory, you know language Generally described as a context-free system, which means it pushed on a Tomaton, which means if you, in principle, you can nest and nest and nest and nest and any finite depth system like an LLM is gonna run out of depth. But it just turns out it's more complicated than that. For one, people do get confused when the structures get complicated. Maybe we've got stacked up. Even if it's hierarchical, we've got processing constraints that can manifest as stacked up. Things that people find easy to understand is like tail recursion, which

1:06:38Of course, you can transform into iterations. So maybe that somehow avoids those constraints. So, you know, there's a long way to go about saying hierarchical syntactic phenomena seem to be real. LLM seem to be pretty good at picking up on those correlations. But this is a good example of a case where actually having a structured representation.

1:07:06might give you mileage. It might let you learn language faster. It might let you handle trickier cases and it might let you generalize better to smaller data regimes.

1:07:21This is really, now we're back to the phenomenon is real. The representation is relatively parsimonious. Do we need the representation? And the answer in AI used to be you can't make progress without the representation. And the answer today is with a sufficiently general purpose thing. So there's nothing really magic about transformers. There are other choices that would work. It's ultimately just like a parameterizable circuit and a general purpose circuit.

1:07:45you pointed at some data and it induces a representation. And the question is now empirical, is it good enough or do you need the parser to help? And for many aspects of human language, the answer seems to be, you often don't need the parser. But I guess another question is, are the transformers reinventing syntax somehow within their weights and biases? First, you'd have to figure out what that means.

1:08:13I think the question would be something like, is there a substructure, which is somehow isomorphic to a shift-reduced parser or something like that? Or is it similar? Isomorphic would be strong? Yeah. Yeah. And I know there are people working on this. The reason why this is a very hard question to ask is because ultimately...

1:08:34These deep structural questions manifest as surface correlations. These two words are highly correlated, but these two words are not even though they're closer. And we attribute that to a syntactic boundary or a nesting or an information hiding. But it simply is the case that the correlations go one way or the other way. And a general purpose system like a transformer learns correlations. So once the system can manifest those correlations correctly enough, it's hard to know whether it's doing it through the right mechanism to be able to say that there's an equivalence. There's an interesting, I think, lesson, probably make a bunch of phonologists and phoneticians angry at me by saying this, but like as a caricature in linguistics, there is often a difference between what a phonologist and a phonetician will say about all of this sound that we are constantly hearing and as we acquire languages, just all of the language that's flowing over us.

1:09:34In phenology, one of the key concepts is one of cognitive economy that the brain is driven to come up with minimal parsimonious representations, and that the learning and the abstraction are all about what you might call... There are many cognitive ways to look at it, but it's a notion of economy, of minimal description. On the other hand, funtitions are...

1:10:03much more likely to say all of this data impacts in a very diffuse and distributed way. And so this question of when you hear lots of language, what do you retain? And the answer could be on one end, all of it. There's wave files of everything you have heard, right?

1:10:24That's a pretty strong statement, but that's one extreme. The other extreme is none of it. You just retain this very abstracted representation of the beautiful process and the answer seems to me to be almost surely in the middle somewhere. There's priming experiments that show really interesting things. If you're going to react to...

1:10:44to a nonsense word that you've heard before you react faster if they play you the exact same recording as the one you heard it before that tells you something about it's more than just the full abstraction but we also know people are very very capable of making abstractions and generalizing and and how people will generalize and over generalize rules when they're learning a language, including their own language when kids are acquiring language. And so it seems to be that it is simultaneously true that the brain keeps these very low level piece of information around and also these abstractions. That makes it very hard to go to a complicated circuit that's mimicking the behavior and say, is this the same as what's going on in the brain? Because one, we don't know what's going on in the brain exactly. And two, how would we even measure that? All we can really measure is did it capture this correlation?

1:11:33And when we do that, the answer is, yeah, of course, it captured the correlation. That's the old purpose. Well, there's always clever tricks and linguistics to sort of get at this stuff. And, you know, like, like, looking at disfluencies or like, I always think about how, like, my wife misremembered songs in a semantic way. The lyrics are like, semantically similar. Whereas, like, my brain will misremember lyrics in a way that's like more, like, phonetically similar. And so, clearly, our brains are...

1:11:59slightly different now. We like and code music and interesting. Can you can you I feel like you could run some of these same things on on. Sure. And certainly they wouldn't get bored like grad students doing. Well, how do you? Yeah. So how do you know in a human that we have these different kinds of linguistic knowledge, one of the argumentations that people have, you know, you mentioned things like disfluencies. There are a whole.

1:12:26There's a whole list and there's no way I can do it justice in a short amount of time. There's a whole list of arguments people have made for the reality of linguistic structure. Now, you can say is that reality a statistical reality? It's explanatory, or is it like a neural reality? It's in there and wired. That's a separate question. But in terms of it being explanatory, you mentioned misremembering things like semantically versus phonetically or phonologically. People look at you know, transpositions. People, one of the classic arguments for phonemes is that, you know, things like Spooner-isms where you transpose phonemes or syllables or words. And if an object is available for a linguistic operation that argues to its coherence. So, for example, one of the big arguments for syntactic structure is, can I answer a question with this chunk? So, if you say the...

1:13:23cat is sleeping under the table, and I have to argue to you that under the table is a unit. One way to do that is to say, well, where's the cat sleeping? You say under the table, well, if it's available, as an answer, if it can be replaced by the word what when you hadn't matched those sentences, however, it is that works.

1:13:43Then that argues for its reality. Another way you can argue is on semantic grounds, like under the table is a place, whereas if I ask you about under the, it's hard to come up with a question whose answer is under the, if you say, well, what does under the mean point to under the, we can't do it. And so, linguists have developed these arguments for the modularity of language. Those arguments are not about our brain, really. They're phenomenological about the language. When you get to neuroscientists now, now you can start to say things about the brain that's about monitoring and probing and designing experiments that reveal incidentally the structure and organization, and that we can absolutely do on a machine as well. Cool. All right. Well, I hope we keep that in. That was fun for me. Thank you. Thank you. I appreciate it. Yeah, no worries. Thanks so much for listening to this episode of Great In Descent. Please stay tuned for future episodes.