Machine Learning Street Talk (MLST) - Every Exponential Ends _ Silicon Valley Forgot _ Adam Becker

0:00Hi, I'm Adam Becker. I'm an astrophysicist and journalist, and I'm the author of two books. Most recently, I've written More Everything Forever, which is about the horrible ideas that tech billionaires have about the future and why they don't work. And my first book was called What is Real? And it's about the sorted untold history of quantum physics. I'm also the host of the podcast, Dreaming Against the Machine, where we try to think about what a better future could look like outside of the grip of the tech industry. And you should listen to this episode of Machine Learning Street Talk, because Tim and I have a great conversation about why some of the most powerful people in the world are wrong about so many things. By the way, many folks in my Discord really loved your previous book, by the way, so quite a few people said that. Well, that makes me really happy. I mean, you know, I hope I'll probably write, you know, more books in the future, but the first book's always special. Awesome. Awesome.

0:57Right. Here we go. Adam Becker, it's a huge honor to have you on MLST. Welcome. Thanks for having me, Tim. It's a pleasure to be here. I found out about you when I read an article on the Atlantic that you wrote last year. I think it was called The Useful Idiots of Doomsaying. Yeah, the Useful Idiots of AI Doomsaying. Yeah, exactly.

1:20Tell me about that. Well, maybe before we get there. So after reading that article, I immediately reached out to you because I thought it was great. And I discovered that you'd written this book more everything forever.

1:31And in a minute, actually, I'm going to read a paragraph from page 28 because I think that sums up the book quite nicely. But I should say at the beginning that we have a very technical audience and I've interviewed many, many folks from the EA community and the rationalist community and so on. So we try to be multidisciplinary. But I think this is actually a great opportunity for you as a physicist to litigate some of these ideas and kind of go from first principle. So anyway, tell me about your book. Sure.

2:00I mean, you did just sum it up in a way in that the book is me litigating a bunch of these things from first principles. I've lived in the Bay Area now for almost 15 years, and I have been to a lot of parties out here where I've heard a lot of people say a lot of bizarre things about the future of technology, AI, you know, space colonization, that kind of thing. And I always just kind of rolled my eyes. But as time went on, I realized, wait, these ideas are actually really influential. And yet there's no good evidence for them. And in most cases, a lot of good evidence against them. And so the book is my attempt to lay out this set of beliefs that are surprisingly influential within the tech industry and are promoted both by tech billionaires and by subcultures that are funded by tech billionaires like rationalism, effective altruism,

2:59And go after these ideas, both in a scientific and philosophical way, ideas like the singularity, you know, an AGI, apocalypse, space colonization, you name it.

3:14In a way, we live in crazy times, right? And large language models are now getting better all the time. And you know, you could almost be forgiven for having this kind of techno utopian view of what's to come. But I wanted just to read a tiny bit from your book, because I think this sums it up quite nicely.

3:31You said that the technologists, they make all of these problems into problems about technology. All the ills of the world will be solved when the singularity arrives, or when super intelligent AI solves them for us, or when we go into space.

3:44Global warming can be solved with nanotechnology, illness and death. All the other problems that come with having a body can be solved by transferring your mind into a computer. Social problems and political problems, like the problems created by tech companies themselves, are dismissed as irrelevant or unimportant when compared to the more urgent problems, like avoiding the creation of an improperly aligned AI or the polite of a hypothetical unborn quadrillions of humans that could live on the other side of the cosmos.

4:13a billion years from now. It's a philosophy made by carpenters insisting the entire world is a nail that will yield to their ministrations. Yeah, that's about right. That's about it. So where do we start with this story, Adam? I mean, I always feel like the right place to start is this idea of the singularity because I feel like that's where all of this sort of comes from is this idea that there is this future coming near as Ray Kurzweil says or nearer now and and that, you know, super intelligent AI and the rate of.

4:59acceleration in the advance of technology is just gonna keep going faster and faster until we have unimaginably advanced technology that solves every single problem and you know Kurzweil is best known as the sort of evangelist for this idea. But his evidence for it is really bad and then the evidence for it in general is just pretty terrible. It's not really an idea that makes much sense.

5:29Well, so how does Kurzweil make the argument? I mean, in your book you told the story of how his father died of a heart attack when he was quite young. So there seems to be a similar theme here that a lot of people, they start to fear dying either individually or collectively. It becomes an obsession. And he had this idea that maybe one day he could upload his mind into a computer and live forever.

5:54Yeah, yeah, that's exactly right. And I do think a lot of it is motivated by fear of death, which, you know, yeah, I don't want to die either, especially not anytime soon, but there's a difference between, you know, a healthy fear of death and letting it sort of run your life. But yeah, I mean, Kurzweil basically takes Moore's law, right, this exponential increase in the number of transistors that you can cram into the same area on a silicon chip.

6:23uh and generalizes it and says oh you know this is not just a technological phenomenon that lasted for about 50 years from the late 20 20th into the early 21st century instead this is a general trend in technology something he calls the law of accelerating returns and not just technology but actually biology as well So he traces it back at least as far as the start of life on earth and and in some places I think even claims to trace it all the way back to the Big Bang with the organization of inorganic systems as well. He just thinks that there is a trend toward greater complexity and intelligence that runs through absolutely everything and that is pointing to a time in the very near future. He has repeatedly said the year 2045.

7:16is when the singularity arrives and our technology becomes unimaginably advanced. And the evidence he marshals for this is pretty weak. He says, you know, you can pick out particular points of interest in the history of life on earth and the history of human technology. And when you plot them on a chart, you get, you know, a straight line on a semi log chart. And so that means that you've got an exponential trend but the fact is that you know those points are cherry-picked and he's got a problem that I think most of us have when we look at history you know unless you work really hard history looks sort of logarithmic right you've got this

8:05uh clearer view of what's been happening in the recent past and then the more distant past is more distant and so you can't see it as well and so you end up with this sort of um you know uh logarithmic view of the past that is something that I think Kurzweil is mistaking for a true exponential trend. Yes, and I think you gave an example in your book that you could look at, maybe this was Kurzweil's example, you can look at lily pads and then they grow exponentially and then they cover the size of the pond and then that's it. But more broadly though, there are limits. Yeah, exactly.

8:46Yeah, what interests me is that we spoke into Chomsky about this and he said after Newton exercised the ghost but left the machine intact, you know, that the whole enterprise of science stopped being about trying to understand how the universe actually worked and it was more about making sense of the universe. I mean, David Krakauer said to us that science is like poetry, you know, it allows us to give the universe meaning and make it make sense to us. But reality is protein, isn't it? So what Ray Kurzweil did, was he selected a bunch of things that were relevant to humans. So he didn't select, you know, things that might have been more globally important to our success. And then he kind of fitted it on a graph. And this isn't just, this is a very natural thing, right? The world is complicated. And to make it make sense, we kind of select things that privilege the way we think about the world. Yeah, absolutely. No, it's a very understandable mistake.

9:38In the book, I will make the analogy to looking out from the top of a skyscraper, right? You see the stuff closer to you more clearly and the stuff that's further away and less directly impacting you is smaller and further away and harder to see. But yeah, I mean, it's a natural mistake. And also, like many other people, and as you were alluding to with the lily pad example that I was talking about in the book that I pulled from Kurzweil, he forgets that the one thing that you can always say about any exponential trend, the one thing that's always true, is that they end.

10:17Uh, you know, yeah, he has the lily pad example to give an idea of how exponential growth works. He says, you know, the, if the lily pads double every day and on day 29, they cover one half of the pond. Well, then on day 30, the pond is full and like, yeah. And then on day 31, the lily pads don't grow anymore because they filled the pond. There's no more pond.

10:43And even Gordon Moore himself said, oh yeah, Moore's law isn't going to last forever. Moore's law has to stop sometime in the 2020s because eventually you get down to the size of individual silicon atoms and you can't really make transistors out of silicon that are significantly smaller than silicon atoms. But I suppose it is a bit of a straw man just to make the sigmoid case. So what they would say is that the actual exponential curve is the stacking of sigmoids and in technology, whether it's Betamax or whether it's film processing technology, what tends to happen is you get these disruptions. So you find a divergent stepping stone and usually a different people, you know, different group of people situated somewhere else, they find a new way of doing things and it just keeps going. Yeah. Yeah. Yeah. No, that's exactly what Kurzweil says. He says, you know, the exponential trend is itself made by a bunch of sigmoid curves. And each curve lets you get further up the overall exponential trend.

11:38Sure, yes, he does say that. But as I say in the book, there is absolutely no reason why that is something that's always going to happen. And indeed his own sort of generalization of Moore's law when he puts that together, it ignores much of the early history of computing where stuff like that did not happen. If you go back to, you know, analog computation devices from a thousand, two thousand years ago, they don't fit on his trend. And there's no technology coming up that looks like it's going to save Moore's law in the future. And that's just one trend.

12:12Kurzweil wants to say that those sigmoids save you and get you a continuing exponential trend in every single area of, you know, human technology and human growth. And that's, I'm sorry, just wish-casting. Yes, but there does seem to be an obsession with infinity and keeping going. I mean, you gave another wonderful example of Jeff Bezos. And apparently he said he was very, very scared of stasis. So for him, death is the lack of growth, right? Yeah.

12:42You know, there's a wonderful bit in your book where you said, you know, okay, so he's saying like we need to keep using more and more energy every single year. And you said, yeah, you know, maybe we can extrapolate this and possibly for another 3700 years, we can still use more energy. But if you think about it, that's actually less time than when the the great pyramid of Giza was created. And at some point, it must end. Yeah, that's exactly right.

13:04Yeah, because, you know, Bezos says we have to get off Earth because at some point in the next 300 years if we keep using energy and keep growing our energy usage at the same rate that we're currently at, eventually we'll be using all of the energy that the Earth gets from the Sun. And that's true. It's three or 400 years somewhere in there. He doesn't mention that also we'd be at that point generating so much waste heat that we'd be boiling the oceans. But, you know, whatever.

13:31And so then he says, so therefore we have to go into space in order to be able to keep growing our energy use. But if you go into space and you even spot Bezos a faster than light drive for free, if you keep that exponential growth in energy use going, eventually, as you said, in less than 4,000 years, you're using all of the energy available in the observable universe.

14:00And that's just not going to happen. Yeah, it's so interesting. And this seems to be an example of folks that wants to maximize human agency. Maybe we'll get to the AI agency in a little bit. But it's almost, you know, like Sam Altman the other day, he said, well, to succeed in the modern world, you need to be high agency. This seems to be the new term that she's using. By the way, it's great that I'm talking to a physicist here because we love talking about the philosophy of agency. But very, in a very basic term, I think of it as the ability to you, the the extent to which you are the cause of your own actions so that the cause or origination and possibly the extent to which you can control the future so causal efficacy but I'm talking to a physicist now and and you know that we're not really the cause of our actions maybe we're a causal conduit so they imagine a world where we could collectively

14:51basically imprint our will on the universe. We could escape our kind of causal, you know, the causal clutches of our embedding and we can just go on forever and reach to the stars. What's wrong with that? So many things. I mean, you can get into questions about like free will and consciousness and I'm not going to do that. What I will say is, you know, I I actually, the thing about that that I want to focus on is the same thing I focus on in the relevant part of my book. They just don't understand that we are not going to be going out and imprinting ourselves into the cosmos at large. That is not the future of humanity.

15:39You know, there's a part of, there's this book by Will McCaskill, this effective altruist, right, what we owe the future. And if you take a look, he's got this afterward for it that's on a website that he runs, but he's got a QR code for it at the end of his book. And he talks about how, you know, a good future for humanity would involve colonizing the observable universe and, you know, sending ships off to, you know, all the galaxies that we can reach in our universe and like intergalactic colonization. And that is just not going to happen. Like we're not leaving the solar system. The stars are simply too far away, it's too hard to get there. And it is too difficult to get ourselves up to any reasonable fraction of the speed of light. And even if we did the number of planets.

16:36that are remotely like Earth appears to be pretty low. I mean, it's just space is such an inhospitable place. We have evolved to be suited for this planet. And there are not other places like it that are waiting for us. This is our only home.

16:55Yeah, I mean I was inspired by James Lovelock's Gaia theory which was you know kind of stimulating people to think about the earth as a system rather than us as individuals Yeah, there's a bit of a spectrum here right because some of them perhaps in this case they think that us as individual bodies we can kind of disconnect ourselves from the system and go somewhere else. But it goes even further than that in Silicon Valley. Some of them think of us as causal patterns or as memetic programs that could be uploaded to computers and whatnot. But you see the pattern, though, that what they're doing is they're increasingly abstracting us and our place in the system that we exist in. Absolutely. Yeah. No, they don't want to be bodies. They want to be systems of pure information. And in that sense,

17:42What is it? Adrian Daub, I think, said that Silicon Valley is a place that likes to pretend that it doesn't have any history. And I don't even think it was the first one to say that. But, you know, there's a tendency to ignore prior art in these areas. And for, you know, this sort of dismissal of the body, I mean, this is an idea that goes back over 2,000 years, right? You see shades of this in Play-Doh and stuff like that. But the fact is that we are our bodies.

18:11You know, you want to look at the best science available. We are not our, we are not patterns in our brains. We are not even our brains. We are our bodies in our environment. And there is no good reason to think that you can abstract a human consciousness out of our physical selves. You know, I almost said physical substrate, right? But that's a word.

18:36and a phrase that comes from this computational analogy for cognition, and I think there's been a real kind of ignorance or amnesia or just convenient forgetting that it's an analogy, right? Cognition is not computation. Computation has some features that are like cognition, and the brain does things that look kind of computer-ish under certain circumstances, but neural networks on a computer are not the same as what our neurons do.

19:05not even close. Absolutely. So we're getting into functionalism here, you know, which is this idea that essentially you can describe our function and behavior, you know, maybe to a very high resolution in a computer program. And the important move here that they make, because they're not saying that the metaphor is the same thing. They're saying that the description could be alternatively physically instantiated in silicon and that would no longer matter. Yes, that's right. And I don't think there's any good reason to believe that this is true, right? I mean, the brain is fabulously complex and the actions of an individual neuron are quite a bit more complicated than a single neuron than a single node in a neural network in a computer. And and we do not have a good understanding of how it is that the brain does what it does. And.

20:05And we don't have a good understanding of how it is that the brain is connected to the body and what parts of the body are important for our experience and consciousness. I mean, it's very much putting the cart before the horse. We would need to really have a much better understanding of what's going on up here and in here before we could hope to uh, reproduce it with enough fidelity in a computer, and I would also argue that there's no good argument that computers, as we currently have them, are gonna be capable of that kind of thing in the right kind of way, even if you subscribe to a kind of functionalism, which I'm not sure is correct.

20:46Yeah, I think it also gives a good insight into the model that the folks have in Silicon Valley because it comes back to this agency thing again. So, you know, you could argue that agency requires, you know, like the good regulator theorem, the brain is not just a complete model of the world. You could argue that it's a kind of input output processing system. So it doesn't really work if you take it out of the environment because, you know, a thought experiment is I take a copy of you and I put you on Mars, are you the same person and they'll get into this identity problem.

21:16philosophy. But the real problem, though, is that, well, maybe the way that you think is literally, you know, maybe you're like an LLM and the people you talk to, the things you look at on Twitter, all of this forms you. And if you take you out of the environment, maybe it's not you anymore. Yeah, I mean, I think that that our environment is an important part of who we are. I think that one of the moves that I see over and over again from these guys is this idea that intelligence is an individual trait as opposed to something collective and social. And I don't think that that's right. And I also, for what it's worth, don't think that we're actually very much like LLMs. I think that if nothing else, the fact that an LLM

22:13takes in more text than any human could in a human lifespan in order to be worse at language in many ways than a five-year-old. And five-year-olds take in far, far less and do far more with it. That right there is a good indication that we're pretty different. I know, but I should push back a little bit because I don't know whether you've played with Mythos before it got taken offline, but even though we can criticise the functionalism thing and we can talk about Chomsky's competence versus performance, there's a difference. But what we do see though is that these models can do things that we do extremely well and they seem to be getting better all the time. Even though they do a different type of syntactic processing than we do, arguably it's

23:01in some ways better than we do. So something is kind of validating their functionalist hypothesis. Yeah. I mean, there's definitely something going on there, right? And then the question is, okay, what is it? What's the significance of that? I would argue that we're learning some interesting things about how language works and what you can get out of the sort of mathematical structure and statistical properties of language. But, you know, I don't remember who said this, but people speculate about whether, you know, LLMs are alive, but you don't speculate or alive, you know, conscious, but you don't hear people speculating about whether systems with similar architectures that

24:02don't work with language are conscious. So nobody's asking if Alpha Fold is conscious. And I think that says a lot about how we as humans see systems that process and use language or emit language as it were.

24:25Yeah, I mean without spending too long on consciousness because we can get stuck. Yeah, we can get stuck there all day. That's right. You're kind of talking a little bit about what I would call AI psychosis. So there was that, you know, there was that Google engineer famously with the Lambda model, was it a couple of years ago? And so yeah, we anthropomorphize these models just like the ELISA system many, many years ago. Yes, indeed.

24:47Yeah, I mean, there are folks that seriously are, you know, taking seriously the idea that these models might have some kind of phenomenal experience rather than just simulating. I mean, and the AI psychosis goes far broader than this, but this is concerning to me. What do you think about that? I mean, the AI psychosis stuff is really, really terrifying to me because I really do think that this is taking that sort of power.

25:15Paratoilia word that I'm almost sure I'm pronouncing incorrectly. But it's it's taking this human tendency to see patterns, especially like human patterns, where there aren't any like seeing a face where in random noise and stuff like that. It's taking that and and almost exploiting it, right? Because we're we're not only attributing agency to these things, but in the case of AI psychosis.

25:43I mean, I don't know exactly what's going on there, but it certainly seems like it's people attributing agency to these things and then getting caught in a kind of feedback loop where these systems are sort of regurgitating themselves, like regurgitating the user back to themselves. Right. Because, you know, that's that's what they do in a way. What Shannon Velour talks about the idea of AI as a mirror.

26:13And I think that that's a really good analogy. It sort of reflects ourselves back to us and in this case, I think it's happening in a really unhealthy way. I agree with that. The thing is, though, using this technology every day, it's getting better. Now you can have a memory system. So every time it does something wrong, you say, no, you should have done that. And over time, you're building a kind of semi-lacrum of yourself and it increasingly does the right thing. And we can talk about the intelligence thing. I mean, yeah, it is a grotesque metaphor that you can take something which I believe is a physical property of stuff in the universe and we can create.

26:53an abstraction that seems to work reasonably well for abstract domains like playing chess and so on. And these models are becoming quite adaptive and they have a lot of capabilities and so on. And you see it's just so deceptive. It might be the most deceptive time in human history because it seems like it is intelligent. And it is actually automating meaningfully human labor and lots of jobs and stuff like that. So how do we make sense of this?

27:22It's a good question. I don't want to diminish what LLMs are capable of, but I keep thinking about calculators. Calculators took something that, and I mean like pocket calculators, right? They took something that we thought of as this inherent human activity of that kind of computation, like four function mathematics.

27:52and automated it. And it used to be that to be a good mathematician or a good physical scientist, you had to be good at doing that kind of work with pencil and paper or in your head. And then that stopped being true. And in some ways, that was a loss of certain things. But on the other hand, it allowed for a kind of mathematical research that could not have been done before, right? You know, you'll hear sometimes mathematicians say things like computers are like telescopes for mathematics. And, you know, but but again, I mean, not to repeat myself, but I do think that the difference here is that it's language, not math. And so that makes it feel like.

28:52something is sort of thinking and conscious and talking to us. And again, that's not to diminish it. I'm not diminishing the functionality of calculators either. But it is quite deceptive. And I think in terms of how to make sense of it, I think we just have to keep in mind what these systems are at the end of the day.

29:20for predicting the next word or whatever in the sequence that they've been given. And it turns out you can get pretty far doing that. You can. And I would push back a little bit on the stochastic parrot thing, even though that's technically true. Internally, they are acquiring during their training process some kind of coarse grained abstraction, some kind of structure, which allows them to...

29:48extrapolate generalize call it what you want. So, you know, it's different to us, but there's something there. But I think the $2 million questions are at the moment, it needs human supervision.

29:57And these folks say, well, we just scale is all you need. Yeah, at the moment, it needs human supervision. And at the moment, it just generates loads of spaghetti garbage code. And it just creates, basically, it creates more problems than it solves. But it's deceptive because most people can't see the problems. But all we need to do is keep scaling it. When GPT7 comes out, it will actually refactor all that code. Maybe we can RL train it to refactor it. And we just have to kind of And it's very dangerous keeping going because now we're messing all of our code bases up and we're creating all of this slop everywhere. But just, just hold on, boys. Just wait for a couple of years and the next version will kind of bring it back in check. What do you think about that? I just don't think that that's true. I don't think that there's, I mean, I could be wrong, obviously, but I don't see good evidence for that, right? I mean, I think that these systems are, unless there's some sort of fundamental breakthrough, right? Something more than just scale.

30:51These systems are always going to require human supervision because they are always going to end up hallucinating. You know, I mean, that's inherent to the way that they work. I say this in the book, but you know, I don't love the word hallucinate.

31:06I know there's been a lot of pushback because it's like, oh, hallucination implies a sort of anthropomorphization of these systems. And I don't love that either, but that's actually not my main problem. My main problem with the word hallucination is it implies that when a hallucination occurs, something different is happening than its normal functioning. And that's not the case. They really only do one thing. And when they're hallucinating, they're doing the same thing that they're doing when they get it right.

31:31And so I think that unless we have some sort of major, major breakthrough, and I mean, it would probably have to be a breakthrough that makes, you know, LLMs themselves look like Eliza. You know, short of that kind of really fundamental breakthrough, I don't see us getting around the need for human supervision on these things.

31:55So and if anything as they get better it's going to get harder to discern when they've made these mistakes even though they're gonna keep making them and that's quite dangerous as you said.

32:05i know and ironically people aren't really talking much about the hallucination now because they're a gentic and they can fix their own stuff you know it's it's more like you know if you analogize it like a database query or a program interpreter it's only as good as the program or the database query so you know it will basically do what you tell it to do and this is this is the gap right because if these things could be alive if they did have agency what would what would that mean.

32:31Let's wire it up in a loop and let's show it a load of video frames in a sequence and we'll give it a basic prompt like do stuff do something interesting what will happen.

32:40Basically nothing nothing interesting will happen right so the more you understand the domain and you put a very specific program in there you can get it to do a specific thing you can get it to hill climb towards a specific goal to solve a specific problem but the framing always comes from us so it's kind of doubtful whether we would overcome that make maybe we will but it's kind of doubtful but but but then there's this thing which is the first or second step fallacy right which is.

33:04As you said, Eliezer Yudkowski, he said that we're only one or two steps away from inventing AGI and it's going to run away. Why does he think that? I mean, first of all, he didn't say one or two. He actually said zero to two. He's not sure that we need any. But yeah, I mean, why did he say that? I mean, you should ask him. But he believes.

33:32in something like a singularity. He believes that if you just throw enough computing power at a machine learning system of the right type, then it will become conscious, wake up, whatever term you want to use.

33:58become intelligent and then use that intelligence to increase its own power, which will increase its intelligence further. And this will create a sort of feedback loop and you'll get an intelligence explosion. And then he believes followed shortly by the end of the world. Can we unpack this a little bit? So you were speaking there a little bit about instrumental convergence, which is this idea that basically it's It's instrumental sub goals towards whatever it's doing will kind of converge on things like power seeking and bad things that we don't want. So it's always going to kill us all in almost every scenario. Yeah, that's what he believes. Yeah. Yeah. Can you explain that in a bit more detail? Sure. Yeah. So the idea there is whatever goal it is that the AI has.

34:50when it achieves this sort of basic level of intelligence, whatever that goal is, that goal will be served better by having more intelligence. And that intelligence in Yadkowsky's worldview is something that you can increase by throwing more computing power and memory and whatnot at it. And the idea there is, No matter what your goal is, it could be something in the famous thought experiment from Bostrom. It could be something as silly as creating more paper clips, or it could be keeping people engaged in chatbot conversations, or it could be making as much money as you can on the stock market. No matter what

35:43your goal is the thought with this notion of instrumental conversions is all of those goals will lead you towards seeking more power, more intelligence, more physical and computational resources in order to better achieve those goals and prevent barriers from being placed in the way of achieving those goals. I don't believe that's really true, but not in the way that they would need it to be in order for this argument to go through, but that's the claim.

36:17Yeah. And it seems to be an abstraction problem again, because in the biological world, intelligence and agency are convolved in a very complicated way. Whereas in this sense, we're now abstracting them, because there's also this orthogonality thesis as well, which is that intelligence and final goals are disconnected from each other. But more broadly, on intelligence, it's almost like it's this magical abstract substrate that you can... It's on a single axis, and you can just have more of it and more of it. But that's not really how intelligence works, is it? No, it's not. Not at all. Intelligence is quite a bit more complicated than that. And it's not a single valued thing. In fact, I'm not sure that we have a great definition of intelligence. And I also think that it's very clear that there are a great number of goals that don't work this way. I mean...

37:14Maybe it's just because it's like I woke up in a cynical mood this morning, but I'm not convinced that the goal of being happy is one that is served well by being more intelligent or powerful. Yes. For example, or to pick a somewhat less cynical example, if intelligence were really so broadly useful for every single goal, You would see more of it in the natural world. It would be something that would evolve, you know, more often and more broadly. Trees have goals and they get there in very different ways. And giving them a more complex nervous system or a nervous system at all would not help them with those goals. It would hinder them. Yeah. I mean, it's also quite debatable whether

38:09things in the real world even have goals. So, you know, we always, as we were saying with science earlier, we need to disentangle how the world really works from how we understand it to work. So we take an intentional stance and we say, okay, well, there's a complicated blob of stuff over there and I'm going to interpret its behavior as having this simple goal. And it might be a cartoon, right? It might not be like... Yeah, that's right. Yeah, no, no, no, it's not even really right for me to have said that trees necessarily have goals. But the point...

38:36the point remains. Yeah, exactly. Like this is seeing the world in terms of goals is itself a kind of mistake. And in some ways related, I think, to the mistake of seeing intelligence as this single-valued thing that you have more or less of. Right? There's this very simplistic worldview at work here.

38:56Yeah exactly and but the thing is that you could make the argument that some folks in that community they've moved on past the kind of the Bostromian Yudkowsky view of this you know monomaniacal supercoherent AGI and now they're a little bit more kind of thinking okay it might be really illegible and it might disempower us but it might not just be this thing that has a single goal. Yeah I mean I'm not.

39:26particularly concerned about that either, honestly, simply because the systems that we have and the systems that seem to be in the near future are not that kind of thing. Right. You know, I mean, and I also think that we are building these systems to, you know what? No, never mind. I'm not going to continue that thought. I think that was wrong. But yeah.

39:57I mean, I'm a writer, right? One of the nice things about writing is you get to try out a bunch of ideas, look at them on the page, and then say, okay, you know what, that one's good, that one's good. Let's get rid of that one, that one, and that one. They're no good. But yeah, but instead we're doing that in real time. As a quick aside, thank you for not being on autopilot. I'd much rather that thoughts come into your mind and you say them because it means that you've not said them before. That's great. That's what we want. Well, yeah. I mean, look.

40:24I could just sit here and read to you from my book, but there's an audiobook, which you can buy wherever audiobooks are sold, please do. I don't think they're being... I mean, it's easy to be cynical and say, oh, follow the money. And as you outlay in your book, there are lots of wealthy billionaires putting lots of money into this and easy to be cynical.

40:50I think there's this anthropic fiasco recently. It doesn't make sense, make it make sense. Why they're making all of this money? Why would they sabotage themselves? I don't think they're being grifters, but what is going on? Are they true believers? What's happening?

41:07I think that a lot of these people are true believers. Yeah. Like, you know, I mean, in the book, I certainly don't go easy on, say, Eliezer Yadkowski. But one thing I will say about Yadkowski is he's certainly a true believer. You know, he believes the stuff that he's saying. The same goes for, you know, like people like Nick Bostrom or Toby Ord, right?

41:31And I think that that's also true of a lot of the people working at places like Anthropic. They really believe that AI alignment is the problem of our time. I just think that they've made a mistake. I think that that's not true and that there are other problems that are significantly more pressing and that...

41:53You know, as I say in my book, many of the biggest problems of our time are not problems amenable to solutions with technology. They are social problems that require social solutions, which is not to say that technology doesn't play a role. It's just that you can't solve them purely with technology. You know, climate change being the big one, the climate crisis is a problem that absolutely does require certain technologies to exist. Most of the technologies that we need Exist already and the problem is one of actually deploying them and persuading people to switch and actually like finding ways to get this stuff out there. That's a social and political problem that requires a social and political solution. So yeah, and then but then you have someone like Sam Altman who I don't know what's going on in Sam Altman's head. I don't know.

42:49if he's a true believer in whatever, but he has gone out and said things and he's not the only one who said things like this, that, you know, AI is going to solve the climate crisis. And like, I think that in the unlikely event that we built, like a really intelligent conscious machine intelligence based on existing technologies in the near future. Well, if we did that and asked it to solve climate.

43:17crisis, the minute we turn it on, it would say, well, you shouldn't have built me. Look at my carbon footprint. Are you implying, though, in a sense, because the wording of that Atlantic title was very interesting, you know, useful idiots. So in a sense, are you implying that they are genuine? But even though this is a culture that valorizes individual agency, and you're saying they actually don't have individual agency, they're being kind of parasitized by some kind of emergence you know, force above them that perhaps they don't understand. I, yeah, except I wouldn't even, I would get more specific instead of saying an emergent force above them that they don't understand. I would say the venture capital startup ecosystem of Silicon Valley. Like, it's, yeah. You know, I mean- But even on that though, it's not a conspiracy, is it? No, no, no, no.

44:09They also probably don't, you know, they probably think that they're just investing in technology and they're going to make lots of money and so on. But I feel like you're making a slightly different point, which is that there are machinations of power that lead in a certain direction. Yes, exactly. And I do think that, you know, for the nice things that I could say about rationalists and effective altruists that, you know, they are generally earnest and true believers.

44:35That one of the many places they fall down and I'm not saying this is the only place they fall down as a you know, this is all in my book, but they I don't believe that they have a good understanding of power. And I think that that's what leads to this. So yeah, no, I don't think that there's a conspiracy. I don't think there has to be conspiracy. Instead, what you have is these these true believers running around saying like AI is really, really important. It's coming soon.

45:03and it could be so powerful that it could lead to the extinction of our species if we're not really careful and don't devote a lot of resources to solving this problem. And then you also have in an adjacent place, ideologically and physically, this giant system of power and capital that needs the promise of perpetual growth in order to maintain a lot of that power and capital.

45:32And so it sees this narrative that is also a familiar narrative to the leaders of that system of power and capital, the venture capital system of Silicon Valley. That narrative is familiar to them because all of the people in this story, including me and I think you were raised on a certain set of science fiction stories, some of which are in the bookcase behind me about, you know, What computers would be able to do and what AI is and would be able to do what space colonization would look like and so on and so forth And so they see this story coming out of these subcultures some of which were funded by other people in the tech industry and

46:17And they they see this stuff and they it's legible to them because they they've read the same science fiction. They say, oh, this is another story about perpetual growth. And again, I'm not saying it's a conspiracy. I'm not even saying that the venture capitalists aren't true believers. I imagine that a good chunk of them are maybe even most are all well, probably not all of them, but maybe even most of them. But, you know, there's not.

46:44There's not a conspiracy at work here, but it turns out that, and I think this was predictable, that running around saying that AI is coming and is going to be extremely powerful, and if we get it wrong, we'll all die, but if we get it right, we'll have the power of the gods and be able to expand out into the cosmos forever. That's a story that's very compatible with the desire for perpetual economic growth and resource usage growth.

47:14that, that is so desirable to the leaders of Silicon Valley. Yeah. And I mean, I guess like we could talk about, you know, the Yanis Farah Fakas idea of the techno capitalist machine and neoliberalism and all of this kind of stuff. But I'm interested in the AI safety thing in particular because you You sketched out the entire story, right? Of, you know, the Extropians and, you know, where Yudkowski apparently used to be a singularitarian himself. And these were early days when, you know, they were just on online forums and just talking about this stuff. Yes. So how did the, you know, and then we've got the whole effective altruism story. How did these things meet? How did all of this get entangled up? Oh, God, well, I mean...

47:58You don't want me an autopilot here, but my standard answer to a question like that is, you know, that's very complicated. And and I could write a whole book about it. But. But without just reading to you from my book, I mean, the the answer is these were groups of people who found each other through the Internet, mostly who were all spending a lot of time thinking about how do we get.

48:25this future of technology that we have all come to believe is going to happen based primarily on the science fiction that we've read. And, you know, I as a sci-fi fan, I can understand why you might think that the future is going to contain the, you know, some or most of the elements in the science fiction that you've read if you just, you know, put your nose to the grindstone and whatnot. But.

48:53But as a physicist, I can tell you that a lot of those things are not going to happen. But these were people who generally were not thinking about it in that way. We're just thinking, yeah, this stuff is going to happen because I can't think of a good argument that it definitely won't, which is not how the world works and not how the development of technology works. And oftentimes there were good arguments that these things won't happen and they were just ignorant of those arguments. But.

49:23You know, I mean, the reason I'm talking about that rather than talking about like, okay, well, this is how the Extropians and singularitarians got together and then the rationalists came out of that and blah, blah, blah, blah, blah, is that's a complicated historical story and anthropological story. But the root cause of it, I think, is this sort of shared belief in a world as revealed by primarily science fiction.

49:51Yeah, and maybe we should talk about the utilitarian thing as well with effective altruism. So, you know, I listened to Will McCaskill. He was on Sam Harris' podcast recently. And after kind of hearing all of this stuff about AI safety and, you know, thinking some of it was quite strange, I thought he was a really nice guy. You know, I could tell that he was genuinely trying to do good in the world. And they've got so many, you know, really, really good activities going on. The thing that I don't understand is how that AI risk then came into it. Now you gave the example of Peter Singer, who was an inspiration and there was that thought experiment with, would you help a child in some dirty water? Is it worth more than the cost of your clothes to help that child? If that's a yes, why not then help children elsewhere? So you see, what they do is they abstract this concept of a utility function. And as we were saying before, when you start abstracting things, then you get into trouble.

50:46Yes. Yeah, that's exactly right. No, I mean, I think that there is something very appealing about a utilitarian approach to ethics and the world from this sort of engineering mindset and scientific mindset even, certain kinds of scientific mindsets, definitely engineering mindset. But this idea that everything can be abstracted and quantified and the things that you can't do that with don't really matter.

51:15That is a story, you know, a story that you can turn ethics and making difficult decisions about life into questions about numbers is always going to be an appealing story to, you know, a fairly large number of people. I mean, I suppose a side effect is Will McCaskill, I think he interviewed him for the book, right? I tried to. Will McCaskill agreed to an interview and then backed out.

51:41And my fact checker ended up talking to him, although I don't know. He got confused, I suppose, the McCaskill, not my fact checker. McCaskill seemed to have thought that my fact checker worked for my publisher and not for me, which I'm not sure how he made that mistake because he... McCaskill worked with the same publisher and knows that publishers should know that publishers never pay for fact checkers, but in any event.

52:09No, I didn't talk with McCaskill. I read a bunch of his writing and I talked with his colleagues, but McCaskill himself did not talk to me. Okay, but in your book, you moved on to long termism as beings.

52:20one of the sort of dangerous side effects because I think there was a quote in the book where, you know, McCaskill said that now is the thin end of the wedge. So obviously it's a good thing if humans all have utility, we should have many, many more humans that they should be all over the universe and even digital versions running on servers all over the universe. And the problem is when you when you start extrapolating forwards, it means that the effective value of us compared to that big future light cone of humans, you know, we're not very valuable.

52:50So that kind of diminishes many of our current ethical concerns here on Earth. Is that roughly right? Yeah, that's roughly right. Yeah. I also want to go back to your question of, okay, how did the AI risk stuff get into all of this effective altruist stuff? And there the answer is...

53:09I mean, basically Nick Bostrom read a bunch of Yadkowski got into the AI risk stuff from there and then started convincing people like Toby Ord and Will McCaskill that, yeah, this AI risk stuff is really important and is, you know, this overriding ethical concern. And I will say, you know, there's even statements that some of the people in this community have made about like, okay, you know, other existential risks, that's sort of the gateway drug to caring about AI risk, which is like the real part of the matter. And I just think that's misguided. But is your position essentially that there are certain mind worms, for whatever better term, that are quite dangerous, right? Because they can be kind of repurposed and, you know, there can be dangerous externalities and so on. So I mean, what's your prescription?

54:07My prescription is, yeah, I mean, well, let me let me back up and say, yeah, I do think that this is kind of a brain worm for people of a certain technical bent. Part of the reason I ended up writing the book is that I myself have a fairly technical bent and sort of see myself in a lot of these people and feel like, OK, you know, if I were just a little bit different.

54:33or if my life had gone a bit differently, I could see myself having been one of these people. Maybe that's wrong. Maybe I don't know myself correctly or don't understand exactly the appeal of this stuff. But I feel like it's a pleasingly compact and tractable view of what the world is and how its most complicated problems work, which I would also argue is where a lot of the misunderstanding of power comes from.

55:03So my prescription, I think would be to I mean I could be flip and say read my book but I would say that that my prescription is really to take more seriously the people who study the relevant areas of inquiry That that the people in these spheres tend to ignore they tend to ignore you know, political science, sociology, they tend to ignore anthropology. They tend to ignore, you know, questions about racism and bias and sexism and whatnot. They tend to... And they tend to get stuck in particular pieces of group thing, some of which...

56:00get really pernicious and dangerous. Like there is, as I describe in my book, a surprising amount of credence within this community of effective altruists and rationalists for ideas like human biodiversity, which is a piece of junk science that tries to give a genetic basis for racism.

56:28And it has been roundly rejected and disproven by the scientific establishment, even though genetics itself comes from a long history of racism. And yet within this community, it's an idea that people feel they have to take seriously a lot of the time, even though there's nothing serious there.

56:49Yeah, I wanted to touch on that because there are folks in this community that have been making these arguments. I'm not sure where Tesco Reel came from. Maybe it was Tim Nickabrew, but there's folks like Emily Bender and Emily Torres as well.

57:03To be honest, the discourse has been really bad. So a lot of folks just think that they're arguing in bad faith and they just dismiss what they're saying. And that's why I feel that we've been able to have a great conversation because we're just talking about it from a technical point of view. We haven't really been, I mean, I don't know much about social science, but do you think that, you know, why is there such an impedance mismatch? Why is it so difficult to have this conversation rationally? Oh man, I mean, I think that I mean, that's a good question. I mean, it's one of the things I was hoping to try to address with my book. Could I give you one example? Sure, yeah, give me an example. I've seen presentations from Timnit and she's very quick to go to your racist, basically, and eugenics and all of this kind of stuff. And it's quite heavy going. It's quite scary stuff.

58:00And maybe we should just start there. So is that part of the problem? Is it just because it's a little bit kind of too intense? Yeah. I mean, I think that the that I mean, look, you know, I think that that my sympathies lie significantly more with people like Timnit than with someone like Yadkowski. And and my issue is like, I think that what's going on.

58:34is you have people in the effective altruist, rationalist community who, you know, want to or feel that they should be able to entertain ideas like human biodiversity without.

58:59recognizing that actually this is an idea that has been entertained for a very long time and has a really horrifying history and has no good science behind it. And then don't understand why someone gets upset at the prospect of, you know, having a dispassionate rational argument about whether or not certain people are fully human and entitled to, you know, the same rights and have the same inherent abilities.

59:28And this is, you know, this goes back actually to that thing I was attributing to Adrian Daube earlier, this sort of like looking at things outside of the context of history. There is no way to do that. History is something that we all live in. We, you know, as the meme says, we live in a society, right? And I think that it is irrational to assume that someone is not going to take that history and that context into account when having a conversation on a particular subject, especially if that subject ends up being related to whether they and their friends and family are going to be treated like people. So I think that that's where a lot of this comes from is there's a group that

1:00:25feels that rationality means not taking particular pieces of context into account. And another group that's like, why wouldn't we take that into account? And of course my sympathies lie much more with that second group. And it's funny as well, because speaking personally, I'm a bit of a weird guy in the tech community because I do believe in physical situatedness. I think it's about how you got there. You know, you can't just abstract yourself out of the city. Yeah, yeah, yeah. And also, I think that the fundamental impedance mismatch, just as we were saying earlier, is folks like Timnit are talking about super agency, not individual agency. So she's not necessarily saying you're individually racist. She's saying that you are basically caught up in this larger system of power dynamics that you're not aware of and you don't have control over. Yeah, I think that's probably right, although I don't know.

1:01:22I don't want to speak for Timnit, but I don't know that I would want to make such a sharp distinction between individually racist versus caught up in this system that you're not paying attention to because not paying attention to it can be a form of racism. It's certainly a form of privilege, right? Yes. And just for the benefit of the audience, can you just briefly make the arguments why...

1:01:49you know, like the eugenics and the racism thing, like what's the main argument there? I mean, there's a couple of things, right? Some of it is what I was just talking about, that this community seems to really entertain hypotheses, not seems to, does entertain hypotheses like human biodiversity in really unhealthy and unscientific ways. But the other thing is this idea of intelligence, right? It's the idea of measuring intelligence, intelligence as a single valued thing.

1:02:18is itself an idea that basically comes out of eugenics and racism. IQ tests, which are taken quite a bit more seriously by people like Yadkowski than they should be, are something that was developed essentially as a tool for eugenics. And when you look at the modern AI, industry and the field itself of inquiry, you see sort of things that are descended from that history present here today, including, you know, people using eugenicist arguments, not even from old school eugenicists back from, you know, when IQ tests were first developed 100 years ago, but from modern day.

1:03:14purveyors of bogus race science and racism, you will see people pulling a language from those people when trying to define things like intelligence, like in academic papers and stuff like that. So yeah, that is sort of the really short argument. There's a longer and more developed version of it that I give some of in my book and you can find elsewhere as well.

1:03:43Yeah, and I think it's ironic as well because in my opinion current AI has proven that intelligence is not this thing that we thought it was so yeah in some sense that the models do have intelligence and they still don't have the thing we want and and why is that it's because we thought the intelligence was like we have this core knowledge we have these these kind of abstractions in our brain and intelligence is just traversing the combinational closure and the more intelligent you are the faster you can do that. And what we found with these models is actually it's all about how you frame the question. It's all about the information you have at your disposal and there are many, many different perspectives on different phenomena and folks with different perspectives can use their intelligence and they can get to different answers. So it's very situated and it's not quite as abstract and pure as we thought it was. Yeah. No, I think that's exactly right.

1:04:34Um, and again, you know, I think that this comes of, I mean, in my book, I call it humanity's denial, right? Um, this idea that you don't have to pay attention to things like history, sociology, politics, um, when again, we, we do live in a society.

1:04:53So I wanted to touch as well on this idea of recursive self-improvement. So this originated with a guy called I.J. Good. And Kurzweil certainly made this argument. So he said, well, the first AGI, it's going to take us a long time to build it. But then we can use the AGI to help us build the next one. And the next one will be built in a fraction of the time. And it will just keep getting better and better. And we're seeing hints of this in AI. Like there's undoubtedly there's some kind of adaptive self-improvement, certainly when you're doing hill climbing towards a particular task. But what do you think about the broader kind of idea? I think that it's predicated on an idea like two problematic ideas, right? Both of which we've already talked about. The idea that intelligence is like a single valued thing. And the idea that intelligence is something that you can equate with or is directly proportional to computational power of some kind.

1:05:48I don't think that either of those things are true. And more generally, yeah, I think, well, no, I'll leave it at that. Yeah, I think that it basically comes from those two things. And the case for those two ideas is pretty weak. Yeah. Why are we not going to space? I mean, space is pretty bad. It's really easy to die in space. There's a joke that I made in my book. It's one of those things when you're writing a book, Uh, or at least when I'm writing a book, I'll make jokes in the text and then some of them don't make it through my edits and then some of them don't make it past my editor. And then I'm always surprised at which ones actually make it into the final book. And one of the jokes that's in the book is, you know, I quote that famous Elon Musk, quote, uh, it'd be cool to die on Mars, just not on impact. And I say, well, you know, if we take Musk at his word, the good news for him is it's really easy to die on Mars. Um,

1:06:45because Mars is a really horrifying place. There's, you know, the radiation levels are too high. The gravity is too low. There's basically no air and the dirt is made of poison. And yet, Musk is right that Mars is the next most hospitable place in the solar system after the Earth.

1:07:07the only real competition it has is the moon and the only thing about the moon that's better than mars is that it's really close by which makes it much easier to get there much easier to communicate with people who are there other than that the moon is pretty much worse in every way there's there's no air uh and um and although the dirt on mars is more poisonous the dirt on the moon is actually it's something that became such a big problem that every Apollo mission was basically at its operational limits just because of the dust, because moon dust is really sharp, abrasive and electrostatically charged. So it just sticks to everything. And, you know, I mean, the fact is that there there's nowhere in our solar system that has anything like.

1:08:03the physical attributes that we would need to be able to live there or to be able to make it a habitable place. Musk talks a lot about terraforming. We're not terraforming Mars. If we had the technology and know how we would need to do something like that, then solving climate change here on Earth really would be a technological problem, and you could solve it very easily.

1:08:29Um, he talks about the need for Mars as a lifeboat for humanity in the event that something horrible happens to earth like an asteroid as big as the one that killed off the dinosaurs 66 million years ago, uh, impacting earth again, uh, there was nothing that that could really happen to earth that would make it as bad as Mars is to live on short of the actual physical complete destruction of the earth. Um, because even that day when that asteroid hit 66 million years ago was nicer day for life on earth than any day on Mars in the last you know two or three billion years at least and we know that because mammals survived and birds survived and all sorts of life survived that day whereas there's no mammal then or now that could survive unprotected on the surface of Mars the it's

1:09:21It's just horrible. And then like farther out in the solar system is even worse. We're not leaving the solar system because the speed of light is just too slow and we're not getting past it. We know that we can't go faster than the speed of light. We have really good science and really good experimental evidence to that effect. You know, we've tested the speed of light limit over and over again in particle accelerators. That's a lot of, you know, particle accelerators get.

1:09:46particles up to almost the speed of light, we know that our theories of, you know, relativity work in that domain. We know that you cannot get a spaceship going that fast. And we know that if you got a spaceship going anywhere near that fast, it would have enormous problems with radiation and shielding. And we think worlds like Earth are pretty rare. So yeah, we're not going anywhere. I could also explain a bit about why AI data centers in space are pretty laughable, too. But I don't know if you want me to get into that. Oh, yeah. I mean, that's low hanging fruit. I mean, I never quite understood that one that there's there's all sorts of places. So many problems. It's just crazy. My favorite is when people say, oh, but, you know, it's easy to cool them because space is cold and like space is a vacuum. Vacuum is a perfect thermal insulator. Like, you know, good luck getting rid of that heat. You're going to have to have radiator veins the size of a city.

1:10:43i know i mean part of me is thinking it's great that we do have people that just have insane ambition because you know musk doesn't doesn't actually. Realize his most ambitious goals but he has done some pretty amazing stuff star link is very useful sometimes doing ambitious goals means that you find interesting new stepping stones and you know that some but there's also the angle of sometimes if you want to hire very talented people and you want to build something you need to create a shared myth maybe you're a true believer maybe it's just some kind of systematic thing.

1:11:12Maybe, yeah, but I think that those myths end up having harmful consequences if they are so detached from reality, right? You know, Musk... has created, I mean Musk did not originate the myth of colonizing Mars, but he is, you know, in the same way that Kurzweil didn't create the idea of the singularity, Musk didn't create the idea of colonizing Mars, but he's certainly, you know, Mr. Occupy Mars these days, and it creates this idea that we don't need to care about the problems here on Earth because we're leaving, you know, and that's just not going to happen.

1:11:46you know, these goals, even if they're not possible to realize, have consequences. And the consequences that they have can be really horrifying. I mean, that's a lot of what my book is about is like, okay, these are why these goals don't work and the consequences trying to pursue them have. And just doing a bit of psychoanalysis here is your kind of prognosis that There's an element of seeking purpose. So even folks in working open AI or whatever, they want to feel that they have a grand purpose and they're benefiting humanity. And is that a similar thing with these billionaires that they want to kind of feel that they're doing great things for the world? I do think that that has to be part of it. I mean, again, we can't get inside their heads. We don't know for sure. But there are so many things that money can't buy and one of them is a sense of purpose, right?

1:12:35um and this sort of myth making gives them a sense of purpose and and you know one of the most common responses i get to. What i wrote in my book is you know do you really think that these guys actually believe this stuff you know why would they it's it's all just you know a ploy for them to make more money and and.

1:12:55what I respond to with that is like, yeah, okay, I don't know what's going on in their heads. Maybe some of them are just being cynical, although there's sometimes evidence like with Bezos, we know that he's always been very, very interested in colonizing space because he's been talking about it since he was in high school. And so that suggests he's not just being cynical, but I also think that when people ask me that question, what they forget and what I usually respond with, is the fact that these myths are useful for getting these guys more money and power makes it more likely that they earnestly believe these things, not less likely. If you had your way, what would you do about all of this? If I had my way...

1:13:45I would tax billionaires out of existence. I don't think that that's a just or reasonable or fair distribution of resources. These people did not make that money on their own. They needed the rest of us to get there. And by hoarding those resources, they are creating massive power imbalances that erode at the fabric of our democracy and at our shared sense of truth as a society.

1:14:15And that makes it very difficult to live with one another. And we need to find a way to live together in harmony. And I think that allowing these kinds of power imbalances make that impossible or next to impossible. So yeah, I think it should be illegal to have that much money. And also with current AI, I mean, just speaking from personal experience, it's really helped me.

1:14:41doing lots more work. I'm doing the work of many, many people and, you know, it's made me more consistent and I'm making better decisions and, you know, so I can kind of see where Sam Outman is coming from, you know, it's democratizing. Now you can quit your job, you can start a business, you can use AI to help you.

1:15:00That's the sales pitch. But from your perspective, I feel that you're saying actually, structurally, this is going to create huge inequality. And it's going to be, you know, especially when we have a two-tier system when most of us can't access the frontier AI only some people can. So how do you see this rolling out? I think that I don't have a great answer to that question because I don't think that it's written yet, right? We don't know.

1:15:26there are so many choices that we are going to have to make along the way. You know, I can tell you something about what the tech billionaires want, but just because they want it doesn't mean that it's going to happen. I mean, hell, they also want AI data centers in space, and that's not going to happen. So, yeah, I think we definitely need to be aggressively regulating this industry. And I don't just mean AI, I mean tech.

1:15:56And yes, the tech titans are going to push back on that and that doesn't mean that they're going to win. And yes, the Trump administration isn't going to want to do that. But you know what? The Trump administration is not going to be forever. So yeah, I think it depends on what we allow these guys to get away with. And isn't it interesting? So so Bernie Sanders was talking about nationalization and now Trump.

1:16:22has done this export control on Mythos and on GPT 5.6. I mean, this is unimaginable, because when Trump came in, he said, no, go on boys, just scale, scale, scale, beat China. So this is a really weird situation. I mean, do you think that they've been influenced by the AI risk lobby, or do you think that it's just cynically KYC they want to control it? What's going on? Well, I mean, Bernie's definitely, I think, been influenced by the AI safety, AI doomer, lobby.

1:16:52And then as for Trump, I think that somebody dangled a bunch of money in front of Trump and showed him something shiny. I mean, you know, they jangled their keys in front of his face. That's generally my theory of how Trump makes decisions that aren't in his immediate obvious self-interest. OK. Any final thoughts on where this is going to go? It's a very unpredictable time, isn't it? It is a very unpredictable time. I'm hopeful.

1:17:19I really am. I think that I think that we are actually going to end up finding a way to regulate these guys. The question is how and when and what consequences that like and what they, you know, what that struggle ends up looking like. And that I don't know, but the public sentiment around AI is really bad. The public resistance to building data centers is just getting stronger.

1:17:49And there is also very clearly a financial bubble here. And when that pops, I don't know what's going to happen, but it's going to be pretty interesting. Yeah. And it might pop quicker now just before the IPO with all of this fiasco. Yes, indeed. Yeah. Yeah. Especially with the SpaceX IPO not having gone particularly well. Adam, it's been an honor having you on the show. Thank you so much. And folks at home, read this book. It's a very, very good book. I enjoyed reading it. Thanks for having me. This is great to be here.