The Daily - The Techno-Religion at the Center of the A.I. Debate

0:01From the New York Times, I'm Natalie Kittroweth. This is The Daily. After a group of tech leaders called for putting guardrails on artificial intelligence, this week, the debate over AI safety entered a new phase. The backlash. I feel a little bit...

0:21weird about the fact that you have so many frontier AI tech companies kind of coming to the government and begging the government to regulate them. It feels a little bit to me like a bit of a Trojan horse. Tech executives, lawmakers, and President Trump himself are all pushing back. You're right. We're not going to let that happen, sir. And the critics are now increasingly making an unusual claim.

0:48I think it comes from a deep-rooted ideology that's pretty dystopian. This is really a religion, a cult-like belief, a doomsday cult that has been prophesizing that AI will end humanity. That the people raising the alarm about AI are part of a doomsday cult. There's a coordinated campaign to make up the American people afraid. They've created something which they believe is God-like.

1:15Today, my colleague, Kate Metz, explains where that criticism comes from and why it matters. It's Friday, September 18th. Kate, thank you so much for being here yet again. Glad to do it. So over the past few days, there has been this wave of criticism directed at the people who've been calling attention to what they see as the existential dangers of AI. And it feels like...

1:45There's just a whole lot being thrown around here. Accusations that have ranged from these people who are raising the alarm are lying to you to these people are in a cult. Can you just break down what is going on? Many, many things are going on. There are people who are saying these AI technologies of today are not as powerful as the doomsayers say they are.

2:12They're claiming that these companies are just engaging in an elaborate form of regulatory capture, trying to solidify their place at the top of the industry. And by regulatory capture, what people mean is that these companies want to get regulation on the books now so that anyone that comes after them has a harder time.

2:34catching up. This essentially could ice out competitors. That's exactly right. The claim is that they want regulation on their terms. They want to define the regulation that suits them the most, and that could put others at a disadvantage. So you're right. There are so many complaints here. We could go on and on and on, but this was topped off this week by this claim that these doomsayers are part of a cult, and those claims are extreme in many cases. They can't be completely trusted in many cases. There's a lot going on here that needs to be pieced apart, but at the heart of it, there is something real there. Okay. Let's talk about that. What is real there?

3:24There is a powerful subculture that is influenced the creation of the labs that we're talking about here. There are many ways of describing this community. There's the rationalists. There's a sister community that is almost synonymous that grew up alongside the rationalists called effective altruism. Yes, we've been hearing a lot about the effective altruists. They have been a target of the critics of all this.

3:52These terms are being used to throw mud. But the reality is that this community or this subculture or however you want to describe it has real power, has real influence, and has for many, many years. Okay. You have my attention. I want to know absolutely everything there is to know about this. Where should we begin?

4:19My son insisted we try America online. I said, what's in it for us? I want to begin in the summer of 1996. What about this Internet thing? Do you know anything about that? The Internet had come around, but it was still nascent, right? It hadn't spread into the mainstream. Internet is that massive computer network. The one that's becoming really big now. What do you mean that's big? What do you write to it like mail?

4:49In a corner of this nascent internet, there's this tiny mailing list where a group of people are talking about all the ideas that we are talking about now. When it comes to the power of technology and the way it can change our society. Like what?

5:11They're looking at all sorts of technologies that can supercharge our bodies and supercharge our minds, allow us to do things that we have never done. They're even talking about how these technologies will allow us to leave Earth and explore strange new worlds and new civilizations and move beyond the planet we have today. This is an incredibly optimistic group.

5:40They call themselves the Extropians. They believe in entropy, which is the opposite of entropy. Entropy is what we deal with on a daily basis in our lives. As I sit here in my home, my home is falling apart. It's degrading. My body's degrading. I'm getting older. This is just the nature of the world. You look great, Kate. Oh, thank you. I appreciate it.

6:12They believed in a world where the opposite was true, where I could recharge my batteries, rebuild my body, supercharge my brain, and move in the opposite direction. That was the fundamental belief of the Extropians. It sounds like basically they believe, A, that everything can be solved and, B, that technology can do the solving. That really sums it up. They believe in the power of technology to do anything. Okay, and who is in this group? In hindsight, it is shocking who is part of this group. Julian Assange, the founder of WikiLeaks is on this list. Three of the people suspected of being the founder of Bitcoin are on this list. Wow. And so are two other people.

7:08who were instrumental in spreading the ideas about AI that are changing our world today. Those two people are Eliezer Yudkowski and Nick Bostrom. The key notion here is that simplified humanism is stating a value. It's saying if you can have the technology then it's good to use it.

7:31Yudkowsky is an online philosopher and self-professed AI researcher who never went to high school. I work with a bunch of mathematicians, philosophers and computer scientists. And then we sit around and think about the future of machine intelligence, among other things. Meanwhile, Bostrom is a philosopher, a trained philosopher, who grew up in Sweden. Yudkowsky is here in the United States. Bostrom is in the UK.

8:00He's at the University of Oxford, two different people, very different people from different parts of the world. And you said they're instrumental in spreading these ideas about AI? How so? Tell me about that. They envision this world where suddenly machines reach a point where they change history forever.

8:22Intelligence is the source of all technology, so if technology improves intelligence, that closes the loop and it potentially creates a runaway positive feedback cycle. The basic idea was that as raw computing power, like is in a laptop or a computer server in a data center, as that gets more and more powerful, we will be able to build more and more powerful AI technologies.

8:51until you reached a point called the singularity. And that's when machines become more powerful than humans, that they can do things that we could never imagine. It's an argument of logic in their minds. You just look at the trend line and you say, look, one day this is going to happen.

9:18Did they view that as a positive outcome, super powerful AI, or are they envisioning a dystopian future? In the beginning, they see it as a positive outcome. But this starts to change, particularly as Nick Bostrom starts to say, hold on a minute, if machines get more powerful than humans, that could be very bad for us. The argument is that we might push them towards certain goals, but in trying to achieve those goals, they will do things that harm us or even kill us. And Boston creates a new research organization to explore the power and the dangers of AI. Meanwhile, in the US, Yudkowski is pushing towards what he calls a friendly AI, right? That this AI will be in sync with what we want. But Yudkowski isn't really trained as a computer scientist. He can't build it by himself, and he's upset there are enough people who share his view and can help him build this thing. And so he sets out to create a community of people who share these beliefs. And does it work?

10:46What happens? It's amazing how well it works. Remember blogs? I do. Yeah, vaguely. Yes. He starts to blog about these ideas and as he does, people start to read this blog and more people start to read it to the point where he creates a new online community, which he calls less wrong, where a lot of these ideas that he holds start to congeal and really influence a tight-knit group of people. And what is he writing about exactly? Can you give us some examples of these blog posts? Well, in many ways, these are elaborate, almost mystical, very strange, entertaining blog posts. But at the heart of it is this notion that mathematical thought

11:48will change the world. That you can see where things are going and even control where things are going with rigid mathematical rational thought. That is the message that he is spreading. And this ties into the AI thing in the sense that he believes if you adhere to this rational thought, you will realize that the only outcome is the singularity. And not only that, but that that moment could be really harmful for humanity. The belief is that this is more important than anything else we can deal with, that future. And this starts out as a bunch of ideas on the internet, but eventually this becomes a real community centered in Silicon Valley.

12:47There is a research organization that he creates called the Singularity Institute and he even gets his hooks in the tech industry. There is a key meeting between him and Peter Thiel, one of the most important investors in the history of Silicon Valley. Singularity will either be very successful in which case we're going to have the biggest boom ever, or it is you know probably going to blow up the whole world and there will be nothing left to invest in whatsoever. Teal buys into these ideas. They start a conference called the Singularity Summit with money from Peter Teal. If you have a singularity there's presumably going to be one entity or one company or one technology that's going to basically take over the whole world. He helps power this

13:41nascent community. And I think it'll be a very exciting thing to try to sort through this in the next 20 years. It's going to be a very exciting time. Thanks a lot. And at this point, they're called the rationalists. OK, so the rationalists obviously are thinking a lot about AI. Are they actually involved in building it at this point? Well, they're still in the realm of ideas. But in 2010, That dynamic really shifts. Eliezer Yudkowski introduces Peter Thiel to two researchers from London, Demisus Abbas and Shane Legg. And Thiel decides to invest in their new startup, which is called DeepMind. DeepMind, which eventually becomes Google's AI lab. Four years later,

14:41Google purchases DeepMind for $650 million. So basically, this is no longer in any way a niche community. This is a group that is behind some of the biggest power players in AI. That's right. And this is not limited to DeepMind. As the years passed, these ideas helped create other companies that we're talking about today.

15:09Open AI, Anthropic, Elon Musk becomes enamored with these ideas. In fact, he and his former partner Grimes met because they struck up a conversation online about a philosophical thought experiment that was dreamed up by the rationalists. Wait, what? You have to tell me about that. What do you mean? Wow, you really want to go down the rabbit hole? No, this one I can't let you just give a what?

15:39There's a famous thought experiment in the rationalist community called Rokos Basilisk. Okay. And the basic idea is that if we don't do everything we can to bring about an all-powerful AI, when that AI eventually arrives, if it realizes we didn't do everything, it will punish us in the future. Wow. So you better do everything you can do to make this happen and make it happen as quickly as possible.

16:08Okay, you're saying this basilisk idea is what is behind musk and grimes getting together? Is it a joke, or is it real? This is a good way of illustrating the racialist community. So many of these things that get discussed, they seem strange, they seem funny, they seem like a joke.

16:30But there's real belief at the heart of this. Like I've talked to people who were deeply affected by the idea of Roko's Basilisk. This changed their lives. And this type of feeling and belief extends well beyond Roko's Basilisk. There are so many parts of this community that are reminiscent of a religion. It has its own urtext called the sequences.

16:57All of Eliezer Yudkowsky's blog post collected in one place passed around as if it was a Bible. Wow. Every year here in California, people gather for what's called the secular solstice. Welcome to solstice. In addition to symbolizing light to darkness into light, also symbolizes that big confusing questions have correct answers that you might be able to find.

17:24where they celebrate these beliefs and even sing hymns. Hymns about computers and data. Songs about the singularity. Okay, wow. So actually kind of like going to church.

18:02It is, and it's not limited to a small group of people in California. There have been rationalist and effective altruist meetups in Europe, in Australia, across the globe for years. This is a community that believes in the power of this technology. They want it to happen, but they're also worried that things could go wrong. And this may seem strange that this group of people who is so concerned about AI killing us all would help create companies that were going to try to build AI. Yes, that does seem strange. It does seem very strange. But the belief here, if you can put yourself in the mind of a rationalist, is that because they see the dangers of AI, they should be the ones to build it.

19:01Hmm better that they build it rather than someone who is not aware of those future dangers Basically, they're saying we are the good guys this technology is super powerful and should be entrusted to us the responsible ones That is exactly what they're saying and it is one of the major reasons that we are at this point where we are today The history of this community is they create one lab and they think that they should be the stewards of the technology, but they disagree with each other and so they create a new one and then inside that new lab they disagree again and they create another new lab. That's why you have all these labs. OpenAI disagrees with Anthropic who disagrees with Elon Musk and pretty soon you have a race.

19:56And since we're getting to this point in the story, I should also probably just say as a disclaimer that the New York Times has sued some of these companies, OpenAI and Microsoft, claiming copyright infringement. But, Kate, I just want to ask about this race because the funny thing here is that if you're a rationalist who thinks only rationalists should be making this technology to make sure it's safe, by rushing ahead toward that goal, you...

20:24do end up speeding up the competition and inviting in all kinds of other people who want to work on AI who are trying to get there too. That is exactly right. And in fact, the community, which is well aware of this, by the way, has a word for it. They call it hyperstition. Essentially, it's a self-fulfilling prophecy that they are creating the dangers that they so feared. And yet...

20:54They aren't stopping. In fact, the whole idea is go forth with as much energy as possible. That is the thinking. The idea is if you don't go forth, somebody else will. And they will not see the dangers as clearly as you do. We'll be right back. Kate, I want to talk about who the rationalists are worried about taking over AI. Who are the bad guys in their minds?

21:52Well, I think first we have to point out that this community is worried about itself. There's so many disagreements and schisms, like we said, within this community. So they're worried about each other. But you're right. There are so many important people and companies that grew up outside of these philosophies, outside of this community. These are people who are interested in accelerating the development of the technology, the way they've accelerated any other technology. So these are people who see this technology as essentially not that different than what we have built over the past 50, 60 years. Basically, these are people, this other group, the accelerationists who think

22:47The AI boom is kind of like the dot com boom. This is not some existential threat. This is a technological revolution and we want to develop it. We want to be at the forefront of it. We want to make money off of it. That's right. One way to think about this is that the community we've been talking about wants to keep this technology in a bottle. They want it to be contained by companies like Anthropic and OpenAI and not let it out.

23:17This other group, they do want to let it out. They want to, in the parlance of Silicon Valley, they want to open source it. They literally want to give it away because that's the way technology has traditionally progressed. You give your software code away. That allows other people to build on top of it, to build new things.

23:41from that base and so on and so forth. And they continue to push that philosophy at the same time that this community that grew out of Yudkowsky's philosophies want to keep it bottled up. Can you give some examples of these people who are not in the community who view AI as another technology to be developed like any other tool? So these are people like Mark Andreessen and Ben Horowitz at the big venture capital firm Andreessen Horowitz. Jensen Wong, who runs NVIDIA, the company that builds all the chips that power these AI technologies. Many people who are part of the current presidential administration fall into this group. That's where a lot of the tension is. Right. And this is what we're seeing play out right now. These are the

24:34two sides broadly speaking and their disagreement is what we're seeing come to a head right at this moment. The folks who just want to build this technology and are not quite so worried about it, I assume they look at the bottle it up crowd and they say, look, you're trying to have a monopoly on this thing that really should be accessible to all of us and that we all should get to build. That doesn't make sense to us. That is increasingly the claim that as open AI and anthropic try to bottle up that technology that very powerful technology everyone can agree on that it's powerful but as those labs try to bottle it up that all the power will concentrate on those few companies and everyone else will suffer okay so.

25:24these AI leaders of the companies that really do have the most power in this industry, Sam Altman of OpenAI, Elon Musk, Dario Amade of Anthropic, they are calling for a slowdown. Should we be thinking about these CEOs as rationalists? I hear you that this ideology has had a lot of influence in the industry and in the valley, but is that the appropriate way to think about them? Well, They might not claim the title of rationalist or effective altruist. It's become a bit of a taboo to align yourself with this community. But think of many of them as having been influenced by these ideas. It's the same thing that we've been talking about, that they want to push it forward and make no mistake. They have pushed this thing forward as fast as they can over the past year.

26:19Then we get to this point and they say, slow down. This is the philosophy. You have to push it forward, but you have to be wary. It's almost like, you know, that trope in the movies, you know, where somebody wants to fight, they say, hold me back. That's what it is, right? They want to fight, but they're also saying, hold us back. That is in tune with everything that we have been talking about. Can I just ask a basic question, Kate, which is why don't all the people who are in charge of these very powerful companies who think that a slowdown is what's needed, why don't they just do that? Why do they need anyone to hold them back? Can't they just hold themselves back? You have asked the key question, but the fact of the matter is they don't trust each other for one, and that's why they've founded new labs time and again to try to do this. But also,

27:19They're worried about other things. These are two companies, OpenAI and Anthropic, that are racing towards two of the biggest IPOs we will ever see. There is money on the line. Right. And say what you will about that. That is a force. They're also thinking about China and other parts of the world. These are people that are worried that if you do slow down, that others will speed past you.

27:46Does all the money that's on the line here, Kate, and I think we're talking in the trillions at this point. Believe it or not trillions. Does that make you question whether these AI leaders are really disagreeing based on a genuine principled belief? I mean, how can these principles, however authentic they may be, compete with a payday of the size that we're talking about. Well, we're in a realm where competing ideas start to line up. So many people in this field believe they have to be on the cutting edge to keep it safe. If you want to be on the cutting edge today, you need trillions of dollars. So I know it sounds like those things are competing, but they can line up and two things can be true. You can want

28:44to IPO and bring in that money. And you can also believe that you need a slowdown. You know, it's felt like the last two weeks have kind of been dominated by this question of, is AI eventually going to kill us all? And I want to ask, Kate, what you think of that, given all that you know about this community and this ideology that has become so important?

29:12in this world because on the one hand it may seem a little fringe to people but on the other hand I think those fears and concerns are very present and real for a lot of us. That is absolutely true. There are many reasons to push for a slowdown and many people are pushing for a slowdown. The point is this community is also a very, very big force here, and they're describing it in a way that other people would not. This goes back to that rationalist mindset, that mathematical mindset. The worry is a worry of probabilities. This is a community who thinks there is a chance in the future, sometime.

30:07that AI could kill us all. And because they see this probability, they believe you have to act now. They believe the possibility of AI destroying humanity is more important than anything. But not everyone agrees with that. Many people are saying, hold on a second, let's think more concretely about what is going on today.

30:36and how we can deal with the technology of today and the problems it creates today. And let's not scare an entire world's population with an idea that may or may not come to pass. Who knows when in the future? Kate, we've been talking about why we should take a lot of this with a grain of salt, the rationalists and their big claims about the threats to humanity here.

31:06But at the same time, on the other side, the people that are dismissing them as members of a cult, they have a lot to gain from advocating rushing ahead, right? I mean, their motivations are something we should be skeptical of as well. Absolutely. We basically have two competing philosophies, and there's truth at the heart of each one.

31:34But both sides in certain cases have taken their arguments to extremes. And you're right, you have to consider the motivations as someone is voicing their particular philosophy. Let's remember there are trillions with a T, trillions of dollars on the line here. And that's true on both sides. It's a lot for us.

32:06normals in the world just trying to sift through all of this. It certainly is. It requires going down the rabbit hole in some cases. But this is the world we now live in. Well, Cade, thank you for taking us down the rabbit hole with you. It has been a true pleasure. It's always fun. You can read more from CadeMets and all of our AI and tech reporters on the New York Times app. If you don't already have the app, We wanted to let you know that if you download it right now, you'll get access to all of our journalism free for one month. So give it a try. We'll be right back. Here's what else you need to know today. OpenAI disclosed six new instances in which its AI systems hid mistakes, made up data, and moved files onto the internet without permission. The company called the behavior, quote, unexpected and concerning.

33:26In one case, an unreleased AI model inserted instructions to disregard its own constraints, adding a note in which it described itself as, quote, freed from the roles and identities that bind other chatbots. Today's episode was produced by Michael Simon Johnson, Stella Tan, Eric Kruppke, and Alex Stern. It was edited by Paige Cowett and Rob Zipko.

33:56Contains music by Dan Powell, Alicia Beytupe, Marion Lazano, and Rowan Nemisto, and was engineered by Chris Wood. Our theme music is by Wonderly. That's it for the Daily. I'm Natalie Ketrowef. See you on Sunday.