#189 - Nate Soares - Why Superintelligent AI Could Kill Us All

**Nate Soares** (0:00)
The most likely outcome if we do this is that we die not because they hate us, but because they don't care about us at all. We're racing to replace ourselves as the smartest creature on the planet. I'm worried about these companies making AIs that can make smarter AIs, that can make smarter AIs leading to recursive self-improvement that could kill literally everybody on this planet. And I'm worried that this could happen in a side of three years. We're doing our best to make these AIs safe, and there's a 75 to 90% chance we succeed. Only a 10 to 25% chance that this kills every single human, right? And I'm like, that's crazy. These guys have no idea what they're doing.
They don't have blueprints, they don't have plans, they don't have engineering designs, they're cowboys, they're yoloing it.

**Peter McCormack** (0:39)
I feel like I'm compelled to get on the other side of this table and join you. Because if you're right, then we lose everything.

**Nate Soares** (0:49)
The bad news is that the bus is careening towards a cliff. All right, but the good news is that the driver is asleep.

**Peter McCormack** (0:58)
This show is brought to you by my lead sponsor, IRON, the AI cloud for the next big thing. IRON builds and operates next generation data centers and delivers cutting edge GPU infrastructure, all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more, which is I R E n.com. Right, Nate, you wrote a pretty provocative book. If anyone builds this, we all die or everyone dies.
I know you've probably been asked this a lot of times, but just outline your thesis.

**Nate Soares** (1:36)
AI companies are racing to make machines that are radically smarter than any human. These AIs are grown like an organism. They're not programmed like old school computer programs. We don't put in a prime directive. They don't have to do exactly what we say. They do their own weird thing. They already have the opportunity and the means to escape labs and replicate themselves and start pursuing their own weird goals. There's not smart enough to do that yet. AIs today are safe because if they try to take over the world, they would fail, not because they're the sort of entity that never would try if they could.
Humanity is racing to make machines that are much, much smarter than us. We're racing to replace ourselves as the smartest creature on the planet.
That's a crazy thing to be doing on its face. If you look at the technical details about how we don't know how to make these AIs care about us, the most likely outcome if we do this is that we die not because they hate us, but because they don't care about us at all. They go off and turn the whole world into data centers and use up all the resources that we were using to grow food.

**Peter McCormack** (2:51)
Was Methos the first warning sign of approaching something that is a little bit more than humanity can handle?

**Nate Soares** (3:00)
I mean, if you were paying attention, ChatGPT was a warning sign. If you were paying attention, the attention is all you need paper was a warning sign. But this is another clear warning sign that has started to get even more people on board and I expect we'll see even more and more clear warning signs going forward.

**Peter McCormack** (3:16)
Okay. The book you wrote, help me understand because I don't want to test it. I want to test your thesis with you and I know other people, there's been criticism but there's also been people who've agreed with you. I've got a foot in both camps.
But is it testimony or is it persuasion? Just to lay that with the second part, you should probably introduce and explain your first book. Did the first book compel this book?

**Nate Soares** (3:44)
What do you mean by my first book here?

**Peter McCormack** (3:46)
The Guilt Shame.

**Nate Soares** (3:47)
The Guilt Shame book? No, that was a whole separate thing. That was actually just collected from a series of blog posts I wrote back when there were still bloggers.

**Peter McCormack** (3:55)
But do you understand why I've connected the two?

**Nate Soares** (3:57)
No, not really.

**Peter McCormack** (3:58)
Because you talked about what you should do, and you should live how you want to be. You don't have to do these things, but you're in a world now where there's a should.

**Nate Soares** (4:15)