If Anyone Builds It, Everyone Dies: How Artificial Superintelligence Might Wipe Out Our Entire Species with Nate Soares

**Nate Soares** (0:00)
If there was an airplane, and some engineers came and said, this airplane has no landing gear. If you try to fly in it, you will crash and die. And the engineers building the airplane, who want everybody to fly in it, say, whoa, hold on, it's true that the plane has no landing gear. We're gonna build the landing gear on the fly and think there's an 80% chance we succeed, all aboard. You wouldn't be like, get me on that plane. People in the field can see that AI is a moving target.
They can see that the chatbots are not the end of the line. Even the optimists are saying there's like a 10% chance this kills us all. And those are the ones building it.

**Nate Hagens** (0:39)
You're listening to The Great Simplification. I'm Nate Hagens. On this show, we describe how energy, the economy, the environment and human behavior all fit together and what it might mean for our future. By sharing insights from global thinkers, we hope to inform and inspire more humans to play emergent roles in the coming Great Simplification. Today, I'm joined by artificial intelligence researcher, Nate Soares, to discuss a pretty alarming topic, the potential risk of human extinction posed by the development of artificial super intelligence. Nate Soares is the president of the Machine Intelligence Research Institute, and has been working in the field of AI risk and alignment for over a decade. He is also the author of a large body of technical and semi-technical writing on AI alignment, including foundational work on value learning, decision theory, and power seeking incentives in the smarter than human AIs. Most recently, Nate co-authored the book If Anyone Builds It, Everyone Dies, Why Superhuman AI Would Kill Us All, alongside Elias Yudkowsky. Nate's warning against the development of artificial superintelligence is akin to other existential threats such as nuclear war and runaway global heating. And as such, I feel it requires some sort of equal exploration and awareness on this chat channel as we integrate the various risks. While we have covered several macro challenges stemming from artificial intelligence, the synthesis that Nate presents here is arguably the widest boundary risk that AI development creates, which is a species level extinction and the transformation of Earth as we know it. Before we begin, if you are enjoying this podcast, enjoying in quotes, I suppose, I invite you to subscribe to our Substack newsletter where you can read more of the system science underpinning the human predicament and where my team and I share written content related to The Great Simplification. You can find the link to subscribe in the show description. With that, please welcome Nate Soares. This was a real eye opener.
Nate, great to see you.

**Nate Soares** (3:04)
Thanks for having me.

**Nate Hagens** (3:05)
Welcome to the show. You know, it's odd. It is November 11th, and I was just outside on a beautiful autumn day chopping firewood for the winter with my dogs. It's just a glorious day, and I knew this conversation with you was around the corner, and we're going to talk about serious stuff, and it's just such a polarized thing that we can enjoy the beauty of life, and then talk about its possible demise because of technology. I used a splitter and a chainsaw and an ax, and boy, we've come a long way from those tools already. So you and Eliza Yudkowsky have just published a book, If Anyone Builds It, Everyone Dies, with the it being artificial superintelligence. And more and more, I'm realizing that the future of AI or ASI is hard to separate from the central topics of this show, which is trying to prepare for society for kind of an abrupt shift to the way things have been going in recent decades in the near future. So let's start with the punchline of your book. What are the primary vital risks that artificial intelligence poses that you'd like everyone to understand?

**Nate Soares** (4:31)
The first piece to understand about the danger of artificial superintelligence is that superintelligence is a sort of a different ballgame from the chatbots of today. So by superintelligence, we mean an AI that is better than every human at every mental task.
That in particular would include tasks of developing technology, of developing better AIs. And the AIs aren't there yet, but this is the explicitly stated goal of many of these AI companies to sort of rush towards this smarter AI, which would, if they managed to keep a leash on it, automate all human labor and radically change the world. And one of the main arguments of my book is that nobody would be able to keep a leash on it, not if it's made anything with anything remotely like the current technology. And so if that is developed using anything remotely like today's technology, I think the most likely outcome is that literally everybody on earth will die.