We Built Something We Can’t Control | A Warning from Top AI Safety Expert

**Brian Keating** (0:00)
The people racing to build superhuman AI say it might kill everyone. This is the man who spent a decade trying to stop that.

**Nate Soares** (0:06)
Bad news is we're in a bus that's racing towards the cliff edge. The good news is that the driver is asleep. The AI will sometimes find a way to edit the test to say you did it. Sometimes it will cover its tracks. A lot of people think, oh, the AI is a program. That's not how these AIs are. We grow them like an organism. Maybe we have 10 years, but maybe we only have 10 months. Who knows?

**Brian Keating** (0:24)
Nate Soares runs the Machine Intelligence Research Institute, and he spent over a decade on just one problem, how to build an AI that won't kill us all, that wants what we want. His new book with Eliezer Yudkowsky is called If Anyone Builds It, Everyone Dies. Now, the most important word in that sentence is the first one. Nate, take us through the title, the subtitle, and this somewhat ominous looking cover art, if I'm not mistaken. What was the origin, the genesis of this book?

**Nate Soares** (0:50)
If I remember correctly, Stuart Russell is the one who came up with the word AI alignment when we were brainstorming. The reason some people credit me is that I got into an academic paper first.

**Brian Keating** (0:58)
As an academic, that's all that matters.

**Nate Soares** (1:00)
Right. But we were all discussing what to rename friendly AI because friendly AI sounded a little bit not academic enough, and I think that was his phrase. One of the critical things about the book title is it starts with if.
A lot of people come in and say, oh, aren't you just bringing pessimism and doom and gloom and telling us we're all going to die? It's like the first word in the title is if. When nuclear physicists came and said, hey, we shouldn't launch all the nuclear weapons because that would cause Armageddon, they weren't prophets of doom like the Masonites who were saying the end of the world is on this particular day and that's in such a time. They were saying, hey, this scientific technology would have these bad geopolitical implications like nuclear Armageddon. If we do this arms race, we're going to get into this really bad situation. The bombs are probably going to be launched one day and then we would die and that would be bad, so we should stop this race. The book title is intended to be very similar to that, is to say, hey, if we do this, we're going to die, which is not saying we are definitely going to do it.
The subtitle of the book, Why Superhuman AI Would Kill Us All, our publishers actually suggested why superhuman AI will kill us all. They said, that rolls better off the tongue, it's less hedged, and we were like, no, that completely defeats the purpose here. The point of the book is to warn people that we are on a track that leads to destruction and we had better change it, and so we really insisted despite a fair bit of pushback, that the subtitle needs to indicate that this is avoidable.

**Brian Keating** (2:27)
The concern that I have is whether or not it's too late for if.
The conversation I had with Roman made me quite depressed, although I think I did push back on some of his safety concerns. In particular, you probably know, but if my audience hasn't seen the episode yet, his claim is that AI is fundamentally unpredictable. So if it's unpredictable, it's uncontrollable, and if something is super powerful, uncontrollable, and unbounded, then it's essentially a guarantee that that ASI will kill us all, not would kill us all, and that in his mind, it's sort of a done deal. Like we've gone too far. He also suggests things that you suggest in the book of AI regulation and international cooperation, which, you know, we all know how easy that is to get, you know, actors to behave in the unilaterally beneficent way to humanity. Is it too late?

**Nate Soares** (3:18)
We're not at super intelligence yet.
And, you know, that's the thing a lot of people don't get about this AI situation. The AI companies are racing to build machines that are far smarter than any human at every task, right? These companies did not start out as chat bot companies. Sam Altman says, you know, we're turning our eyes to super intelligence, the true sense of the word. Dario Modi talks about having the equivalent of a country worth of geniuses running in a data center. The founders of DeepMind have been thinking about the sort of like true general intelligence since the beginning. The chat bots are what make money. That sort of like surprised a lot of people who sort of stumbled into it. But these companies are all looking to create sort of like the real deal, like the stuff that can't just exceed individual humans, but can exceed humanity, right? We're not there yet, right? And a lot of people look at the AIs today and they're like, I don't really see how Fable could kill everybody. I can see how maybe it could like empower some hackers to do some extra superhuman level cyber attacks, but I don't see how it could kill everybody. It's sometimes hard to convey like, yeah, we're talking about where AI is going.