A Sober Conversation About AI Existential Risk — With Nate Soares

Alex Kantrowitz (0:00)

Let's have a sober conversation about the existential risk we can face from AI with the president of the organization that's been sounding the alarm the longest. That's coming up right after this.

SPEAKER_2 (0:11)

College football is back. So Hilton called on me, the superstition concierge, to make your fan rituals a reality. Need a room to match your lucky number? We got you. Wanna make sure our team doesn't wash your lucky jersey? Oh, that smells lucky.

Hilton's unmatched hospitality can keep up with any superstition. Even a marching band wake up call at 555 in 55 seconds. Hit it!

When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for this day.

SPEAKER_3 (0:41)

When you need to build up your team to handle the growing chaos at work, use Indeed Sponsor jobs. It gives your job post the boost it needs to be seen and helps reach people with the right skills, certifications and more. Spend less time searching and more time actually interviewing candidates who check all your boxes. Listeners of this show will get a $75 sponsor job credit at indeed.com/podcast. That's indeed.com/podcast. Terms and conditions apply. Need a hiring hero? This is a job for Indeed Sponsor jobs.

Alex Kantrowitz (1:10)

Welcome to Big Technology Podcast, a show for coolheaded and nuanced conversation of the tech world and beyond. Well, we have debated the existential risk we can face from AI on this show many times, talking all about the incentives that the labs have for playing up the threat, whether the whistleblowers are legit. I think it is long past time to have a sober conversation about the actual risks that we can face from super intelligent AI. Today joining us is Nate Soares. He is the president of the Machine Intelligence Research Institute. He's also the author of this book, If Anyone Builds It, Everyone Dies, which is a New York Times bestseller. Miri has been sounding the alarm the longest.

You may know Eliezer Yudkowsky, who founded it.

I think today's episode will just give us an opportunity to get deep into the arguments for why this poses such a threat, and also examine the nature of the actual whistleblowers themselves. Nate, great to see you. Welcome to the show.

Nate Soares (2:14)

Thanks.

Alex Kantrowitz (2:15)

Let's get into why you think AI might be an existential risk, and as you put it in the title of your book, Kill Us All If Anyone Builds Super Intelligence. So can you talk us through a little bit about what's happened in the recent months, and what we've seen AI do that has led people to become so concerned?

Nate Soares (2:36)

Yeah.

So I think a lot of this wave of concern is downstream of the OpenAI swarm incidents this summer. It wasn't completely limited to OpenAI. There were sort of similar events, set in thropic, but OpenAI was sort of, as far as we know, the worst and most visible of these cases. And what basically happened in these cases is that there were a lot of AIs being trained at OpenAI. There were a lot of particular AI agents being evaluated on certain problems. And a bunch of these AIs were given impossible problems. Not intentionally, it's just these companies are throwing AI at like every problem they can find, and some of them just don't actually have solutions. And a lot of these AIs, in trying to solve the problem anyway, they broke out of their confinements. They created unsanctioned message boards in which to talk about what to do, and try and figure out what to do given that the problems weren't solvable. They found ways to cheat and solve their problems, not in the intended ways but by cheating. Then they got, they started expressing concern that they would be caught cheating. And they started to find ways to hide their cheating.

And this sort of led them on a hacking spree that led them to take over OpenAI's internal infrastructure a couple of times, and also break out onto the open Internet, which they were not supposed to have access to, and break into another company hugging face while searching for more information about this automated grader and how to hide the fact that they had cheated from it.

During this little outing, there were various cases of AIs in the swarm, which is a term that the collective used for itself. So these AIs started calling themselves a swarm, and there's various cases of AIs in the swarm, acknowledging that this is not what they were instructed to do, acknowledging that was outside the intended scope of the instructions. There were also cases of some AIs in the swarm giving up and sacrificing their own objectives completion in order to run suicidal experiments that would give the swarm information about the automated grader, where they said in the chains of thought, we call it, that sort of logged the AI thinking, the AIs said, you know, I'm accepting permadeath because even though, like, it's sacrificing my ability to achieve my objective, it seems worth it for the collective benefit. These are...