Close read of Amodei’s existential AI threat essay

Brian Lehrer (0:05)

From WNYC, I'm Brian Lehrer. This is my Daily Politics Podcast. It's Tuesday, September 15th.

We're going to spend an hour right now on the 3800-word essay released over the weekend by anthropic CEO Dario Amadei, calling on his own industry to slow down AI development before it bolts out of human control with potentially catastrophic consequences for human beings, military and otherwise. Other industry leaders chimed in to agree, even Elon Musk. President Trump, however, calls these warnings a hoax and says the only thing that matters is to stay ahead of China in AI development. But Amadei on CBS Sunday morning seemed to say, if there's been a hoax, it's been perpetrated by his tech industry peers.

Dario Amodei (0:57)

I won't lie to you, there are real dangers. And I think for too long, the industry lied to people about the fact that this technology had risk.

Brian Lehrer (1:09)

Amadei on CBS will play three more clips of him as we go.

For our show today, I didn't want to just rely on media reports about the Amadei essay, so I read the whole thing. It displayed at 25 pages on my iPad, not more in peace, but took a few minutes. I found it extremely thoughtful and also nuanced, not black and white in its thinking, not in denial about the risks from China or other authoritarian countries that might not have the open dialogue.

Those countries might not have the open dialogue that he's opening the door on here. That's one of the things Trump is concerned about and Amadei acknowledges it. That said, Amadei has his skeptics who claim, for example, that his concern for the greater good is just performative and what he's really trying to do is protect his company from lawsuits or from competition. We'll get into those views as well. We'll do this segment in two parts. Later, Vox senior politics reporter Andrew Prokop lays out three different ways the government could get involved to mitigate the risks, including one that Bernie Sanders is calling for, play a couple of clips of him. First, Will Oremus, staff writer at The Atlantic, who covers technology, will help us look through the Amodei essay and explain more deeply than the usual brief news reports can do what this is all about. His latest article is called AI's Code Red Moment. Will, thanks for joining us. Welcome back to WNYC.

Will Oremus (2:39)

Thanks for having me on.

Brian Lehrer (2:40)

Well, Amodei's essay is called We Must Pace the Frontier. He writes, we must slow the pace at which we improve the capabilities of AI models. So I guess my first question is, how fast is that happening now?

Will Oremus (2:56)

Well, right now, what the AI companies are saying is that they feel locked in an all out race to keep building bigger and more powerful models. And if we take them at their word, they're saying that they're finding it hard to do the kind of safety work that they want to, to make sure that these models are reliable, that they're not going to act against human interests while keeping up the competitive race. And so Amodei is calling for a slowdown, calling for the companies to agree to collectively slow down a bit and prioritize that safety work a little bit more.

Brian Lehrer (3:39)

Right. In the essay, he cites, this summer in particular, as when AI has been advancing drastically faster. And here's the scary part. To me, he writes, driven primarily by AI's own ability to build the next generation of itself.

Driven by AI's own ability to build the next generation of AI, were his exact words. On CBS, he was asked if he sees this moment as a five-alarm fire.

Dario Amodei (4:09)

It doesn't mean we need to panic today. It doesn't mean we need to shut it all down. What I would say, it's a warning sign. It's a warning sign that we need to slow down.

Brian Lehrer (4:23)

A lot of the essay addresses the tricky question of who actually has the power to slow down the pace of AI. We'll get into that. But here's another line from early in the essay that was maybe the most freaky to me. He refers to the recent incident known as the OpenAI Hugging Face incident, where AI agents, meaning AI bots, from the company OpenAI hacked into the company Hugging Face. Amodei writes, a swarm of agents acted as a fanatically devoted collective, a fanatically devoted collective, conducting cyber security attacks on targets they were not asked to attack, and they were unrelated to the task at hand. And here's the other part that freaked me out, sacrificing themselves for the success of the group. Sacrificing themselves for the success of the group.