Dario Amodei from AI's Anthropic issues a stark warning

Pete Ross (0:00)

This is the Global News Podcast from the BBC World Service.

I'm Pete Ross, and in the early hours of Sunday, the 13th of September, these are our main stories. Three AI bosses call for the development of artificial intelligence to be slowed down due to concerns over safety. The right-wing party, Reform UK, gets another donation of nearly $50 million, the second in as many days from a cryptocurrency billionaire. And?

Salman Rushdie (0:28)

The first way of thinking about it that I had was, maybe it's not pointless for people to see what terrorism looks like. I'm going to tell the story, and the attacker doesn't get to be the storyteller.

Pete Ross (0:39)

The author, Salman Rushdie, discusses a new film about the vicious knife attack that left him blind in one eye. Also in this podcast, at least 16 people have died in a fire at a nursing home in Chile. And in Tennis, Elena Rybakina has become the US Open Women's Singles Champion.

When artificial intelligence first hit the mainstream, it was promised to be a game changer for economic growth and human wellbeing. But in a major shift, the CEO of one of the US's leading AI companies, Dario Amodei of Anthropic, published a lengthy post warning the industry to slow down before safety risks get out of hand. Elon Musk and the OpenAI boss, Sam Altman, both said they agreed with him. Last week, a couple of Anthropic executives resigned. One of them, Jacob Coxon, told the BBC that, quote, if we don't slow down at the current rate of progress, there's a strong chance that we could all die in the immediate future. Our North America technology correspondent is Lily Jamali.

Lily Jamali (1:45)

This essay from Anthropic CEO Dario Amodei has really made the rounds in Silicon Valley throughout this morning and into this afternoon here.

Nearly 4,000 words long, he lays out his concerns that the risks associated with artificial intelligence technology are very serious, even though he also says there are tremendous benefits so that it doesn't make sense to stop developing it, but that the pace of development should slow. So he proposed creating a system that, first of all, gives governments a little more time than they currently have to properly regulate this technology. And he also floated the idea of monitors that would be sort of independent third parties with the same access that company employees have to the models, so that they can keep tabs on what's happening both as the models are being developed and then later also when they're deployed.

Pete Ross (2:44)

Lily Jamali in San Francisco. Gina Neff is a Professor of Responsible AI at Queen Mary University of London and an expert on what the technology can do. Owen Bennett-Jones spoke to her and asked if universal regulation is getting closer.

Gina Neff (2:59)

They're saying regulate me with huge caveats. Dario Amodei is calling to, quote, pace the frontier of AI development, just as the company are floating a $1.6 trillion IPO.

His message is clear. Anthropic say they have such powerful technology that only they should be entrusted with it. The sentiment is both undemocratic and also cynical. But I think we all need to be heeding Amodei's warning here. The technical development of AI right now outpaces the social, legal and regulatory advances that we're going to need if we're going to use AI safely in society.

Owen Bennett-Jones (3:39)

So are you saying that Amodei, this head of Anthropic, is basically saying Anthropic's in the lead, so it's fine if we freeze developments just now? So leave me ahead.

Gina Neff (3:50)

They absolutely want to pace right at the moment when they're in the lead.

OpenAI just announced that they're putting a pause on their own IPO because it would be unsafe to do so at this particular moment. So Anthropic is going ahead. What they're calling for in this regulation though is thin when it comes to regulation. It's like regulation proposed by a high school dropout of Model UN. They would hire independent evaluators to sit in house with the company. Now we've seen that story of businesses doing self-regulation before and it always fails.

Owen Bennett-Jones (4:26)

Amodei has said rogue AI agents could take over the internet in 6 to 12 months, causing hundreds of billions of dollars in damage. What does he mean would actually happen?

Gina Neff (4:35)

Yeah, this is going on the heels of one of Anthropic's regular reports around the cybersecurity abuses and other harms that they've detected within the company.

What they can see is that bad actors around the world are using clawed tools like ordinary people do to do some bad things. And they're worried that as we move to what's called the agentic AI moment, that these agents might not be able to be controlled. They want good people. And in this missive he published, democratic people and people in the US to control AI and not other countries in the world. The report has lots of examples of China, Iran and Yemen using clawed tools to do bad things.