AI firms are going back on their safety promises

**SPEAKER_1** (0:00)
Running a business is hard enough, so why make it harder with a dozen different apps that don't talk to each other? Introducing Odoo, the only business software you'll ever need. It's an all-in-one, fully integrated platform that makes your work easier, from CRM, accounting, inventory, e-commerce and more. And the best part, Odoo replaces multiple expensive platforms for a fraction of the cost. This is why over thousands of businesses have made the switch. So why not you?
Try Odoo for free at odoo.com. That's odoo.com.

**SPEAKER_2** (0:30)
For decades, companies built their procurement and supply chains around cost and efficiency.
Today, that's no longer enough. Wars, trade conflicts and the regionalization of trade have changed the rules. Companies still need to control costs, but they also need to reduce risk and respond faster to disruptions and shocks. GEP helps the world's leading enterprises do exactly that. Through its AI-native quantum intelligence platform and services, GEP helps companies gain visibility, predict and manage risk, and make smarter capital investments across their global value chain. More than 7,000 employees across 30 offices support over 1,000 organizations worldwide. GEP combines agentic AI with three decades of procurement and supply chain data and expertise. Learn more at gep.com.

**Meghan McCarty Carino** (1:25)
The world's top AI labs are kind of slacking when it comes to safety. From American Public Media, this is Marketplace Tech. I'm Meghan McCarty Carino.
For the last couple years, the Future of Life Institute has put out a semi-annual report card for the big AI companies. The nonprofit is focused on reducing existential risk, the threat that rogue, super-intelligent AI could destroy civilization. And it asks a panel of independent experts to assess companies like Anthropic, OpenAI and Google on criteria like how well they test models before release, what their whistleblower policies look like, or if the models are causing current harms, like assisting in wrongful deaths or military operations. And the latest report shows scores slipping, even just since last winter, as companies weaken voluntary commitments. Anthropic, which has made safety a core part of the company's identity, received the best grade, but it still only got a C+.
OpenAI and Google got Cs, Metta improved its grade to a D+, and XAI got an F. Here to explain is Sabina Nong, an AI Safety Investigator at the Future of Life Institute.

**Sabina Nong** (2:55)
What we have observed is actually two really concerning trends. The first trend is we have seen that companies that are racing the hardest, including Anthropic, OpenAI, Google DeepMind, and Metta, have either weakened or avoided their prior commitments to pause unilaterally once certain thresholds have been hit. And they have instead replacing these unilateral pledges with, for example, competitor contingent conditions, or they have just completely voided it from their latest updated safety frameworks. And I think the second most concerning trend that we have been observing is that there have been a lot more real current harms that have been attributed to the AI systems that are developed by these companies. So some cases include the allegations of wrongful death and self-harm lawsuits. And we have also seen that industry have been pivoting to embracing defense contracts, while a lot of the companies previously have promised that they will not engage in military use of AI.

**Meghan McCarty Carino** (3:53)
How do you think the competitive commercial landscape for AI is playing into what you're seeing?

**Sabina Nong** (3:59)
I think this is, from my personal perspective, one of the leading reasons that companies are actually racing towards superintelligence, but also at the same time, not necessarily investing enough in frontier AI safety management. Some of the cases that we are seeing right now is that companies that are racing the hardest, especially those that are pushing the frontier through certain technical paradigms such as recursive self-improvement, which is basically using AI systems who improve their own AI systems, are actually not having very credible strategies of making sure that these technologies remain under human control.

**Meghan McCarty Carino** (4:34)
Right. I mean, we've seen companies, namely Anthropic, sort of publishing warnings about the dangers of recursive self-improvement, the need to slow things down, also having sort of published accounts of some things that have gone wrong in their training. What's the concern here? It seems like Anthropic at least is being transparent about those things.

**Sabina Nong** (5:01)
I think this is a really good observation. I think not only Anthropic, but also other companies have been calling a lot more for a certain pause to happen. But if you closely look at their language, all of these promises kind of come with a certain condition, which is that if our competitors are not going to do it, then we're not going to be doing this by ourselves. And I think there is a certain fallacy with this logic in and of itself, because these companies are actually very actively pursuing the frontier of these capabilities of the emerging AI technology, including, like I've said, recursive self-improvements, which has been commented by reviewers to really be leading us towards a irreversible loss of control risk. It is really time that companies are living up to their promises that they have set out a long time ago, where they have promised to say, like, if certain capability thresholds are going to be met, then it is the responsibilities of every developer within the ecosystem that they need to stop unilaterally right now. Instead of saying that, okay, if we're not leading, then we should not really be putting into the mitigations that should necessarily be there.