Even Other AI Labs Are Rallying Around Anthropic’s Slowdown Proposal

Nathaniel Whittemore (0:00)

This weekend, the AI discourse took a major step forward, not only when Anthropic CEO Dario Amodei released a new essay called We Must Pace the Frontier, but when the leaders of most of the other labs came out publicly to support it. More than we've had before, the letter contains a set of specific proposals, that if nothing else, give us something much more tangible to debate than the vagaries of general AI risk. It's early, but it feels to me like a major inflection point, like the labs recognizing that the days of them being the sole arbiters of how fast AI moves are coming to a close. And within that, the negotiations on what the next phase means beginning.

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

Welcome back to the AI Daily Brief. Quick note, if you have been anywhere near basically any media source, you will know that the conversation is entirely around this new essay from Anthropics CEO Dario Amodei. So unsurprisingly, this will be a main only episode. Another coverage right now is heavily slanted towards some of these big political societal safety type of questions. But in between, I'm trying to give you as much practical stuff as I can to keep the balance. This is simply where the industry discourse is right now and what I think is important for everyone to understand and have a stake in the conversation around.

Sometimes in AI, there are a million things going on, and trying to keep track of it all without sinking is like skipping rocks across a pond. In other cases, there is just one thing, one conversation that is absolutely dominating, and that's what we had this weekend. The AI safety discourse which ratcheted up last weekend, crescendoed on Saturday with a new 3,000-word blog post from Anthropic CEO Dario Amodei called We Must Pace the Frontier. It is the clearest and most specific call yet for a shift in how the frontier AI labs operate and attempt to slow the pace of development of AI to something more manageable. As you might imagine, the piece absolutely dominated discourse in the AI and extended political circles over the weekend. And part of that was due to how it was received by other AI leaders. The post begins with a recitation of Dario's argument for why AI is worth the risk. Dario says that he believes that AI could dramatically raise the quality of human life, that it could accelerate economic growth, cure most major diseases, and as he puts it, usher in a renaissance of democracy and freedom. He personalizes it by pointing out, as he has in essays past, that his father died of a disease that was cured just a few short years later. Carefully wielded, he writes, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity.

However, it also comes with risks like loss of control, misuse in cyberattacks and bioterrorism, and the potential for significant economic disruption. He writes, along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic, not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers while building it too fast is reckless. We have sought, he says, a middle way to show that it's possible to build carefully and succeed commercially and to make safety something on which AI companies compete.

However, he continues, two recent developments have changed his perspective. The first is an acceleration in AI development due to the early stages of recursive self-improvement, i.e. AI's ability to build next-generation models more quickly. Left unchecked, he says, RSI could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all. Dario's second opinion-changing concern was the hugging face incident, which he characterized as a quote, swarm of agents essentially acting as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand.

It's easy to dismiss this incident, he wrote, because no one was hurt and the economic damage was minimal. But in my opinion, a swarm that possessed greater capabilities, but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it's my worry that in 6 to 12 months, such a swarm could be capable of taking over the entire Internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage, and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.