Does intelligence need a hard cap?

Casey Newton (0:09)

This is Platformer Plus. I'm Casey Newton. The following column was created using a synthetic voice clone made by 11Laps.

In today's episode, does intelligence need a hard cap? Calls for a new kind of slowdown were the talk of The Curve, plus slinking back to X. This is a column about AI. My fiancee works at Anthropic. See my full ethics disclosure at Platformer.News slash Ethics. Today, let's talk about a new front in the conversation about whether to slow the pace of AI development. An emerging push to consider limits on just how smart a large language model can be. I spent the past weekend at The Curve, an annual conference that brings together executives from top AI labs, leaders of nonprofit organizations, government officials, and a small contingent of journalists. Inside a converted hotel in Berkeley, the group spent a few days loudly debating questions of economics, politics, and safety. And while the conference has had one eye on existential risk for each of the three years it has existed, the questions felt notably more urgent this year. I see two explanations. One is the ongoing fallout of the OpenAI hugging face incident, whose implications have alarmed people across the industry. The other is recent blog posts from OpenAI and Anthropic outlining their progress toward recursive self-improvement, AI systems that can research and train their successors, leading to ever faster releases and increased risk that the systems will escape their control.

At The Curve, I heard multiple speakers say we may want to limit how intelligent a future system is allowed to become. Depending on how far labs take any restrictions, the result could be a de facto ban on systems reaching superhuman intelligence. Whether that seems meaningful to you depends on whether you believe recursive self-improvement or superintelligence is even possible given the architecture of the current models. And a lot depends on how you would define and assess intelligence here, but it was the most striking idea I heard at a conference full of noteworthy discussions, and its airing in the relatively friendly confines of The Curve may be a preview of a more public discussion sometime soon. Over the weekend, though, these comments were made at sessions being conducted under the Chatham House Rule. Accordingly, I can't identify the speakers, but the fact that there appears to be agreement on this subject among speakers at The Curve struck me as newsworthy. So what does it mean to place limits on intelligence? The speakers I heard did not offer many details, though we can gather clues from other remarks made recently by AI executives. For one, Anthropic CEO Dario Amadei has called for some kind of speed limit on recursive self-improvement. For another, the company's responsible scaling policy, which has been copied in some form by most of its leading rivals, seeks to impose limits on the training and deployment of more powerful systems as they develop new capabilities. Other approaches could involve restrictions on using frontier models to conduct AI research, limits on how much compute is available to systems and how many copies of itself a system is allowed to run, or preventing labs from deploying models past a certain level of capability.

That last one is arguably already in action.

Notably though, any effort to restrict models advancement in this way would require enforcement capabilities that don't exist yet.

Individual labs can't impose restrictions like this, nor can individual countries and the current US government actively opposes them. There are other, easier methods to manage A.I.'s development. Anthropic has already adopted the idea of embedded evaluators, and OpenAI has said it will follow. An antitrust waiver that explicitly allows labs to collaborate on safety questions might be useful enough that it could overcome fears that it would allow leading companies to consolidate their power even further. Still, speaker's comments over the weekend suggested to me that nothing proposed so far meets their own definition of enough, including the morally binding accord AI leaders signed last week with the president. For the moment, lab leaders and the US government differ sharply on one all-important question, how close are we to actual danger? The former say we may see a catastrophe as soon as next year. The latter has vacillated between creating a de facto licensing regime for frontier models and publicly encouraging US companies to go even faster. But the Trump administration remains frustratingly obtuse on the question of where all this is headed, even as more warning signs blink red. A few days before speakers mused about the need to limit intelligence, the president was insisting that everyone start calling it superintelligence now. But superintelligence is already worth of more than a branding exercise, and the limits that matter most today are among the officials at the White House.