The AI Industry Asks Government to Slow It Down

**Nathaniel Whittemore** (0:00)
Today on the AI Daily Brief, we're talking about what some are calling the new AI pause letter Pacing the Frontier and what it says about where we are in the history of this AI transformation.
The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.
All right, friends, quick announcements before we dive in. For those of you who are looking to keep your AI skills up this summer, come check out our latest free training program. You can find it at summeradventure.ai and it is a choose your own adventure with a full range of different AI skills available to you. Come join a couple thousand AI friends and have an adventure.
Welcome back to the AI Daily Brief. Today was actually supposed to be a prerecord over here. I'm with KPMG at their annual Tech Symposium event, and we have a really fire operators cut show about understanding and maximizing token budgets coming. But then we got the latest open letter in the long lineage of AI open letters, and the sheer volume of the conversation around it, I think demanded that we dive in a little bit. Now, before we get into the Pacing the Frontier letter, we have to go back a couple of days to where we left off. At that point, basically every major company in the tech industry had signed an open letter urging the Trump administration to protect open source AI. The effort was led by NVIDIA and Microsoft and supported by several dozen companies across both big and little tech. Most notably, after a brief delay upon its arrival, OpenAI signed the letter, but Anthropic remained conspicuously absent from the petition, thus leading to the title of yesterday's show on the AI Daily Brief. The debate was, of course, raging in the context of the Trump administration narrowing in on a framework for AI safety testing. The June executive order set a deadline of August 1st for the creation of the framework, so this is the critical week to impact the policy that would govern AI deployment and model releases in the US.
On top of that, there has been a raging debate about what to do about Chinese AI, inspired first by sensational headlines around GLM 5.2 and then Kimmy K-3. There have been rumblings that the Trump administration was debating a ban on Chinese open source. And what's more, this was not just happening behind closed doors. This was clearly a live debate with multiple factions within the White House advocating their policy views in public. We got tweets on either side of it from Treasury Secretary Scott Besson and Commerce Secretary Howard Lutnick. There is, in other words, and I think this is important, a feeling of a crescendo moment right now.
The AI of 2026 is very clearly not the AI of 2025 The policy implications of this 2026 AI are clearly much more significant than anything we had before. So far, it's been fits and starts and bumbling and reaction, but a lot of that is now coming to a head in a way that has fairly big implications for how AI in 2027 happened in the United States and beyond.
Now, after I recorded yesterday's show, of course, Anthropic broke their silence and formalized their position on open-weight models with a blog post that was personally attributed to CEO Dario Amadei. Amadei noted the recent accusations that Anthropic is lobbying for a ban, writing, Anyone who has read my past writing should know that I don't regard such bans as a useful measure, but let me state it clearly so there is no doubt, Anthropic has never advocated for a ban on open-weight models. Still, while Dario acknowledged that open-weight models are a valuable public good, he reiterated his core concerns about making powerful models freely available. Indeed, he wrote that protectionist bans do nothing to address the two quote-unquote nightmare scenarios he's most concerned about. He claims his primary concern is the risk that authoritarian governments could use powerful AI to achieve total military dominance or deep repression of their people. Amadei wrote that the Chinese government is clearly the most capable in this regard, but they are not the sole concern. In this context, Dario noted that open weights are almost entirely irrelevant. The more dangerous models might actually be trained in secret and used solely by authoritarian governments for weaponry and domestic surveillance.
His secondary concern is the use of powerful AI to carry out cyberattacks or create biological weapons. In his estimation, widely distributed open weight models do carry a much higher risk than proprietary models in this aspect. That's because open weights make it difficult to apply guardrails or monitor use. Still, Amadei noted that the policy being discussed would be pretty ineffective, commenting, Banning the use of these models by US businesses does nothing to address the risks, because bad actors are unlikely to be legitimate US businesses. It would protect US. AI companies from competition, but that has never been my goal. Dario then lays out three policy proposals. First, double down on enforcement of export controls to stop the flow of powerful chips into China. Two, crack down on industrial scale distillation operations. And three, introduce mandatory safety testing for all models both open and closed. Now, as part of this, interestingly, he refined the argument around distillation a little more. He noted that the concern is that distillation allows for much more effective training runs, allowing Chinese labs to more easily catch up to the US frontier. Amadei reiterated, We should have policy interventions to deter industrial scale distillation. A blanket ban on open weights models is neither the correct remedy nor something we have called for. Now, touching on his reasons for not signing last week's open letter, Amadei wrote, I agree with much of it. Open weights expand access to the AI economy, they strengthen competition at least for some use cases, and they give customers greater control. Concerns about distillation should be addressed through targeted legal and commercial frameworks. But I don't agree with the letter's assertions that open weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true. Questions like this should be empirically answered by rigorous pre-release testing not assumed in advance. Making it clear, he concluded, to summarize my in Anthropics position, we have not and are not advocating for a ban on open weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial scale distillation, and requiring safety testing of all sufficiently capable models, open and closed.