What if Humans Weaponize Superintelligence, w/ Tom Davidson, from Future of Life Institute Podcast

**Nathan Labenz** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, I'm sharing a cross post from the Future of Life Institute podcast, featuring a conversation between host Gus Docker and Tom Davidson, senior research fellow at the Foresight Center for AI Strategy, on a topic that deserves far more attention than it currently receives, the risk of AI-enabled coups. This cross post came about after I listened to Tom's appearance on the 80,000 Hours podcast, which was also excellent. I was planning to do my own original follow-up interview, but for the second time recently, Gus beat me to it, and as always, he did an excellent job. So I thought I could save Tom some time by cross-posting, and also felt that this was the perfect episode to follow on our most recent one on AI whistleblower protections and support. At a high level, Tom's analysis is a sort of reframing of the risk that humanity could lose control to AI systems. Historically, lots of AI safety theorists have worried about scenarios in which AI systems rise up against or otherwise supplant humans as the primary architects of the future. This is a possibility that I have always taken seriously, even when it seemed unlikely. But as you'll hear, Tom shifts the focus to a highly related problem that on reflection does seem almost strictly more likely, at least in the near term. The use of increasingly powerful AIs by human actors to consolidate power in ways that would have been impossible with previous technologies and which could prove similarly devastating. Importantly, Tom emphasizes early in the conversation that he does not think that anyone at leading frontier AI companies are explicitly planning an AI-enabled coup today. Rather, the risk emerges from the interaction of powerful incentives, rapidly advancing capabilities, and the natural human tendency to want more influence to achieve one's goals. Step by step, without any single flagrantly malicious decision, we could find ourselves in a world where the traditional checks and balances of democratic society have been quietly circumvented by those with exclusive access to transformative AI. These sorts of possibilities are more familiar, and therefore perhaps less entertaining to imagine and debate. But the very real historical precedent for humans using new technologies to concentrate power is a strong reason to take this concern super seriously as well. As you'll hear, Tom walks through the specific capabilities that would enable these scenarios. AI systems that match human leaders in persuasion and strategy, superhuman cyber attack capabilities, and fully autonomous military robots that outperform human warfighters. He then also segments the threat landscape into three distinct models. First, singular loyalties, where AI systems deployed in government and military roles are made explicitly loyal to individual leaders rather than institutions or the law. Second, secret loyalties, where backdoors or hidden allegiances are embedded in AI systems that appear to serve legitimate purposes. And third, exclusive access, where a small group gains control of dramatically more powerful AI capabilities than anyone else has. One scenario that Tom describes in detail is that of a US-based AI company integrated into the military, developing sleeper agents. Those are AI systems that behave normally until triggered to act on hidden loyalties at a critical moment.
And then, if that's not all scary enough, there's the possibility that AIs could automate AI research itself, which in the most extreme case could allow an AI company to go from market leader to global hegemon by converting a small initial lead into a decisive strategic advantage. Throughout the conversation, Tom grounds these scenarios in historical precedent, from traditional military coups to recent patterns of democratic backsliding in countries like Venezuela and Hungary. He notes that the US has seen increasing polarization, erosion of democratic norms, and concentration of executive power, all trends that AI could dramatically amplify. And of course, one can't miss that the presidents of both Russia and China wield extremely concentrated power already and appear likely to do so for as long as they remain individually capable. Tom's assessment is that there's roughly a 10% chance of an AI-enabled coup in the next 30 years, up from a baseline of perhaps 2% without AI. And he sees this risk as being concentrated in the period when AI becomes extremely powerful, but before we've had the chance to develop robust governance structures. Which, if one listens to the likes of Dario, Sam Altman, and Demis, could be coming quite soon indeed, such that decisions being made today about AI development and deployment could determine whether these scenarios ultimately come to pass. The mitigations Tom proposes amount to a defense-in-depth strategy. System integrity measures to prevent secret loyalties, requirements for distributed control of military AI systems, transparency requirements for frontier AI development, and establishing clear rules that AI systems should follow the law, rather than individual commands. He also suggests that as we hand off more government and corporate functions to AI, we could perhaps program these systems to actively maintain democratic checks and balances, potentially making future societies more resistant to coups than today's. There is a lot more here, and I really think it's worth giving all these possibilities a serious ponder, particularly as a counterpoint to those who have worried about the dangers of open source models. I take those issues super seriously too, but this conversation convinced me that we need to start taking concentration of power scenarios just as or even more seriously, while the window for establishing norms and safeguards still remains open. Now, here's Gus Docker's conversation with Tom Davidson of the Foresight Center for AI Strategy, from the Future of Life Institute podcast.