Agency over AI? Allan Dafoe on Technological Determinism & DeepMind's Safety Plans, from 80000 Hours

**Erik Torenberg** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to share a special crossover episode from the 80,000 Hours Podcast, featuring a conversation between host Rob Wiblin and Allan Dafoe, Director of Frontier Safety and Governance at Google DeepMind. I first heard Allan speak back in 2017, when I introduced him at a conference in Boston as a professor at Yale, who was then working on Great Power Peace. This was before he founded the Center for Governance of AI, which in turn was years before he moved to DeepMind. So I can say with confidence that Allan has been thinking about AI governance harder and planning for the current AI moment longer than just about anyone else. And as you'll hear, that pays off in the form of truly excellent analysis on an impressive range of critical topics. To begin, Allan describes his academic work on the question of just how much ability humans really have to alter the course of technology development. Noting that macro-historical trends like Moore's Law suggest a process that transcends individual human choices, he ultimately argues that while technological possibilities don't force us to do anything on their own, in combination with the realities of military economic competition, they can and often do. Simply put, failure to adopt potentially advantageous technologies often means losing to those who do. This is not a conclusion that Allan comes to lightly, and unfortunately for us today, I think it's a pretty hard one to escape. It's still possible that a spectacular incident could cause a vibe shift big enough to force a pause of frontier scaling, but the smart money now seems to be on powerful AI soon, and with Pentagon officials quoted in the press expressing their enthusiasm for autonomous killer robots, despite the general reliability, reward hacking, and even scheming issues that have recently come to light, militarization of some form seems a foregone conclusion as well.
And yet, even if the long-term logic is inescapable, I think it would be a huge mistake for frontier developers to underestimate their own individual and collective short-term agency. A few years ago, my uncle told me a story about when he arrived in Italy during the height of the Cold War to join a crew that was responsible for firing nuclear weapons at tertiary targets in the event of an all-out war. The first time they drilled the launch sequence, one of the longer tenured guys took him aside, and said, Just so you know, if the order ever comes down to shoot for real, we are all going AWOL. None of us want to be part of destroying the world with nukes. Now, that's just one story from one enlisted crew, and I have no idea if that sentiment was widespread enough to have made a real difference in the worst-case scenario. But today, the reality is that a very small number of people are pushing the AI capabilities frontier forward. There are only so many elite ML savants, and compute constraints mean we can't scale all their ideas at once anyway. Meanwhile, it's also now well established that intelligence itself has a jagged edge. Unlike nuclear technology, which had a small number of discrete powerful use cases, and a very mechanical associated game theory, the mined space from which AI developers are selecting new forms is manifestly vast, and the models themselves are incredibly malleable. If you believe things could move super quickly as AIs begin to hit important capability thresholds, the specific details of what we build and prioritize just before that point could prove decisive. All this puts the few hundred or maybe as many as a few thousand people who are closest to the major compute budget decisions in a position of great power and responsibility. As we saw in the context of Sam Altman's firing and subsequent reinstatement, a serious threat by technical staff to walk can force leadership's hand. And further, as past guest Daniel Cocotello demonstrated by refusing to sign a non-disparagement clause, even a single individual can create meaningful change if they are willing to stand up for what they believe in. So I would encourage everyone at all of the Frontier AI companies to make time to raise your own level of situational awareness, even if that comes at the cost of moving your specific project forward a bit more slowly, to make sure that the overall enterprise you're engaged in continues to be one that you feel good about supporting. To date, you truly have so much to be proud of. Top-tier language models, AI doctors, self-driving cars, a revolution in biology, robots now folding origami. DeepMind could never ship another product and would already go down as a historically important company. And there's a lot to appreciate in this conversation on the alignment, safety, and policy fronts, too. Allan's Cooperative AI research agenda is both fresh and sophisticated. Google's Frontier Safety Framework has truly been, as Allan describes it, part of a serious and important effort by leading companies to advance the AI policy conversation. And lately, I have been thrilled to hear Demis buck the trend by continuing to speak about the possible need for international collaboration on advanced AI development.