Helen Toner: OpenAI Reflections, Adaptation Buffers, and AI in Warfare

**Erik Torenberg** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with Helen Toner, Director of Strategy and Foundational Research Grants at CSET, the Center for Security and Emerging Technology, and author of a new sub stack called Rising Tide. Helen is best known to the general public for her role as an OpenAI board member responsible for temporarily firing Sam Altman in late 2023 But she's been feeling the AGI, or at least the need for society to invest in preparation for the possibility of transformative AI, since way back in 2016 when she started working on AI policy full time. That's a full five years before joining the OpenAI board in 2021, when it's worth noting OpenAI had already launched GPT-3 as an API product, already taken $1 billion in investment from Microsoft, and was increasingly recognized by those in the know as a leader in the generative AI wave. Certainly by that time, OpenAI had plenty of access to super talented candidates for its board.
With that context in mind, and again, remembering that her blog is called Rising Tide, despite what you might have heard elsewhere, it probably should not surprise you to learn that Helen is definitely not an AI de-cell or even especially hawkish on most AI safety issues. On the contrary, she argues in early posts on her sub stack that non-proliferation is the wrong approach to AI misuse and instead promotes the concept of adaptation buffers, the notion that society broadly has a critical window of opportunity, to adapt to new AI capabilities between the time when they're first demonstrated, typically at high cost in terms of both R&D and compute, and when they later become widely accessible, typically at much lower cost, as we've recently seen with companies like DeepSeek dropping the cost of frontier reasoning capabilities. While her focus today is on other things, I couldn't resist asking Helen some OpenAI related questions, and I appreciate her willingness to engage despite having addressed these issues in multiple forums already, including especially an episode of the TED AI Show, which we'll link to in the show notes. The only truly new detail that you'll hear in this conversation is her assertion that media reports suggesting that some sort of Q-Star breakthrough in reasoning had led to the board's decision were quote unquote, totally false. But nevertheless, I think it's important that Helen and other former OpenAI team members continue to speak candidly about their experiences with the company and its leadership. As Helen notes in another of her first blog posts, everyone's timelines are dramatically shorter than they used to be. What passes for long timelines in AI circles today would have been quite short not many years ago. And given this new short timeline's consensus, scenarios like former OpenAI researcher Daniel Cocotello and team's recent AI 2027 reflect not just one of the shorter timeline forecasts, but if I'm reading between the lines effectively, a warning about how OpenAI leadership might fail to act responsibly around the time of AGI by abandoning its principle of iterative deployment, keeping the best models for its own internal use, plus maybe that of the US government, and aiming for a sort of AI takeoff via the automation of AI research. That's a warning, by the way, that's become a bit more credible this week with the news that OpenAI has indeed announced that GPT 4.5 will be deprecated from the API. All that's enough for me to feel strongly that it's important for Helen to use appearances like this to continue to remind Washington decision-makers that OpenAI's CEO was not consistently candid with its board, and also for me to applaud moves like the amicus brief recently filed to the Elon Musk First OpenAI Lawsuit by 12 former OpenAI team members, who argue that nonprofit promises were central to OpenAI's early hiring success, and that the nonprofit should not cede control of the company at any price, a development that happened after I recorded with Helen and on which I hope to do a full episode soon. Of course, the stakes are only rising from here. With OpenAI and other AI companies seeking Pentagon contracts and special legal protections, Helen's latest research out of CSET, with Rhodes Scholar and former Navy Aegis Operator Amelia Proboscow on AI for Military Decision Making, is super important and a super attempt to map out how AI systems have been and are likely to be used, and how that may diverge from how they actually should be used given their current limitations. Among many other interesting details, I was amazed to learn that some nations, including most prominently Russia, currently have published military doctrines about AI, which seem to be fundamentally out of touch with current AI systems' lack of reliability and total lack of adversarial robustness. This too is something that Washington decision makers probably can't be reminded of often enough, as they seek to develop autonomous killer robots. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just drop us a comment on YouTube. Of course, we welcome your feedback too, as regular listeners will know. While I believe that there's probably some non-trivial and irreducible risk associated with developing advanced AI at all, it's my sense that much of the extreme AI risk we face today in fact exists because key decision makers, under intense and growing pressure, seem fairly likely to make some very bad mistakes. If this show can do anything to contribute to a positive future, I hope that it can help people start thinking about those critical but avoidable failure modes sooner and better, so that we can minimize the extreme downside risk and get to live in that age of AI-provided abundance that we've been promised. If you think I can be doing a better job, I encourage you to reach out either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy this conversation. Looking back on OpenAI and looking ahead to adaptation buffers and military use cases, all amidst shorter and shorter timelines to AGI. With Helen Toner, from the Center for Security and Emerging Technology, and author of the new blog, Rising Tide. Helen Toner, Director of Strategy and Foundational Research Grants at CSET, the Center for Security and Emerging Technology, and author of a new sub stack, Rising Tide. Welcome to The Cognitive Revolution.