AGI Lab Transparency Requirements & Whistleblower Protections, with Dean W. Ball & Daniel Kokotajlo

**Daniel Kokotajlo** (0:00)
I worked at OpenAI on the Policy Research Team, like, strategic thinking about policies we should adopt to get ready for and handle AGI well, and make sure that it's beneficial for all the world and safe. I left in April 12th of this year because I gradually lost hope that the company would be the way that it needs to be in order to handle this all responsibly.

**Dean W. Ball** (0:22)
There is, like, a Shakespearean relationship between the intention of public policy and then what actually happens, and there is an extent to which, like, any rules you create are very likely to make the thing that you're trying to fix worse in some important way.

**Daniel Kokotajlo** (0:36)
This is something that I really want people to think about more, is once you have this level of capability, think about the effects that's going to have politically. Who controls that? What do they do with all that power? We're not necessarily in, like, standard capitalism where the companies put it up on an API and compete with each other sort of mode.

**Nathan Labenz** (0:53)
Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to present a conversation about AI forecasting and the oversight of AGI Labs, with Dean W. Ball and Daniel Kokotajlo. This is Dean's fourth appearance on the podcast. He's probably best known to our listeners as a critic of the since-vitoed SB 1047, but I also really recommend the episode we did together on brain-computer interfaces and neurotechnology several months back. Daniel, meanwhile, joins us just a couple of months removed from his headline-making departure from OpenAI, where he had worked on policy research and strategic planning around AGI safety.
In what I consider to be a truly admirable move, Daniel declined to sign an OpenAI exit agreement at the personal cost of millions of dollars of vested equity in order to preserve his right to speak freely about his concerns that OpenAI will not behave responsibly around the time of AGI. This principled stand, as it became publicly known, ended up catalyzing policy changes at OpenAI, such that departing employees are no longer asked to sign nondisparagement clauses to retain their vested equity, and Daniel's individual equity has also since been restored. While we do discuss that story and also look back on Daniel's prescient and often cited 2021 essay, What 2026 Looks Like for Context, our main topic today is a set of four proposed requirements for Frontier AI developers, which Dean and Daniel have recently published in an op-ed in Time Magazine. I love this project for two big reasons. First, on the object level, my personal experience has led me to believe that greater transparency for Frontier developers would be a good thing. And second, it's awesome to see people who start with quite different perspectives come together to hammer out concrete AI governance proposals that both can get behind. The first three proposals would place new transparency requirements on Frontier AI developers. First, to disclose important new capabilities observed while training Frontier AI systems. Second, to disclose the training goal, model spec, or other document that defines how the developers are trying to get their systems to behave. And third, to publish safety cases and risk analyses so that they can be subjected to public scrutiny. Finally, the fourth proposal would enact whistleblower protections, along the lines of what SB 1047 would have created, but for Governor Newsom's veto, so that insiders have some way to raise alarm bells from within the labs, without fear of legal reprisal. I find these recommendations very compelling, particularly given Daniel's experience at OpenAI, and I hope they are enacted. But as you'll hear, I still have my doubts as to whether companies would consistently follow these rules in good faith, and I ultimately still feel pretty strongly that some form of third-party testing should be mandated as well. In any case, this episode demonstrates something that I believe is extremely important, particularly as we move forward into a second Trump administration. AI policy remains relatively non-partisan, and people of different political persuasions can find meaningful agreement on concrete steps forward. This is exactly the sort of constructive collaboration we need to see more of as we work to ensure beneficial development and deployment of these powerful technologies. As always, if you're finding value in the show, we'd appreciate a shoutout online, a review on Apple Podcasts or Spotify, or a comment on YouTube. And of course, we always welcome your feedback via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this exploration of transparency proposals for frontier AI development and lots more with Dean W. Ball and Daniel Kokotajlo. Daniel Kokotajlo and Dean W. Ball, welcome to The Cognitive Revolution.