The OpenAI board member who fired Sam Altman says we still can't trust AI CEOs

SPEAKER_1 (0:01)

Hear that?

SPEAKER_2 (0:02)

That's me in Tokyo learning to make sushi from a master.

Helen Toner (0:05)

How did I get here?

SPEAKER_2 (0:06)

I invested wisely. Now the only thing I worry about is using too much wasabi.

Get where you're going with SPY, the world's most traded ETF. Getting there starts here, with State Street Investment Management.

SPEAKER_1 (0:17)

Before investing, consider the fund's investment objectives, risks, charges and expenses. Visit statestreet.com/im for perspectives containing this and other information. Read it carefully. SPY is subject to risks similar to those of stocks. All ETFs are subject to risk, including possible loss of principal, and the loss of capital.

Nilay Patel (0:33)

The I'm Nilay Patel, Editor in Chief of The Verge, and Decoder is my show about big ideas and other problems. Today, I'm talking with Helen Toner, the Executive Director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life. Helen is a well-known figure in the AI safety and policy community, although she is perhaps best known for having served on the board of OpenAI when she was one of four board members who voted to fire Sam Altman. That incident, now sometimes referred to as the blip, ended with Sam coming back as CEO, Helen resigning in a new board of directors. OpenAI has since restructured entirely. But it's worth remembering what Helen and the board were supposed to do. OpenAI's nonprofit board existed specifically to make sure the company didn't betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull that lever, that accountability mechanism broke. And here we are. Now, the question of AI safety is everywhere. This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked into real companies. The CEOs of those frontier labs have since called for a slowdown and asked Washington for help.

The White House's answer was a voluntary accord, one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI's biggest competitor. Helen has seen up close what happens when safety commitments run headlong into commercial incentives. As she put it to me, if our hopes for the future depend on any one person being trustworthy, we're in big trouble. So we talked about what we might do instead, what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like, including how to think about China, which is the issue that looms large in every policy conversation about AI right now, especially now after the Jensen-Wong affair. Just for clarity, Helen and I talked before NVIDIA opened its investigation and Jensen ultimately resigned, so you're not going to hear us talk about that directly. Before we get going, do you have a favorite and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay, Helen Toner, the Executive Director of CSET and the future of AI safety. Here we go.

Helen Toner, you're the Executive Director for the Center of Security and Emerging Technology at Georgetown University. Welcome to Decoder.

Helen Toner (2:51)

Thanks so much.

Nilay Patel (2:52)

I have wanted to talk to you for 10 million years. I am so excited you're here.

You've been involved in a lot of AI safety and ethics ideas for a long time. You're obviously on the board of OpenAI during what everyone has come to call the blip. And now you're talking a lot about AI safety, which I have 10 million questions. But let's just start at the start. The Center of Security and Emerging Technology, people call it CSET. What is CSET?

Helen Toner (3:15)

CSET, we are, well, it's awesome to be here. I'm really excited to talk to you as well.

CSET, we're a team inside Georgetown, but we're really not academics. We're about 50 people. Been going since 2019 Basically in 2019, a group of us decided that in the national security space, sort of the security space more broadly, there's a lot happening, and a lot of emerging technologies that were becoming very relevant, where some policymakers who had very important jobs did not have the time, did not have the expertise to kind of get across what was going on with those emerging technologies. Top of our list in 2019 was AI already. So basically, our mission is to do analysis that is really rigorous, but also actionable for policymakers, and analysis that is both kind of grounded in the technology, and also grounded in real national security expertise and understanding and issues. And so we produce long reports, we go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff.