Anthropic Researcher Says AI Has Over a 10% Chance of Killing All Humans

Nathaniel Whittemore (0:00)

This week, an AI researcher went mega viral announcing his resignation from Anthropic, arguing that both it and OpenAI were effectively gambling with our lives. Another still employed AI researcher chimed in to agree and decided to add that he thought that there was greater than a 10 percent chance that AI kills us all. Now, doom prognostications are nothing new around AI, but something has shifted to make the message hit different this time. 200 million views on X and dozens of mainstream media outlet interviews later. Today, we're going to unpack what changed. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

All right, friends, quick announcements before we dive in. Such as, for example, the multiplayer AI sprint for teams. If you haven't yet, this is my big prediction for where I think agents are going this fall, and as a totally free four-week self-directed sprint that you and your team can do to get out ahead of it. Last note, today is a main only type of episode. The plan is to be back with our normal headline's main breakdown tomorrow.

Yesterday, a pair of posts on X escaped their proverbial containment, jumping aggressively from the AI community to dominate discourse even in the broader world. We're going to discuss those posts, the issues that surround them, the responses, and the underlying concern. But first, I want to make one request. Anyone who has interacted with modern media in any way, shape, or form, will feel on some level how much we are pushed to feel outraged. In the world of algorithms, different political positions are not disagreements to be discussed, but legitimate reasons for loathing the people who hold those different opinions. This is in large part shaped, I believe, by the easy equation of people being angry means they spend more time on your app. But the net result is a lot of us feeling a lot more angry all the time and not being particularly willing to engage with people who think differently than we do. When it comes to AI, this phenomenon is cranked to 11 Part of that is that the stakes are presented as so dramatic. Case in point, I am literally talking over a mainstream article whose headline is Anthropic Insiders Warn AI Could Kill All Humans. And part of that is because this particular debate is not about the facts of today, but what might be in the future. It is, in other words, an unwinnable debate, where the opposing positions, whatever they may be, are by definition unfalsifiable. That means all we have is the argument, and so the argument gets intense. So my request is to try, hard as though it might be, to not succumb to the instinct to outrage, to listen to the other side without being angry, even if that listening produces no change in what you believe. The more calmly and thoughtfully we can have this particular conversation, the better I believe the likely outcomes. I know this is not easy. In fact, I'm sure many of you are already feeling your blood boiling simply by me applying equivalents of both sides. When you think it's insane, either A, that I could countenance the deniers when the stakes of this crisis are literal civilizational collapse, or B, that I could coddle these doomsday zealots who have no proof to back up any of their positions. And so with that dramatic beginning, let's actually talk about what happened.

Like I said, two posts on X this week went absolutely giga viral. The first was from a researcher named Jacob Coxon. He wrote, I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving super intelligence and gambling with our lives. Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains and progress is not slowing. The people building AI earnestly believe that it could kill us all by the end of the decade. That is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible. But I hear the same people express fear privately. No other human activity poses this level of danger. A common response is, if they truly believe this, then why are they still building it? At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to get there first. They believe no one else will act responsibly, and they must do it themselves despite the risk. Accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company's slack. Attempting to speed run alignment should require extraordinary confidence that there are no better trajectories available. I am optimistic about the potential for coordination. Warning shots like the hugging face attack have made pacing agreements between US labs more viable. I don't feel we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities. If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a super-intelligent RL run without a rigorous understanding of its mind? Should you put your head down because it's happening anyway, or take this moment to call for different conditions?