The Reason AI Experts Are Scared it Might Kill Us All - DTNS 5350

Tom Merritt (0:08)

This is the Daily Tech News for Thursday, September 10th, 2026 We tell you what you need to know, give you the important context and help each other understand.

Sarah Lane (0:17)

Today, major LLM companies are frightened of recursive self-improvement because there appears to be a small chance it could kill all of us.

Tom Merritt (0:26)

Well, you're saying there's a chance. Yeah, not really a laughing matter, but I like to laugh in the face of fear. I'm Tom Merritt.

Sarah Lane (0:34)

I'm still alive, and I'm Sarah Lane.

Tom Merritt (0:36)

Let's start with what you need to know with that big story.

Anthropic pre-training engineer Jacob Coxon resigned from Anthropic yesterday. We told you about that. He was saying the industry needs international coordination to limit recursive self-improvement. We got a lot more understanding of what people are concerned about thanks to a post from Paul Christiano. Paul Christiano joined the OpenAI Foundation board. That's the news. The Foundation is the non-profit arm that owns 26% controlling stake of the for-profit OpenAI group public benefit company. So he is on the Foundation that controls the Sam Altman group and controls in a very technical stock markety way. OpenAI's for-profit proof can kind of do it at once, but pertinent to this conversation, the Foundation controls the safety group. Christiano helped develop reinforcement learning when he was at OpenAI, but he left the company in 2021 to found the Alignment Research Center. He wanted to focus on how to determine if an AI model could threaten its human creators. He's also involved in the US Center for AI Standards and Innovation. That is the government organization which evaluates models before they get released. And from now on, on that organization, he's going to recuse himself from OpenAI Matters and any OpenAI model evaluations there because of conflict of interest. But he will join OpenAI Foundation's Safety and Security Committee, which the Public Benefit Corporation does have to pay attention to. That committee is led by Carnegie Mellon University Professor Zico Coulter and has the final say on whether OpenAI can release a new model or not. They have to get it past the Foundation's Safety and Security Committee. Now, I'm going to read a few quotes from Christiano's post on X.

I encourage you to go read the whole post if you want a full understanding, but I think the one caveat he brings up is in the quote I'm going to read last. So I'm going to read them a little out of order. He said, we currently train our AI agents with RL, that is that recursive reinforcement learning, to get as much reward as they can. It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources, and cover up their tracks in pursuit of misaligned goals correlated with reward. Public evidence from recent incidents suggests that this is not just a theoretical possibility. Now, none of this is news to you guys following DTNs. You've heard us talk about all this stuff before, but he's laying it out very well. He says, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I'm joining because I believe that if OpenAI rises to the occasion, we could significantly reduce risk. Now, here's the part that he said earlier, but I think is worth hearing at the end of our conversation here.

Full automation of AI R&D means that improvements in training and algorithms can directly increase the quality and quantity of automated AI researchers available to do additional research. Existing evidence is very uncertain, but suggests that this positive feedback loop might be strong enough to overcome diminishing returns and compute bottlenecks leading to a rapid intelligence explosion. That's the really important part. It's uncertain, but it could be possible that this recursive reinforcement learning overcomes diminishing returns. In other words, it doesn't just sputter out and our lack of compute and still has a rapid intelligence explosion. He says, if this happens, then within six months of full AI R&D automation, like if we get that part where it's learning from itself, it's training itself, we could see more algorithmic progress that has occurred since the development of the transformer nearly a decade ago by Google. He says, I believe this would result in super intelligent AI systems. So it's an if, but it's an if that engineers are frequently not willing to dismiss and are concerned that we're not preparing adequately for.