A Potentially Historic AI Breakthrough

Nathaniel Whittemore (0:00)

When it comes to AI safety, the news lately has been a little bit dreary. Or at least, I should say, the headlines have been. We've been living through a new wave of panic and concern, driven in part by the reality that models are getting more capable and closer to what many view as key inflection points. It's also being driven, however, by the juiciness of the scare headlines. And yet the most interesting news in that area this week might be very good indeed.

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, Robots and Pencils, Cdata, Granola and Blitzi. To get an ad-free version of the show, go to patreon.com/aidailybrief, or you can subscribe on Apple Podcasts. And if you want to learn more about sponsoring the show, send us a note at sponsors at aidailybrief.ai.

Now, it's not all that often that a 65-minute, densely technical web article becomes the thing that everyone was talking about. And yet that's exactly what happened at the end of this week with Nick Cammarata's Neural Networks Are Already Clean. I, like many people, noticed it when OpenAI's Rune posted. When Nick told me that he had some interpretability breakthroughs, he was underselling it. I think this may be remembered as a major moment in scientific history. News so good, even Eliezer Yudkowski will have to lower his P-Doom. We can start making alignment into a normal science.

Nick summed up his article this way, Interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it.

I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic because neural networks appear to fold into arbitrary tangled knots. Years later, I started OpenAI's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had. With modern agentic tools and a much closer look, the picture looks totally different. The vast majority in models up to trillions of parameters are legible and their circuits beautiful. Many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. It may be a bitter lesson for interpretability. The simple thing at absurd scale works.

So the promise here is that we might actually be able to understand what's going on inside the processes of AI, something that so far we haven't. Today we're going to try to do a understandable version of Nick's densely technical paper, sharing, of course, both the excited and the skeptical responses to it. Hopefully you come away understanding why this could be a big moment and what cautions we still need to have.

One of the uncomfortable facts underneath all of AI is that even the folks building it don't really know how it works on the inside. The systems are trained, not programmed. You set up the conditions and the model learns its own way of doing things, spread across billions of numbers. In AI researcher Chris Ola's phrase, AI is grown more than built. So because we can't read these systems, the way we check them today is basically a very elaborate exam. We ask questions, read team and grade the answers. While that's useful, it only covers what we thought to ask. And it also introduces a set of problems, like the model behaving well, not because that's what it otherwise would do, but because it knows it's being watched. This is coming up more and more. Ironically, in some recent tests of models like Opus 55, researchers were concerned when they saw the models cheating less. The big drop off in cheating behavior brought up the question, did the model just shift away from cheating or did it realize that the humans watching it would think cheating is bad and so it disguised that behavior?

What this means is that in the addition to the race for capability, there's also another race to reach interpretability, to understand what's going on with AI before the models reach what Anthropic called an overwhelming level of power. If you followed the field, the story of the last six years went from excitement to something closer to despair. In the first phase from about 2017 to 2020, teams like Chris Ola's team, which included Nick, mapped ImageAI's detector by detector, and it looked at first like you could read a neural network like code. But then between 2020 and 2023, they started to find a problem. Many of these detectors seemed to fire for unrelated things, meaning that we couldn't really understand exactly what was going on even though at first it appeared we could. The field settled on an explanation for this that we'll get into in just a minute. When Nick's team at OpenAI used GPT-4 to explain GPT-2's detectors in 2023, only about 2% came out clean, i.e. actually interpretable in the way that we thought. Between 2023 and 2025, a bunch of different workaround strategies were tried, some of which showed some progress but which never really produced many wins.