Why "Hidden Reasoning" in Filler Tokens Changes AI Safety Forever

**SPEAKER_1** (0:00)
So imagine you're interrogating a suspect in a high stakes investigation. You ask them a critical question.
Instead of answering, they just stare at you and slowly count out loud, like one, two, three, four.

**SPEAKER_2** (0:13)
Scalling.

**SPEAKER_1** (0:14)
Yeah, exactly. They do this for a full minute, then suddenly they snap out of it and give you the exact, precise location of the hidden money.
Like what on earth were they doing in their head while they were counting? Because today, we're actually seeing frontier AI models do exactly this.

**SPEAKER_2** (0:33)
Yeah, it's wild.

**SPEAKER_1** (0:34)
These frontier large language models are performing complex multi-step reasoning that is completely hidden from the output. They're just using seemingly meaningless sequences of filler hokens.

**SPEAKER_2** (0:46)
Strings of dots.

**SPEAKER_1** (0:47)
Right. Dots or letters are just simple numbers.

**SPEAKER_2** (0:49)
Which, I mean, that presents a massive problem for anyone deploying these models in production.

**SPEAKER_1** (0:53)
Absolutely.

**SPEAKER_2** (0:54)
Because this phenomenon entirely breaks behavioral chain of thought monitoring. Like we've spent years relying on prompting models to think step by step.

**SPEAKER_1** (1:04)
So we can actually read their reasoning.

**SPEAKER_2** (1:06)
Exactly. We want to audit their logic, ensure they aren't going off the rails. But how do you audit an AI agent or secure a computational pipeline when the model is thinking in silence?

**SPEAKER_1** (1:17)
You can't. I mean, it's bypassing your security and observability layers right out in the open.

**SPEAKER_2** (1:20)
Yeah. But the solution we are exploring today is a newly developed unsupervised decoding pipeline. And it reads the model's internal residual stream.

**SPEAKER_1** (1:30)
It just bypasses the surface tokens entirely, right?

**SPEAKER_2** (1:33)
Exactly. It ignores the dots and the numbers, and it recovers these hidden intermediate values with an astonishing 80 to 95 percent accuracy.

**SPEAKER_1** (1:40)
Welcome back listeners to the Neural Intel Podcast. Let's dive into today's topic. As always, we'll focus on the technical details and implications of the technology we discuss. To stay updated on the latest in AI and ML, visit our blog at neuralintel.org, and check us out on YouTube, Apple Podcasts, and Spotify.
Keep the discussion engaging and interesting, and let's focus on the technical aspects.

**SPEAKER_2** (2:03)
We are pulling our insights today from a really fascinating 2026 paper. It was presented at the Mechanistic Interpretability Workshop at ICML.

**SPEAKER_1** (2:11)
Right. Titled Reading Between the Dots, Decoding Hidden Computation Across Filler Tokens.

**SPEAKER_2** (2:16)
Yeah. It's a collaborative effort by Kaylee Brauer, Claudio Meringue-Fordun and Samuel Marks. And they're representing research from Harvard, the Cambridge-Boston Alignment Initiative, MIT and Anthropic.

**SPEAKER_1** (2:28)
So for the architects, researchers and CTOs listening, our mission today for this deep dive is to unpack the mechanistic reality of how LLMs use these filler tokens.

**SPEAKER_2** (2:38)
We are going to explore the math behind the hidden states, dive into the tensor manipulations that prove what's happening, and discuss exactly how you can build observability tools for models that attempt to hide their internal reasoning.

**SPEAKER_1** (2:52)
Because to really grasp the magnitude of this paper, we kind of need to look back at the historical context of chain of thought prompting, right? Like, oh, key.

**SPEAKER_2** (3:00)
Yeah, absolutely. A few years ago, when researchers published the Let's Think step-by-step paper, the whole industry had this collective realization. We found that forcing a model to generate intermediate English tokens dramatically improved its performance on math and logic.

**SPEAKER_1** (3:14)
Like, it needed to talk it out.

**SPEAKER_2** (3:16)
Exactly. The assumption that crystallized from that era was that models were fundamentally bound by their output sequence.
We just assumed the model needed the semantic structure of English or, you know, a formalized language to hold a thought.

**SPEAKER_1** (3:31)
We thought of the context window as a literal readable scratch pad. Like, if the model didn't write down 2 plus 2 equals 4 in the text, it couldn't use the concept of 4 in the next step.

**SPEAKER_2** (3:42)
Right, that was the governing paradigm for a long time. But this paper completely shatters that illusion of statelessness.

**SPEAKER_1** (3:48)
It really does.

**SPEAKER_2** (3:49)
The researchers didn't just ask the models open-ended questions. They designed highly specific, controlled tasks to test hidden computation without that explicit chain of thought.

**SPEAKER_1** (4:00)
They wanted to see if the model could do the heavy lifting in its hidden latent space, right?

**SPEAKER_2** (4:04)
Yeah, provided it just had the time, meaning the token generation steps, to actually do it.

**SPEAKER_1** (4:09)
So they used four task families where the intermediate computational steps are like mathematically undeniable. First was one fact addition, a prompt like what is the age at which Mozart died plus 93? The model has to retrieve Mozart's age, hold it in memory, and then add 93