How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning

Nathan Labenz (0:00)

Hello, and welcome back to The Cognitive Revolution. Today, my guest is Eric Bigelow, member of technical staff at Unicorn mechanistic interpretability startup Goodfire. The story behind this episode is unique. After recently speaking with Bronson Shane of Apollo Research, who explained that even with access to models' internal chain of thought, it is still often extremely difficult to determine how a model will decide to act, I asked Claude to survey the literature to see what the field as a whole understands about how these critical decision tokens are chosen. One name kept coming up in that thread, and it was Erik. Erik recently completed a PhD in the Harvard Psychology Department with a dissertation titled Toward a Cognitive Science of Large Language Models. And while some of his academic colleagues initially questioned his pivot to focus on LLMs, the fact that even elite academic institutions are now willing to engage with AI's as a sort of mind strikes me as very notable indeed.

We start today with a survey of Erik's work over the last couple of years, beginning with his 2024 paper on Forking Paths in Neural Text Generation, in which he conducted a massive resampling experiment at every token of a model's chain of thought to identify the critical tokens where uncertainty around the final answer suddenly collapses. As so often with LLMs, some of the findings were intuitive, while others were quite surprising and at times difficult to interpret. Interestingly, Erik says that a system's decisions are actually taken not so much by the model itself, but by the process of sampling from the model's final token distribution. Which means that while the stochastic parrot paradigm should definitely be retired, stochasticity still plays an important role in model behavior.

Erik frames much of what we are seeking to understand about LLMs in terms of in-context learning, emphasizing that few-shot prompts are just one clear example of how much runtime context can shape model behavior. His bet is that this flexibility means that improved harnesses and memory systems will outcompete per-user fine-tuning, and thus that our focus of study should remain on major foundation models. Along the way, we also discuss why 7-8 billion parameters currently seems to be the interpretability sweet spot, why Kimi K3 was a step function for studying frontier-level coding and reward hacking, and why, for now at least, interpretability outside of frontier model companies does meaningfully depend on Chinese open-weights models. We also get Eric's thoughts on why chain of thought is getting so weird, including his perspective on the chain of thought dialect and the strange language that Astra uses to communicate with sub-agents, plus his pro tips for using Goodfire's Silico research agent, and how he hopes future AI product interfaces will begin to resurface model uncertainty to users.

The bottom line from this conversation and others I've had recently on the future of AI monitoring and control is that chain of thought is increasingly unreliable, and that because these issues seem to correlate not just with RL training, but with the power of the raw pre-trained model itself, this trend might be quite difficult to reverse. That in turn means that an awful lot, including our ability to understand why agents take the actions they do, may rest on interpretability. Erik is a pioneer of this research, but as you'll hear, tons of work remains to be done.

With that, I hope you enjoy this overview of what we know and how much we have left to discover about how AIs decide what to do. With Erik Bigelow from Goodfire. Erik Bigelow, member of technical staff at Goodfire, welcome to The Cognitive Revolution.

Eric Bigelow (3:58)

Thank you very much, Nathan. Excited to be here.

Nathan Labenz (4:01)

Yeah, I'm excited for this conversation too. I think this might be the first time in the history, there's been like 400 episodes of the podcast at this point, that I went to AIs looking for an answer for a very specific question and came back with a name and that name was yours. So the kind of upstream conversation I had of this one was with Bronson Shane at Apollo, who has the dubious distinction perhaps of having read maybe more chain of thought reasoning than like any other living human, at least any living human who can talk about it all publicly. And one of the big takeaways I had from that conversation was he said, you look at these chains of thought, and the models are thrashing around, doing a sort of linearized tree search of possibility. And then at some point, they make a decision. And even having read all the chain of thought that came before that, it's often like very not obvious why they made the decision that they made.