Untangling Neural Network Mechanisms: Goodfire's Lee Sharkey on Parameter-based Interpretability

**Nathan Labenz** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with Lee Sharkey, Principal Investigator at Mechanistic Interpretability Startup, Goodfire, about fascinating recent work that he and co-authors have done to start moving beyond analysis of the concepts that neural networks represent between layers, and actually begin decoding how they compute within and across layers. We begin by discussing why an understanding of concepts isn't enough. On one level, this needs no justification. An approach that explains only the meaning of the intermediate results between layers, while it does take a serious bite out of the black box problem, leaves the layers themselves as smaller but still unexplained black boxes. But more concretely, research has also highlighted important weaknesses of the feature-centric approach. Conceptually, a sparse representation of features inherently loses a lot of potentially important information, which is encoded in the structure of how features are clustered together and otherwise meaningfully arranged in space. For example, in some networks, the days of the week are represented not by seven random directions in activation space, but by a set of directions that lie together in a plane, such that a simple rotation operation can act as a sort of next-day function that converts a given day of the week into the next. If history is any guide, there is presumably a lot more such critical complexity to be discovered. And so, with this motivation in mind, and taking inspiration from Sparse Autoencoder and similar techniques that effectively separate clean concepts out of their usual state of superposition, Lee and team have similarly set out to decompose a neural network's parameters, which of course are used to process layer inputs into outputs into simpler subcomponents that they hope will correspond to interpretable mechanisms that the network has learned. Their first approach published earlier this year was called Attribution-Based Parameter Decomposition. Lee describes this as a sort of model unmerging. If you imagine, counterfactually, having started with a giant mixture of experts model and then merging all the experts into a single instance of the architecture, parameter decomposition would be the process of unmerging to recover those hypothetical original experts. Personally, I visualize it as expanding a neural network, which of course has width and depth defined by its architecture, into a third vertical dimension, where each vertical slice consists of a simpler subnetwork that's presumably needed only a small fraction of the time, such that when all the vertical dimensions are again collapsed down to a single model, they all add up to the original network weights. Whatever mental model you prefer, amazingly, by constructing the loss function that incentivizes three things, first, faithfulness, or the accuracy of the sum of the sparse networks back to the original full network, two, minimality, or the idea that as few network slices as possible should be active for any given input, and three, simplicity of the discrete subnetworks derived from the process. You can actually get this to work. I really do think it's worth taking a moment to appreciate how incredible this is. As Lee says, channeling Ilya, sometimes the models really do seem to want to learn.
That said, given the complexity of the target, you shouldn't be surprised to learn that this first method was still far from ideal. In addition to being computationally expensive and requiring massive memory for all those parameter copies, training success was extremely sensitive to hyperparameter choices, and there were conceptual issues with the gradient-based way that performance was attributed to parameters. Specifically, in some cases, critical components that had approached a local performance maximum would have a near-zero gradient, and would be misclassified as unimportant. These weaknesses naturally inspired the second method that we discussed, stochastic parameter decomposition. For efficiency, instead of describing network subcomponents as sparse instantiations of the original network architecture, the new approach breaks each weight matrix into rank-1 components, which are matrix operations that read from one direction in activation space and write to another, and which can later be regrouped into larger units. Also, to better identify which subcomponents really matter, they replaced the gradient-based approach with a novel approach that uses stochastic masking to help the network learn to predict each component's causal importance, effectively identifying which elements the overall network can't do without. Overall, it's a more scalable, stable, and accurate approach, and on toy problems with known ground truth answers, it is able to successfully recover the expected mechanisms. Of course, as with any branch of interpretability, plenty of work remains. One big question is how can we algorithmically group the rank 1 subcomponents into semantically meaningful mechanisms? Much as sparse autoencoders require a feature labeling process, we still need to figure out how the rank 1 components ladder up to conceptually intuitive transformations. And of course, there will be plenty more challenges with scale. But as Lee explains, the promise of this work is tremendous. This kind of understanding could enable everything from surgical unlearning of capabilities to the identification and interpretation of novel scientific insights that models may have learned, including and perhaps especially from non-language data. All things considered, this conversation is a great chance to visualize what's going on inside neural networks, to ponder how much we've learned and how little that still seems to explain, to marvel at the fact that gradient descent and backpropagation can effectively optimize such complex architectures even under such complicated multi-part constraints, and to appreciate the combination of brilliance and plain hard work that goes into figuring all this out. Right up there with understanding our own biology, and arguably no less important or challenging, understanding how and why neural networks do what they do is one of the grand scientific challenges of our times. So I really hope you enjoy this conversation about decomposing neural networks in parameter space with Lee Sharkey of Goodfire. Lee Sharkey, Principal Investigator at Mechanistic Interpretability Startup Goodfire, welcome to The Cognitive Revolution.