Understanding the inner thoughts of AI

**Hannah Fry** (0:00)
Welcome to Google DeepMind the Podcast. I'm Professor Hannah Fry.
What if you were to peer inside the mind of AI? You wouldn't find fully formed thoughts or intentions written in plain English, just vast arrays of numbers combining together in ways that somehow produce intelligence. How? We genuinely don't know.
And that is the problem a field called interpretability is trying to solve, mapping meaning onto those numbers, shining a light inside of the black box. In this episode, I am joined by Neil Nanda, who leads the Language Model Interpretability Team here at Google DeepMind. Thank you so much for joining me. Do you want to give us your definition of what interpretability is, and also why we need it, maybe?

**Neel Nanda** (0:44)
Sure. So, interpretability is kind of the neuroscience or the biology of AI. And it's trying to understand how these things work, often called opening up the black box. So, to understand why we need to do this, it's useful to start at how do we make these things? How do they work? And in particular, the neural networks are more grown than designed.
No one designs what a network like Gemini should look like. Instead, we have these enormous mountains of data, and we have this flexible learning algorithm, the neural network, that starts just kind of doing stuff randomly. But then we keep giving it a bit of data, and then giving it a nudge to do a bit better next time. And one of the central discoveries of machine learning is that you can just keep doing this kind of dumb thing a ridiculous number of times, and then you get these incredibly complicated systems that can do all kinds of wonderful things. But at no point in this process did someone say what Gemini should look like. It just emerged from this stacking of millions of nudges.
And I think that's quite a good analogy here to evolution. No one designed the human brain. Instead, over hundreds of millions of years, organisms were nudged, as it were, towards survival by natural selection, and these small nudges accumulated over time into the rich complexity of the biodiversity on the world today.
And the job of a biologist is essentially to reverse engineer what evolution has learned. Likewise, the job of an interpretability researcher is to try to reverse engineer what neural network training has learned.

**Hannah Fry** (2:35)
So what got you into this, then? How did you come to be part of the interpretability community?

**Neel Nanda** (2:41)
I think there were two main factors for me. A safety factor and a scientific factor. So, on the safety side, I think AI is progressing extremely fast. I think it's pretty plausible that in the next decade or two, we'll have human level AI, AGI. And I think this has a lot of potential to be extremely good for the world, but also it's a pretty dramatic change. And I think changes like this come with a lot of risks. And it's pretty core to making this responsibly, that we try to understand how to do it safely. And the more we understand about a system, the better a place we're in.
The more we can understand why it does what it does, debug issues, flag risks in advance, et cetera.
The scientific motivation is, I'm kind of a scientist at heart. I want to understand things. And I find it extremely annoying that in modern machine learning, people just don't really understand the systems.
And I know, it just seems like obviously the most important question is how do these things work? What is going on? And I get paid to try to answer this question, it's great.

**Hannah Fry** (3:59)
What was the original goal, though, of interpretability? I mean, did people ever really want to and or expect that you could properly connect up the dots from the micro level to the macro level?

**Neel Nanda** (4:11)
You know, if you ask five interpretability researchers a question, you'll probably get six different answers.
But at least in mechanistic interpretability, the subfield that I've spent a lot of time working in, I'd say there was this dream that we could fully understand the model or get as close as we could. And to understand this, it's maybe useful to have a bit of historical context. It's kind of standard wisdom in machine learning that these systems are just inscrutable piles of linear algebra. We don't know how they work. They're black boxes, but they can do things. So let's just use them. And there was a series of really exciting work, especially from Chris Oehler, then at OpenAI, finding that this wasn't true. You could do things like find a neuron in a model that led up on pictures of dogs, and another one that led up on pictures of dog ears that made the dog one light up more. And it just seemed like, ah, it could have been completely unintelligible, and we can actually understand so much.