Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Lenny Rachitsky (0:00)

One of the big questions on people's minds these days is AI doing bad things. As someone that watches these models think and operate, why is it so hard to catch it thinking badly, trying to deceive us?

Emmanuel Ameisen (0:09)

We train Claude to be helpful, harmless and honest. Is it the case that it's behaving well, but if you look inside, actually it's like, oh, I'm only behaving well because Lenny's watching right now.

Lenny Rachitsky (0:18)

People's full-time jobs is studying the model and how it thinks, how it operates. How long has this been a skill set and a job?

Emmanuel Ameisen (0:24)

One of the most important things to realize, if you like use language models or think about them, it's just this like very bizarre fact that to make them is not to understand them. All these models do is just a lot of mathematical operations on many numbers. If you look at it, it makes no sense. You just have a list of 100 numbers, you know, the model is doing math, and then you just try to see if there's patterns.

Lenny Rachitsky (0:42)

You are figuring it out the same way we figure out how the brain works, looking for patterns. You show this image, this part of the brain lights up and you're doing the exact same thing with AI. Wow, this is so fascinating.

Emmanuel Ameisen (0:50)

There was this model we were training, it would go crazy. It would do some math, it would get to an answer, I think it was 12 It would immediately go back and say like, oh my god, the answer actually is for sure 36 It would do math again, it would get to 12 Slowly, it would start descending into madness, writing in all caps, literally saying, I'm possessed by a demon.

Lenny Rachitsky (1:07)

Humans, if you really boil it down, we're basically next token predictors ourselves. We're just trying to figure out the next thing to say, Dario is always out there, this is really dangerous, we gotta be really careful. And everyone's just like, no, he's just saying that for marketing reasons. And now it's like, wait, shit, it's really dangerous.

Emmanuel Ameisen (1:21)

Take the idea that this technology is going to keep improving seriously.

Lenny Rachitsky (1:25)

Do you think about consciousness as a part of this, or do you think AI is conscious?

Emmanuel Ameisen (1:29)

Oh, man.

Lenny Rachitsky (1:33)

Emmanuel, thank you so much for being here and welcome to the podcast.

Emmanuel Ameisen (1:37)

Thanks for having me, Lenny.

Lenny Rachitsky (1:39)

So you work on interpretability at Anthropic. Explain what that is, explain what it is you do.

Emmanuel Ameisen (1:45)

Yeah.

So I work in a team of interpretability researchers, and our goal is to understand how LLMs like Claude work by looking inside them. One of the most important things to realize if you like use language models or think about them is just this very bizarre fact that to make them is not to understand them. You make them by building this architecture which is like a scaffold, and then you pass a bunch of data through them and they grow in inscrutable ways. So that's why it's even a thing that we could be trying to do to study like how these models work. And so a lot of our research has been on, well, when the models are about to produce one more word, there is actually a whole universe of stuff that happens, and we try to kind of untangle as much of it as we can.

Lenny Rachitsky (2:41)

There's going to be a lot of mind-expanding, I don't know, elements to this conversation, I expect. One is along these lines, this point that you make, that models are not built, they're grown.

Speak more to that.

Emmanuel Ameisen (2:55)

At a really high level, the way a model like Claude works is you kind of train it in these successive phases, like the first phase, which people refer to as pre-training, is you just take a large corpus of data, and over this corpus of data, you sort of feed the model as input, you know, the, let's say, first half of a certain document, you ask it to predict the word that comes next.

Then you do that again and again and again and again for sort of an ungodly amount of data.

And after you do that, there's then this other phase where, and you sort of have this fine-tuning phase where maybe instead of arbitrary data, you have a corpus of examples of like how the model should behave, and that's where you train it on. So you have examples of dialogue of like helpful assistant telling you, you know, your favorite recipe or solving coding problems. And then there's sort of like further details here that I'm just going to skip for a second. But then, okay, so what does that mean that you're doing this process? Like what is this thing that you're sort of training? All models are, is you know, they take this input, which is a bunch of words. They turn these words into numbers and then they do some math on these numbers and they get some intermediate results. And then they do more math and they get some intermediate results. And they do this again and again and again and again many times. And then at the end, they have a final result, which then convert back into a word. And so the math that they do, these computations, that's the part that's grown. That's the part where it's just like through learning to do this task better, we sort of like update parameters in one direction or other. And then they learn some formulas. We have no idea what they are. And the goal of my team is to sort of like study, you know, both sort of like what these formulas mean in the end, and also what these intermediate results are. Can we decode them and can we see like what the model is doing as it's doing all of this intermediate stuff.