Richard Sutton – Father of RL thinks LLMs are a dead end

**Dwarkesh** (0:00)
Today, I'm chatting with Richard Sutton, who is one of the founding fathers of reinforcement learning and inventor of many of the main techniques used there, like TD learning and policy gradient methods. And for that, he received this year's Touring Award, which, if you don't know, is basically the Nobel Prize for Computer Science. Richard, congratulations.

**Richard Sutton** (0:17)
Thank you, Dwarkesh.

**Dwarkesh** (0:18)
And thanks for coming on the podcast.

**Richard Sutton** (0:20)
It's my pleasure.

**Dwarkesh** (0:21)
Okay, so first question. My audience and I are familiar with the LLM way of thinking about AI. Conceptually, what are we missing in terms of thinking about AI from the RL perspective?

**Richard Sutton** (0:33)
Well, yes, I think it's really quite a different point of view and it can easily get separated and lose the ability to talk to each other. And yeah, large language models have become such a big thing. Generative AI, in general, a big thing, and our field is subject to bandwagons and fashions, so we lose track of the basic, basic things. Because I consider reinforcement learning to be basic AI, and what is intelligence, the problem is to understand your world.
And reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do.

**Dwarkesh** (1:20)
I guess you would think that to emulate the trillions of tokens in the corpus of Internet text, you would have to build a world model. In fact, these models do seem to have very robust world models, and they're the best world models we've made to date in AI, right? So what do you think that's missing?

**Richard Sutton** (1:38)
I would disagree with most of the things you just said.
Just to mimic what people say is not really to build a model of the world at all. I don't think, you're mimicking things that have a model of the world, the people. But I don't want to approach the question in an adversarial way. But I would question the idea that they have a world model. So a world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don't have the ability to predict what will happen. What we want, I think, to quote Alan Turing, what we want is a machine that can learn from experience. Where experience is the things that actually happen in your life. You do things, you see what happens, and that's what you learn from.

**Dwarkesh** (2:28)
Yeah.

**Richard Sutton** (2:29)
The large language models learn from something else. They learn from here's a situation and here's what a person did. Implicitly, the suggestion is you should do what the person did.

**Dwarkesh** (2:38)
Right. I guess maybe the crux, and I'm curious if you disagree with this, is some people will say, okay, so this imitation learning has given us a good prior, given these models a good prior, but reasonable raise to approach problems. As we move towards the era of experience, as you call it, this prior is going to be the basis on which we teach these models from experience because this gives them the opportunity to get answers right some of the time, and then on this, you can train them on experience. Do you agree with that perspective?

**Richard Sutton** (3:12)
No, I agree that it's the large language model perspective.

**Dwarkesh** (3:16)
Right.

**Richard Sutton** (3:16)
I don't think it's a good perspective.

**Dwarkesh** (3:18)
Yeah. Here's why.

**Richard Sutton** (3:20)
So to be a prior for something, there has to be a real thing. I mean, a prior bit of knowledge should be basis for actual knowledge. What is actual knowledge? There's no definition of actual knowledge in that large language framework. What makes an action a good action to take? You recognize the need for continual learning. So if you need to learn continually, continually means learning during the normal interaction with the world. So then there must be some way during the normal interaction to tell what's right.
So is there any way for it to tell in the largest language model set up, to tell what's the right thing to say? You will say something and you will not get feedback about what the right thing to say is. Because there's no definition of what the right thing to say is. There's no goal. If there's no goal, then there's one thing to say, another thing to say, there's no right thing to say. So there's no ground truth. You can't have prior knowledge if you don't have ground truth. Because the prior knowledge is supposed to be a hint or an initial belief about what the truth is.