Francois Chollet — Why the biggest AI models can't solve simple puzzles

**Dwarkesh Patel** (0:00)
Okay, today, I have the pleasure to speak with Francois Chollet, who is an AI researcher at Google and creator of Keras.
And he's launching a prize in collaboration with Mike Knoop, the co-founder of Zapier, who we'll also be talking to in a second, a million dollar prize to solve the ARC benchmark that he created. So first question, what is the ARC benchmark? And why do you even need this prize? Why won't the biggest alum we have in a year be able to just saturate it?

**Francois Chollet** (0:27)
Sure. So ARC is intended as a kind of IQ test for machine intelligence. And what makes it different from most LM benchmarks out there is that it's designed to be resistant to memorization. So if you look at the way LMs work, they're basically this big interpolative memory. And the way you scale up their capabilities is by trying to cram as much knowledge and patterns as possible into them.
And by contrast, ARC does not require a lot of knowledge at all. It's designed to only require what's known as core knowledge, which is basic knowledge about things like elementary physics, objectness, counting, that sort of thing. The sort of knowledge that any four-year-old or five-year-old possesses.
But what's interesting is that each puzzle in ARC is novel, is something that you've probably not encountered before, even if you've memorized the entire Internet. And that's what makes ARC challenging for LLMs. And so far, LLMs have not been doing very well on it. In fact, the approaches that are working well are more towards discrete program search, program synthesis.

**Dwarkesh Patel** (1:41)
So first of all, I'll make a comment that I'm glad that as a skeptic of LLM, you have put out yourself a benchmark that...
Is it accurate to say that suppose that the biggest model we have in a year is able to get 80% on this, then your view would be we are on track to AGI with LLMs? How would you think about that?

**Francois Chollet** (2:01)
Right.
I'm pretty skeptical that we're going to see LLM do 80% in a year. That said, if we do see it, you would also have to look at how this was achieved. If you just train the model and millions or billions of puzzles similar to ARC, so that you're relying on the ability to have some overlap between the tasks that you train on and the tasks that you're going to see at test time, then you're still using memorization, right? And maybe it can work. Hopefully, ARC is going to be good enough that it's going to be resistant to this sort of attempt at brute forcing.
But you never know. Maybe it could happen. I'm not saying it's not going to happen. ARC is not a perfect benchmark. Maybe it has flaws. Maybe it could be hacked in that way.

**Dwarkesh Patel** (2:49)
So I guess I'm curious about what would GPT-FI have to do that you're very confident that it's on the path to AGI.

**Francois Chollet** (2:58)
What would make me change my mind about that, Alain, is basically if I start seeing a critical mass of cases where you show the model with something it has not seen before, a task that's actually novel from the perspective of its training data, something that's not in the training data, and if it can actually adapt on the fly.
And this is true for Alain, but really this would catch my attention for any AI technique out there. If I can see the ability to adapt to novelty on the fly to pick up new skills efficiently, then I would be extremely interested. I would think this is on the path to AGI.

**Dwarkesh Patel** (3:37)
So the advantage they have is that they do get to see everything. Maybe I'll take issue with how much they are relying on that, but let's suppose that they are relying, obviously they're relying on that more than humans do.
To the extent that they do have so much in distribution, to the extent that we have trouble distinguishing whether an example is in distribution or not. Well, if they have everything in distribution, then they can do everything that we can do. Maybe it's not in distribution for us. Why is it so crucial that it has to be out of distribution for them? Why can't we just leverage the fact that they do get to see everything?

**Francois Chollet** (4:10)
Right.
You're asking basically what's the difference between actual intelligence, which is the ability to adapt to things you've not been prepared for, and pure memorization, like reciting what you've seen before. And it's not just some semantic difference. The big difference is that you can never pre-train on everything that you might see at test time.