François Chollet: The ARC Prize & How We Get to AGI

**François Chollet** (0:00)
Hi everyone, I'm François. I'm super excited to share with you some of my ideas about AGI and how we are going to get there. This chart right there is one of the most important facts about the world. The cost of compute has been consistently falling by two orders of magnitude every decade since 1940 There's no sign that it's stopping anytime soon. And in AI, compute and data have long been the primary bottleneck to what we could achieve. And in 2010, as you all know, with the abundance of GPU-based compute and large datasets, deep learning really started to work. And all of a sudden, we are making fast progress on problems that had long seemed intractable across computer vision and natural language processing. And in particular, self-supervised text modeling started to work. And the dominant paradigm of AI became scaling up at LMP training. And this approach was crushing almost all benchmarks. And remarkably, it was getting predictably better benchmark results as we scaled up model size and training data size with the exact same architecture and the exact same training process. That's the scaling laws that Jared told you about a few minutes ago. So it really seemed like we had it all figured out. And many people extrapolated that more scale was all that was needed to solve everything and get to AGI. Our field became obsessed with the idea that general intelligence would spontaneously emerge by cramming more and more data into bigger and bigger models. But there was one problem. We were confused about what these benchmarks really meant. There's a big difference between memorized skills, which are static and task-specific, and fluid general intelligence. The ability to understand something you've never seen before on the fly. And back in 2019, before the rise of LLMs, I released an AI benchmark to highlight this difference. It's called the abstraction reasoning corpus, or ARC1. And from at that time, back in 2019 to now, with a model like GPT-45, for instance, there's been a roughly 50,000 X scale-up of basal LLMs. And we went from 0% accuracy on that benchmark to roughly 10%, which is not a lot. It's very close to 0, if you take into account the fact that any one of you in this room would score well above 95%.
So, to crack general fluid intelligence, it turns out we needed new ideas beyond just scaling up pre-training and doing static inference. This benchmark was not about regurgitating memorized scales. It was really about making sense of a new problem that you've never seen before on the fly. But then, last year, in 2024, everything changed. The AI research community started pivoting to a new and very different pattern, test time adaptation, creating models that could change their own state at test time to adapt to something new. So this wasn't about querying preloaded knowledge anymore. It was really about the ability to learn and adapt at inference time. And suddenly, we started seeing significant progress on ARC. So finally, we had AI that was showing genuine science of fluid intelligence. So in particular, in December last year, OpenAI previewed its O3 model, and they used a version of it that was fine-tuned specifically on ARC, and that showed human-level performance on that benchmark to the various time. And today, in 2025, we have suddenly moved on from the pre-training scaling pattern, and we are now fully in the era of test and adaptation. So test and adaptation is all about the ability of a model to modify its own behavior dynamically based on the specific data it encounters during inference. So that covers techniques like first-time training, program synthesis, train-of-thought synthesis, where the model tries to reprogram itself for the task at hand. And today, every single AI approach that performs well on ARC is using one of these techniques. So today, I want to answer the following questions. First, why did the pre-training scaling paradigm not get us to AGI? If you look back just two years ago, this was the standard dogma. Everybody was saying this. And today, almost no one believes this anymore. So what happened? And next, does the standard adaptation get us to AGI this time? And if that's the case, maybe AGI is already here. Some people believe so. And finally, besides the standard adaptation, what else might be next for AI?
And to answer these questions, we have to go back to a more fundamental question. What is even intelligence? What do we mean when we say we're trying to build AGI? If you look back over the past decades, there's been two lines of thoughts to define intelligence and to define the goals of AI. There's the Minsky style view. AI is about making machines that are capable of performing tasks that would normally be done by humans. And this echoes very closely the current mainstream corporate view that AGI would be a model that could perform most economically valuable tasks. 80% is often quoted as the number. But then there's the McCarthy view. That AI is about getting machines to handle problems they have not been prepared for. It's about getting AI to deal with something new. And my view is more like the McCarthy view. Intelligence is a process and skill is the output of that process. So skill itself is not intelligence. And displaying skill at any number of tasks does not show intelligence. This is like the difference between a road network and a road building company. If you have a road network, then you can go from A to B for a specific predefined set of A's and B's. But if you have a road building company, then you can start connecting new A's, new B's on the fly as your needs evolve. So intelligence is the ability to deal with new situations. It's the ability to blaze fresh trails and build new roads. So attributing intelligence to a 3D, a crystallized behavior program, a skill program, that's a category error. You are confusing the process and its output. So don't confuse the road and the process that created the road. So to formalize this a bit, I see intelligence as the conversion ratio between the information you have, mostly your past experience, but also any developer-imported priors that the system might have, and your operational area over the space of potential future situations that you might encounter. And that's going to feature high novelty and uncertainty. So intelligence is the efficiency with which you operationalize past information in order to deal with the future. It's an efficiency ratio. And that's the reason why using exam-like benchmarks to evaluate AI models is a bad idea. They are not going to tell you how close we are to AGI. Because human exams weren't designed to measure intelligence. They were designed to measure task-specific skill and knowledge. They were designed according to assumptions that are sensible for humans, but not for machines. Like, for instance, most exams assume that you haven't read and memorized all the exam questions and the answers beforehand. So if you want to rigorously define and measure intelligence, here are some key concepts that you have to take into account. The first is the distinction between static skills and fluid intelligence. So between having access to a collection of static programs to solve known problems versus being able to synthesize brand new programs on the fly to face a problem you've never seen before. And of course, it's not a binary, it's not one or the other, there's a spectrum between the two. The second concept is operational area. For a given skill, there's a big difference between being skilled only in situations that are very close to what you've seen before and being skilled for any situation within a very broad scope. For instance, if you know how to drive, you should be able to drive in any city, not just in a specific geofence area. You can learn to drive in San Jose and then move to Sacramento and you can still drive. Again, there's a spectrum there, it's not binary. Lastly, you should look at information efficiency. For a given skill, how much information, how much data, how much practice did you need to acquire that skill? Of course, higher information efficiency means higher intelligence. And the reason these definitions matter a lot is that, as engineers, we can only build what we measure. So the way we define and measure intelligence is not a technical detail. It really reflects our understanding of the problem of cognition. It scopes out the questions we are going to be asking, and so it determines the answers that we are going to be getting. It's the feedback signal that drives us towards our goals.