The Benchmark With No Instructions — ARC-AGI-3 (winning team!)

**Tim Scarfe** (0:00)
But we have the benefit of millions of years of evolution, right? So it's almost a bit unfair that we're expecting AI algorithms to be able to do all of that.

**Dries Smit** (0:09)
Exactly.

**Jeroen Cottaar** (0:10)
I think it boils down to one of the big open questions in the field, like is language critical to intelligence?
I will say, let's say, at least when I'm playing the games myself, and I think that goes for all of us, I'm using language. My name is Jeroen Cottaar. I have a background in physics and mathematics, and I've been working in the industry for about 15 years, with the last five years specializing more towards AI.

**Michal Tesnar** (0:30)
It was immediate to him what the goal was. He recognized the pattern, and I showed him the game within three seconds. He finished the first level.

**Stefano Viel** (0:38)
Often the agent starts thinking that reducing the energy bar to the minimum is the goal, or that stepping 10 times in a region is the goal, which for a human is kind of clear that it's not the actual goal. My name is Stefano.
I study computer science, machine learning. I did some research in reinforcement learning at EPFL.

**Tim Scarfe** (0:58)
I guess the million dollar question though is, do you think it's possible in principle to do really well on Arc-Agi-3 and be no closer to AGI?

**Dries Smit** (1:07)
Yes, I do think it is possible.

**Benjamin Crouzier** (1:10)
I don't know. I guess the main idea is a bunch of bright people in the room and do good research together.

**Jeroen Cottaar** (1:17)
We transformers can't plan, but they can do a very good job of pretending, essentially. That is in a sense indistinguishable.

**Michal Tesnar** (1:24)
36% might be misleading as a number if you don't look behind it. So what it really measures is action efficiency.

**Dries Smit** (1:31)
My name is Dries Smit. I studied electrical and electronic engineering. I specialized in reinforcement learning for the past five or six years now. And I've been doing on the research side Arc-Agi-2 and 3 and also some quant work in between. Yes, perhaps that tells something about the benchmark. Some leakage of human priors into the games.

**Michal Tesnar** (1:52)
Hello, I'm Michal.
I do master's in data science and I'm super excited about Arc and I love solving it. In the example that we saw, we have a maze, right? There's no native primitive in a newborn that says this is a maze.

**Dries Smit** (2:10)
Okay, so this is one of the official Arc-Agi games. This is one of the first games that was released. It's called LS20. And here you can see a 64 by 64 frame, which is typically how the game observations is presented to the agent. So you get an initial frame, and each of these pixels can take one of 16 colors. And now you need to figure out what the goal is, and even how the player moves, whether it's a specific player or object that's moving around or some other more abstract game.
You need to figure this out.

**Jeroen Cottaar** (2:42)
So very often our harnesses will break here.
So they actually got the idea right. You need to move to the exit with the two things aligned. But then they think they should win here. But actually you need to move up one more move. But it is quite common actually that they fail to beat the level from this point, because they cannot think that you should, they think, no, but that's outside the maze, so you can't move there. And the human would just, whatever, just give it a go and see what happens. But somehow we can't get our harnesses to also do that.

**Benjamin Crouzier** (3:11)
The best recipe we have today to build the Intelligent Systems is scaling up these large language models. I think Sam said you should definitely not be trying to train LLMs yourself. I don't recommend it.

**Jeroen Cottaar** (3:24)
Why not?

**Benjamin Crouzier** (3:24)
I think the easiest is just go work at Anthropic.
That's the easiest.
I have a sense, I cannot prove anything, that this is the time things start to get weird. I think he's right about that. Most fundamentally is I want to understand intelligence, how the brain works, and how to build intelligent machines. It's a big mystery. I guess when I was young, I was excited about both computers and brains. Computers, I feel now, we have a clue how it works, and I did computer science studies, so that mystery is kind of gone, but the mystery about brains is very much alive. I have a strong sense that most of societies bottlenecked on cognitive label and insight and intelligence. And so if we have more intelligence, that would be a big unlock for many things.