How the ARC Prize is democratizing the race to AGI with Mike Knoop from Zapier

**Elad Gil** (0:05)
Hi, listeners, and welcome to No Priors. Today, we're talking with Mike Knoop, the co-founder and head of AI at Zapier. Mike co-founded the company in 2011 and was an early adopter of the power of AI in the enterprise.
Recently, he's joined forces with François Chollet to launch a competition to accelerate progress towards AGI called the ARC Prizes. Mike, welcome to No Priors, and maybe you can start off just by telling us a little bit more about what you're up to on the price side. It sounds really exciting.

**Mike Knoop** (0:29)
Yeah. Thanks for having me. I'm super excited. I've been a No Prior listener since literally episode one. So finally excited to get on and introduce yourself. So I'm Mike. I'm one of the co-founders of Zapier. I've run and advised all of our AI projects over the last two years or so. My day job has been building AI at the application layer for Zapier. But my nights and weekends have been more interested in this AGI research and progress. In fact, this kind of curiosity goes all the way back to my college days, pre-Zapier.
I think actually this is one of the reasons why Zapier was so early into some of the AI stuff was this curiosity in AGI.
The Chain of Thought paper that came out in Jan 2022 was what shook me loose. I was running half the company actually at that point and I gave up my exact team role to go back to being an IC and answer for myself, how close are we to AGI? As it turns out, we are not that close.
My belief is that AGI's progress has really stalled out over the last four or five years. I think there's a handful of reasons for that. I think the biggest one is that the consensus definition of what AGI is, the definition of it is wrong. I think we're measuring the wrong things. This leads people to think that we're closer to AGI than we actually are. This causes AI researchers and generally the world to be over-invested in exploiting this large language model paradigm and regime, as opposed to exploring new ideas, which are desperately needed. And frontier AI researchers also basically completely stop publishing. The GPT-4 paper had zero technical details. The Gemini paper had zero technical details on the longer context stuff.
And I just wanted to help fix this. I wanted to see if there was something I could do to accelerate. So yeah, I'm excited to share. We just launched ARC Prize. It's a million dollar plus non-profit public challenge to beat François Chollet's ARC AGI eval and open source the solution to it and open source the progress towards it. ARC AGI, to the best of my knowledge, is the only true AGI eval that actually exists in the world and measures a actually good definition, correct definition of what AGI is, which we can talk about.
There's an AI lab called Lab 42 out of Switzerland that's been running a small annual contest over the last four years to try and beat this eval. And state of the art today is 34%.
State of the art four years ago when the source introduced was 20%. So we've made very, very little marginal progress towards it. And this was pre-L and pre-scale, right? So it's like it has successfully resisted the advent of scale and LMs. The ARC AGI actually looks like an IQ test if you go look at some of the puzzles. Maybe we can overlay some of the puzzles and show some stuff.

**Elad Gil** (3:08)
Yeah, could we actually get into that? I'd love to hear sort of what you view as the consensus definition of AGI today.
What's wrong about it and then what do you think is the right way to measure or calibrate against that?

**Mike Knoop** (3:18)
Yeah, the sort of consensus definition that I think is most popular in sort of the AI industry right now is that AGI is a system that can do like the majority of economically useful work that humans can do. I think Vinod gets credit for coining this one and you know, I think it's a useful definition actually. You know, look, I spend my day job building application and there is legitimate economic value that is sort of unlocked by the current regime with language models.
However, I don't think it's a good AGI definition though. You know, I think it's a good definition of systems that are useful and economically useful, but you know, I kind of joke that like I think it says more about what many humans do for work than it does about actual general intelligence.