From Math Olympiads to Navier-Stokes: How Fast Is AI Progressing? with Greg Burnham

Sam Charrington (0:00)

AI's progress in mathematics has been remarkably fast. Not too long ago, Frontier models struggled with grade school math. Now, they're contributing solutions to problems that have resisted mathematicians for decades, including, most recently, the Navier-Stokes equations. But as AI moves from tests we already know the answers to, to more open-ended research challenges, it gets harder to understand what these systems are truly capable of, and what their progress tells us about where AI is headed. My guest, Greg Burnham, leads capabilities research at Epoch AI, where he and his team are developing new ways to track AI's rapidly evolving strengths and limitations.

Here's Greg on why measuring and understanding AI's capabilities matters so much right now.

Greg Burnham (0:44)

AI capabilities, writ large, have gotten to such a point where what they're good at and bad at is starting to have real impact on the world, like on things we care about for other reasons. We're past the academic exam phase of understanding AI capabilities. And for that, we need high quality benchmarking, high quality evaluations. And one of my big questions is, just can AI come up with new ideas? If AI systems gets sort of superhuman at this, that's a big deal for the world, both in terms of economic and scientific progress, but also in terms of becomes harder to predict what AI systems are capable of. We use math as a testbed for this.

Sam Charrington (1:28)

I'm Sam Charrington, and this is The TWIML AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

Let's talk a little bit about AI capabilities research broadly, how you pursue that research, and why you think it's important.

Greg Burnham (1:58)

We're getting to a point in the trajectory of AI overall where AI capabilities can have a real impact on the world. So, maybe the transition could be characterized as school to work.

In maybe even up through 2025, most ways that we tested what AI could do looked more like human exams that you might give to kids or maybe advanced graduate students. I was going to say. Right, right. The kid exams, it is remarkable.

Sam Charrington (2:38)

The bar exams, the medical exams.

Greg Burnham (2:42)

And these paths, AI was perfectly good at these already, end of 2024, beginning of 2025 So, we really saw this transition happening over time, to be clear.

But this is certainly the year where AI capabilities are impacting work activities that humans were engaged in anyway, for their own purposes. And so, as these capabilities grow, it's just important for us to keep tabs on them. They grow very rapidly in some qualitative sense. We see no measurement we have shows any slowdown in how AI is getting better and better at any tasks that we're able to measure. And so, we think that just understanding the impact AI will have on the world, it's important to know what it can do, what it can't do, and keep tabs on how that trajectory is going.

Sam Charrington (3:41)

I want to push back on one thing you said, in terms of we're not seeing any slowdowns that can be looked at broadly, like in terms of maybe the number of benchmark data points that we're looking at. Certainly, we're seeing this broad performance.

But is it also true that on a particular, I feel like on a particular benchmark, we're getting the, for expected reasons. When we went from GPT-2 to GPT-3, we're taking these huge steps up these benchmarks, and now there's just a lot less room left, and we're making more incremental progress. Do you see that as a slowdown in AI performance, or would you characterize it differently?

Greg Burnham (4:29)

I would say, well, I'd say there's two issues here. One is, on the benchmarks we have, what does performance look like? Two is, are there things our benchmarks don't measure, that maybe there has been less or could even be negative, improvement on, and is there some reason why it's the things that are hard to measure where there's been the least progress? But on the first point, because this is a very, this is a much more cut-and-dried statistical point, we absolutely see continued progress on benchmarks, like a benchmark that was challenging for GPT-3, many years, whatever, six years ago, is now completely solved, aced by GPT-4, benchmarks challenging for GPT-4, same for GPT-5, now we've got GPT-6. So we do some statistical work to try to aggregate benchmarks over time, because the benchmarks that people used to track progress years ago, now every AI model off the shelf that you've heard of, that you might use, is too good at them. So people make new benchmarks, and then you have to do this stitching of like, okay, well, a model that was pretty darn good at this benchmark, was not so good at this next one, and you can use like, you can stitch those together and get this sort of composite score.