Can Open Models Keep Up With the AI Frontier? | Nathan Lambert

Nathan Lambert (0:00)

When I think about model progress, it comes down to different forms of capital, and that is compute data and talent.

Lisha Li (0:06)

How do you see that playing out in terms of like, who's gonna win in the next couple of years?

Nathan Lambert (0:10)

The Chinese are so good at building models. To say that China is anything but equal in talent, you're gonna have a hard time. What they don't have is compute.

Lisha Li (0:16)

Most people regard distillation as sort of like the main lever for them to do this.

Nathan Lambert (0:21)

People distill my products every single day. They take it, they strip it down to its raw parts. That's called competition. Previous post-training was much more reliant on the supervised fine-tuning. Now this has morphed a bit into environments where it's RL heavy. But it's just like hard to map this type of post-training, which is you get the completions and the score goes up to what is going on in reasoning today.

Lisha Li (0:41)

The discourse now is just so much fear on like how much the frontier is just hidden and also breaking away.

Nathan Lambert (0:46)

The AI capability plot doesn't necessarily worry me, but the AI capability plot in the pretty much non-existing preparation of cross-public cybersecurity and all of these things.

SPEAKER_3 (0:56)

The frontier AI labs are spending extraordinary amounts of money to build better models. So how does open source keep catching up? In this episode, a16z partner Lisha Li sits down with AI researcher Nathan Lambert, founder of the new nonprofit research lab Trillium, to unpack the race between open and closed AI.

They discuss why open AI and Anthropic still have major advantages in tripute, capital and speed. But why Nathan believes the technical talent behind open models, particularly in China, is often underestimated. They also separate the reality of model to solution from the rhetoric around it, and look at how post-training, data, and RL environments are becoming increasingly important sources of progress.

And finally, they turn to recursive self-improvement. Nathan explains why Trillium plans to document how AI is actually being used to improve AI. From the tasks models can already automate, to the places they still get stuck. And why making that progress visible could lead to a much more grounded debate about how quickly the Frontier is really moving.

Lisha Li (2:00)

So Nathan, really happy to have you here. Thank you for joining in person. Super timely, given your launching Trillium, which I'd love to talk about. But more sort of on a macro level, I think we chatted a lot about open source, and that's a lot of what you care about. And what we've been hearing a lot in the public discourse is like, hey, we have to paste the Frontier, the Frontier this, the Frontier that.

And I think an arc of the stuff that I'm really trying to ask experts about is what is in the Frontier, what are we pasting, what's there? And then more importantly, kind of threading it with open source and open weight models, like can they keep up with the Frontier? And I think this matters very much and is something that you're an expert on. So maybe to sort of start on that arc, sort of how you would break down what's gotten us to the Frontier for both closed labs and open source models. How does it break down, in your mind, maybe first at a higher level of pre-training, mid-training, post-training, how post-training has evolved, which is something you've done a great amount of education to the public on.

Nathan Lambert (2:58)

When I think about model progress, it comes down to different forms of capital that you put into them, and that is like compute data and talent. And all of the labs have various amounts of this, and the closed frontier labs of OpenAI, Anthropics, Google's have clearly had the most of these. And what we see is like the different abilities of the labs to organize these forms of capital, and making the org chart work for building models is really hard, which is why you have companies have to do complete resets from the bottom up. And then it actually kind of works sometimes. A company like Meta is like totally reset, and they're like on a positive trajectory again. Seriously.

I think I'm very interested in these human aspects of why competition drives the frontier and why the org chart is so important, because we see these things, and this comes into play in the open and closed dynamic as well, where it's like the closed companies, the true frontier labs, which I say, if somebody wants to work at the frontier of AI, mostly tell them to work at Anthropic and OpenAI, which is the easiest bet. I think Meta, Google, SpaceX AI, I think you still might be there, but it's harder to see from the outside. And these companies have just taken the most ambitious bets, the earliest in order to get there, and that's why OpenAI has done things like start the reasoning model revolution. And I think that kind of breakthrough, and that beautiful plot with the inference scaling laws is peak OpenAI. It's like they're really good at getting this plot that just changes how everyone thinks about everything.