AI Models Will Learn at Test Time by Next Year | Rohan Anil artwork

AI Models Will Learn at Test Time by Next Year | Rohan Anil

MTS

November 5, 2026

Former Anthropic pre-training researcher and Core Automation co-founder Rohan Anil explains how compute islands allow decentralized GPU clusters to train frontier AI models without colocation.

0:00

Rohan Anil

The reason it's jagged is that we don't have mechanism in the architecture and optimization for models to be able to learn from its own mistakes much quicker. And today we have to wait for labs to do the pre-training and RL cycle every game on, with the right environments and pre-training data to solve those tasks. We are designing the algorithms that work at test times. We will try our best, and I think we can get it done next year.

0:30

SPEAKER_2

And we are back. We are live in the studio with Rohan Anil, who's co-founder of Core Automation. We've had Jerry Torek, the other co-founder of Core Automation on before. He was great. This is one of my favorite interviews ever.

I think this one might even be better. So Rohan, welcome to MTS.

0:46

Rohan Anil

Thank you for having me and I'll try.

0:50

SPEAKER_2

So I'm curious to start. You left Anthropic to join or start Core Auto. No one ever leaves Anthropic.

What made you leave Anthropic?

1:00

Rohan Anil

Oh, I think it's a meme that people don't leave Anthropic. I think Anthropic is one of the best places to do research. I think I really love the pre-training team that I worked on. I think it's just, I think people have different motivations in life. Mine was to focus on the next frontier, which I think we're building at Core Auto.

We have had models that when you scaled up, brought us agents and brought us the world that we have today. What is the next piece of technology that can make it extremely useful in a way that changes our lives? I mean, we're all going to work. We're still using the AI. We are actually working even harder than we worked before. So what is missing in these models? And it's the ability to learn at test time. And that's something that we're working in.

1:51

SPEAKER_3

Yeah. So tell us about, you recently wrote the Hitchhiker's Guide to Training on Compute Islands. For those unfamiliar, what are Compute Islands and how do we maximally utilize compute?

2:04

Rohan Anil

Yeah. So at a very high level, either you have compute to train a frontier model or you don't. I think from training perspective, the most valuable computation is frontier training. And if you can get to frontier, people care. Like the top two models matter and everything else don't matter as much or have a very limited shelf life. When we started the company, it was quite clear to accrue a lot of compute.

It's not just the money. It's about co-location. It's about data centers. It's able to have that much power. And physical constraints come in the way of delivery of many of these things. But on the other hand, we were getting a sufficient number of thousands or hundreds of GB300s, hundreds of P300s, or thousands of them at different locations that we could get.

And if you had a training algorithm that can actually use them to get to Frontier, then you can unlock compute wherever it is, and you can also get to Frontier much quicker.

In some sense, this is, as I put it in the blog post, it's not new. People who worked in this field have been working on this entire, another set of area of this local SGD federated learning. And it's not as popular, but there is an emerging area of training in a distributed set of islands, or compute islands. And if you can do that well, as well as perhaps training otherwise, then it flips the question of, like, you could use any type of chips. In our experiment, we used two types. One is Grace Blackwell, and the other one is just Blackwell. So one is NVLink 72, and the other one is NVLink 8 They use completely different parallelism. They have slightly different numerics due to the parallelism configuration, completely different batch size. And what we proved, our main result is we get similar, slightly better result when we scale up on a very efficient frontier training scaling model.

4:05

SPEAKER_2

Yeah, so...

Yeah, what happened to the old dream of training AI models using normal compute in people's computers? I remember this was a thing that some people were working on back in like 2022 Maybe it just became like unfeasible due to the scale of the models that are currently being trained. But like, you know, why wouldn't it be possible in theory, you know, in the same way that, you know, people using their ordinary computers in a distributed way can be used for like seating torrents, why can't they be used to train models?

4:36

Rohan Anil

Read the other 15 minutes of this episode here.

$15 gets you this episode and 99 more. Credits never expire. After checkout, one link brings you back here to read it.

Not what you expected? Full refund within 14 days.

Thousands of transcripts fetched by people building searchable podcast archives

Fetch transcripts with code

The same credits work from a script or an AI agent. The demo key returns a sample episode in full, no card needed:

request
curl -H "x-api-key: pt_demo" \
  https://spoken.md/transcripts/1000651996090

Markdown with the speakers named, for your notes, your knowledge base, or anything that makes HTTP calls.

From $0.10 per transcript. No subscription. Credits never expire. Prices exclude VAT, added at checkout for EU customers. Not what you expected? Email us within 14 days with 20 or fewer credits used and we refund the pack in full.

Using your own key:

request
curl -H "x-api-key: YOUR_KEY" \
  https://spoken.md/transcripts/1000799939013