Millions of Small AIs Could Beat One Giant Model | Andrew Trask artwork

Millions of Small AIs Could Beat One Giant Model | Andrew Trask

MTS

October 7, 2026

Andrew Trask, Senior Research Scientist at Google DeepMind and Executive Director of OpenMind, discusses the limits of embedded evaluators, double-blind frontier model testing via confidential compute, and why ensemble routing could decentralize the AI landscape.

0:00

Andrew Trask

From a theoretical perspective, implies that actually when you think about economic competition, meaning the most value for the lowest price, we're not gonna end up in a world with one giant model. We're gonna end up winning in a world with millions of models that we ensemble and route to, depending on the prompt. And that that's actually gonna outcompete the other systems.

0:21

SPEAKER_2

All right, we are back. We're live with Andrew Trask, who is a senior research scientist at Google DeepMind and leader of the OpenMind community, a trader and executive director.

We have a lot to talk about, mostly evals. So we'll talk about it with Andrew today. Welcome back.

0:38

Andrew Trask

Glad to be here.

0:40

SPEAKER_2

So I think the last time we had you on MTS was before the great independent evaluator vibe shift of about three weeks ago. And Dario, Sam, Elon all came out in favor of independent evaluators embedded in Frontier Labs.

Many policymakers have come out in favor of this. How do you think this affects the landscape of evaluations?

1:04

Andrew Trask

You know, I think it's a meaningful step forward, but it's not getting us all the way to the finish. So there's a problem that the evaluation ecosystem faces, which is sometimes just using the public products themselves is not enough to answer all the questions that we need. And so you need to have access to APIs that might not have been built for public consumption. And this has hindered the external email ecosystem from being able to answer some of the more important questions about frontier AI systems, their impact in the world, their behaviors and capabilities, etc. The solution to that, or one solution to that, is to give access to internal APIs that say employees use to do their work, which are quite a bit more flexible, but also come with some risk. But the problem that shows up if we stop there is that it's fundamentally a low-scale approach. Meaning, if you think about all the many, many ways in which AI could impact the world, the main important question to be able to ask, having a few dozen folks at desks with corporate laptops instead of the frontier labs, is unlikely to reach the level of scale and external website that's needed in the end. There are the solutions to this though, and this is why I say it's a meaningful step on the path forward.

If I had to hedge a guess, where we'll end up is that the folks that are embedded are actually setting up endpoints, setting up dashboards, setting up these longer term access systems, that then enable a much wider community of external folks to ask things on a more continuous basis, to explore lots of different questions going into the future, and to feel confident if they're actually attached to the real systems because embedded evaluators made sure everything was hooked up properly.

2:44

SPEAKER_2

Right.

So you just wrote this article a couple weeks ago called How to Scale the Impact of Embedded Evaluators. Can you explain the thesis of this article for the audience?

2:54

Andrew Trask

Yeah, sure. So some of it is similar to what I just described, that embedded evaluation means that we can ask and answer a much more diverse set of questions than a public product could enable us to ask and answer, which is fantastic. It's exactly how it should work. The evaluator should be able to query user logs. They should be able to query training runs and code and bring in third-party data and all things that internal evaluator can do.

But because the fundamental scaling challenge of how many desks and corporate laptops does Anthropic and OpenAI are going to manage in that very open-ended way, isn't necessarily going to scale to the global impact that AI is going to have in every country and jurisdiction and corner of the world. But it's a nice bridge. They can go in, they can set up endpoints, they can set up these pipes that enable outside folks to check in on the frontier systems on an ongoing basis. And so basically that paper describes a certain set of pipes that we work on in OpenMind for facilitating that kind of external access.

3:55

SPEAKER_2

How worried are you about a few different evaluators having a disproportionate amount of control? Like there's Meter, Redwood, Palisade, Apollo. There's only a few of these evaluation orgs that are fully prepared to do frontier model evaluations. They all come from the same social scene, social milieu of like the Bay Area rationalists. They've been criticized for this by the Trump administration, among others.

Some people say like they're evil EAs, and some people just say like, oh, well, if we're going to have like embedded frontier lab evaluators, we should make sure that they are coming from like diverse backgrounds basically. So what do you think?

Read the other 21 minutes of this episode here.

$15 gets you this episode and 99 more. Credits never expire. After checkout, one link brings you back here to read it.

Not what you expected? Full refund within 14 days.

Thousands of transcripts fetched by people building searchable podcast archives

Fetch transcripts with code

The same credits work from a script or an AI agent. The demo key returns a sample episode in full, no card needed:

request
curl -H "x-api-key: pt_demo" \
  https://spoken.md/transcripts/1000651996090

Markdown with the speakers named, for your notes, your knowledge base, or anything that makes HTTP calls.

From $0.10 per transcript. No subscription. Credits never expire. Prices exclude VAT, added at checkout for EU customers. Not what you expected? Email us within 14 days with 20 or fewer credits used and we refund the pack in full.

Using your own key:

request
curl -H "x-api-key: YOUR_KEY" \
  https://spoken.md/transcripts/1000793730929