Why Frontier AI Evals Are Falling Behind | Vincent Chen
MTS
October 8, 2026
Vincent Chen, co-founder at Snorkel AI, announces an expansion of Snorkel's Open Benchmarks Grants program to $30 million to keep evaluation infrastructure in line with accelerating frontier model capabilities.
Vincent Chen
The core challenge we're trying to solve with this program and that we maybe have a call for the field to help address is that the pace of AI development is far outpacing that of measurement. And that's a real risk. If the tests that you're using to evaluate these models are too simple, too static, you fall into patterns of benchmarks.
SPEAKER_2
We are live with Vincent Chen, who is a co-founder at Snorkel AI and leads Open Benchmarks grants, which is Snorkel's program funding open data sets and evaluation infrastructure for frontier agents. Vincent, welcome.
Vincent Chen
Hey, thanks for having me.
SPEAKER_2
So I hear there's some stuff that's going on at Snorkel.
Vincent Chen
Yes.
SPEAKER_2
Do you want to tell us about it?
Vincent Chen
Yes, I do. Just a few minutes ago, we officially announced that we are expanding our Open Benchmarks grants program to a $30 million commitment. Wow. Actually, just hit go here.
This is a program that we're expanding to really help evals catch up with the pace of accelerating AI development. We see a lot of really exciting progress.
Our claim is really that evals need to keep up with the pace in order to be useful. These are increasingly going to be important for policy decisions, economic decisions, even personal decisions about how we spend time and interact with these agents. But again, development is far outpacing, how quickly we can build these evals. As a part of the grant, we're going to be supporting a few key pillars. One, we're finding more open benchmarks, more benchmarks that cover the full space of capabilities in the Jagged Frontier. Two, we're introducing something called the Open Benchmarks Red Team. The intention there is to really make sure that we're helping benchmark builders investigate weaknesses and actually patch them up in a really collaborative way. And three, we're launching our first ever Snorkel Research Fellowship. And again, the intention there is really to support independent contributors who are excited to advance the science and research behind eval, which I can go into in a second, but we think has a lot of really exciting new ground to break. So yeah, we're really excited to break the news here on MTS.
SPEAKER_2
Well, we're really excited to have you.
What should we start with? There's a lot to dive into. I guess we could start with the Snorkel Research Fellowship.
Vincent Chen
Sure.
SPEAKER_2
So it's supporting independent contributors developing new eval methods. What do you hope people apply with? Like, what's your request for startups here?
Vincent Chen
Yeah, well, the program has a few different requests for startups to use your words, right? The first is request for new types of benchmarks.
We laid out a little bit of a think piece or a vision on, hey, where we think capabilities are going and what types of evals need to complement them. And we came up with three axes that we think are actually well articulated to capture the space of evals that are needed. One is environment and input complexity, right? The real world is so much more complex than just a simple text prompt. How do you actually capture and represent that in your evals and environments that you're trying to measure? Two is autonomy horizon. So this is to say, agents are expected to work for a very, very long time now. How do you make sure they're doing so reliably? And three, output complexity. A lot of these types of jobs or a task that you're doing, again, it's not just a simple answer that comes back. It's a whole artifact. It's a whole program that comes out. And so one request is requests for new benchmarks that cover those axes.
Aside from that, we're really looking for just smart, motivated, independent benchmark builders who are excited to push the frontier, build new methods, build new benchmarks. We have a research fellowship program where folks are able to join us in our offices or remotely once a week and access our paid expert contributor network, work with our team directly. And so think of that as just like a one-click deeper way to work with us in building these types of benchmarks.
SPEAKER_2
Totally. Tell us about the Open Benchmarks Red Team. What is this like?
What is Red Teaming benchmarks like?
Vincent Chen
Yeah, great question. Benchmarks have become extremely challenging to build. The complexity of data is increasing. You could imagine, take Open AI's crazy dump of math groups last night.
To actually verify those with manual experts would take many, many, many years of human expert time. And so as a result, we also understand that, with the difficulty of building these benchmarks, it's really hard to keep them really high quality and continuously maintained. And we want to make it easier for folks who are building benchmarks to do that. So the goal of this Red Team program is, we'll work with you on your benchmark, we'll apply a lot of our internal tech and research know-how to help you identify weaknesses, we'll surface those weaknesses to you and then help address them. And we'll do so in a really collaborative way so that you can fix those and make sure your benchmark is fundamentally trustworthy, which is really, really important.
Read the other 8 minutes of this episode here.
$15 gets you this episode and 99 more. Credits never expire. After checkout, one link brings you back here to read it.
Not what you expected? Full refund within 14 days.
Thousands of transcripts fetched by people building searchable podcast archives
Fetch transcripts with code
The same credits work from a script or an AI agent. The demo key returns a sample episode in full, no card needed:
curl -H "x-api-key: pt_demo" \
https://spoken.md/transcripts/1000651996090Markdown with the speakers named, for your notes, your knowledge base, or anything that makes HTTP calls.
From $0.10 per transcript. No subscription. Credits never expire. Prices exclude VAT, added at checkout for EU customers. Not what you expected? Email us within 14 days with 20 or fewer credits used and we refund the pack in full.
Using your own key:
curl -H "x-api-key: YOUR_KEY" \
https://spoken.md/transcripts/1000793834534