The Best Way to Test New AI Models

Nathaniel Whittemore (0:00)

We are currently in the fall glut of new models. Over the past several weeks, we've gotten a slew of new closed models at the frontier, Opus 5-5, Astra 6, Sol 6-1, with more on the way. And from those same labs, we've also gotten more cost-effective and faster models like Sonnet 5-5. And then as if all that wasn't enough, we've also gotten a number of new open-weight models which are increasingly relevant for lots and lots of different types of businesses. And of course, every time one of these models gets released, it's released with benchmarks. But benchmarks don't really tell us much about how it's going to be relevant for us personally. For that, you need to build your own personal AI benchmark, a system to tell if and how a model matters for your own personal AI stack. And that is the goal of today's operator-focused episode.

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

All right, friends, quick announcements before we dive in. This is a recording of the webinar that was hosted by me and Nufar Gaspar last week. Given that already we've gotten another couple new models this week, I think the system that is discussed here continues to be extremely pertinent. Hopefully you find this useful and tomorrow we will be back with a normal episode. But for now, let's talk about building a personal AI benchmark.

Nufar Gaspar (1:23)

We prepared some very fun materials to you, so we will share everything at the end. You'll see throughout what are the materials. But basically everything that you see me doing and talking about, you will get everything in order to try and do a similar process for yourself. And do leverage the Q&A box because we have a large number, so we're unable to open microphones. But Dan is here and Nathaniel is here, and everybody will try to cater to your questions as best we can. Without further ado, Nathaniel, some motivation on why we are here.

Nathaniel Whittemore (1:51)

Yeah. Thanks everyone for being here. I think for those of you who are regular listeners of AIDB, you'll know that this is a soapboxy issue for me when it comes to benchmarks. Every time we get a new model, which by the way, is now I think 11 days on average for a frontier model. One of my standard cautions is basically first to be skeptical, of the published benchmarks. Not so much because we think that any of these companies are lying, that's not the nature of what they do, but more because most of these benchmarks are now in the training data sets, and there's part of the canon that goes into them. It's just there's limited value of benchmarks, and the longer they've been around, the less value there is. Now, lots of people are continuously creating new benchmarks to try to improve that, and it's actually a pretty fertile environment right now, much more so than six months ago. But still, there are just real limits to that. On top of that, even if we assumed that all of the benchmarks were perfect and accurate, we are now at a stage where so much is about not strictly better or strictly worse determinations of model performance, but how a model feels and how it fits into the overall model stack that you're putting together. And there are going to be times, I guarantee, where the thing that is technically state the art on some benchmark is worse at the version of the thing that you're doing that's lower on the charts, right? This is particularly the case for things like writing, where so much of it is incredibly subjective. So all of that is to say that there's kind of no shortcut solution to figuring out how a new model is or isn't going to fit into your process other than using it. And so one of the things that I've talked about in the past on the show is trying to put together your own benchmark that has your most common use cases. And Nufar has very kindly gone out and put together a more significant, full, thoughtful version of that so that I don't have to just keep repeating this without being able to actually give you a resource to help. So that's, I think, couldn't be more pertinent. We've, I think we're on model three this week. We got Claude five, five, soul six, one. We didn't technically get Gemini for Argon. We just had it announced, but it's a time right now where I don't see this slowing down despite us pacing the frontier now.

Nufar Gaspar (4:01)