Sonnet 5 review: I ran 64 generations to find out if it's worth it

**Claire Vo** (0:00)
We've got a new model, people, and it's from Anthropic. Now, is it Mythos? No. Is it Fable? No, but it is Claude Sonnet 5 Anthropic is claiming it's the most agentic Sonnet model yet, and we will get Opus level tasks at Sonnet level prices. Now, I've been testing a lot of models, and I'm starting to get bored of doing the vibe check. What I want to start developing is a set of benchmarks. We can regularly test these new models against that you'll care about. So today, I'm going to be introducing the How I AI Bench, a set of AI and Claire Vo graded benchmarks that are going to tell us if this model and any model is good at writing PRDs, solving bugs, and one-shotting designs. I'm going to show you exactly how I built this benchmark using Claude Code, and we're going to see on a blind test what comes out on top. Let's get to it. This episode is brought to you by Runway, a new kind of creative platform that has everything you need to generate any image, video, or piece of content you want, all in one place. With Runway, it's now possible to go from initial idea to a finished deliverable in a matter of minutes. From turning low fidelity product shots into campaign ready imagery all the way through putting together big brand films, Runway can help your team scale for creative ambitions while keeping your budgets and timelines from doing the same.
Runway brings together the world's most advanced AI models, which is why enterprises like Microsoft, Robinhood, Amazon, and Adobe, along with studios like Lionsgate and Legendary all use Runway to ship real work every day. Try it yourself at runwayml.com/howiai, promo code howiai.
Quickly, before we get to our evals, let's just talk about the headlines of Sonnet 5, this new model. Anthropic is pitching it as close to the performance of Opus 4.8, but much less expensive. So as you can see here, it's not quite at this 69% on agentic coding, SWE-bench Pro, or the 82% on terminal bench 2.1, but it's not that far behind. And I suspect that most of us are not going to notice the difference. It's also supposed to be really good at computer work and knowledge work. And so this should be an everyday model that people reach for. In my episode with Felix from Anthropic, he says that we're all abusing Opus and we should definitely be using the Sonnet models more. And we are going to put Sonnet 5 to the test against that proposition. Now, what do they say that Sonnet 5 is really good at? Well, it's really good at agentic tool use. So you're going to get slightly longer running tool runs, longer running sessions than you would with Sonnet 4.6 at a lower cost than doing the same comparable task with Opus. So you're going to see here, you know, Sonnet 4.6, a lower pass rate on these long running tasks. Sonnet 5 getting pretty close when you have extra high reasoning on, and then Opus of course has the highest pass rate, but it's also much more expensive. That holds true also with computer use. So as you see, Sonnet 4.6, not bad, about 80 percent pass rate. But when you want to get past 80 percent into really successful computer use, browser use, etc., which is what I've been doing a lot lately, you're going to get a slightly cheaper experience, but almost as good as Opus 4.8 when you're using Sonnet. And then the headline seems to be, it's much more affordable than Sonnet. So it's going to be $2 per million input tokens and $10 per million output tokens at least through the end of the summer. And then it's going to go up a little bit. So if you want to test this model and you want to test it at launch prices, get that done now.
So as I said at the beginning of the episode, I'm a little tired of doing me sort of like one-off vibe checks. Sure, I can put this into Cursor, into Claude Code, one-shot a landing page, and kind of say, what do I think? And I've done this for a couple of models. I've done it for GPT 5.5. I've done it for open weight models like GLM 5.2, but I've always felt like my feedback on these models is kind of soft. Yes, we put it against like specific workflows, but I don't like that it's not repeatable. And I don't like that we're not testing it over time. What do I like about this process though? I do like that it is a Claire Vo benchmark. I have a perspective, I have a point of view of what's good and bad, and I don't want to lose that Claire Vo taste by doing an LLM in the loop or an AI as judge on these benchmarks. So I'm gonna show you how I built and will build the How I AI Bench and on a blind kind of taste test, how these models did across a couple use cases. Okay, what's really fun is the evals are not quite done running. So they are running in a sub agent right now for the final scores. So I will actually be surprised at the end of the episode about what I think of Sonnet 5 amongst all these other models. But I just want to show you how you can build your own evals benchmark for you to assess whether or not these new models are really working in your favor.