Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test?

Claire Vo (0:00)

I was massively prepared for Anthropic to drop Opus 5.5. I was even prepared for OpenAI to drop another model. I got up early this morning because I had a little bit of early access to Opus 5.5 to record you all an amazing Opus 5.5 review. And guess what? They both landed this morning. So now I am, despite having in the can, you will see it later, a great Opus 5.5 review. I am just going to go ahead and do this one live. And I have never done anything live. So I am going to talk to you all about these new models. There's actually three that came out today. Opus 5.5 from Anthropic, GPT-6 Sol, a GPT-6 Luna. There's like price wars going on. And I haven't done an episode on the How I AI bench. And I just updated it for these new models. And we're actually going to vibe check this baby live together as a group. And we're going to go through very quickly what these models are, what they're saying about the models. And then I'm going to vibe check live. We're going to do the blind taste test bench. We're going to surprise myself live. You all can see my complete internal inconsistencies. And we're just going to see how it goes.

So Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. It was able to have some early testing of all these models. So I have a good sense of what I think they're good at, but I did think it was an important moment right now. We've had Astra come out, Fable's been refreshed. I just wanted to completely rethink the How I AI bench. And if you've watched any of our old How I AI bench model episodes, they're really focused on two things, PRDs and prototypes. And I just think that these models are getting smarter, more agentic. And so I actually built out a much broader benchmark for this blind taste test. I'll walk you through all how that works. And I want to get your feedback on if the comparisons are useful. And we're actually going to blind taste test it live. You all can judge how vibey my vibe check is. This is just how we make decisions about things. So it is imperfect. Does have an LLM as judge in the middle, but we're gonna do it live and we're gonna see truly which model I love. But before we do that, let's just go through the highlights. Okay, so these are not the frontier front two models. These are not the Astras and the Fables, but Opus got a refresh, Sol got a refresh and then Luna got a refresh. All of them cheaper and faster. So these are gonna be your daily drivers for common coding and knowledge work tasks.

They're, here you can see like the stack rank of how expensive these models are. Opus 5.5 lit twice as expensive as GPT-6 Sol. And the pricing wasn't out when I tested these, and this is really gonna impact how I think about using these models. Again, Opus 5.5 was cheaper than Opus 5 and Fable 1, to which they compare the intelligence. But Sol, you know, my favorite, my babe, that is I'm excited about because GPT-6 Sol is my favorite and cheap. Okay, so the real thing that they're focusing on, not just cutting the cost, but also cutting on cached inputs, and just speed and token efficiency. And so you're going to see both sort of like output drop, token use drop, and cost drop. It's really, really nice. And then caching, really great, especially if you're resending contacts, we've seen a lot of caching savings when we use all these models at chat PRD. And so definitely if you're building on these models, make sure that your caches are optimized and that you're using, taking advantage of that because it can save you a lot of money. And if you don't, as I learned, it can cost you a lot of money. The other thing that I wanted to call out, and again, I hate this Claude generated presentation, but this is something that I really want you all to focus on, is this left side on Opus 5.5. Opus 5.5 is the first Opus level model that has shipped with the fabled level kind of like cyber and bio guardrails. And so you should be able to fix your own bugs in Opus 5.5, but you won't be able to do like cyber security work or bio work, like make good choices. It'll kick you down to Opus 4.8 or block you. So just know that there's those controls there. You know, Anthropic is saying that this is supposed to line model. It's the one that was built when after like pacing the frontier. So it had external evaluators. One thing that I've seen in Opus that you should expect to see from a lot of Anthropic models is it continues to be a little bit of like a conservative scold. You'll see that in my Opus 5.5 review. So, you know, it will tell you no. It'll tell you like I shouldn't do that. You won't get prompt injected. So that's good.