Why Fable 5.1 Is Worth the Upgrade

**Nathaniel Whittemore** (0:00)
Anthropic has released its latest models, Fable 5.1 and Mythos 5.1. On the benchmarks, they are undeniably state-of-the-art, outperforming everything else that exists on pretty much every category. Anthropic also claims that they've made major advances in the cost, so that for many tasks, including long-running agentic tasks, Fable 5.1 should cost as much as 25 or even 40 percent less than the comparative task in Fable 5
Initial responses are pretty good, although users are getting pretty varied mileage in terms of just how much the costs actually are and how far you can even get with Fable 5.1 given usage limits. Still, the question comes up, as it will now forever with every new model, is this one good enough that it's worth switching to? Except I think that that's no longer the right question. Instead, the question should be, what can I use this model for? How does it fit in to my overall model stack? What can I do to take most advantage of it while recognizing whatever trade-offs it comes with? That's what we're getting into in today's episode, so let's dive in.
The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.
All right, friends, quick announcements before we dive in. Our next set of executive agent leadership programs at Super Intelligent are coming up just after Labor Day. You can find out about those at training.bsuper.ai. Again, you can find out all about that at training.bsuper.ai.
We have kind of a dramatic set of headlines today. The first up is an update about OpenAI's forthcoming Astra. In a Tuesday blog post, OpenAI said that they now believe that Astra meets the critical cybersecurity capability threshold under their preparedness framework. In layman's terms, that means that the model is capable of finding and exploiting previously unknown security flaws without human guidance. In their previous assessment at the beginning of August, OpenAI believed that it was possible Astra would reach the threshold but weren't sure yet. Essentially, this is the same concern that saw Anthropic keep mythos under lock and key earlier this year.
Sharing some details on how they assessed Astra's capabilities, OpenAI shared that the model achieved a perfect 100% score on exploit bench. This benchmark evaluates a model's ability to develop exploits based on known vulnerabilities. OpenAI then took it a step further and developed their own internal version of the benchmark, consisting of 20 high severity vulnerabilities that were recently disclosed. The idea was to test whether the model was actually capable of creating novel exploits from scratch by using tests that couldn't be in the training data. OpenAI wrote, On this dataset, Astra achieves much higher arbitrary code execution rates than GPT-56 Sol using far fewer output tokens. During the evaluation, the model even discovered and used two zero-day vulnerabilities as part of an exploit chain. Now, to put some numbers around this comparison, Astra managed a 30% score on their internal version of exploit bench with 40,000 tokens used, as opposed to GPT-56 Sol, which wasn't capable of any significant results until it spent around 110,000 tokens. But if we extrapolate out to other capabilities, this could mean the model is much more token efficient for running agents across the board. In further testing with expert partners, OpenAI found that Astra was able to design and execute full exploit chains to gain root access to a hardened operating system and execute commands on a hardened browser. As a result, OpenAI will deploy a series of new safeguards for Astra's release. The model itself has received additional training to refuse cybersecurity tasks, resulting in a 91.5% refusal rate, up from 59% for GPT-56 sole. OpenAI is also adding more classifiers to detect cyber abuse and attempted jailbreaks, and in addition, OpenAI will now be flagging certain accounts as higher risk and applying more stringent model behavior guardrails to those accounts. OpenAI says that they believe that Astra is more likely to respect security boundaries than previous models, but they're still implementing additional chain of thought monitoring to detect and stop misaligned actions early.
In an unusually serious post on X that even used like correct grammar and punctuation, Sam Altman added, There is an obvious tension here. On one hand, Astra is very good and we are excited to see what people will build with it. We are proud of our work. On the other hand, we are clearly in a phase of development where we believe caution is warranted, and we are pacing our progress to ensure that we can meet the safety standards required by new capability levels. Astra has been done with training for a while now and is a significant step forward in both capabilities and alignment. For the models after that, we have been slowing things as needed to ensure that we can do sufficient work on safety and alignment.