Fable is Back: Here's What You Should Try First

**Nathaniel Whittemore** (0:00)
Today on the AI Daily Brief, Fable 5 is officially coming back. Before that in the headlines, the quest to cut inference costs. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.
Welcome back to the AI Daily Brief Headlines Edition, all the daily AI news you need in around five minutes. We kick off today with a story that is very of the zeitgeist that we are living in right now. OpenAI has found a way to slash their inference costs in half, sort of. This headline from the information grabbed a lot of attention, and understandably so. Everyone right now is looking for new approaches to token efficiency, and the implications of these searches have huge impacts on the business models and the companies that are shaping AI and the larger market structures they're operating in.
Now, when it comes to this article specifically, the details do suggest that it might be a smaller breakthrough than it appears at first. The claim is that OpenAI researchers have discovered a new optimization technique that cut their inference requirements in half for existing models. When the technique was applied to ChatGPT users who weren't signed into the service, OpenAI was able to serve that entire user base segment on just 100 GPUs. The OpenAI source didn't disclose what the technique was. The information speculated it could be quantization, cache optimization, batching queries, or routing queries to a lower power model. Notably, none of those techniques would improve service for OpenAI larger models without compromises. The universal truth that there is no free lunch remains, and most attempts at optimizing inference come at the expense of model quality.
Now, there's also the question of what it means that OpenAI is testing this technique on a tiny batch of their least engaged users. That might be a totally reasonable starting point, just the first test of many, or it could be a cautionary approach that implies that there's some risk of quality degradation. The TLDR is that while there seems to be something interesting here, we probably shouldn't treat it like some sort of silver bullet to resolve the compute crunch. Still, the information Stephanie Palazzolo is convinced that OpenAI is onto something. In an accompanying video, she said, This is a very important secret sauce for them that they don't even want to tell other OpenAI employees about because if these things leak, it can quickly be picked up by other labs, which can also then use that to lower their costs. This is something they're holding very close to their chests.
Now, many pointed to a new research paper from DeepSeek, which open-sources a speculative decoder system called dSPARK that can speed up inference by 85% during testing on small models. Now, it's unclear how dSPARK impacts costs, but it is a reminder that inference optimization is not even close to a solved problem, meaning that, theoretically, huge gains from some novel technique are definitely plausible. And of course, even if OpenAI hasn't found a way to boost efficiency by 50% across the board, any sort of gains here could still be a very big deal. OpenAI's army of free users are a significant drag on profitability, so anything they can do to cut inference costs to that user base could really move the needle. And given the particular audience, certain types of quality reductions may be more tolerable. Everett Randall of Benchmark Ventures has been talking about a phenomenon he's calling the AI Mom Test. He recently said, There's nothing my mom actually asks of her AI products that needs to be done by the frontier or even a near frontier model. And it seems to be at least initially like that could be the group that this new technique addresses.
Certainly there is a lot of chatter out there about innovations and new approaches in this area. AI aggregator Andrew Curran tweeted, I'm posting this prediction now so I can quote it later. There has been a significant breakthrough in architecture, specifically around memory efficiency, not by one of the big labs but by a team that was spun out of OpenAI. They will probably announce it soon.
Now in addition to labs finding new approaches, companies themselves are also finding new more efficient architectures. 20minutevc's Harry Stebbings tweeted, In the last 24 hours, I have had five founders message me of varying sized companies, some 10 person startups and one 200 billion dollar public company. All of them stated they have been able to cut inference spend by 75% or more with little effort, no performance change and better latency. The times, they are a changing.
Now, speaking of innovations in this new token efficiency era, Vibecoding platform Base44 has launched their own AI model in an attempt to shore up the business. The model is called Base1 and follows the same playbook as Cursor's composer. Namely, Base44 has taken an open source base model and applied their own fine tuning using training data from hundreds of millions of user interactions on their platform. CEO Maior Shlomo laid out the strategic thinking in a few different ways. Firstly, Base44 is making the bet that rarely trained models can be competitive with the frontier. This bet appears to be paying off for Cursor with most viewing Composer 2.5 as good enough for common tasks. Right Shlomo? General models need to be good at everything. They need to understand many programming languages, many workflows, many domains and many kinds of reasoning. However, Base44 only needs their model to be good at building web apps. Now of course, Base44 also views the model as a cost control measure. Shlomo wrote, It gives us more control over cost, latency, reliability and quality while still letting us use the best external models where they are the right fit. Finally, he writes, The model gives Base44 a way to utilize platform data to improve the product. The idea is similar to the Harness model pairing that OpenAI and Anthropic have pursued with their own coding platforms. Base44 believes they can develop the model and harness in tandem to deliver strong platform-specific results. As AI becomes a bigger part of how software is created, writes Shlomo, owning more of that intelligence becomes just as important as owning the infrastructure around it.