Inference Race: OpenAI Cut Inference Costs in Half. AMD and Cerebras Split AI

**Ksenia** (0:00)
In the span of one month, two separate announcements rewrote what it costs to run AGI models, and none of them involved installing a new chip. OpenAI reportedly cut the inference cost for logged-out for logged-out chatgipd traffic by more than half for an optimization it has not yet disclosed. It then described how its own model rewrote the production code that serves it, producing costs by another 20%.
And AMD and Cerebras announced that one AI answer would begin on an AMD supercomputer and finish on a Cerebras system. The computation would change processor architecture before the model says a word for claimed five-fold gain and efficiency. How are these two announcements connected? Because it's about one thing. Inference was never a fixed cost. There is still an enormous amount of efficiency hiding inside every AI answer. And the companies that find it first will serve more intelligence from the same silicon than their competitors can. So, the real question of this episode is where that remaining efficiency lives.
Welcome to Attention Span by Turing Post. My name is Ksenia. And let me show you why the AI industry is taking Inference apart. When you send the prompt to an Outer Progressive Language model, Inference proceeds in two main phases. The first phase is called Prefuel. The model reads the input tokens together, runs them through its layers, and prepares the internal state needed to begin the answer. For an agent inspecting a repository, the code, instructions, and previous conversation all belong to Prefuel. The model can process many input tokens in parallel through large matrix multiplications. Prefuel largely determines how long you wait before the first output token appears. During Prefuel, the model also builds a Kiwi cache. This is not a database of facts or long-term agent memory. At every attention layer, the model turns each front token into reusable key and value vectors. The cache stores those vectors for this request, so generation can consult them without running the whole prompt again. After Prefuel, the model can emit its first output token. The code then feeds that new token back through the layers, consults the cached state, and adds new entries to the cache and chooses the next token. Conventional autoregressive generation has a dependency across output tokens, so the answer advances step by step. Hardware performs many operations in parallel inside each step, and providers can batch users together. Yet, each sequence must advance through the loop. At low batch sizes and strict latency, moving model weights and reading the growing KVK Sheet can dominate the RFMEC. Decode determines how quickly the answer continues after it has started. This gives us two user-visible delays. The weight for the first token and the weight between tokens. Improving one does not automatically improve the other. Most serving systems have traditionally placed both phases on the same accelerators. GPUs can perform both with batching and scheduling to keep them busy. The problem appears under mixed traffic. A long prompt demands a burst of parallel computation, while active requests need another token every few milliseconds. Prioritize the prompt and existing answers can stutter. Protect the decodes and the new request waits longer for its first token. The phases also want different provisioning. Prefuel capacity follows input length and arrival rate. Decode capacity follows output length, concurrence and latency. Their ratio changes across applications and throughout the day. This does not make GPUs poor inference processors. One shared pool is simply being asked to satisfy two different latency problems at once. And here we need to discuss two different computes. Install Compute and Effective Compute. Install Compute is the processors, memory, networking and power a company owns or reads. Effective Compute is the useful inference work that capacity produces at the required latency and quality. Better batching, caching, routing or numerical precision can allow the same fleet to handle more requests. Phase desegregation goes further by changing where each part of the request runs. This differs from model parallelism, where one operation is divided across mostly similar devices and every token travels through that distributed model. Phase desegregation splits the requests in time. One serving pool completes pre-fill, the key vCache crosses the boundary, and the second pool continues decode. The provider can then choose different hardware and scale each phase independently. AMD and Cerebras add another choice, a different processor architecture for each pool. So what do they suggest? AMD Helios is a rack-scale reference design containing 72 MI455X GPUs and up to 30 WTB of HBM4 memory. The proposed system uses that capacity as a prompt engine. Helios can process many requests, work through long context, and build a key vCache for each one. Cerebras has not named the exact WSE configuration for this service. Its current WSE 3 shows the architectural idea. Most of a silicon wafer becomes one processor with 900,000 AI optimized cores and 44 gigabytes of distributed on-chip SRAM. Large models may still use multiple systems and external model storage, so the entire trillion parameter model does not sit inside one wafer. The relevant advantage comes from dense local memory, on-wafer communication, and an execution style Cerebras has optimized every fast token generation. In the proposed workflow, Helios performs pre-fuel, and then Cerebras performs decode. The logic of requests remains continuous while its execution changes hardware. Cerebras plans to deploy Helios reps inside its own data centers. So this is one operator running two architectures in a single service rather than a federation across separate clouds. AMD and Cerebras have not yet disclosed the interconnect, transit format, end-to-end handoff latency, or the required ratio of Helios reps to Cerebras systems. Existing disaggregated systems send the Kiwi cache and request metadata to cross this boundary. And long context caches can reach gigabytes. That yellow payload is the most important object in the story. If it moves quickly, each system can concentrate on its face. If it moves slowly, the network gives back the time the specialized hardware is saying. The announcement describes the division of labor while those integration details remain private. As always, it's important to look into the history and see where the novelty is coming from. In 1964, the CDC 6600 became the fastest computer in the world and is commonly described as the first successful supercomputer. Its designer, Seymour Cray, used one central processor for scientific arithmetic and ten peripheral processors for input, output, and operating system work. Different jobs went to different processors, with the coordination caused between them. CPU and GPU systems later made heterogeneous computing ordinary. AI serving is applying the same logic inside one model request. In 2023, the vLLM project introduced page detection. Conventional serving systems wasted GPU memory for reserving contagious space for unpredictable kivicaches. Page attention divided the cache into blocks borrowing from virtual memory. Its experiments reported 2-4 times more throughput than earlier systems at similar latency, simply through better management of temporary state.