Engineering Persistence: How MLX-Engine v1.8.5 Solves the KV Cache Rewind Problem

**SPEAKER_1** (0:00)
You know, usually when we talk about a hardware bottleneck in AI, there's this expectation of heat, like actual physical palpable stress on the machine.

**SPEAKER_2** (0:10)
Right, yeah, like you can feel it.

**SPEAKER_1** (0:12)
Exactly. You fire up a complex, agentic workflow locally, and you immediately wait for the fans on your rig to sound like a, well, like a jet engine taking off.

**SPEAKER_2** (0:22)
Oh, absolutely. You just expect the GPU to start melting straight through your desk.

**SPEAKER_1** (0:26)
Yeah. It's a very visceral experience. And I think in the engineering world, we've kind of conditioned ourselves to equate massive, localized AI compute directly with, you know, massive thermal and raw processing overload.

**SPEAKER_2** (0:40)
We really have. I mean, if the fans aren't screaming, is the model even thinking?

**SPEAKER_1** (0:44)
Right, exactly.
But here's the hook for today's deep dive. When you step into the world of modern, open source inferencing architectures, specifically when you're trying to run persistent, multi-step agentic loops, suddenly the bottleneck isn't just about raw compute power anymore.

**SPEAKER_2** (1:00)
No, it's not.

**SPEAKER_1** (1:01)
It's not thermal throttling. The problem is memory, specifically the KV cache.

**SPEAKER_2** (1:05)
Yeah, that's the real silent killer right there.

**SPEAKER_1** (1:07)
Yeah, it really is. Because modern open source LLM architectures use these incredibly clever attention tricks, right, like sliding windows.

**SPEAKER_2** (1:16)
Right, to save RAM.

**SPEAKER_1** (1:17)
Exactly. But those exact same memory saving tricks completely break the ability to easily rewind and reuse that QZ cache.

**SPEAKER_2** (1:25)
Which is, I mean, it's devastating for multi-step workflows. It turns these agentic loops into incredibly resource heavy, painfully slow processes.

**SPEAKER_1** (1:35)
It creates a structural nightmare for local hardware. The machine spends more time trying to remember what it was just doing than actually doing it.

**SPEAKER_2** (1:42)
Exactly. It's just spinning its wheels.

**SPEAKER_1** (1:44)
But there is a solution.

**SPEAKER_2** (1:45)
Yeah.

**SPEAKER_1** (1:45)
And that's what we're digging into today. We're looking at the newly released MLX engine, specifically version 1.8.5.

**SPEAKER_2** (1:51)
Which is a massive leap forward.

**SPEAKER_1** (1:53)
It really is. It solves this issue by checkpointing and swapping 256-token KV cache blocks to a highly optimized temporary disk-backed LRU store.

**SPEAKER_2** (2:03)
An LRU store.

**SPEAKER_1** (2:04)
Yeah. And we are talking about reducing extra RAM usage by up to 82%.

**SPEAKER_2** (2:08)
Which is just wow.

**SPEAKER_1** (2:10)
And drastically speeding up processing, essentially turning solid state drives into high-speed pseudoram for AI.
Welcome back, listeners, to the Neural Intel Podcast. Let's dive into today's topic. As always, we'll focus on the technical details and implications of the technology we discuss. To stay updated on the latest in AI and ML, visit our blog at neuralintel.org and check us out on YouTube, Apple Podcasts, and Spotify.

**SPEAKER_2** (2:37)
It's great to be back. And this topic is just particularly crucial right now.

**SPEAKER_1** (2:41)
Oh, totally.

**SPEAKER_2** (2:42)
Because we are moving way past the era of just generating a single text response to a single prompt.

**SPEAKER_1** (2:48)
Right. The chatbot days are kind of table stakes now.

**SPEAKER_2** (2:51)
Exactly. We are deep in the era of agentic loops. We're talking about software that writes itself, analyzes its own output, realizes it made a mistake.

**SPEAKER_1** (2:59)
Corrects that mistake.

**SPEAKER_2** (3:00)
Right. And iterates over and over.

**SPEAKER_1** (3:02)
So for the technical CTOs out there, or the MLX engineers listening, the architects trying to build these autonomous systems without paying massive cloud API tolls, this is it.

**SPEAKER_2** (3:11)
This is the infrastructure discussion that dictates whether your product actually works in production, or if it just OGM crashes at 2 a.m. Yeah.

**SPEAKER_1** (3:18)
Out of memory crashes are the worst. So, let's unpack this context crisis. And I think we have to start by grounding this in the physics of the models everyone is actually deploying right now.

**SPEAKER_2** (3:29)
Good call.

**SPEAKER_1** (3:30)
We're looking heavily at the architecture of Qwen 3.5, including its 3.6 iterations and Gemma 4

**SPEAKER_2** (3:38)
Right. These are arguably the heavyweights in the open source arena at the moment.

**SPEAKER_1** (3:41)
For sure.

**SPEAKER_2** (3:42)
They're incredibly capable, but they represent a major divergence in how we handle massive context windows.

**SPEAKER_1** (3:49)
Awesome.

**SPEAKER_2** (3:50)
Well, let's take a step back and look at the underlying math of the context window.

**SPEAKER_1** (3:53)
Yeah.

**SPEAKER_2** (3:54)
As sequence links is scaled up from 4K to 32K up to 128K and beyond.

**SPEAKER_1** (3:59)
Yeah, they're getting massive.

**SPEAKER_2** (4:00)
The size of the KV cache grows linearly.

**SPEAKER_1** (4:03)
Wait, let me pause you right there, because we throw the term KV cache around a lot. For the researchers listening, we know it's the key and value matrices. But for someone focused strictly on the orchestration layer, let's break down what that physically looks like in RAM.