Reiner Pope – The math behind how LLMs are trained and served

**Dwarkesh Patel** (0:00)
Today, I'm interviewing Reiner Pope, who is CEO of MatX, which is a new chip startup. Previously, he was doing TPU architecture and many other things at Google.
This is a very different format for my usual interviews. This is going to be a blackboard lecture, we're going to get up in a second. We, in fact, built this whole new studio with specifically this format in mind.
And so, it's a pleasure to get to inaugurate it with you. We're going to be talking about model architecture, MLN for many other things. And the reason I think it's an important topic is because once you actually understand how training and inference actually work in a cluster, as we'll see a lot of things about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are. Fundamentally, also, why AI progress is the way it is, start making sense and you need to understand the details to get there and you need a blackboard to understand the details. So, Reiner, thank you so much for doing this.

**Reiner Pope** (0:50)
Yeah, very happy to be here.

**Dwarkesh Patel** (0:52)
Just a heads up, this is a lecture with graphs and equations and all that stuff. So, if you can, I would really recommend watching it on a video platform like YouTube. Okay, full disclosure, I am an angel investor in MatX, but that's not related to this podcast.
Reiner, maybe to kick us off, I'll ask this question. So, we have a couple of companies like Clod and Codex and Cursor are offering something like Fast Mode, where for 6X the price, they'll give you streaming tokens at 25X the speed. Mechanically, I'm curious what's going on here. Why is it the case if you can pay more to get the faster latency?

**Reiner Pope** (1:27)
Two, could you keep going?

**Dwarkesh Patel** (1:29)
Could you pay 100X more and somehow get even faster speeds or much, much faster speeds?
Three, could you go the other way? Could you have something like Clod Code Slow Mode, where if you are willing to wait for minutes on end, you could get even cheaper prices. So, maybe this will help motivate the kind of analysis that you'll be doing through the lightshower.

**Reiner Pope** (1:49)
Great. I mean, to jump to a little bit, to jump to the conclusion, the big effect is batch size, but what we're going to do now is quantify exactly what that looks like and what its implications are on latency and cost. There's going to be another effect which is, you can call it speculative decoding or multi-token prediction. We can maybe come back to that later, but I think the first thing that we'll talk through is batch size. So, what I'd like to introduce is sort of the two principles of analysis. Firstly, we're going to look at a roofline analysis of how a runner transformer model on a cluster of chips. We'll take a sort of, let's say, a Blackwell NVL72 cluster, so a rack of 72 GPUs.
So, the roofline analysis means we look at memory bandwidth and compute performance. Then, the other side of that is that we're going to look at just two simple factors of the model which are the time to operate on the weights, and then the time to operate on the context, the KB cache. So, let's jump in. What we're going to try and do is we're going to try and estimate the time that it takes to run an inference of a certain shape. Now, we're not perfect here. We can't exactly predict the time and so instead we're going to approximate. And so, we're going to say that the time must be greater than or equal to a certain quantity. And so, we're going to consider two different aspects. We're going to look at the time for it takes to do the memory fetches and then the time it takes to do the compute.
And it will turn out that this actually gives us a very strong predictive power even with a simple one. So, one by one, what is the time that it takes to do the compute?
So, there are really two things I need to do in the compute. I need to multiply by all of the active parameters and then I need to do some work on the attention. So, multiplying by all of the active parameters, I have a certain batch size that I'm running, and then I've got a number of active parameters in my model, and then I'm just going to divide this by the compute throughput, which is the flops of the chip. So, this is a hardware concept.