Ep. 030 - Long Live the Short King: Why 4-HI HBM Wins (Memory) | Myron Xie, Jordan Nanos

Jordan Nanos (0:05)

Hear ye, hear ye. Long live the short king. We are back for another episode of SemiAnalysis Weekly. I got Myron laughing already. We're going to talk about why four high HBM wins in this episode.

Put an article just yesterday, I think two days ago, probably when this episode comes out, and called a few pretty big changes in the industry. Everything from the HBM suppliers all the way to the Nvidia GPUs that this is going to impact, and potentially other accelerators we will discuss, and we'll dig into some of the motivations for this change and why people are revisiting the balance between capacity and bandwidth. Myron, how are you doing, man?

Myron Xie (0:52)

I'm doing very well. So yeah, this is a fun article to write, and I think the thesis has challenged a lot of conventional assumptions on HPN content per accelerator.

So to give listeners some background, the trend over the last few years has been every major chip or AI chip generation over generation, it packs in more HPN content per chip. So they add more stacks of HPN. And then one of the big changes is also they make the stack taller or with more layers. So back in the Hopper or Ampere era, 8 high was expanded. And then with Blackwell Ultra, they moved to 12 high. So it's having 12 layers of DRAM dies in a HPN stack. So you get, you know, basically you increase your capacity by 50% over 8 high.

And then the presumption from there was that you keep going taller. So the industry was contemplating 16 high for, you know, going to 16 high, so 16 DRAM, laser DRAM in a cube with HPM 4E. And Nvidia, when they originally previewed Rubin Ultra to people at GTC last year, they said that Rubin Ultra would have one terabyte of HPM per package, per chip. And to get there, they'd have four compute dies, you know, 16 stacks of HPM 4E. That's, and then each stack of, each 4E die, HPM 4E die has 32 gigabits or 4 gigabytes of DRAM capacity. And then they'd make it 16 high. So that's how you get to one terabyte. But now, what we understand is you're fast forward a year and a half later, Rubin Ultra is going to just have 192 gigabytes of HPM 4

So how do you get from one terabyte to 192 gigabytes is, first of all, it's no longer going to be for compute dies. It's just two compute dies per package. It's no longer going to be HPM 4e. There might be a HPM 4e version, but it's going to be HPM 4 at the start, and HPM 4 is only three gigabytes per die, and it's going to be eight high. And then this, and you will also notice that this is even lower than HPM content than, say, Blackwell Ultra and conventional, what we call, vanilla Rubin, which are both that 288, because they use 12-high HPM.

So why is this happening? The big reason, it's really primarily motivated by supply. So as I think many readers are aware, in a very big shortage of memory supply in DRAM and DAND, but let's talk about DRAM, there's very little wafer capacity being added. And then as the, I guess, number of accelerator shipments go up, the number of DRAM HPM demand keeps going up, and it's very hard to just keep up with that demand. So what the suppliers and also customers like Nvidia, Google Broadcom, et cetera, realizes that we have X amount of, we have secured this much HPM. But the logic we have secured it, and call us it, we secured it TSMC, is that if we take the HPM supply we have, make it, ship it in 12 high cubes, it's like, it's not enough to like, it's not enough supply to like ship all that logic. So let's ration our DRAM supply, or our HPM supply, and we ship it in eight high cubes. So you get more from, you get more cubes from, I guess, the implied wages you get. And then that helps balance the equation more. And that's what Nvidia has realized.

But of course, you're like going downgrading from eight 12 high HPM to eight high HPM. I mean, does that have performance impacts? Like, how does that affect your performance in TCO? And I think one of the big things is the, I guess, in terms of how important is HPM capacity for the inference.

Specifically, there's a threshold that you need. Like, you need to have enough HPM so you can hold the model weights, hold enough, have enough capacity to support KVCache for a lot of users so that you can batch effectively. But once you get beyond that threshold, what we find is that the returns to that are diminishing. And because HPM is getting much more expensive from next year, as the shortage gets reflected into prices, the cost of additional HPM that you might not need is also much higher. So the penalty to having too much HPM capacity becomes much worse.