Ep. 038 - GLM 5.3 on AMD, NVIDIA, and Chinese Chips From Day One (InferenceX) | Kimbo Chen, Wenyao Gao, Jordan Nanos

Wenyao Gao (0:06)

Hello, everyone.

Jordan Nanos (0:07)

Welcome back to SemiAnalysis Weekly. This week, I'm joined by Wenyao and Kimbo. We're going to talk about everything, Glm 5.3. Guys put out a great article recently.

The model is about two months old, and we've got everything from InferenceX performance results to model architecture, post-training pipeline, and just like what people are using it for that we can dig into. Spoiler alert, lots of discussions around cybersecurity capabilities with this model. Kimbo Wenyao, welcome to the show, guys.

Kimbo Chen (0:31)

Thanks for having me, Jordan.

Wenyao Gao (0:33)

Excited to be here.

Jordan Nanos (0:35)

I can hear it in your voice, guys.

Okay. The title of the article was how GLM 5.3 sparse attention affects HBM and memory usage. For those who don't know, maybe we can start, Kimbo, with you with just an introduction. What is sparse attention? Why do we care about HBM memory usage when we talk about these new open models? Especially ones that are coming from Chinese companies like ZAI.

Kimbo Chen (1:02)

Yeah. Sparse attention is a category of attention mechanism, where in contrast to linear attention, where you try to modify the core operation of the attention, which is usually called a scale dot product attention. Linear attention tries to change the fundamental algorithm to linear time algorithm, whereas sparse attention chooses to change the way you select the number of tokens while keeping the core attention operation.

The idea is to invent some sort of token selection algorithm, so that regardless of how long the sequence, the context becomes, you will be attending to only the most important tokens, the most relevant tokens at every forward pass.

Jordan Nanos (1:53)

Awesome, yeah, so this makes the model a lot more efficient, right? And a lot of different labs are actually pursuing things like this to make attention a little bit more efficient. Can you maybe give a little comparison of what ZAI is doing with the GLM models to some other lab like DeepSea, Kirkini?

Kimbo Chen (2:08)

Yeah, so like you said, it's almost in some sense inevitable because just physics, we just don't have that enough memory and don't have that enough compute. And GLM uses, since GLM 5, they use the DeepSea sparse attention, the attention mechanism DeepSea invented actually for their V32 model. And they opted that instead.

And the core components are, you have a lightning indexer, which is the token selection component, and then the sparse attention. The sparse attention is based off of the more standard multi-latent attention, multi-head-latent attention invented by DeepSea at a popularized on DeepSea V3. So that's GLM. And for DeepSea, even though they invented DSA, a DeepSea sparse attention on V32, they continued innovating, they continued changing their attention variants. So for V4, for the V4 series, they invented CSA, compressed sparse attention and heavily compressed attention, which are most roughly speaking, building on top of DSA and further adding more token compression modules. And for V41 Flash, they simplified the HSA and CSA into like a slightly different design. And yes.

Jordan Nanos (3:42)

Yeah, sorry, I mean, don't mean to interrupt you, but let's say that the reason they're pursuing this is in the name of efficiency, right?

Like the, I think people speculate as to why the Chinese labs may care about this more than the American labs due to restrictions that they have on hardware. But roughly the outcome from all of these pursuits of sparse attention is that the cost to serve tokens on this chart, a million tokens goes way, way down. Like it's really, really cheap to serve if you have the ability to use sparse attention or like additional ways to like HH stuff like that, right?

Kimbo Chen (4:22)

Yeah, totally. It's just infeasible to fit that many tokens with a dense, like a dense attention anymore, given like our usage goes easily above a million token context for like a single session.

So, yeah.

Jordan Nanos (4:46)

Okay, Wenyao, let me bring you in here. When it comes to actually serving this model and trying to analyze the cost to serve a million tokens at 150 tokens per second, this obviously involves us running benchmarks that simulate a workload that a real provider like ZEI or somebody else who's downloading the model and serving it, like one of the inference endpoint providers would actually be incurring. In other words, they want to serve dozens or hundreds or even thousands of current users on a single model endpoint. Can you talk to me about that restriction that Kimbo just mentioned?

If you're going to support all these users with a million token context, it's going to have a lot of pressure on your system, like the resources that you have available, like memory, right?