Vera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture Analysis

Skyler Monroe (0:10)

Hey everyone, welcome back to the AI Hardware Show. I'm your host, Skyler Monroe, and if you're into chips, silicon, and the hardware powering the AI revolution, you are absolutely in the right place. Huge thanks to our sponsors today. Ada, helping businesses integrate AI into their real workflows. Ago Consulting, that's all go your go to for silicon development from AI chips to full sock design. And Zen Semiconductor, building the processors and AI fabric behind modern data centers with ventures like their Sierra RISC-V CPUs and Loom AI fabric, plus Ada for deploying AI at scale in any business. Today, we've got a packed show. We're diving deep into Nvidia's Vera Rubin architecture, and what it means for inference economics, China's Biren is pushing into optical interconnects.

Samsung wants your memory to start doing math. There's a wild story about H200 chips showing up in China, and Intel just hit a major milestone on high-end EUV that has the whole industry paying attention. Let's get into it. Alright, first up, and this one is genuinely exciting for anyone who tracks Nvidia's roadmap, we're talking about the Vera Rubin Nvl72, specifically how it stacks up against the Gb200 Nvl72, and what the total cost of ownership picture looks like for inference workloads. This comes from the folks at Semi-Analysis, and it's a deep, meaty breakdown. So let's try to unpack the highlights. So first, what even is Vera Rubin? It's Nvidia's next-generation architecture, named after the legendary astronomer. And when we talk about the Nvl72 configuration, we're talking about a rack scale system, 72 GPUs tied together at the rack level, designed to operate almost as one giant compute organism, rather than individual cards talking over a network. Now, the headline technical feature that Semi-Analysis is calling out is the 3-bit LUT-based Tensor Core. Let me break that down, because it's actually pretty fascinating. Traditional Tensor Cores do matrix math in standard numerical precisions FP8, INT8, that kind of thing. What Nvidia is doing with Rubin is introducing lookup table-based computation at 3-bit precision.

Think of it like replacing a calculator with a cheat sheet. Instead of computing every multiplication on the fly, you pre-store the answers and just look them up. At 3-bit, your table stays small enough to be practical, but you get enormous throughput because you're essentially replacing multiply accumulate operations with memory lookups. The silicon behind this is the SM140. They're calling the streaming multiprocessor design Fineman, another great scientist namesake. The SM architecture is essentially the fundamental compute building block on any Nvidia GPU. So evolving the SM to support LUTPEP-based tensor operations is a pretty significant architectural shift, not just a die shrink or a clock bump. Now here's where the inference TCO total cost of ownership's story gets really interesting. When you're running large language models at scale, your two biggest cost levers are performance per megawatt and performance per dollar. Power costs money every single month. Capital expenditure on the hardware is a one-time hit, but depreciated over the system's life. It has to pencil out. Semi-analysis is reporting that Vera Rubin Nvl72 makes significant gains on both axes compared to Gb200 Nvl72. And that's not a given. It's actually harder than it sounds. The Gb200 Nvl72 was already a monster in both performance and efficiency, so leapfrogging it isn't trivial, but the rack-scale design philosophy helps a lot here. When you design the whole rack as a unit-powered delivery—cooling, interconnect—you eliminate a ton of overhead that you'd otherwise pay at the individual card or node level. On the software side—and this is something that doesn't get enough attention—semi-analysis is also highlighting improvements in the software stack. PyTorch integration, Vllm support, and OpenA, ITriton compatibility. Rubin apparently has public software support already taking shape, which is a big deal, because it means developers can start optimizing for it before it even ships widely. That's a lesson Nvidia learned well. Hardware without software is just expensive art. The practical implication here? For hyperscalers and cloud providers running inference at scale—think serving millions of AI queries per day—the move from Gb200 to Vera Rubin isn't just about raw speed. It's about whether they can fit more tokens per second into the same power envelope. And based on what Semi-Analysis is showing, the answer looks like a clear yes. That has real downstream effects on pricing for AI APIs and the economics of the whole AI services industry. Really compelling stuff. Vera Rubin is shaping up to be a serious step forward. Okay, story two? And this one is part of a much bigger trend that I think is flying under the radar for a lot of people. China's GPU developer Biren is expanding its AI chip R&D into optical interconnects and advanced cooling. This comes from Digitimes, and on the surface, it might sound like a niche technical update, but there's a lot of strategic depth here. So first, who is Biren? They're one of China's more serious domestic GPU plays. Founded around 2019, they've been trying to develop high-end AI accelerator chips that can serve as an alternative to Nvidia hardware, in a world where US export controls are increasingly restricting what China can buy.