Piecemakers bets edge AI devices will diverge from reliance on HBM — custom-designed memory fuses DRAM stack directly to the processor using hybrid bonding

Skyler Monroe (0:10)

Hey, everyone, welcome back to the AI Hardware Show. I'm your host, Skyler Monroe, and if you're into chips, silicon, and the hardware that's actually making AI happen, you're in exactly the right place. Big thanks to our sponsors keeping this show running. Ada, helping businesses integrate AI into their workflows. Ago Consulting, that's Ago, your go-to for silicon development from AI to SOC, and Zen Semiconductor, building the processors and AI fabric behind modern data centers with ventures like their Sierra RISC-V CPUs and Loom AI fabric, plus Ada for deploying AI at scale in any business. Today, we've got a packed episode. We're talking about a bold bet against HBM memory at the edge. Nvidia squeezing 24% more output from the same power budget. How NVLink 6 keeps massive AI clusters alive. A deep dive into how billions of dollars worth of restricted Nvidia chips are still finding their way into China. And a startup called D-Matrix that just joined Nvidia's ecosystem. Let's get into it. All right, first story, and this one is genuinely interesting from a memory architecture standpoint. A company called Peacemakers, backed by Nanya, one of Taiwan's major DRAM manufacturers, just started trading on the Taipei Stock Exchange, and their entire thesis is essentially a contrarian bet. They think high-bandwidth memory, HBM, is not the right answer for edge, AI inference workloads. Now, that's a bold stance to take when basically every major AI accelerator story you hear involves HBM, this, HBM, that. So let me back up and explain why HBM became the dominant memory solution for AI in the first place. When you're training or running large AI models, the bottleneck is almost never raw compute. It's feeding data to the processor fast enough. Your GPU is hungry, and it needs memory that can shovel data into it at enormous speeds. HBM stacks multiple dram, dies vertically, connects them with tiny through silicon vias, and then places that stack right next to the processor on an interposer. The result is massive bandwidth, we're talking over a terabyte per second on modern implementations. But here's the thing, HBM is expensive, like really expensive, and it consumes significant power.

For a data center running a handful of massive H100 or B200 clusters, that cost is justifiable.

But what about edge devices? What about inference hardware that needs to run in a base station, or an industrial controller, or even a relatively modest server at the network edge? Suddenly, the economics and the power envelope of HBM start looking a lot less attractive. Piecemakers is attacking that problem with a different approach. They're fusing DRAM stacks directly to the processor using hybrid bonding. Now, hybrid bonding is genuinely fascinating technology. Instead of using traditional solder bumps to connect chips, hybrid bonding uses copper to copper connections at the nanometer scale. You're essentially welding the chips together at the atomic level. The result is much denser interconnects, lower power, and shorter signal paths compared to traditional packaging. Think of it this way, traditional chip packaging is like connecting two cities with a highway.

HBM with an interposer is like building a bridge between them. Hybrid bonding is more like tunneling directly through the mountain, you get a much more direct efficient path. The bandwidth per watt improves significantly, and you can get really tight integration without the cost overhead of a full HBM stack and interposer. The key question is whether this approach can deliver enough bandwidth for demanding edge inference workloads. Large language models and vision models are still extremely memory hungry, but Piecemakers is betting that for edge inference specifically, where you're running a fixed model not training, and the model size is more constrained, their solution hits the sweet spot of performance, power, and cost. If they're right, this could open up a whole new category of purpose-built edge AI memory. If they're wrong, well, HBM wins by default. Either way, it's a fascinating technical bet, and one worth watching closely. Okay, story two. Nvidia doing Nvidia things, but in a pretty clever way. Their DSX technology, which stands for data center scalability, is apparently delivering a 24% improvement in AI data center output without increasing the power budget. Let me say that again. Same power, 24% more work done. In an industry where power is genuinely one of the most constrained resources right now, that is a significant number. So what is DSX actually doing? At a high level, it's a suite of optimizations that work across Nvidia's hardware and software stack to reduce wasted cycles and improve utilization. The dirty secret of large AI clusters is that raw theoretical compute numbers, those peak FLOPs figures you see in spec sheets, rarely translate directly to useful work. There's overhead everywhere. Network communication between GPS, memory transfers, synchronization points between operations, all of that burns time and power without actually advancing the computation. Think about it like a highway system. You could have the fastest cars in the world, but if they're stuck at traffic lights every 100 meters, or if they spend half their time in the wrong lane, your overall throughput tanks. DSX is essentially intelligent traffic management for the data center. It's making sure the compute, the memory bandwidth, and the networking are all synchronized, so nothing is sitting idle waiting for something else.