Joo-Young Kim, HyperAccel CEO on AI Chips, LLM Inference & Efficiency

Grace Gong (0:01)

Hi, Joo-Young. Welcome to Venture with Grace.

Joo-Young Kim (0:04)

Hello, everyone. I'm Joo-Young Kim, CEO of HyperAccel.

Grace Gong (0:08)

I'm so excited about our conversation, but before diving, I want to give a quick shout out to our amazing sponsor. This episode is brought to you by Nebius, the ultimate cloud for AI innovators. Nebius provides AI infrastructure you can count on, combining reliability and speed with flexibility and engineering support unmatched by hyper-skillers. AI leaders like Meta, Shopify, and Higgsfield already partner with Nebius to run their AI workflows. Plus, venture-backed startup can save up to 150K on compute costs when they apply for access. Visit nebius.com or nebius.com/startup to learn more.

I want to start with your journey because obviously you went to the MIT of Korea, and I know that a lot of the innovation, that's where a lot of the innovations are coming from Kaist, and you were visiting Scholar and eventually work at Microsoft before starting your company. Maybe we could start from what were some core lessons that you've learned early on in your career shaping into who you are today.

Joo-Young Kim (1:11)

Before I joined Kaist, actually I used to work for Microsoft for several years, working on AI accelerator for the data center application. That was FPGA based, but kind of one of the oldest in the world, like even before the TPU comes out. So I saw that time a lot of opportunities in the AI hardware acceleration for kind of a coming compelling workloads.

That time I saw a lot of opportunities there. I learned a lot on the software optimization and deploying hardware at scale. And I then came back to Korea as a professor there. So I also heard that the transform model becomes a big trend in the AI algorithm side. So I decided to work on the transformer side because the model side is really big. So I thought like I can actually contribute to this because I'm a hardware engineer. So that time with my graduate student, I prototyped the FPGA based server system that accelerates the transformer inference end to end.

I saw the big boom with the ChatGPT at the end of 2022 So I found the company in January 23

Grace Gong (2:43)

I feel like not many company can say they want to go against Nvidia. Maybe we could start from the kind of like, obviously nowadays many company are, I guess it's really difficult to go against the giants in general, but what are some insights that you have seen to give you the confidence of going after this category?

Joo-Young Kim (3:08)

Yeah.

GPU is really a great computing platform for performance. It's a general proposed AI accelerator.

You can use it for training as well as the inference. But one thing is that GPU is actually not designed for LLM inference. LLM inference has unique workload characteristics, and the GPU itself is a really para-processing machine, but it's not really designed for the LLM inference because it needs to generate the token one by one, and the effective memory bandwidth is a really important metric there. But GPU has very complex memory hierarchies. So I saw a kind of opportunity there. So if we make a really specialized and optimized machine for LLM inference, then we can actually get into the market. And especially if you look at the AI era, that's a really big boom that changes everything. So we thought that there is a spot for our specialized silicon for this upcoming era.

Grace Gong (4:28)

I think there's a concept that one is like pre-fuel, one is decode. And assuming you guys are going after the decode aspect, maybe if you want to dive into more about these two AI inference stages, and how are you guys competing in that category?

Joo-Young Kim (4:51)

Right. So as you mentioned, LLM inference has two broad stages. The first one is the pre-fill that processes the input tokens in parallel. That's more like compute-oriented one. And the other is the decode that generates the output token one by one. And when it does that, you actually have to read the entire model and key value cache again and again. So it's really memory bandwidth limited. So those two are really different, but you have to actually handle those two together in the inference side.

So of course, GPU can do that very well, but it has like low hardware utilization problem. So we actually wanted to solve that with our own LPU technology.

So I can actually go over the kind of technical highlights here.

The first one is the big specialized engine for the inference processing. So unlike the conventional MPU and GPU has like thousands of small cores in the array, but they have very complex memory hierarchies. So it only achieves like 50 to 60 percent of the memory bandwidth utilization, because there are so much data movement overhead because between the thousands of small cores and the big global buffer. So we actually wanted to solve this with the chip level architecture called Streamline Data Flow Architecture. So in this architecture, we simplified all the data paths from the memory channels to the cores. So in summary, we have the big core that can actually generate the token by itself, and on top of that, we simplified all the data flow really like streamlined. So all the data only flows once, and each of the big core generators, the token, by doing this, we can actually utilize up to 90 percent of the memory bandwidth given. So that's the kind of core architecture of our LPU, and it can actually solve both of the pre-fill and the decode side, and that's the kind of fundamental thing of our architecture. And regarding the pre-fill, we actually add good batching capability at the token level. So those two are kind of technologies behind our LPU to address decode as well as the pre-fill.