Inferact: Building the Infrastructure That Runs Modern AI

**Simon Mo** (0:00)
Our goal is to make VLM the world's inference engine, really push the capabilities on the open source front, and then build a universal inference layer. That means we'll have the runtime to power any new model on new hardware for new application, be able to tailor that to extreme efficiency and support all the AI work we're going for. I fundamentally believe that open source, especially how VLM itself is structured, is critical to the AI infrastructure in the world. And what we want to do with Inferact is to support, maintain, steward and push forward the open source ecosystem. It is only that VLM, when VLM becomes a standard and VLM help everybody to achieve what they need to do, then our company in the sense have the right meaning and to be able to support everybody around it.

**SPEAKER_2** (1:01)
What if the hardest problem in artificial intelligence isn't training smarter models, but simply keeping them running? For most of the history of computing, once a system was built, the hard part was over. You wrote the program, pressed run, and the machine behaved predictably. Even early machine learning followed that pattern. Inputs were standardized, workloads were regular, the computer did its job and stopped. Large language models quietly broke that assumption. Every request is different. Prompts can be a sentence or an entire archive. Outputs can end instantly or stretch on indefinitely. Thousands of users can arrive at once, each making incompatible demands on the same hardware. And all of this has to happen in real time on GPUs that were never designed for this kind of unpredictability. Over the last few years, this problem has moved from obscure to essential. As models have grown larger, more diverse, and more deeply embedded into products, the challenge of running AI systems has started to rival the challenge of building them. That's where the tension lies.
A public story of AI progress is about better models and bigger breakthroughs, but underneath it is a quieter systems problem. How do you schedule chaotic requests efficiently? How do you manage memory when you don't know when a conversation is actually finished? And what changes when AI systems stop behaving like single turn tools and start acting like agents that think, pause, and interact with the world over time? This episode focuses on the hidden layer. We examine why inference, the act of running trained AI models, has become one of the most complex and important problems in modern computing and why open source infrastructure is increasingly central to solving it. Matt Bornstein, general partner at Andreessen Horowitz, is joined by Simon Mo and Woosuk Kwon, co-founders of Inferact and creators of the open source inference engine VLLM. This is a conversation about the infrastructure beneath AI and why it may matter more than the models themselves.

**Matt Bornstein** (2:55)
We are here today with Simon Mo and Woosuk Kwon, lead contributors on the VLLM open source project, and co-founders of Inferact, a new AI inference company. Super excited to have you guys on the show today. Thank you. Thank you so much for coming. We're going to talk a little bit about VLLM, the open source project. We're going to talk a lot about inference and what inference technology really is, and then we'll talk a little bit about Inferact, the new company. To start, can you talk a little bit about where VLLM came from? What is it? How did you start it? And why is it such an exciting project?

**Simon Mo** (3:25)
Thank you for having us. VLLM projects started from, actually, Woosuk's a prototype project at UC Berkeley doing his PhD, and grow into today's open source project on GitHub for inference runtime for everybody. Maybe Woosuk can talk a little bit about the page attention paper.

**Woosuk Kwon** (3:42)
Oh, yeah. So basically, I think it kind of started in 2022, when Meta released the OPT model as open source. I'm not actually sure how many people actually remember the model nowadays, but it was kind of one of the first open-weight large-engaged models to reproduce GPT-3. And our lab tried to create a demo service to run the model and to demonstrate it for a broader audience. And yeah, like it was working, but super slow. So I started a small side project to optimize that demo service. That was kind of at the beginning. And then initially I was thinking that it may only take like a couple of weeks to optimize the service and to win. But it turns out that it actually has a lot of open problems inside of it because this, you know, autoregressive language model is pretty different. Actually, it was pretty different from other traditional like ML workloads. And it wasn't actually, it was kind of like a brand new. And this like outside this Frontier Labs, back in the day, I started to work on it and it became a research project. And we wrote a paper and it even became a like a like open source project, pretty well defined open source project as more and more people got interested in it.