The Engine Powering Open-Source AI

**Matt Bornstein** (0:00)
The fun thought experiment is if GPUs dropped in price by 99%, then do we get back to a real open source world?

**Simon Mo** (0:07)
If moderation is never solved, in the future, people will go to open way by default, because that is where you know for sure you can control your guardrail for trusted use cases.

**Elena Burger** (0:18)
And can you talk about where Vllm sits in that stack?

**Simon Mo** (0:21)
Vllm is an inference engine. It is kind of like databases and operating system and other critical software to power AGI. NVIDIA, AMD, Google, their newest ship will make sure Vllm can run on them. And in a lot of cases, they use Vllm as a benchmark, while bridging almost a 10x gap. For proprietary models, there is a regular mode and a fast mode. But for open way, every provider can offer potentially even 10 different levels of speed.

**Elena Burger** (0:49)
Five years from now, open source AI models, have they closed the gap with frontier models?

**Simon Mo** (0:55)
Capability-wise, I don't really see a big gap. Not even today, because...

**SPEAKER_4** (1:01)
Open source AI has become one of the most important forces shaping the industry. But making frontier models available to the world requires far more than releasing model weights. It requires an entirely new infrastructure layer.
In this episode, Elena Burger and Matt Bornstein are joined by Simon Mo, co-founder and CEO of Inferact, to discuss how open source inference became critical infrastructure for AI. Why enterprises are increasingly turning to open weight models. And what the next generation of AI systems will require. They also explore model licensing, inference economics, and the future of open AI.

**Elena Burger** (1:41)
Today, we're here with Simon Mo, co-founder of Inferact and a lead maintainer of VLLM, the open source inference engine, now running on half a million GPUs at any moment. We're also joined by Matt Bornstein, an a16z general partner.
Simon, Matt, thank you so much for joining us. I think first we should start with open source AI and kind of the more recent history of open source AI. So VLLM actually has its origins kind of back in 2022, pre-Chat GPT, and your team set out to make a slow open source demo faster and instead just found this pile of unsolved problems. So can you talk about what means serving an LLM so fundamentally different from the ML workloads everyone already knew how to run? Yep.

**Simon Mo** (2:32)
Good to be here.
So serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs. And it is a computationally intensive process that will require a lot of engineering and ensuring that for each request, user can see the LLM's response quickly and efficiently. So this typically means we need to handle differences in input distribution, how long each request is, output distribution, which is non-deterministic, and batching and scheduling a lot more in the core of the inference engine.

**Elena Burger** (3:13)
Yeah. And so the team, the project has been around for about four years now, but the company is a little bit more recent. So I want to throw this out to either of you, Matt. I know you've known the team and observed the team for a very long time. So at what point did you see this sort of transitioning from being a much beloved open source project to critical infrastructure and then a company?
Yeah.

**Matt Bornstein** (3:39)
I mean, I think you have to go back a little bit, right? Open source was the norm for AI models early on, right? I mean, we literally have this company called OpenAI, which it's become a little bit of a joke. It's not as open as it once was or not nearly as open as it once was. But early on, all the frontier AI work was being open source or at least released into open weights, which is a little bit different than true open source. And people could mostly run these models sort of on hardware or computers that they already had. I'm curious, Simon, like maybe you remember like what was like the first model that like you actually needed to go out and get special software and like a special set of computers to be able to run?

**Simon Mo** (4:14)
Probably BERT, and before that, it was like ResNet for computation, like images, computer vision, classification. So ResNet already need to run on NVIDIA K80, which is kind of one of the first SKUs on AWS and other places.
But even at this point, ResNet, you can still kind of run on a commodity, even CPU devices, it's just very slow. But for BERT, where running at it is like, wow, you have to run it on GPU to make it anything faster and efficient for anything translation or any task. So that was before 2020 even.