Does AI Understand the Machine It Runs On? | Inside OpenAI

Ksenia (0:00)

When OpenAI talks about democratizing intelligence, these are some of the people who have to figure out how to make that happen. Welcome to the Inference show. My name is Ksenia, and today I'm talking with Philippe Tillet and Matthew Ferrari, two people whose work becomes part of your experience every time you send a request to ChatGPT. Phil is the creator of Triton, a programming language that lets people write efficient GPU code without having to become CUDA specialists. He joined OpenAI as an intern in 2019, before there was an API or a product, and he spent years working on training efficiency. So he has watched this whole operation change from the inside. Now the models are surprising him with their own optimizations, and even changing what he needs from the tools he helped build. He also has a philosophical side with an interest in Bertrand Russell and philosophy of mind. Matt is a phenomenally bright engineer, and very much a practitioner. He works across the inference system, looking at how requests are distributed, how hardware is used, and how models can help investigate problems. One thing he explains particularly well is why making one piece faster might barely improve the experience for the person waiting for an answer. You have to understand how everything works together and decide where your effort will make a difference. And that's what interested me about their collaboration with AI. Coding is great, but when did they realize a model could help them think through the engineering too? We recorded this interview on Astra's launch date with our examples coming from Sol. But the question applies to every new model. Once you've built something more capable, how do you make that capability available to more people? Phil and Matt, take us into the work behind that. Enjoy.

Ksenia (1:48)

Hello, guys. Thank you so much for joining me for this interview. It's a very fascinating area of research. I wanted to address the blog that you posted in July, and there you wrote that efficiency has been central to distributing the intelligence to a wider audience. Can you elaborate? What does it mean for you?

Philippe Tillet (2:06)

Efficiency and democratizing intelligence is a big part of our mission here at OpenAI, and obviously, we have a limited finite number of compute that we have to distribute between everyone that we want to serve. Every time we make our inference stack faster and more efficient, this allows us to distribute our products more broadly to free users and to pay users and to everyone, and this is why we're pushing so hard on this.

Ksenia (2:32)

Phil, you joined OpenAI in 2019 You were an intern and then you were an author of Triton at that time. I know it's a wide question, but what has changed for you as an efficiency and inference specialist, thinking about 2019 and now?

Philippe Tillet (2:47)

Yeah, a lot. So when I joined in 2019, inference was not a thing. Like OpenAI didn't even have an API or a product. I joined to do pre-training work. So I spent my first four or five years actually working on pre-training efficiency, all the way from GPT 3.5 to our latest models. And over the past few years, I've seen like inference become more and more and more important as our products have hit the market and the industry has grown.

Ksenia (3:16)

And so it's so casually.

Philippe Tillet (3:18)

Yeah, it's been maybe this sensitized a little bit.

Yeah. And I think just the scale has become like mind-blowing. When I joined, we had one cluster of 10,000 V100s that we were using for pre-training. And now like it would be a drop in the bucket today. Yeah, that's been quite a journey.

Ksenia (3:39)

Matt, what does excite you the most since you joined when it was already all happening?

Matthew Ferrari (3:45)

Yes.

Ksenia (3:45)

What excites you the most about Inference?

Matthew Ferrari (3:48)

Yeah, so I think it's changed so dramatically. It's so crazy to say that only in one year's time to see it transform so dramatically. When I joined, a lot of the work that we had mentioned in the blog post that you spoke of, it looked so dramatically different than it does today. There was a lot of tuning kernels by hand, investigating efficiencies, maybe one or two layers deep into the stack. But now, with all of the tools that are available, Codex, GPT 5.6 Sol, enables you to do so much more and dive so much deeper into the performance. So the work has dramatically changed, and also the impact that you're able to have just as one person working on efficiency end-to-end. So I think it's been really exciting.