The AI Infrastructure Stack with Jennifer Li

**SPEAKER_1** (0:00)
Thanks for listening to the a16z AI Podcast. I hope you're enjoying the summer so far. And in the spirit of summer, we're taking a break from our usual format to run an episode that originally aired on the Complex Systems Podcast, which you can find online at complexsystemspodcast.com, and we'll also include subscription links in the show notes. Now, as you'll figure out pretty quickly, we're re-airing this one because it's a great discussion with a16z general partner Jennifer Li about the changes AI is bringing to the infrastructure layer.
Jennifer gets into the nitty-gritty of enterprise infrastructure, but also touches on the often unspoken reality that infrastructure ultimately powers products that are supposed to improve people's lives. It's a wide-ranging and insightful discussion that you'll hear, hosted by Complex Systems host Patrick McKenzie, after these disclosures. As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security. And is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.

**Patrick McKenzie** (1:12)
Welcome to Complex Systems, where we discuss the technical, organizational, and human factors underpinning why the world works the way it does. Hi, everybody. My name is Patrick McKenzie, better known as Paddy O'Levin on the internet. And I'm here with Jennifer Li, who's a general partner at a16z.

**Jennifer Li** (1:34)
Hi, Patrick. It's great to be here. Thanks for having me.

**Patrick McKenzie** (1:37)
When in the course of human events, it becomes necessary for one man to read a disclaimer, a decent respect to the opinions of mankind compels him to read exactly the text written by compliance. Please note the content here is for informational purposes only, should not be taken as legal business tax or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures. Thanks very much for coming on. So one of the things that I think is underappreciated when there's a major shift in how people consume software is that all software is built on many layers of infrastructure, both software infrastructure and hardware infrastructure. We seem to be in the opening innings of what I think will ultimately be a transformation of society due to the AI shift, but which is certainly impacting the software and hardware stack we work on top of. That seems to be your primary investing focus at a16z. So we'd love to just chat with you about what are the things that people might not see behind the chat windows when they're using these products.

**Jennifer Li** (2:38)
Amazing. This is my favorite topic. My everyday life is living and then breathing through infrastructure changes, and I feel like my whole career has been waiting for this moment where every single piece of the stack is shifting.
It's definitely a very exciting time for me. As you said, people probably see a lot of the products, whether it's consumer or B2B, just around the capability of the models. But I think you'll realize how much infrastructure is being built or needs to be advanced to support this new type of workloads. And I'm very excited to dive into this.

**Patrick McKenzie** (3:10)
Sure. So I guess we can start close to the metal or close to the user, whichever you prefer.

**Jennifer Li** (3:14)
Let's start with the user. I came from a product background and I always like to think of a problem when it's close to the user. So let's start from there.

**Patrick McKenzie** (3:21)
Sounds great. So presumably, there's the application layer these days and then a model layer beneath it, sometimes the same people. And what we've seen in previous iterations of this game in SaaS and similar is there ends up being a middleware layer between those two providers as well. Is there a developing middleware in the AI yet?

**Jennifer Li** (3:38)
So the answer is yes. And this middleware, whether we start from the frameworks themselves or connectivity tissues, protocols, pipelines, there are quite a few moving pieces. And I spent a lot of time in thinking about what is the new application stack look like. And to be honest, largely, it's not that dissimilar to the current application stack, where you still have databases, you still have CDNs, content delivery systems, you still need the front end client and servers. A lot of these things are not changing. However, given now we have this new modality or like new capability called AI and AI agents, it does put a lot of pressure in thinking about what does real time and low latency workload look like when we're, let's say, delivering large amount of image and video inference or audio inferences, and where does that capacity and also capability come through. So that's an area where I have been spending quite a bit of time. Maybe to dive into one area specifically, like I think a lot of the applications we're seeing today are mostly centered around large language models, and whether they're delivering already amazing answers to your questions around a product, a PDF documentation, retrieving information and knowledge. But I think a very underestimated and very appreciated area is this diffusion model world, where lots and lots of creative tooling are being revamped and reimagined. Well, we know large language models are really powerful, but these diffusion models are incredibly creative, and they're incredibly diverse too, given the different genres and capabilities that produces either great graphics or really imaginative images. And it's a huge infrastructure challenge to deliver low latency and high quality outputs. This is where I'm seeing on the Infra layer, the capabilities are bifurcated, or at least how the infrastructure stack is being built is bifurcated based on modality, where language models, I think we know, are more expensive or transformer models, let's put it that way, more expensive to train, but they're a lot more economical to inference versus diffusion models needs several steps to inference. And also we know these multimodality assets or multimedia assets are much more expensive to deliver to end users. So there needs to be quite a bit of optimization done at the infrastructure layer to make sure the steps are well optimized and also deliver through very low latency, very fast throughput way to not having the user, let's say, wait for 30 seconds for a video to be generated. And even just compare the last two years, the optimization has really created new experiences for users and we're definitely happy to dive into that.