REPLAY: Scoping the Enterprise LLM Market

**Naveen Rao** (0:00)
I have been in this field for a while, just from a sheer interest standpoint. I spent 10 years in industry as a computer architect and software architect, and then went back to get a PhD in neuroscience for the reason of can we actually bring intelligence to machines and do it in a way that's economically feasible. Overall, I'm very happy and excited about where the world has gone. I mean, being the sideshow is never as fun, to be honest with you. You can get passion from that because I think it actually, when everyone's telling you you're the sideshow, you kind of have to be passionate to keep going. And I think you can use that as strength, but really the whole point of having that passion, that strength is to make something that's meaningful and something that's lasting, something that really does change the course of human evolution.

**Derrick Harris** (0:44)
Hi there, and thanks for listening to the a16z AI podcast. I'm Derek Harris. If you're listening to this episode on the day it published and you're in the United States, happy Black Friday. Now, this is actually a re-airing of our first episode from back in April, featuring myself, Databricks VP of AI Naveen Rao, and a16z partner Matt Bornstein. A lot might have changed in the AI world since then, but this discussion about the state of enterprise LLM adoption and the overall market demand for LLMs remains both valid and insightful. All the background you should need is that Naveen has been in the AI space for more than a decade, building both custom chips and models, and that we recorded this on the heels of NVIDIA's GTC event in March, so we naturally kick off the discussion on the topic of NVIDIA. As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.

**Naveen Rao** (1:58)
NVIDIA has also been really on top of each trend. So, kudos to them for doing that really well. I mean, they've been able to like see the trend for whatever thing it is, low precision, tensor cores, what have you, and execute extremely well. So, it's just a formidable competitor for anyone to go up against. And then, you know, everyone talked about the sort of kudos, lock-in, that sort of thing. I actually don't think that's the reason anymore. I think it's just they've become the gold standard, and it introduces risks to move to any other hardware platform.
We're always looking at new hardware to see if we can find a better TCO. Basically, like, effective FLOPS per dollar is the number I look at. And it's hard because they do build a good part, and, you know, we can extract a lot out with the mature software stack that exists. That's really what keeps it locked in, is just the maturity.

**Derrick Harris** (2:52)
When you say you look at other hardware platforms, are you at liberty to divulge what those are? My brain immediately goes to what the cloud platforms are building. Obviously, there are some startups experimenting in the space, but I'm curious what you're looking at.

**Naveen Rao** (3:04)
We talked to all of the above. But at this point in time, it's still very hard to move away from NVIDIA because if we're trying to go build models for some purpose, that represents the shortest path to the goal, if you will. Anything else introduces some friction at this point. Now, I think by the end of the year, that might actually change. There might be some other players that are capable of getting to the end goal without so much friction. We're building our software stack to make that really easy for our customers, really deliver the best TCO to our customers through a stack that they already know how to use. We got a lot of folks who are building on top of the mosaic Databricks stack. Yeah, if we can extract away a lot of these hardware details, we can make it such that our customers can have more choice.

**Matt Bornstein** (3:50)
Naveen, on that point, language models have, for the most part, standard around the transformer architecture now, obviously. It seems like that's creating an opening for chip companies to tailor their products to a more homogenous set of workloads. Do you think that's true? And if so, do you think it's a good thing or a bad thing for the industry?

**Naveen Rao** (4:11)
It's definitely true. If you go back five, six years, you had to support so many different families of neural networks. There were common nets and RNNs and LSTMs and this, that and the other thing.