Open Models and Maturation: Assessing the Generative AI Market

**Matt Bornstein** (0:00)
When you talk about multimodal, what that really means is starting to combine all of these things into, say, one model that can understand images and text as input modalities, for instance, or one model that can output both images and text as output modalities. I think there's pretty widespread belief that the architectures kind of converge from any of these models over time. For instance, we see transformers playing an important role in image generation now, which wasn't the case a year ago.
So it's likely that these architectures will converge more over time, and you'll see these models become more popular as these cases get a bit more mature, and it becomes more obvious why you would actually want multiple modalities to one model.

**Derrick Harris** (0:37)
Hi, you're listening to the a16z AI podcast, and I'm Derek Harris. This week, I'm joined by a16z partners Matt Bornstein and Guido Appenzeller to revisit some of the themes of a post they published in January 2023, titled, Who Owns a Generative AI Platform? While some things have remained constant, like NVIDIA's ownership of running these workloads, a lot more has changed. So we decided now is a good time to take an account of the space.
And although we did succeed in our attempt to publish this before OpenAI released GPT-5, we did end up recording before the announcement of the multimodal GPT-40 for Omni, as well as the related spate of new features for ChatGPT. Although that doesn't change anybody's stance on multimodal models, they seemed like a future last week and they still do today, it does seem like we'll now see adoption ramp up and some legitimate killer apps for multimodal models should begin to emerge. But we actually start off discussing the emergence of highly capable and competitive OpenLLMs before stepping back and having a broader discussion based on the themes of last year's piece. And then a look at what's to come and where there's ample opportunity for entrepreneurs.
As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.

**Guido Appenzeller** (2:11)
Tropics cloud has made a comeback, right? After cloud one and two were not particularly impressive. Cloud three is very impressive, and we now have two close source models set up. But I think the far more interesting trend has been that we've seen open source models, in terms of quality, narrow the gap to the close source models, right? Between Llama and Mistral and Databricks, we're seeing some very, very competitive models that at this point sometimes rank higher than some of the GPT-4 checkpoints. Not the best ones, but some of the GPT-4 checkpoints.
And that's sort of really, really exciting development, I think, because it gives people more options. I think the really interesting question is why that has happened.
So I think the main reason why is probably that everybody essentially trains on the same data, and today models are data-limited. In the early days of LLMs, we were mostly limited by how much compute we can throw at the problem, how many parameters we can have in our model. Let's really shift it. I think today, the training data set sizes are getting very large. We're talking like north of 10 trillion tokens. And there really isn't that much textual, preserved human knowledge on the planet to consume here. We're basically reaching the limits of everything we can find on the internet, all discussion forums, all Wikipedia, all books.
And if everybody at the end of the day is training on the same data set, or like very similar data sets, and everybody is roughly using the same model architecture, the net result is just that we're seeing these different models converge in terms of the quality of responses that they offer.

**Derrick Harris** (3:47)
Is that surprising? Like was that the expected outcome that we would see happening? It seems like they got really good a lot faster than maybe was expected, or maybe people would have foreseen.

**Guido Appenzeller** (3:57)
I think it's not completely surprising. I think one thing that we learned, which specifically was I think pioneered by Meta, was this idea of overtraining models. We had this idea that with Chinchilla optimality, there's a certain ratio between tokens and the model size. I think we've seen that we can create better models in the sense that they're smaller and still give very high quality answers by overtraining these models to a much larger degree. And the net result of that is that basically the number of parameters becomes a little less important and the amount of training data becomes more important.