Mistral 7B and the Open Source Revolution With Arthur Mensch, CEO Mistral AI

**Sarah Guo** (0:06)
Open source AI models have completely changed the landscape of technology over the past year. One tiny team of ex deep mind and meta researchers in France has made a huge splash recently, Mistral.
This week, Elad and I are joined by Arthur Mensch, the CEO and co-founder of Mistral, who recently released Mistral 7B, an Apache 2 licensed open source model that has changed people's mental models about what can be done with small models. Arthur, welcome to No Priors.

**Arthur Mensch** (0:33)
Thank you for inviting me. Very glad to be here.

**Sarah Guo** (0:36)
OK, so just six months ago, when we met, you were leaving DeepMind to start Mistral.
It takes real guts to look at the scale of dollars and compute that open AI and Google and others have amassed and say, like, we want to play in this game too, and it's important we do. Tell us about the inspiration to start.

**Arthur Mensch** (0:55)
Me, Guillaume, and Timothée were, I guess, pretty early in the field. And it had been 10 years that we had been doing machine learning. And we did know where to start from and how to make a good model with a limited amount of compute and money. Well, not so limited, but at least more limited than where we were coming from.
And so I think that's why I get to start the various companies we were in, move into directions that we hadn't anticipated when we joined the company. And we decided that there would be a very good opportunity for creating something that would be a standalone company in Europe, focusing on making AI better, focusing on making Frontier AI and focusing on making open source AI as a core value.

**Sarah Guo** (1:41)
Maybe we can talk about each of those pieces. So 10 years in machine learning before, you were a co-author on the Chinchilla scaling laws paper. You worked on the sort of mixture of experts ideas early. Can you talk a little bit about what your research directions were at DeepMind?

**Arthur Mensch** (2:00)
Yeah, so I come from an optimization background. So my focus has always been for the last 10 years to make algorithms more efficient and to use better the data that we have. To make models with good prediction performances.
And so when I arrived at DeepMind, I joined the LLM team that was 10 people at the time. And very quickly, I started to work on retrieval of augmented models.
So with a paper called Retro that I collated with my friend Seb Bourgeau, who is still at DeepMind. The point there was to use very large databases during pre-training so that we didn't force knowledge into the model itself and we would tell the model that it would have access to an external memory anyway. And so it was working quite well. We could actually lower the perplexity, let's say. That's what you work on when you make LLMs.
There were some limitations that we that I think the community has started to address quite quite well. And that was at the time when retrieval methods weren't really mainstream. Now they've become completely mainstream. So that's the first project I did. I worked on a sparse mixture of experts, also quite quickly, because that was related to my topic of postdoc, which was optimal transport. So optimal transport is a setting where you have, I guess, tokens you need to assess them, assign them to devices, and you need to make sure that there's some good assignment in between the two of them so that the devices don't see too many tokens. And as it turns out, the way you do it is with optimal transport. Optimal transport is a mathematical framework to do it correctly. And so I started to work on introducing this two sparse mixture of experts. And very quickly, we started to move on to scaling loads. So how do you actually take the method that is working at a certain scale and try to predict how that will evolve with the scale, the number of experts, the amount of data you see.
And so that's a work I've done with many colleagues as well. On how do you adapt the scaling loads for dense parameters, for dense models, to a setting where you want to predict the performance not only with relation to the size of the model, but also the number of experts. I guess that was the second thing I worked on. And then connectedly worked on Chinchilla, which is, I think, a major paper in the history of LLM.
Also with Seb, Jordan, Laurent, many other people. Basically, the story was that everybody was training models on too few tokens because of the paper from 2020 that happened to be not very well executed. And so what we observed is that you could actually correct that. And so instead of training very large models on very few tokens, you should actually grow the number of tokens as you grow the size of the model, which if you think about it, it makes a lot of sense because you don't want to have infinite size model looking at a finite size, a finite number of tokens. And similarly, you don't want to have a finite size model looking at an infinite number of tokens.