Mistral: Voxtral TTS, Forge, Leanstral, & what's next for Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

**swyx** (0:05)
Okay, welcome to Lain's Space. We're here in the studio with trustee co-hosts, Vibhu. Welcome.

**Vibhu** (0:11)
I'm excited for this one.

**swyx** (0:12)
As well as Guillaume and Pavane from Mistral, welcome. I'm excited to be here.

**Guillaume Lample** (0:17)
Thank you for having us.

**swyx** (0:18)
Pavane, you are leading audio research at Mistral and Guillaume, you're a chief scientist. What are we announcing today? We're coordinating this release with you guys.

**Guillaume Lample** (0:26)
Yeah, so we are releasing VoxTrial TTS. So it's our first audio model that generates speech. It's not our first audio model. We had a couple of releases before. We had one in the summer. That was Boxtral, our first audio model. But it was like a transcription model, the ASR. Maybe like a few months later, we released some updates on top of this, supporting more languages. Also, a lot of tables take features for our customers. Context biasing, position, timestamping, and the transcription. We also had some real-time models that can transcribe not just at the end of the web. You don't need to fill your entire audio file, but that can last come in the end time. And here this is the natural extension in the audio, so basically speech generation. So yeah, so we support languages. And this is a pretty small model, 3D model, so very fast. And also, stay up there. Performed at the same level as the best model, but it's much more efficient in terms of cost. And also much, in terms of cost, it's also much, only a fraction of the cost of our competitors. And we are also releasing the work of this model.

**swyx** (1:25)
Yeah. Mammal linked?

**Guillaume Lample** (1:27)
Not this time.

**swyx** (1:28)
What's the decision factor?

**Guillaume Lample** (1:32)
It's a good question. There will be more.

**swyx** (1:36)
Yeah, for one. Any sort of research notes to add on what you could say?

**Pavan Kumar Reddy** (1:39)
Maybe we'll dive into it later in the podcast too, but it's a novel architecture that we developed in-house. We iterated on several internal architectures and ended up with a autoregressive flow matching architecture and also have a new in-house neural audio codec, which converts this audio into all point by herds, latent tokens, semantic and acoustic tokens. And yeah, that's the new part about this model. And we're pretty excited that it came out with such good quality. And Tium was mentioning, yeah, it's a 3B model. It's based off of the Ministral model that we actually released just a few months back and installed Trunk. And it mainly meant for like the TTS stuff, but they need text capabilities are also there, yeah.

**swyx** (2:24)
So there's a lot to cover. I always love anything to do with novel encodings and all those things, because I think that's obviously increase a lot of efficiency, but also maybe bugs that sometimes happen. You were previously at Gemini and you worked on post training for language models. And maybe a lot of people will have less experience with audio models just in general, compared to pure language. What did you find that you have to re-visit from scratch as you joined Mistral and started doing this?

**Pavan Kumar Reddy** (2:53)
At least when it comes to, I think there are two buckets, I guess. The audio understanding and audio generation. The audio understanding, like the Walkthrough models that Keeam was mentioning that we released earlier. The Walkthrough chat that we released, I think, July last year. And the follow-up transcription on the models family that we released in January. That would be one bucket on the generation is another bucket. I think you can also treat them as a unified set of models. But currently, the approaches are a little different between these two. To your question on how audio is fed to the model. In the understanding model, it's very similar to actually the Mistral models that we also released. Yes, that was amazing. Yeah, that was the first project I worked on after joining Mistral. It was pretty nice. And Walkthrough was very similar in spirit, I guess. So we feed audio through an audio encoder similar to images through a vision encoder. And it produces continuous embeddings, which are fed as tokens to the main decoder transformer model. Yeah, and the model output is just pexed. So on the output side, there is nothing that needs to be done in these kinds of models. I guess the interesting part about the generation stuff is the output now has to produce audio. And the approach that we have is this neural audio codec, which converts audio into these latent tokens. There is a lot of existing retrace and a lot of models, which are based off of this kind of approach. And we took a slightly different design decisions around this. But at the end of the day, the neural audio codec converts audio into a 12.5 Hz set of latents and each latent has a semantic token and a set of acoustic tokens. And the idea is that you take these discrete tokens and then feed it on the input side.