Speech Recognition Is Not a Solved Problem — Pavan Muddireddy

Pavankumar Reddy Muddireddy (0:00)

I think a lot of people don't know how audio generation works. Voice is one of the primary ways humans communicate. It definitely came much before text was a thing.

The world looked very different, I think, with neural codecs and autoregressive generation, which was really cool that the field progressed so far, and when we made it work, it was pretty cool. It's an autoregressive architecture, so what you predicted already is, in a sense, committed, so it becomes context and you can't edit it out. It's like a voice, an end-to-end speech model, but primarily to serve as a interface to other systems. Yeah, I'm a research scientist at Mistral. I work on audio research at the moment. I lead the audio research team here at Mistral. I've been broadly focused on multimodal models at Mistral since I joined like two years ago.

Tim Scarfe (1:00)

This episode was produced in partnership with Mistral AI. So, Mistral is a full stack AI company, and you're in the thick of it. You're doing the actual research. What does that actually mean deep down?

Pavankumar Reddy Muddireddy (1:14)

Yeah, definitely. The full stack description of Mistral applies to the whole company as in it's serving needs at different layers of the AI stack, I suppose. I think the company is most well-known for the model layer. Since the original Mistral 7b, we've been releasing open models, both text models and over time, multimodal models and reasoning models. That's one layer that we continue to focus on and one of the most important layer and the place I work on primarily. On top of it, we have offerings at the application layer. We have Mistral Vibe, which is our coding offering, a chat app, and also a command line based interface.

On top of it, we also have Vibe Work, which is a similar agent tech working experience. That's at the product layer. We also have AI Studio, which has our API offerings and workflows, where you can build AI agents easily. We also have Forge, which is a fine-tuning platform, and also other kinds of model adaptations, where you can take some of our non-open models as well and just adapt them to your specific needs. Again, applies to both text models and multimodal models.

Over the last year or so, we also have Mistral Compute, which is the AI Cloud offering, which is bare bones compute, on top of which many of the other work happens, including the research work. It's a full-stack AI company in that we do have some presence and offering across various layers of the AI stack, because most of the enterprises which tries to incorporate AI usually require integration at several points in their stack, and we try to meet them where they are, and give them the efficiency gains or productivity boost that they're looking for.

Tim Scarfe (3:43)

Very cool. What's it like though, just being a research scientist at a frontier AI company? I mean, you must be looking at lots of interesting research that other folks are doing, going to conferences and so on. I mean, how is all of that work being integrated into your workflow?

What does your day in the life look like?

Pavankumar Reddy Muddireddy (4:03)

Yeah, that's a great question. It's really fun to work at a frontier company, which is also a general frontier company as opposed to a task-based company. Again, I think there are trade-offs. It's not as strict. One is better than the other. But for me personally speaking, it's always nice to have visibility into the whole work happening across the board. Even though I focus on audio, I closely model our work on text pre-training, post-training, reasoning, vision. And more recently, we have ventured into robotic foundation models as well.

While I don't work across the board at any given point in time, it gives me good exposure to what's going on. And it also, I think, helps me incorporate the latest techniques because I think at this point, not just for audio, but across modalities and tasks, the approaches are converging. And the paradigm of pre-training, post-training, there are subtle differences and specific nuances to specific niches. But at the same time, the broad techniques are super general. I mean, that's the success of the latest paradigm, I suppose. So, an average day involves doing a lot of my own research work, training models, running ablations, investigating evals and stuff, but also following the work that's happening across the company.

Tim Scarfe (5:44)

Yeah, and I suppose it's cool as well because you folks get to publish a lot of your research, so you can kind of speak openly to the community. And a lot of your models are also open weights as well.