From Voice Agents to AI Avatars with Alexander Smola

Sam Charrington (0:00)

Voice AI has gotten very good over the past few years, but it still suffers from a bit of an uncanny valley problem. Humans are remarkably sensitive to the subtle things that make a conversation feel natural or not. A little too much latency, an interruption handled awkwardly, the wrong tone or emotional response, and suddenly the illusion breaks. And the bar only gets higher as AI systems evolve to incorporate more of our senses, moving beyond text and voice to systems that can see and be seen, introduces a whole new set of challenges around natural interaction.

My guest today is Alex Smola, co-founder and CEO of Boson AI and a professor at Carnegie Mellon University. Our conversation explores what it takes to build these systems, from the models and inference infrastructure needed to make voice work in real time, to emotional intelligence, and ultimately audiovisual avatars. Here's Alex on where he sees all of this heading.

Alexander Smola (0:57)

I would argue that voice is an intermediate stepping stone. You might think, well, you know, what's next? It's clear that eventually we will have avatars. And so I think what this is converging to is that you'll be talking to an aviation that looks and feels like a human. And we're still very, very far away from making this really natural.

So it's quite exciting, actually.

Sam Charrington (1:23)

I'm Sam Charrington, and this is The TWIML AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

When I think about kind of the frontier of voice AI, that most folks probably have access to, I'm imagining it's something like ChatGPT advanced voice mode.

Like, and actually, I want you to react to this. Do you think that actually that's, you know, crap and there are much better systems, and you should point me to, you know, X, Y, and Z, or are there, you know, are there better systems behind closed doors and labs and enterprises or what? But in general, I tend to think that, you know, they've gone through several iterations of it. It does keep getting better, but it's still very infuriating. And in my experience, it really only works in perfect conditions, meaning a silent room, no background noise, being very conscientious about interrupting. If you interrupt, either you have to be committed to plowing forward, or you have to stop and let the thing, you know, catch up to you. You can't like, you know, try to have a natural engagement. It's very brittle, I think.

Alexander Smola (2:48)

So you can definitely do better than that situation.

So fun situation from last week. So I was in Jeju in South Korea for the KDD conference, and we were out at dinner for drinks in a bar.

And basically, I was showing off our system and just me foolishly saying, hey, watch this, let's see what happens. And so what I can confirm is that our model was able to handle people switching to Slovenian and then somebody else to Hindi fairly well, even though the bar was very noisy. I would say a non-trivial amount of the credit definitely goes to the iOS team doing really good noise cancellation and audio separation on the mobile phone. So I don't think that this would have happened just by waving a microphone somewhere. But I think we're on our way. And so basically, you know, having a single microphone will probably never really solve this. You need microphone arrays to really do proper noise cancellation, especially if you have many sources. But that's a solved problem, right? I mean, those things ship.

Sam Charrington (4:18)

Well, I've got a huge microphone array sitting in my laptop here. I don't know if Codex was using it, but like I've also had like kind of eh, experiences with the relatively new Codex voice. Just that interaction just wasn't very fluid.

Alexander Smola (4:35)

Yeah, okay. So that's, I mean, sometimes voice input is good to if you, if you're feeling too lazy to type. I find typing way more precise for technical work. But for, so as in, you know, hey, let's configure our storage array. Well, no, I don't want to talk to you. I want to type. Because here's a very specific serial number and a specific URL, and I want to get this exactly right. So that's where I still prefer text, but maybe this is old school. But for, you know, if you're out in a bar and you're asking for information or advice or something, I think we're getting there. I will give it another year and this will actually become quite ubiquitous. And I'm pretty happy where we are at with our models. I think probably in a year, audio will become pretty much bulletproof.