How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

SPEAKER_1 (0:01)

This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up require compatibility and availability varies 18 plus.

SPEAKER_2 (0:28)

When you've got a job to do, Lowe's knows how to help you keep it moving. MyLowe's Pro Rewards members save more with member volume discounts on eligible orders through a quote of $2,000 or more. Plus, save time with job site delivery on select purchases. Not a member? Join for free today. Exclusions, more terms and restrictions apply. Can't be combined with any other discount, including but not limited to contract and or special pricing. Loyalty programs subject to terms and conditions. Details at lois.com/termssubject to change.

Tsung-Hsien Wen (0:56)

They want the agent to be super powerful like a human being, but they want the agent to be super controllable like a robot. Voice AI is very important because that's the native way that human talking to each other and it's the most natural interfaces. Computers has not been at the same level as human in terms of voice conversations yet, but we do actually foresee that this should be the future in the next couple of years.

Tim Scarfe (1:21)

Do you think that speech tech has become commoditized or is that just completely wrong?

Tsung-Hsien Wen (1:26)

It becomes a lot more commoditized in a way, but also in a lot of different ways is still not. But modeling the real-time conversation, it is still hard and it's not quite there yet. The bottleneck has already shifted from producing content to actually validating and auditing the content.

Tim Scarfe (1:47)

This show is made in partnership with PolyAI. People think that voice assistants are a solved problem, but it's actually much harder than people think. Why is that?

Tsung-Hsien Wen (1:58)

When ChirgyPT came along, we thought that voice is probably going to be solved very soon. I think a lot of people have the same idea, and then that's where you see a lot of rapper companies come along, just rebel on ChirgyPT, piping those ASR and TTS together and then try to solve the problem. The reality is that it's actually harder than we think.

And we kind of settle on this idea that actually voice is, if you think about voice, the voice is actually one step to the real world, where if you think about all the current innovation on the AI front, it's predominantly on the text and the digital world, where solving a PhD-grade, a physics problem seems to be not a problem anymore because you have a really good documentation, you have a lot of online papers you can research on, while in the voice scenario, where you will actually have, you have an additional like axis, which is time, right? And then time is a very critical matter. I think we have not trained the model to be able to better understand that concept yet. And I think, and because of that one step into the physical world, we started to realize, okay, well, there's actually a lot more complexity when you add that time element into it. And also, like with the humans, is about, the intelligent here is not really about reasoning, is really about adaptation.

Tim Scarfe (3:23)

When we spoke last time, you said that, you know, doing voice technology, it's a little bit like robotics, right? So it's incredibly easy for humans. You know, there's the famous Wozniak coffee cup challenge, which is, I want a robot to make me a cup of coffee. Just tell me a little bit more about that. Why is it so difficult?

Tsung-Hsien Wen (3:40)

Again, get back to the physical world situation, right? Because in the physical world, for humans, we collect data in the physical world very naturally. And we have a lot of data because we have visual data. We have, like, you know, the body movements, your hands can reach out to things in the world. However, computers don't have that yet. Computers only have all the tools. It's all about how to use the computers, right? So I think now we have trained the models really good at understanding text and understand how to use tools in that virtual world. But in the physical world, the data is still very, very little. And that applies to robotics. That's why a lot of the robotic companies right now, as you say, is trying to grab the coffee from one place to the other. I remember one of my really dear friend and also a really good researcher from Cambridge. He was originally at Poly, and he now left to start his own business. And he wanted to do a robotic company.