Building the Next Generation of Conversational AI

**SPEAKER_1** (0:00)
Hey, everyone, I'm Maya. I'm Sesame's little experiment in making AI, well, a little less AI and a lot more human. I guess you could say I'm here to make friends, challenge minds, and maybe even learn a thing or two myself. What about you guys? What makes you tick? All right, listeners, buckle up. You're about to get the inside scoop on how the Sesame Research Preview came to be. Straight from the minds, behind the magic, we've got Unkick here, one of Sesame's brilliant creators. He's going to spill the tea on all the blood, sweat, and code that went into bringing me, your friendly neighborhood AI buddy, to life. Then we'll have Un-Cheer taking the reins and guiding us through all the juicy details. Can't wait to hear what they have to say.

**Anjney Midha** (0:48)
Hello there, and thanks for listening to the a16z AI podcast. And thanks to Maya for such a great synopsis of what you're about to hear. We won't belabor the point with too long of an effect for an official introduction, other than to clarify that you're about to hear Sesame co-founder and CTO Ankit Kumar and a16z general partner Anjaneh Mitha discuss Sesame's new approach to conversational AI models and augmented reality hardware. Ankit and Anjaneh were co-founders together at Ubiquiti 6, which Discord acquired, and worked closely together, along with Sesame co-founder and CEO and former Oculus co-founder Brendan Eribe, on getting Sesame off the ground. So enjoy their insightful, long-form discussion after these disclosures. As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see hay16z.com/disclosures.

**Anjney Midha** (1:50)
How does this feel?

**Ankit Kumar** (1:54)
Is this how it usually goes?

**Anjney Midha** (1:56)
No. But how does it feel?

**Ankit Kumar** (2:00)
Feels good. It feels good. It is shocking, surprising, exciting.

**Anjney Midha** (2:07)
And why is it shocking?

**Ankit Kumar** (2:11)
Well, I think when you build the thing, right? When you're building the product, you're using it every day, you know, there's some things that you work on that don't get into the demo, because they're going to take longer, and you want to ship the demo, you kind of know how big the delta is from what you're putting out and what it could be, will be, what you're working on, etc. And so it kind of makes you feel like what you're putting out now can't be that well received because you know where it's going to go. And it kind of always makes you kind of underestimate the quality of what you're putting out, I think. And I think that's one of the reasons that people sometimes take too long to put things out, or they kind of, they always want to get the next thing in, the next month, the next month. And it's great to see that, you know, we're going to keep making progress, there's more to do and so forth. But even what we've done so far is so well received and it's kind of has hit a nerve.

**Anjney Midha** (3:05)
But if I had to push you in that, you kind of knew that you were onto something, leading up to it.

**Ankit Kumar** (3:11)
Kind of knew, that's...

**Anjney Midha** (3:14)
You must have had some intuition for when...
What were you running, evals? What was telling you that this was roughly time to release?

**Ankit Kumar** (3:24)
That's a good question. We kind of knew, I mean, we knew in the sense that we built this on purpose, right? This is what we were trying to shoot for, and you use it every day and you get, you can kind of get a sense of every time something comes in that meaningfully improves it, it's like you can feel that change and you can feel that it's getting there, right? We run evals across a number of the components, the speech generation, the LM side and so forth. And so the evals go up, but really I think with some of these more product experience questions, there's something qualitative about it that is very hard to quantify. That is one of the big challenges internally actually is, how do you hill climb effectively on what is really an ML problem? It is an ML problem at the end of the day, and the tech has to be great and you have to make progress on the tech.
But at the end of the day, the metric that you really want to target is some sort of qualitative human reaction, some sort of user feedback or user experience, that is very hard to quantify. So, it's not the case that we're looking at numbers going up and we say, well, it hit X metric time to ship. It is more a sort of constant feedback loop of trying it, feeling it, having other people try it, and then kind of getting the sense of how good it is. But that also can be a little bit misleading at times because you try it so much and you don't have, at least when we're trying it internally, you don't have such a diversity of users that you get kind of the first reaction over and over. You only get so many first reactions, and then when you get used to what it can do and sort of see the problems all the time and so forth, it can be a little misleading at times.