Your AI Friends Have Awoken, With Noam Shazeer

**Elad Gil** (0:05)
Do you view all this as a path to AGI or sort of super intelligence?

**Noam Shazeer** (0:09)
Sure, yeah.

**Elad Gil** (0:10)
And is that part of the goal? For some companies, it seems like it's part of the goal, and for some companies, it seems like it's either not explicitly an anti-goal, or if it happens, it happens, and the thing people are trying to build is just something useful for people.

**Sarah Guo** (0:22)
What a flex, AGI is a side effect.

**Noam Shazeer** (0:25)
Yeah, well, I mean, that was a lot of the motivations here, because I mean, my main motivation for working on AI, other than that it's fun, well, I mean, fun is secondary. Like the real thing is like, I want to drive technology forward.

**Sarah Guo** (0:42)
This is the No Priors Podcast. I'm Sarah Guo.

**Elad Gil** (0:45)
I'm Elad Gil.

**Sarah Guo** (0:46)
We invest in, advise, and help start technology companies.

**Elad Gil** (0:49)
In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions. Thank you.
Transformers, Large Language Models, AI Chat. These are the foundations supporting today's AI revolution. And this week on No Priors, we have AI researcher, engineer, and inventor, who is a key part to these innovations and is considered one of the smartest people in AI. Noam Shazeer is the CEO and co-founder of Character AI, a service that allows users to design and interact with their own personal bots to take on the personalities of well-known individuals or archetypes. You could have a Socratic conversation with Socrates, or you could pretend you're being interviewed by Oprah.
Or you could work through a life decision with a therapist bot. Character recently raised $150 million from Andreessen Horowitz, myself and others. We talk about how Noam got his start at Google, his groundbreaking AI discoveries and what he's doing at Character. So Noam, welcome to No Priors.

**Noam Shazeer** (1:45)
Hey Elad, thanks for having me on. Hi Sarah.

**Sarah Guo** (1:48)
Good to see it.

**Elad Gil** (1:49)
Yeah, thanks for joining. So you've been working on NLP and AI for a long time. So I think you were at Google for something like 17 years off and on.
And I think even your Google interview question was something around spell checking, an approach that eventually got implemented there. And when I joined Google, one of the main systems being used at the time for ads targeting was like fill and fill clusters and all this stuff, which I think you wrote with George Herrick. And so it'd just be great to get kind of your history in terms of working on AI, NLP, language models, how this all evolved, what you got started on and what sparked your interest.

**Noam Shazeer** (2:16)
Oh, thanks Elad. Yeah, just always naturally drawn to AI. Wanted to make the computer do something smart. Seems like pretty much the most fun game around. Was lucky to find Google early on and really is an AI company. So yeah, got involved in a lot of the early projects there that maybe you wouldn't call AI now, but seemed pretty smart at the time. And then more recently was on the Google brain team starting in 2012 It looked like a really smart group of people doing something interesting.
I had never done deep learning before or neural networks, I guess, as it was called then or whatever. I forget when the rebrand happened. But yeah, it turned out to be really fun.

**Elad Gil** (2:56)
That's cool. And then you were one of the main people working on the transformer paper and design in 2017 And then you worked on mesh TensorFlow, I think, sometime within the following year.
Can you talk a little bit about how all that got going?

**Noam Shazeer** (3:10)
Yeah, I mean, I messed around a few years on the Google brain team and utterly failed at a bunch of stuff until I kind of got the hang of it.
Really the key insight is that what makes deep learning work is that it is really well-suited to modern hardware, where you have the current generation of chips that are great at matrix multiplies and other forms of things that require large amounts of computation relative to communication. So basically, deep learning really took off because it runs thousands of times faster than anything else. And as soon as I got the hang of that, started designing things that actually were smart and ran fast.
But the most exciting problem out there is language modeling. It's like the best problem ever because there's like an infinite amount of data, just scrape the web and you've got all the training data you could ever hope for. And like the problem is super simple to define. It's predict the next word, the fat cat sat on the, what comes next. Like it's extremely easy to define. And if you can do a great job of it, then you get everything that you're seeing right now and more. You can just talk to the thing and it's really AI complete. And so got started around like 2015 or so, working on language modeling and messing with the recurrent neural networks, which was what was great then. And then Transformer kind of came about as someone had the bright idea.