Scaling and the Road to Human-Level AI | Anthropic Co-founder Jared Kaplan

**Jared Kaplan** (0:00)
Hey, everyone. I'm Jared Kaplan. I'm going to talk briefly about Scaling and the Road to Human Level AI. But my guess is for this audience, a lot of these ideas are pretty familiar, so I'll keep it short. And then we're going to do a sort of fireside chat Q&A with Diana. I actually have only been working on AI for about six years. I, before that, had a long career, the vast majority of my career as a theoretical physicist, working in academia. And so, how did I get to AI? Well, I want to be brief. Why did I start in physics? It was basically because my mom was a science fiction writer, and I wanted to figure out if we could build a faster than light drive, and physics was the way to do that. I also was very excited about just understanding the universe. How do things work? How do the biggest trends that underlie sort of everything that we see around us? Where does that all come from? For example, is the universe deterministic? Do we have free will? I was very, very interested in all of those questions, but fortunately, along the way, during my career as a physicist, I met a lot of very, very interesting, very deep people, including many of the founders of Anthropic that I now work with all of the time, and I was really interested in what they were doing, and I kept track of it, and as I moved from different, among different subject areas in physics, from Large Hadron Collider Physics, Particle Physics, Cosmology, String Theory, and on, I got a little bit frustrated, a little bit bored. I didn't feel like we were making progress quickly enough, and a lot of my friends were telling me that AI was becoming a really big deal, and I didn't believe them. I was really skeptical. I thought, well, AI, people have been working on it for 50 years, SVMs aren't that exciting. That was all we knew about back in 2005, 2009 when I was in school. But I got convinced that maybe AI would be an exciting field to work on, and I got very lucky to know the right people, and the rest is history. So I'm going to talk a little bit about how our contemporary AI models work and how scaling is leading them to get better and better. So there are really two fundamental phases to the training of contemporary AI models, like Claude, ChatGPT, et cetera. The first phase is pre-training, and that's where we train AI models to imitate human written data, human written text, and understand the correlations underlying that data. And these figures are very, very retro. This is actually from the playground of the original GPT-3 model. And you can see that, as a speaker at a journal club, you're probably elephant me to say certain things. The word elephant in that sentence is really, really unlikely. What pre-training does is teach models what words are likely to follow other words in large corporate text, and now with contemporary models, multimodal data. The second phase of training for contemporary AI models is reinforcement learning. This is another very retro slide. It shows the original interface we used for sort of Claude zero or Claude negative one back in the ancient days of 2022, when we were collecting feedback data. And what you see here is basically the interface for having a conversation with very, very early versions of Claude and picking which response from Claude was better according to you, according to crowd workers, et cetera. And using that signal, we optimize, we reinforce the behaviors that are chosen to be good, they're chosen to be helpful, honest and harmless. We discourage the behaviors that are bad. So really all there is to training these models is learning to predict the next word and then doing reinforcement learning to learn to do useful tasks. And it turns out that there are scaling laws for both of these phases of training. So this is a figure that we made five or six years ago now and it shows how as you scale up the pre-training phase of AI, you predictably get better and better performance for our models. And this is something that came about because I was just sort of asking the dumbest possible question. As a physicist, that's what you're trained to do. You sort of look at the big picture and you ask really dumb things. I'd heard it was very popular in the 2010s to say that big data was important. And so I just wanted to know how big should the data be? How important is it? How much does it help? Similarly, a lot of people were noticing that larger AI models performed better. And so we just asked the question, how much better do these models perform? And we got really lucky. We found that there's actually something very, very, very precise and surprising underlying AI training. This really blew us away that there are these nice trends that are as precise as anything that you see in physics or astronomy. And these gave us a lot of conviction to believe that AI was just going to keep getting smarter and smarter in a very predictable way. Because as you can see in these figures already back in 2019, we were looking across many, many, many orders of magnitude in compute, in dataset size, in neural network size. And so we expected once you see something is true over many, many, many orders of magnitude, you can expect it's probably going to continue to be true for a long time further. So this has sort of been one of the fundamental things that I think underlies improvements in AI. The other is actually also something that started to appear quite a long time ago, though it's become really, really impactful in the last couple of years, is that you can see scaling laws in the reinforcement learning phase of AI training. So a researcher about four years ago decided to study scaling laws for AlphaGo, basically putting together two very, very high-profile AI successes, GPD3 and scaling for pre-training and AlphaGo. This was just a researcher, Andy Jones, working on his own, with like his own, I think, maybe single GPU back in these sort of ancient days. And so he couldn't study AlphaGo, that was expensive, but he could study a simpler game called Hex. So he made this plot that you see here.