What is Digital Life? with OpenAI Co-Founder & Chief Scientist Ilya Sutskever

**Sarah Guo** (0:06)
OpenAI, a company that we all know now, but only a year ago was 100 people, is changing the world. Their research is leading the charge to AGI. Since ChatGPT captured consumer attention last November, they show no signs of slowing down. This week, Elad and I sit down with Ilya Sutskever, co-founder and chief scientist at OpenAI, to discuss the state of AI research, where we'll hit limits, the future of AGI, and what it's going to take to reach super alignment. Ilya, welcome to No Priors.

**Ilya Sutskever** (0:36)
Thank you. It's good to be here.

**Sarah Guo** (0:37)
Let's start at the beginning. Pre-AlexNet, nothing in deep learning was really working. And then given that environment, you guys took a very unique bet. What motivated you to go in this direction?

**Ilya Sutskever** (0:48)
Indeed. In those dark ages, AI was not an area where people had hope and people were not accustomed to any kind of success at all. And because there wasn't, there hasn't been any success, there was a lot of debate and there were different schools of thoughts that had different arguments about how machine learning and AI should be. And you had people who were into knowledge representation from a good old fashioned AI. You had people who were Bayesians and they liked Bayesian non-parametric methods. You had people who liked graphical models and you had the people who liked neural networks. Those people were marginalized because neural networks did not have the property that you can't prove math theorems about them. If you can't prove theorems about something, it means that your research isn't good. That's how it has been. But the reason why I gravitated to neural networks from the beginning is because it felt like those are small little brains. And who cares if you can't prove any theorems about them? Because we are training small little brains and maybe they'll do something one day.
And the reason that we were able to do AlexNet when we did is because a combination of two factors, three factors. The first factor is that this was shortly after GPUs started to be used in machine learning. People kind of had an intuition that that's a good thing to do, but it wasn't like today where people exactly knew what they need GPUs for. It was like, oh, let's like play with those cool, fast computers and see what we can do with them. It was an especially good fit for neural networks. So that was a very, that definitely helped us. I was very fortunate in that I was able to realize that the reason neural networks of the time weren't good is because they were too small. So like if you try to solve a vision task with a neural network, which has like a thousand neurons, what can it do? It can't do anything. It doesn't matter how good your learning is and everything else. But if you have a much larger neural network, you'll do something unprecedented.

**Elad Gil** (2:53)
What gave you the intuition to think that that was the case? Because I think at the time it was reasonably contrary to think that despite, to your point, you know, a lot of the human brain in some sense works that way, or different, you know, biological neural circuits. But I'm just curious, like, what gave you that intuition early on to think that this was a good direction?

**Ilya Sutskever** (3:09)
I think, yeah, looking at the brain, and specifically the... if you... like, if all those things follow very easily, if you allow yourself, if you allow yourself to accept the idea... Right now, this idea is reasonably well accepted. Back then, people still talked about it, but they haven't really accepted it or internalized. The idea that maybe an artificial neuron, in some sense, is not that different from a biological neuron. So now, whatever you imagine animals do with their brains, you could, perhaps, assemble some artificial neural network of similar size. Maybe if you train it, it will do something similar.
So that leads you to start to imagine, almost imagine the computation being done by the neural network. You can almost think, if you have a high-resolution image and you have one neuron for a large group of pixels, what can the neuron do? It's just not much it can do. But if you have a lot of neurons, then they can actually do something and compute something. So I think it was like considerations like this, plus a technical realization. The technical realization is that if you have a large training set that specifies the behavior of the neural network and the training set is large enough such that it can constrain the large neural network sufficiently, and furthermore, if you have the algorithm to find that neural network, because what we do is that we turn the training set into a neural network which satisfies a training set. Neural network training can almost be seen as solving a neural equation. Solving a neural equation where every data point is an equation and every parameter is a variable. And so, it was multiple things. The realization that the bigger neural network could do something unprecedented. The realization that if you have a large data set, together with the compute to solve the neural equation, that's what gradient descent comes in. But it's not gradient descent. Gradient descent was around for a long time. It was certain technical insights about how to make it work. Because back then, the prevailing belief was, well, you can't train those neural nets or anything. It's all hopeless. So it wasn't just about the size. It was about even if someone did think, gosh, it would be cool to train a big neural net, they didn't have the technical ability to turn this idea into reality. You needed not only to code the neural net, you need to do a bunch of things right. And only then it will work. And then another fortunate thing is that the person with whom I work with, Alex Krzyzewski, he just discovered that he really loves GPUs and he was perhaps one of the first person who really mastered writing really performant code for the GPUs. And that's why we were able to squeeze a lot of performance out of two GPUs and produce something unprecedented. So to sum up, it was multiple things. The idea that a big neural network, in this case a vision neural network, a convolutional neural network with many layers. One that's much, much bigger than anything that's ever been done before could do something very unprecedented, because the brain can see and the brain is a large neural network. And we can see quickly so our neurons don't have a lot of time. Then the compute needed, the technical know-how that in fact we can't train such neural networks. And it was not at all widely distributed. Most people in machine learning would not have been able to train such a neural network even if they wanted to.