John Schulman: OpenAI and recent advances in Artificial Intelligence – #16

**Steve Hsu** (0:05)
Our guest today is John Schulman. He is a research scientist at OpenAI. He's the co-lead for their reinforcement learning team, where he works on designing better reinforcement learning algorithms and also designing better training environments that teach agents transferable skills. And much of the work that he does uses games or virtual worlds as a testbed.
John, welcome to Manifold.

**John Schulman** (0:33)
Thanks for having me.

**Steve Hsu** (0:34)
So I sort of know you through one of your OpenAI co-workers, Sam Altman, and really wanted to have you on the show because Corey and I are both fascinated by what's happening in AI, and OpenAI is one of the most interesting places where that work is going on. Let's begin with your academic background or your childhood and how you got interested in AI in the first place.

**John Schulman** (1:00)
So going back to my childhood, there is a TV show called BattleBots, where people would build these combat robots, which were really RC controlled machines and fight them against each other in a ring. So that really captured my excitement, and I decided to set out and build one of these with a few of my friends. And at least that was a good intro to a lot of engineering and both mechanical and electrical.
So that ended up, that didn't go so far. At least I moved on to other things when I got into high school. But at that point, I started to read a little bit about AI through, for example, Nils Nilsson's textbook, got interested in the topic there. And at the same time, I found Ray Kurzweil's book about the singularity is near, and I found that pretty persuasive and that had a big effect on my thinking going forward.

**Steve Hsu** (2:03)
So of AI people, it seems like I often meet people who are older and maybe live through the AI winter, and they're very pessimistic about what can happen, say, in our lifetimes versus people like Kurzweil who are super optimistic. And so it's interesting to meet younger people like you who maybe cut your teeth already on those ideas, like the singularity and things which were much more kind of futuristic.
What do you think is the breakup among, say, people that you studied with in terms of whether they're sort of AI pessimists or AI optimists?

**Corey Washington** (2:36)
Before you answer that question, John, I want to hop in. My role here is this the audience ombudsperson. So can either you or Steve explain what the singularity is?
And I guess this will come out, but what's Ray Kurzweil's view? And when you said this had an effect on your thinking, did it make you happy? Did it make you scared?

**John Schulman** (2:56)
Yeah. So as for what the singularity is, this is just the idea that AI, mostly AI, possibly other technologies will become more powerful in a way that's self-reinforcing, and you'll have some kind of runaway effect. So in the AI case, the idea is that you get smarter and smarter machines, and then at a certain point, they can improve themselves.
And so the rate of progress increases and eventually things become incomprehensible. And that's why it's called the singularity, because you can't really predict anything about what's going to happen after it.

**Corey Washington** (3:31)
And this is the plot of many disaster movies when the computers take over.

**John Schulman** (3:36)
Yeah.

**Steve Hsu** (3:37)
It could be utopian or dystopian, depending on what they do to us once they take over.
But we'll get into that, I think, later in the podcast. So let's stick with John's background a little bit. So you, I think, attended the same educational institutions that I did. So you were an undergrad at Caltech, and you did your PhD at Berkeley, is that right?

**John Schulman** (3:56)
Yeah.

**Steve Hsu** (3:56)
And were you in computer science the whole time?

**John Schulman** (3:58)
I majored in physics at Caltech, so also like you.

**Steve Hsu** (4:02)
Oh, wow, okay. And then you were smarter than me. You switched to computer science.

**John Schulman** (4:07)
I was always, I mean, I was pretty interested in physics in high school and starting college.
I dabbled in a little physics research early on in my undergrad career.
And then at a certain point, I realized I liked doing physics, but when I would go and read about what new scientific developments were occurring, I was more excited about developments in certain other fields, like neuroscience and AI. So I was kind of split between understanding the human mind and how to build artificial minds. I ended up actually applying to grad programs in neuroscience, and that brought me to Berkeley. So I did some lab rotations in neuroscience, and I was pretty sure I would go into that field. But for my last rotation, I ended up working with Peter Abil, who works on robotics and machine learning. And that was kind of just on a whim because I liked his work a lot, and so I thought I would learn something new by working with him. And then I ended up getting really excited about the projects I worked on with him. And a few months later, I ended up transferring into the computer science program and finishing up my PhD in AI.