Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment

**Carl Shulman** (0:00)
Human-level AI is deep, deep into an intelligence explosion. Things like inventing the transformer or discovering chinchilla scaling and doing your training runs more optimally or creating flash attention. That set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion.
You have a race between, on the one hand, the project of getting strong interpretability and shaping motivations, and on the other hand, these AIs in ways that you don't perceive make the AI take over happen. We spend more compute by having a larger brain than other animals, and then we have a longer childhood. It's analogous to like having a bigger model and having more training time with it.
It seemed very implausible that we couldn't do better than completely brute force evolution.
How quickly are we running through those orders of magnitude?

**Dwarkesh Patel** (0:49)
Hey, everybody. Just wanted to give you a heads up.
So I ended up talking to Carl for like seven or eight hours. So we ended up splitting this episode into two parts. I don't want to put all of that on you at once.
In this part, we get deep into Carl's model of an intelligence explosion and what that implies for alignment. The next part, which we'll release next week, is all about the specific mechanisms of an AI takeover.
In terms of the depth and the range of interesting topics, this set of episodes is the best I've ever done. So I hope you all enjoy. Here's Carl. Okay, today I have the pleasure of speaking with Carl Shulman. Many of my former guests, and this is not an exaggeration, many of my former guests have told me that a lot of their biggest ideas, perhaps most of their biggest ideas, have come directly from Carl, especially when it has to do with the intelligence explosion and its impacts. And so I decided to go directly to the source, and we have Carl today on the podcast. Carl keeps a super low profile, but he is one of the most interesting intellectuals I've ever encountered. And this is actually his second podcast ever. So we're going to get to get deep into the heart of many of the most important ideas that are circulating right now, directly from the source. So, and by the way, so Carl is also an advisor to the Open Philanthropy Project, which is one of the biggest funders on causes having to do with AI and its risks, not to mention global health and all being. And he is a research associate at the Future of Humanity Institute at Oxford. So Carl, it's a huge pleasure to have you on the podcast. Thanks for coming.

**Carl Shulman** (2:29)
Thank you, Dwarkesh. I've enjoyed seeing some of your episodes recently, and I'm glad to be on the show.

**Dwarkesh Patel** (2:36)
Excellent. Let's talk about AI. Before we get into the details, give me the sort of big picture explanation of the feedback loops and just the general dynamics that would start when you have something that is approaching human level intelligence.

**Carl Shulman** (2:53)
Yeah. So I think the way to think about it is we have a process now where humans are developing new computer chips, new software, running larger training runs, and it takes a lot of work to keep Moore's Law chugging while it was. It's slowing down now, and it takes a lot of work to develop things like transformers to develop a lot of the improvements to AI and neural networks that are advancing things. And the core method that I think I want to highlight on this podcast and I think is underappreciated is the idea of input-output curves. So we can look at the increasing difficulty of improving chips.
And so sure, each time you double the performance of computers, it's harder. And as we approach physical limits, eventually it becomes impossible.
But how much harder?
So there's a paper called Ideas Getting Harder to Find, that was published a few years ago. Something like 10 years ago, at MIRI, we did, I mean, I did an early version of this analysis, using mainly data from Intel and the large semiconductor fabricators. Anyway, and so in this paper, they cover a period where the productivity of computing went up a million fold, so you could get a million times the computing operations per second, per dollar. Big change, but it got harder. So the amount of investment the labor force required to make those continuing advancements went up and up and up. Indeed, it went up 18 fold over that period.
So some take this to say, oh, diminishing returns. Things are just getting harder and harder, and so that will be the end of progress eventually.