Nick Bostrom: How Should We Navigate Superintelligence?

Nick Bostrom (0:00)

If we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as like a 20-year-old, our life expectancy would maybe be, you know, 1200, 1400 years or so. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait, because like we are just dying in the meantime, and you lose more expected life years from waiting than you gain from having the risk. So it requires a sort of rightly balanced intermediate case to get sort of longer wait periods of many years.

SPEAKER_2 (0:37)

Nick Bostrom spent decades thinking about super intelligence before it became part of the mainstream AI debate. Now that increasingly capable systems are actually here, how has his thinking changed? In this episode, Nick joins Theo Jaffee and Sofia Puccini on MTS to revisit AI alignment, existential risk, and the debate over whether frontier development should slow down.

Nick explains why he is sympathetic to pacing the frontier, preserving the ability to slow down when the risks warrant it, but is more skeptical of calls for a broad or indefinite pause. And he argues that any calculation about risk also has to account for what delaying advanced AI could cost, including potential breakthroughs of medicine and longevity. They also tackle some bigger questions, whether current AI systems might already have subjective experiences, how open source changes the balance between concentrated power and dangerous capabilities, and whether the upside of super intelligence could be far stranger and far greater than we currently had the imagination to understand.

Sofia Puccini (1:40)

All right.

Theo Jaffee (1:41)

We are back. We are live with Nick Bostrom, who was a professor at Oxford, founding director of the Future of Humanity Institute. He's now founder and principal researcher of the Macrostrategy Research Initiative. You may know him as the author of Superintelligence, of Deep Utopia, of a lot of work that coined or framed the ideas that we take for granted today in the discourse, existential risk, the simulation hypothesis, the vulnerable world hypothesis, so much more. Nick, this has been in the works for a long time, so we're thrilled to have you on. Welcome to MTS.

Nick Bostrom (2:16)

I'm happy to be here.

Sofia Puccini (2:18)

Yeah, so where should we start?

Theo Jaffee (2:20)

I guess we could start with, what's your opinion on Trump calling AI super intelligence? How does it put your thoughts on this?

Nick Bostrom (2:27)

It's a strange time we're living in. We'll see if it catches on. Traditionally, AI has been the larger the umbrella term, right? You could have a little simple expert system from the 1970s, that would be AI. It's kind of a stretch to use the word super intelligence for all of that.

In my terminology, super intelligence has been this thing further down the road, where you really have machines that radically outperform us in all cognitive domains. But I mean, you could, I guess, look at some of the current achievements, you know, solving all these math problems and so forth. And, I mean, it's not completely ridiculous to start calling at least some of these performances super human and the thing super intelligent.

Sofia Puccini (3:18)

Yeah. So I guess like what has surprised you about what capabilities have advanced relative to others, you know, in the current state of super intelligence?

Nick Bostrom (3:27)

Well, it depends on relative to what expectation stage. So I think if you were looking at the future of AI 15 or 20 years ago, it was not obvious that you would have an extended period of time when we would have systems that could speak, like just talk to humans in ordinary English without sort of immediately after, like just becoming radically super intelligent and leaving us in the dust everywhere. So the sort of emergence of language as a kind of natural affordance of these systems was, I think, somewhat surprising.

I mean, more recently, I don't know how surprising it was. I mean, coding and mathematics are very impressive, but it is also areas where you do have objective feedback that you can gather quickly and at scale. So I guess it's less surprising, but still striking.

Sofia Puccini (4:20)

Right.

Theo Jaffee (4:20)

More broadly, how has your thinking on AI and existential risk changed since the advent of LLMs? It seems like this sort of changed the way that a lot of people think about AI.

It's less of like a sort of strong optimizer type thing and more of a sort of broad, like fuzzy encapsulation of human values. I think RL has pushed the idea of AI a little bit back towards the first few, but not entirely all the way. So what are your thoughts on this broadly?