251 - Eliezer Yudkowsky: Artificial Intelligence and the End of Humanity

**Eliezer Yudkowsky** (0:00)
The Hoover Dam wasn't built in a day, and the GMC Sierra lineup wasn't built overnight. Like every American achievement, building the Sierra 1500 heavy duty and EV was the result of dedication, a dedication to mastering the art of engineering. That's what this country has done for 250 years, and what GMC has done for over 100 We are professional grade. Visit gmc.com to learn more. Assembled in Flint and Hamtranek, Michigan, in Fort Wayne, Indiana, of US and Global Resource Parts.

**Robinson Erhardt** (0:30)
As a contractor, I don't pay for materials I don't use. So why would I pay for stuff I don't need in my mobile plan? That's why the new MyBiz plan from Verizon Business is so perfect. Now I can choose exactly what I want and I only pay for what I need.

**SPEAKER_3** (0:45)
Right now, with MyBiz plan, get our best price as low as $25 a line. Visit verizon.com/business to get started today. Price per month with five plus lines includes auto pay and paper free billing and special intro offer discounts. Taxes, fees, economic adjustment charge in terms of client offers on June 10th, 2025

**Eliezer Yudkowsky** (1:00)
I'm worried about the future AIs. I'm worried about the AI that is good enough at AI research to build the AI that builds the AI that is far than us and kills everyone. The AI gets the universe that wants most. That is a universe that does not happen to have us in it. And that is how indifference kills you. If we could just get 50 tries at building super intelligences, you know, it could be like our clever alignment theory didn't work. It killed everyone. Let's build another one. Oh, that one killed everyone too. Wow, the second crazy theory we had didn't work either. The basic description I would give to the current scenario is, if anyone builds it, everyone dies.

**Robinson Erhardt** (1:43)
You founded The Machine Intelligence Research Institute, whose unofficial motto is, the default consequence of the creation of artificial super intelligence is human extinction. And I'm wondering when the idea of the existential threat of AI for humanity first kind of came onto your radar, and whether it was as dramatic a moment as that quote I just gave you.

**Eliezer Yudkowsky** (2:17)
So the general high impact that superhuman intelligence in any form was going to upset the whole human apple cart, the whole world economy apple cart, that it wasn't going to be just another technology, or just another nice thing to have. That was 1995-1996 when I would have been 15 or 16 years old, reading a book by Werner Wienke called True Names and Other Dangers, where Wienke mentioned that every science fiction writer's crystal ball, or even ability to envision a consistent future, breaks down at the point where their scenario has predicted the rise of smarter than human intelligence, because they can't write the smarter than human characters. If they were actually, if you're smart enough to predict exactly where, say, Deep Blue, the ancient world champion artificial chess player, or Stockfish, the modern chess player, if you could predict exactly where they would move on a chessboard, you'd be that good of a chess player yourself. You just always move where you predicted Stockfish would move. So something smarter than you is unpredictable in its details. And that was Wernher Vinge's observation that I came across in 1996 And I was, oh, all right. So transhuman intelligence in any form is the changing of everything. And probably artificial intelligence comes first, though that was just a guess then, a good guess, but a guess nonetheless. I saw that superhuman intelligence was going to be drastically important. I did not then see it as a threat. I thought, if it's smart, it can figure out what the right thing is to do, know the right thing, do the right thing.
Thought it was going to be, like, happily ever after. So the point at which I realized that this line of thought was mistaken, that different powerful intelligences could steer to different places, and that the whole elaborate philosophy I'd built up in my mind about intelligence is figuring out the right thing and doing the right thing, that this elaborate philosophy was mistaken, and that moreover I'd made a, you know, in a way a teenager's kind of mistake by trying to use that kind of philosophical-ish high idealistic thinking to make predictions about a universe that didn't run on philosophy deep down. Realizing that would, I'd say, be the moment of, boy, I sure have been stupid. And then went off to try to not have the default thing happen and not have the world end.