When AI's builders sound the alarm

Reid Hoffman (0:00)

Aria, welcome back from a summer extravaganza of sabbatical with the family in Europe. It's awesome that you're back.

Aria Finger (0:07)

Reid, I am so delighted. And wouldn't you know, the first weekend, I was back on the job. The AI world, of course, explodes with news and Twitter posts. And so we are going to jump right into that. So as everyone saw recently, there have been warnings from researchers at Anthropic and OpenAI that have brought concerns about losing control of advanced AI to a much broader public. Such a broader public that last week, my mom, my 75-year-old mother, forwarded me an email from Heather Cox Richardson and said, have you heard about this Anthropic AI news? It's a big deal. So it has certainly escaped the tech world. Everyone is talking about this. Anthropic head of alignment, Evan Hubinger, even put his personal estimate of AI causing human extinction within the next decade above 10 percent. And now that's his assessment. It's not a scientific consensus. I don't know how he is so certain in particular about above 10 percent, and it is not an official company forecast. But of course, that raises concerns and the warnings concern more advanced self-improving systems rather than a claim that today's chat bots pose this extinction risk. It's just how fast things are improving.

Then Dario Amodei, he proposed pacing the frontier, giving safety work time to catch up with advances in AI capabilities. We've heard this before. You and I have discussed, should we pause? What does that mean for the US versus other countries? Dario's proposals include embedded outside evaluators, coordination among AI labs and democratic governments and international agreements. Anthropic has committed to these embedded evaluators. Sam Altman from OpenAI has said, OpenAI will implement them too. Super interesting to see the heads of the AI labs actually agree on this. Other industry leaders have expressed support for this general idea, but not something specific.

With the accelerationist and de-accelerationist crowds, they've had a field day arguing in public, in private, on Twitter about what we should do. When you think about all of these warnings and proposals, does this feel like meaningful progress and what is missing from this discussion?

Reid Hoffman (2:20)

As you know, what I tend to say is, hey, it's less about acceleration, de-acceleration, and that actually frequently is a dichotomy of various sorts, and either one, especially in extreme being deeply wrong, but more about steering. Now, part of steering is sometimes when you get to a particular curve, you slow down when you go around that curve, but you're slowing down with a purpose of where you're going to, like not I'm just slowing down, I'm going around. Now, you know, obviously you push the metaphor and you say, well, if you're heading towards a cliff, you stop, right? If it's a cliff, but this kind of gets to kind of what are the actual ways I think we should analyze, you know, kind of what are the called safety concerns.

The first is kind of ex-risk, you know, existential risk. And I think that actually, in fact, the kind of the X percent in a decade, et cetera, et cetera, are all in the kind of histrionic category. And they may be, in some cases, deeply well-meaning, like EG., I don't know how to call the percentage. And it's a little bit like, you know, you're exploiting the Hiroshima bomb, and it's going to split the Earth's core, and we're going to be one molten, you know, thing. And I don't know how to articulate percentage, so I just put a solid number on it as a way of doing it.

Aria Finger (3:42)

And if you say 10 percent, people will pay attention.

Reid Hoffman (3:44)

Yes, exactly. And so, whereas, you know, when I kind of chart out the existential risk, I can go, look, there's only a very specific set of scenario where it gets there in that kind of time. There's multiple checkpoints along the ways. There's multiple things we can do. There's multiple telemetry. And so I tend to think that the X-risk is the way of kind of saying the existential risk is the way of saying, look at me, look at me, it's really important. This is a key moment in history. It's really important you talk to me about this. And so I broadly across the whole set, including the Anthropic people who are driving the frontier along with OpenAI on this, which is part of the well-commented irony of all these things.

I didn't think that's well-meaning, but the sky is falling, the sky is falling, the sky is falling. Now we get to the three real ones, which is bio-risk, cyber-risk and jobs.