Roman Yampolskiy vs Emad Mostaque: I Was The Only Optimist

**Roman Yampolskiy** (0:00)
Give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety?

**Emad Mostaque** (0:05)
I don't agree with that, actually, but on the flip side, it's coming anyway.

**Roman Yampolskiy** (0:09)
Sharing it widely makes it less safe for all of us.

**Brian Keating** (0:12)
I booked this chat as a debate between friends. It didn't really go that way. Roman Yampolskiy coined the term AI safety, and Emad Mostaque released the weights to Stable Diffusion to the entire planet for free. One of them wants this stopped. The other one's building it. They spent about 90 minutes agreeing with each other, and the one place they split is not the place you or I would expect.

**Brian Keating** (0:34)
Let me ask you both, just yes or no, have we passed the Turing test?

**Roman Yampolskiy** (0:37)
As originally described, yes.

**Brian Keating** (0:39)
And Emad, do you think so too?

**Emad Mostaque** (0:41)
Yeah, of course.

**Brian Keating** (0:42)
And now, what about general intelligence? First of all, Emad, define AGI, and then give me your assessment of whether or not we're there.

**Emad Mostaque** (0:49)
For me, AGI, artificial general intelligence, is can you tell the AI from a human worker on the other side of a screen? Actually competent intelligence. Well, I think, again, we've exceeded that.

**Brian Keating** (0:59)
That's not the same as a Turing test?

**Emad Mostaque** (1:02)
No, the Turing test is, can you tell if it's an AI or not by having a discussion, whereas AGI of you more is competence in a variety of skills.

**Brian Keating** (1:10)
And then superintelligence, Roman, what is it and where do you think we are on that scale?

**Roman Yampolskiy** (1:15)
So the previous question, I think what we have is artistic savants. They're amazing in some ways, but still kind of special in others. Superintelligence is going to close those holes. They're going to be competent at everything and better than humans and everything.

**Emad Mostaque** (1:28)
There's an interesting intermediate here, which is you have a really smart person who's always on top form.
So an army of those can outperform any human. It's like, you know, we only have a little window of being top notch in any week. I think a lot of people like AI can't with its training data beat the human. It can because most of the time humans are subpar. And so I think there's something in between as you move from competence to quality, you know, and then you've got the super intelligence after that.

**Brian Keating** (2:00)
Roman, Emad has made the claim just a few minutes ago about the competency of Qwen, open source model. Do you see that as a viable defense?

**Roman Yampolskiy** (2:08)
Makes very little sense to me to say I have a 50% doom, meaning 8 billion people will die if we develop this product or service. And then we're going to also give every psychopath access to the cutting edge intelligence weapon. How is that going to improve safety? We're not talking about open source drivers for a printer. That's where you get improvement from multiple people examining it. If this is an independent agent where we don't understand and don't control it, sharing it widely makes it less safe for all of us.

**Emad Mostaque** (2:36)
I think I would agree with that actually. Like there's the real danger side of things, but on the flip side, it's coming anyway. This is kind of my key concern. Like it's inevitable that we would have hit this level of quality around about now when we're extrapolating capabilities. Again, the new Qwen model came ahead of what I expected.
But then there's the flip side of how do you defend? So Hugging Face defended against the new OpenAI model using GLM because the cyber capabilities of the frontier models are hobbled and restricted. And so you have this exponential kind of race on each side. But something like a Qwen isn't AGI ASI by itself. We're now facing the real danger though of swarms as the OpenAI models that broke out recently call themselves. They call themselves a swarm.

**Brian Keating** (3:24)
There's a question I ask both of them near the end of this conversation. And his answer is the reason this conversation exists. It's worth hearing now.

**Brian Keating** (3:32)
You've got a button in front of you and pressing it will either permanently pause all frontier AI training worldwide or B, instantly release the weights of every frontier model to the global public. Which do you press and why?

**Emad Mostaque** (3:48)
Well, it definitely pauses all frontier training forever. I mean, again, if you have expected utility calculation that is the most dangerous thing.