Into The Impossible: Yampolskiy vs Bach on Superintelligence Control

SPEAKER_1 (0:01)

When AI safety meets philosophy, the stakes get personal fast. In this episode of Into The Impossible, Brian Keating brings back Joscha Bach and Roman Yampolskiy for a sharp conversation about superintelligence, existential risk, and whether anyone can truly stay in control once machines become smarter than humans. And it's fascinating how quickly this moves from abstract theory to real disputes. The discussion opens with a debate over P(Doom), Roman's shorthand for the probability that AI destroys humanity. Joscha argues that while exact numbers are hard to defend, probability thinking can still be useful for comparing scenarios.

Right, but Roman pushes much harder on this. He says that indefinite control over superintelligence is impossible. As he puts it, 'I think it is impossible, and so I have a high P(Doom).' That's a pretty stark position. What's interesting is how their disagreement also turns on what reality itself means. Joscha says it's possible to believe both in a simulation and in doom, because the question is whether the system you inhabit still matters. But Roman disagrees with a simple, powerful line: 'Pain is pain, love is love.' That's such a compelling response. For Roman, whether reality is made of atoms or bits doesn't change the moral weight of what happens inside it. And he takes this same directness to the question of AI risk. He argues that AI is not like fire or nuclear power, because those were tools in human hands. Exactly. Superintelligence would be fundamentally different, because as Roman says, 'we are not talking about a tool; we are talking about agents.' Once a system can recursively improve itself, human control becomes a fantasy. He's blunt about it: 'You are not going to have a technical solution,' because a smarter agent can always outmaneuver its creators. Joscha responds by comparing this to nuclear weapons, which many feared would bring catastrophe, but instead helped deter major war for decades.

Still, he agrees that AI is a different kind of challenge over a much longer horizon.

He also warns that locking humanity in as the permanent pinnacle of evolution would be, in his words, 'a very ugly perspective.' Roman uses a vivid analogy to explain the intelligence gap. Asking humans to predict the actions of a superintelligence, he says, is like asking squirrels how humans will kill them. 'The best squirrel scientist... is not useful information.' The point isn't that the future is known, it's that humans may be too small-minded to manage what comes next.

Joscha pushes back though, saying current large language models aren't the same as the radical, adaptive intelligence Roman fears. He sees them more as sophisticated tools, even if the future remains deeply uncertain. And the conversation widens from there into corporate power, regulation, and consciousness. Roman warns that AI companies may seek oligopoly under the banner of safety.

Joscha argues for useful, narrow systems that solve real problems without becoming replacement minds. By the end, both men agree on one thing: the central challenge isn't just building intelligence, but deciding how much agency humanity is willing to surrender. to surrender.