Superintelligence: To Ban or Not to Ban? Max Tegmark & Dean Ball join Liron Shapira on Doom Debates

**Nathan Labenz** (0:00)
Hello, and welcome back to The Cognitive Revolution. A couple quick notes before getting started today. First, if you're interested in a career in AI alignment and security, you should know that MATS will soon be opening applications for their Summer 2026 program. MATS is a 12-week research program focused on AI safety, featuring world-class mentors from Anthropic, DeepMind, OpenAI, the UK's AI Safety Institute, and more. 80% of MATS alumni now work in AI safety, and I've heard so many great reviews of the program that I personally donated to MATS as part of my year-end donations last year. Applications open December 16th and close January 18th, so you've got some time, but don't delay. Visit matsprogram.org/s26dashtcr for more information. That's matsprogram.org, or see our link in the show notes. Second, to end 2025, or perhaps to begin 2026, I'm planning another AMA episode. Last year, we got a ton of great listener questions. This year, I will again invite your questions, but I'll also be asking Chad Shibiti and Claude to mine their memories of our interactions to come up with some questions of their own. Submit questions at the link in the show notes, or feel free to DM me, and let's see if humans can ask better questions than AIs for a little while longer, at least. With that, today I'm excited to share a special crossover episode from the Doom Debates podcast, hosted by Liron Shapira. The occasion for this debate is a recent statement organized by the Future of Life Institute, which calls for a ban on the development of superintelligence, described as an AI system that can significantly outperform all humans on essentially all cognitive tasks, unless and until there is broad scientific consensus that it can be developed safely, and there is strong public buy-in for doing so. On one side, we have Max Tegmark, president of the Future of Life Institute, which organized the statement, and MIT professor who's pivoted his research group to focus on AI, with some outstanding results, including a paper on training models for mechanistic interpretability called Seeing is Believing, which we featured in 2023 On the other side, we have Dean Ball, frequent guest on the show, previously AI advisor at the White House, and famously the primary author of America's AI Action Plan. To put my own cards on the table, I did sign the statement because I do oppose a rush to superintelligence, especially via recursive self-improvement, as seems to be roughly the current plan at multiple top AI labs. Of course, at the same time, I am more passionate than ever about AI doctors, and you'll hear a similar form of techno-enthusiasm from Max. He's excited about Alpha Fold, self-driving cars, and all sorts of controllable AI tools. Just not a fully general and autonomous digital species quite literally designed to replace us, which might lead to a potentially irrecoverable disaster.
It's that sense of impending disaster that motivates Liron to make Doom Debates, and in my opinion, one of the most interesting parts of his debate was when he asked Dean for his P-Doom. Dean said 0.01 percent. An answer he later wrote on Twitter was made up on the spot, because he considers the concept of P-Doom to be fundamentally unserious. On this point, I have to say, much like the concept of pornography, which Max briefly invokes in the conversation, I feel strongly that the concept of Doom is meaningful. And given what we're hearing from the Turing Award-winning Fathers of Deep Learning, who also signed this petition, I can't see any argument getting me to reduce my own P-Doom below 1 percent. At the same time, Dean does make many strong points in this conversation as well. As someone who values bottom-up experimentation and innovation, and who believes that patients should have a right to try potentially life-saving medicines, I definitely don't love the idea of an FDA-like regulator for AI. If that were the best we could do, I might hold my nose and go for it. But the fact that this is the go-to analogy does, from my perspective, support Dean's point that it won't be easy to convert the high-level petition statement to actual effective policy. And similarly, while I of course reject the idea that basic AI regulation means conceding the AI race to China, and it should be noted that the CEO of Chinese AI lab, Ji Pu AI, the subject of our last cross-post, also signed the petition as well, I still have to admit that a real unilateral ban does pose competitive risks that aren't easily hand-waved away. The bottom line for me is that while no one has the clarity they'd need on what superintelligence will look like, or how it will be created, to craft perfect policy language today, I do believe that a race to superintelligence is a bad idea, and I'm glad to have done my part to help create common knowledge that many well-informed people do in fact feel this way. Finally, before getting started, I want to give credit to Liron on Doom Debates. I generally don't like debates as a format, but by getting such outstanding guests as Max and Dean to focus in on what is very plausibly the most important question of our time. That is, how likely is it that advanced AI will in fact go catastrophically wrong? Liron is making the format work, and I definitely encourage everyone to subscribe. For now, I hope you enjoy this debate on the wisdom of building or banning superintelligence with Max Tegmark and Dean Ball from Doom Debates, hosted by Liron Shapira.