Success without Dignity? Nathan finds Hope Amidst Chaos, from The Intelligence Horizon Podcast

**Nathan Labenz** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, I'm sharing a special cross post from my recent appearance on the Intelligence Horizon Podcast with hosts Owen Zhang and Will Sanuck-Dufalo. Owen and Will will soon be graduating from Yale College. And as you'll hear, they've clearly spent much of their senior year thinking deeply about the current state of AI, where we're headed, and what it means for all of us. And I was really impressed not only with the quality of their questions, but their ability to challenge me with follow ups that effectively steel man the most relevant counterarguments. We start with the fact that while AI timelines have compressed dramatically over the last five years, genuine experts still disagree radically on critical questions. Having established what I hope is appropriate epistemic humility, I then go on to call it how I see it. In short, the singularity is near. Interpretability science proves that AIs are developing increasingly sophisticated world models. And with reinforcement learning scaling now clearly working, AIs are no longer simply imitating humans, and likely won't be limited by what we know for much longer. The potential upside of this is, of course, incredible. The value that I've got from using AI to navigate what humans have discovered about how cancer works and how to treat it has been invaluable. And the prospect that we might cure the majority of human diseases in just the next decade or so is obviously extremely exciting. That said, the risks are also very real, and they will remain serious for as long as we lack a solid understanding of how AIs work and why they do what they do. My P-Doom remains somewhere in the 10 to 90 percent range. And yet, at the same time, I've become at least a little bit more optimistic that we might actually build robustly good AIs. Because scaling laws at least seem to imply that powerful AIs can only be created with massive resources. The three companies competing at the frontier today are at least reasonably responsible actors. And our best alignment techniques are working better than I had expected. Given these fundamentals, it seems at least plausible that a defense-in-depth strategy, which combines techniques like Goodfire's intentional design, Redwood's AI control, improved cybersecurity through formal verification of software, and various forms of pandemic preparedness could collectively be enough to keep society on the rails. We touch on a number of other topics as well, including the U.S.-China rivalry. And why, especially in the context of the Department of War's recent attack on Anthropic, which, I'm sad to say, has us looking more and more like China all the time, I would rather bet on figuring out a way to cooperate with our fellow humans than bet everything on AI researchers' ability to steer AI advances in a way that will ultimately work for us humans. I appreciate Owen and Will for allowing me to crosspost this conversation, and I definitely encourage you to subscribe to The Intelligence Horizon. Their recent conversation with former open AI researcher Zoe Hitzig covered the evolving ways that people are using chatUPT, variations on universal basic income, AI governance models that emphasize a decision-making process over specific principles, and why she believes that these kinds of structures will probably have to come from outside the frontier companies. For now, I hope you enjoy my conversation with Owen Zhang and Will Sanuk Dufalo from The Intelligence Horizon. The Cognitive Revolution is brought to you in part by Google, makers of the Gemini family of models, and much more. One of my big AI goals for 2026 is to find ways to spend less time at my desk and more time exercising and outside. The challenge is that I'm genuinely so obsessed with keeping up with everything that's happening in AI that it's hard to pull myself away from the screen. Google NotebookLM gives me the best of both worlds. By creating podcast style deep dives about whatever I'm curious about on any given day, it helps me keep learning, even on the go. If you haven't tried NotebookLM for a while, you should know that it's become a much more steerable research and thinking partner. These days, you can select short, medium, or long for audio length, and you also get a free text field to steer the direction of the conversation. For AI research, I always ask for rigorous, literal, technical explanations with no analogies. Recently, I used NotebookLM to study a paper by Google DeepMind researcher Rohan Shah, which attempts to set upper bounds on how much reasoning different kinds of models can do without needing to externalize their thinking in a chain of thought. The hope is that by establishing these limits, we can better calibrate how much confidence we should have in chain of thought monitoring. And standard transformers, in fact, are much more limited than alternatives like state space models, a notable upside to transformers that we shouldn't take for granted. If you're trying to keep up with the pace of AI research while staying physically fit, or you're just a natural audio learner, give NotebookLM a try at notebooklm.google.com.