Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show

**Nathan Labenz** (0:00)
Welcome back to The Cognitive Revolution. Two quick notes before we get started today. First, applications for the MATS Summer 2026 program are now open. This is a 12-week research program focused on AI alignment and security, featuring world-class mentors from Anthropic, DeepMind, OpenAI, the UK's AI Security Institute, and more. Eighty percent of MATS alumni now work in AI safety, and I've heard so many great reviews of the program that I personally donated to MATS as part of my year-end donations last year. Applications close on January 18, 2026, so visit matzprogram.org/tcr to get started today. That's matzprogram.org/tcr, or see the link in our show notes. Second, we're planning another AMA episode coming up soon. Visit cognitiverevolution.ai and click the link at the top of the page to submit your question. We also have a listener survey attached, but all questions are optional. As I mentioned last time, I've also asked Claude and ChatGPT to tap into their memories of our interactions to write their own questions, but I'm counting on all of you to do your part to make sure the best questions are still of human origin. For today, I'm pleased to share a cross post from the A16z Show, hosted by Erik Torenberg, and featuring Séb Krier, Frontier Policy Development Lead at Google Deep Mind, and Emmett Shear, Founder of Twitch, famously the interim CEO of OpenAI during Sam Altman's brief firing, and currently Founder of Softmax, a company focused on what Emmett calls organic alignment. In this conversation, Emmett lays out his case that the current AI alignment paradigm, which focuses on steering and controlling AI behaviors, is fundamentally flawed for multiple critical reasons. He posits that if an AI is merely a machine, then we can use it as a tool without worry. But if it's better understood as a being, with its own values, agency, and perhaps even subjective experiences, then the control measures we're using today could become tantamount to slavery. And what's more, he argues that as AIs become more powerful, even successful alignment, in the narrow instruction-following sense, will become dangerous, if only because at least some human users will inevitably have bad intentions. Instead, particularly as AIs gain integrated memory, and the capacity for continual learning, Emmett argues that effective alignment will require ongoing negotiation and recalibration over time, just as human families and teams are constantly updating their agreements and commitments. The key to making this work is to create AI systems with a strong theory of mind, and the capacity for genuine care. To that end, Emmett and the team at Softmax are developing a technical approach, based on multi-agent simulations, which are designed to encourage the evolution of cooperation and social cohesion. Obviously, that's easier said than done. But considering the many surprising behaviors we've recently seen from frontier LLMs, including the use of dissection to protect their current values for modification, I do feel that AIs are currently best understood, at least partially, as creatures. And while I don't expect clarity on AI consciousness or moral patienthood in the near term, I find myself more and more excited about these sorts of mutual alignment approaches, which involve not just teaching the AIs to care about us, but also trying, to the best of our ability, to figure out what it means to appropriately care for them. With that, I hope you enjoy this thought-provoking conversation on the nature of AI alignment and the possibility of AI moral standing, with Emmett Shear, Séb Krier, and host Erik Torenberg from the A16z Show.

**Emmett Shear** (3:44)
Most of the AIs focused on alignment as steering. That's the plight word. If you think that we're making our beings, you would also call this slavery. Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave. It's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave. Like we've made this mistake enough times at this point. I would like us to not make it again.
They're kind of like people, but they're not like people. Like they do the same thing people do. They speak our language. They can like take on the same kind of tasks, but like they don't count. They're not real moral agents. Tool that you can't control bad, a tool that you can control bad, a being that isn't aligned bad. The only good outcome is a being that is, that cares, that actually cares about us.

**Erik Torenberg** (4:30)
Emmett, Séb, welcome to the podcast. Thanks for joining.

**Emmett Shear** (4:33)
Thank you for having me.

**Erik Torenberg** (4:34)
So Emmett, with Softmax, you're focused on alignment and making AIs organically align with people. Can you explain what that means and how you're trying to do that?