The Adversarial Mind: Defeating AI Defenses with Nicholas Carlini of Google DeepMind

**Nicholas Carlini** (0:00)
There are lots of lessons we've learned over the years. One of the biggest ones probably is the simplest possible objective is usually the best one. Even if you can have a better objective function that seems mathematically pure in some sense, the fact that it's easy to debug simple loss functions means that you can get 90 percent of the way there. So the accuracy under attack for the type of additional examples you train on usually is 50 percent, 60 percent, maybe 70 percent, and that's much bigger than zero, right? Like, you know, this is good. But as an attacker, what does 70 percent accuracy mean to me? 70 percent accuracy as an attacker means to me I try four times and probably one of them works. The core of security is turning this really ugly system that no one understands what's going on and highlighting the one part of it that happened to be the most important piece. This is important to do to show people how easy it is because the people who know it's easy are not going to write the papers and say it's easy.

**Nathan Labenz** (0:56)
Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with Nicholas Carlini, prolific security researcher at Google DeepMind, who's demonstrated over and over again that despite many attempts and tremendous effort, AI systems still cannot be robustly defended against adversarial attacks. My goal in this conversation was to draw out the mental models, frameworks, and intuitions that have allowed Nicholas to be so consistently successful at breaking AI defenses. And we cover a ton of ground, including the fundamental asymmetry between attack and defense, how visualization helps him understand high-dimensional spaces, how adversarial defenses usually work by modifying lost landscapes and the techniques he uses to get around those challenges, how confident we should be in our understanding of the features learned by interpretability techniques like sparse autoencoders, the relationship between interpretability and robustness, the compute requirements for different types of attacks, how he approached and ultimately quite quickly defeated the tamper-resistant fine-tuning defense that we previously covered in our episode with Dan Hendricks, how models store and can be made to reveal training information, what makes humans more robust than current AI systems, whether the black box characteristics evolved by biological systems might be adaptive for security purposes, and the still quite limited role that today's AIs can play in developing Carlini-style adversarial attacks. Throughout the conversation, Nicholas shares a number of fascinating insights, from his observation that almost everything in high-dimensional space is close to a hyperplane, to his emphasis on starting with the simplest possible loss function, to his practical wisdom about which defenses are worth spending the time to attack in the first place. At the same time, there's an important meta-lesson here about the possible irreducible black box nature of intelligence itself. Nicholas doesn't fully understand why he's so good at this work, and as you'll hear, he chalks a decent part of it up to an impossible to articulate intuition that he's developed over years of experience. Now, as we enter into an era in which reinforcement learning is quickly propelling AIs to human or even superhuman levels of capability in more and more domains, we can only expect more Move 37 type insights from AI systems as well, and we'll face real challenges in determining how much to trust them.
This in turn underlies another important theme of this conversation, which is the genuine ambivalence of the AI safety community toward powerful open source models. It's underappreciated and worth repeating that most AI safety advocates are lifelong techno optimists who, like Nicholas, genuinely fear concentration of power and appreciate both that open source software has been amazing for the world and that open source AI models specifically have been critical to enabling all sorts of recent safety research.
Yet, at the same time, they worry that extremely capable AI systems are coming soon and, in part because of Nicholas' work, strongly doubt that we will be able to make such systems safe enough to be distributed broadly in an irreversible fashion. This is a really vexing dilemma, but with AI being deployed in more and more contexts all the time, my hope for this episode is twofold. First, that highlighting Nicholas' work can help equip policymakers to make informed decisions as they inevitably confront difficult trade-offs. And second, that we might inspire a few talented researchers and builders to meet the market demand and social need for AI security expertise by pursuing their own version of Nicholas' storied career path. As always, if you're finding value in the show, we'd appreciate it if you take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. We welcome your feedback and suggestions too via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, I hope you enjoy this window into the habits of mind that support successful AI security research with Nicholas Carlini of Google DeepMind. Nicholas Carlini, security researcher at Google DeepMind, welcome to The Cognitive Revolution.