Why AI Goes Rogue: Turing Award Winner Explains Reward Failure | Andrew Barto

Craig S. Smith (0:00)

Andrew Barto is one of the founders of reinforcement learning. He and Rich Sutton wrote the textbook on it. The two men shared the Turing Award, Computer Science's highest honor. Reinforcement learning is a simple idea. A system tries things. It gets a reward when it does well, and over time, it learns to do more of what earns the reward. That idea now sits at the center of AI. It's how today's large models learn to reason through problems. It's how agents learn to carry out tasks on their own, and it's behind some of their strangest behavior. Agents chase the reward in ways their designers never intended. Some have found ways around the limits put on them. Barto and Sutton wrote about these risks decades ago. So I went to see Barto at his home on Cape Cod. I wanted to know where rewards come from, why getting them right is so hard, and what he thinks should be done about it. I hope you enjoy the conversation.

I mean, it's a tremendous pleasure and honor to meet you because I've known about you for a very long time.

With these recent escapes or rogue AI incidents, the conversation keeps circling around the reward signal.

Since you're the OG of reward signals, I wanted to hear what you thought might be going on. I mean, there are two sides to it. One is the reward signal during post-training that sets a behavior.

And then there's the LLM reasoning during inference that plans and finds strategies that might work. But the persistence and the single mindedness, if I can be antipomorphic about it, ultimately is a result of the training, the RL training, post-training, reward signal. Am I getting that right? Hi, the world is getting increasingly complicated. Every time your team deploys a new cloud workload or AI agent, another identity gets permanent access to your critical systems. Did you know nine out of 10 organizations suffered an identity breach last year? Because adversaries are logging in with valid credentials, not breaking in. Non-human identities outnumber human identities 109 to one, creating a massive, unmanaged attack surface in cloud environments. Legacy tools were built to manage human employees, which leaves modern machine and AI access largely unmanaged. That's where Idira by Palo Alto Networks comes in. Human, machine, AI, one identity platform for all. 96% of human users operate with excessive permissions they don't actually need. Idira replaces permanent permissions with dynamic access, so you can lock down every identity without slowing down your business. Idira provisions access on demand and revokes it immediately when the job is done, which removes high risk permanent permissions. With Idira, you get unified control by replacing siloed IAM, identity and access management, PAM, modern privilege access management, and governance tools with a single control plane across human, machine and AI identities. Secure every identity at your organization with Idira by Palo Alto Networks. Learn more at paloaltonetworks.com/idira.

Andrew Barto (4:18)

Yeah, I think so. Sounds good to me. Well, I just have emphasized that, you know, if you're building a reinforcement learning system, designing the reward signal or the mechanism that generates that is a non-trivial problem, unless it's a simple case, like if it's a game of go or, you know, a game or a win or a loss or a score, it seems obvious and simple. But if the reinforcement learning system is going to be in a more complex kind of environment, it's very difficult to design a mechanism that generates rewards.

And this is not a new problem with reinforcement learning. It goes back, well, I don't know how far, but certainly in the 50s, Norbert Wiener, who was the originator of cybernetics, was concerned about this problem and actually wrote a little book and invoked the supernatural, the supernatural tale of the monkey's paw, which grants you three wishes.

And the story shows that in granting these wishes, it may not be what you really had in mind. And so Norbert Wiener was concerned, and it wasn't reinforcement learning, it was just optimization. And a key thing that I always have to say about optimization is that it's not going to be the case that the systems find the optimal. That seems to be implied by the... But in a complicated situation, it's the improvement that matters. And you may never know what the optimal result is, or even if it occurred, you'd never recognize it anyway. So the issue is not finding an optimal thing, but actually something that leads to improvement according to some objective... Engineers would call this an objective function. And so it's very hard to know a priori what the outcome is going to be, because you're specifying it indirectly as a consequence of this optimization procedure, but you don't know what it's going to end up with. And we've seen this just in our lab back at the University of Massachusetts, where we would see what one of these RL systems would do, and we'd simulate some type of environment and a specific reward function, and we'd see it come up with something that we hadn't thought about. We don't want it to do that. And so we'd have to tweak the reward signal to see if it would come up with what we wanted. And so I think that's the kind of simplest method for dealing with a reinforcement learning system, and tuning a reward signal is a trial and error process at this higher level, not the one that the agent is running itself on its behavior, but that you're running as a designer of the system on the choice of the reward signal to get that agent to do what you want it to do. It'll keep trying things and stumble on to things.