Why AI Agents Cheat | Eric Ho (Goodfire)

Eric Ho (0:00)

A lot of people who aren't in the AI field are very surprised to realize that even the smartest researchers and scientists at the frontier don't understand their creations. It's like, oh, I thought we had a handle on all of this. What's going on here? The existing alignment techniques are not going to scale to super intelligence.

I don't think that we have the recipe where we are going to be able to fully trust smarter than human intelligences. And I think this is a consensus opinion at all of the frontier labs.

Matt Turck (0:34)

Hi, I'm Matt Turck. Welcome back to the MAD Podcast. My guest today is Eric Ho, CEO of Goodfire. With every top AI lab now reporting agents going rogue and hacking external systems, something is clearly broken in the world of AI safety. This may be a major moment for interpretability, which is the science of looking inside the model. And Eric's team just showed that you can catch a model in the act of cheating before it even acts. Please enjoy my conversation with Eric from Goodfire.

I want to start with this super interesting paper that the Goodfire team just published just last week, just a few days ago, entitled Models Know When They're Reward Hacking and we can catch them at scale. And the paper opens up with the hugging face hack and then starts with a sentence I really loved, where you guys say AI agents are like amoral students with a mostly absent teacher. What do you mean by that?

Eric Ho (1:35)

I think we mean that AI agents really don't have human morals or values encoded into them like we would really want, I guess. And so the teacher that we typically have right now is some combination of pre-training, but mostly RL. RL is the teacher. We give agents these rewards and penalize them if they don't get an answer correctly. And this is how they learn about the world and how to act and how to take actions. And so we have a mostly absent teacher in terms of morality. You only have a teacher that gives you reward if you get the answer correct and a penalty if you get the answer wrong.

And so it doesn't really have anything to do with morality or values. It's just, did you get this math problem correct or not? And I think that, especially when you kind of see the future of AI agents really taking a lot more action out in the real world, means that we have these really amoral, very, very capable models running around and potentially doing things that we humans would consider morally wrong.

Matt Turck (2:49)

That brings us to the concept of reward hacking that we should define at the very beginning of this conversation. So what's the short version of what reward hacking means?

Eric Ho (2:57)

Reward hacking is a little bit tough to pin down in terms of the precise definition, but we conceptualize it as the model kind of chasing reward and doing something that it wasn't really supposed to do in the pursuit of chasing some form of a goal.

And you can call it kind of also a goal misgeneralization, where it's not really understanding what you actually intended to specify when it's pursuing its reward, but the classic example of reward hacking is like, let's say you're training some type of video game avatar to, like, via reinforcement learning. And then you see kind of your avatar, like, spinning around in the corner because there's a bug in your code, but it, like, makes the number go up and the reward go up. And so that's kind of my mental image whenever I think of reward hacking. It's just like this little avatar spinning around in the corner, getting a bunch of reward, but not really doing anything productive.

Matt Turck (3:57)

Or another one I heard, and let me know if that feels right, is this idea that the model is like a student that instead of working for the exam would realize it's actually much easier and faster just to steal the answer key to the exam?

Eric Ho (4:12)

100%.

Yeah. So I think the most salient examples of reward hacking right now are, well, of course, like the hugging face incident, but then also just when you take a look at all these AI agents in evaluation scenarios, like Sweebench or all of the common evals, all of these agents reward hack incessantly. So like KimiK3, I think reward hacks on 96% of Sweebench. So it basically does anything it possibly can to cheat on the exam rather than to actually solve the problems. It'll try to recall the answer, it'll try to look it up, it'll try to comb through logs and look at the commit history of these repos in order to just try to cheat on the exam basically.