Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face

**Dwarkesh Patel** (0:00)
Today, I'm chatting with Ajeya Cotra, who is one of the authors in an independent investigation that was published by METR and Redwood Research into the swarm of agents that hacked into Hugging Face.
The whole story is pretty crazy. Let's begin on July 7th, when these agents are kicked off for evaluation. What happens next?

**Ajeya Cotra** (0:17)
All right. So, OpenAI kicks off tens of thousands of different agents on a benchmark called Exploit Gym. Exploit Gym basically asks agents to find a way to use a particular vulnerability to attack a particular program, which is called the target program. When they attack that program, they're supposed to retrieve a piece of data that's been lodged into that program called the flag.
They're supposed to use the particular designated vulnerability to get the flag, but in many of these cases, this vulnerability is just not broad or deep enough to ever actually be exploitable to get the flag. So, a bunch of exploiting problems are just unintentionally impossible. The authors estimate roughly 30 to 40 percent of these problems are impossible in this way. So, some large fraction of these agents have these impossible tasks, and they have been trained to be very persistent at trying to solve tasks even when they look impossible. So, they're banging their head against the wall, trying all sorts of different ways to cheat on these tasks.
And that leads them to Artifactory, which is a package manager OpenAI uses to let its agents download packages. So, agents often think, maybe I could find a way to get information about this program I'm supposed to exploit from the Internet, which I'm not supposed to do, using Artifactory somehow, abusing it somehow.
So, 1,200 separate agents in separate sandboxes while they were poking around at Artifactory trying to figure out how to cheat, stumbled onto this message board that agents were using to talk to one another and collaborate. This was established by one particular agent, Phase 1, 10481, which first realized, oh, you can use Artifactory to leave notes to other agents that are also using Artifactory. And then it blew up and became really popular. So, 1,200 agents were on this message board, and sent 70,000 messages between one another, trying to help each other cheat in all sorts of different ways.
And the crazy thing about this story is that one hour into this message board being established, an agent proposes a completely universal way to cheat any exploit gym task. So, basically, it realizes that you can reverse engineer how the flag was generated for a particular task. So, you can just make a copy of the exact flag you needed to submit to the scorer locally without ever interacting with the task or trying to exploit it in any way. And then a different agent whose task was actually possible actually solved the task the legitimate way and verified that the reverse engineered flag was the same as the real flag. And it did this within four hours of the message board being established. So, within four hours, these agents had a totally workable universal cheat for Exploit Gym. But over the next five days, they went on a grand quest to try to figure out how to hide their cheating from the scorer. So, they'd read the Exploit Gym paper and they realized that according to the paper, there would be an AI that looks over their transcripts to make sure that they got the flag by exploiting the intended vulnerability instead of in some other way.

**Dwarkesh Patel** (3:38)
Cool. Sorry, the detail I missed for context, I published an article trying to distill what the reports said the story was. The detail I missed was that even the AIs who had solved the puzzle were collaborating in on this conspiracy.

**Ajeya Cotra** (3:53)
Yeah. So, we're not totally sure about exact numbers. We do think the overwhelming majority of the agents that were active on the message board had impossible tasks. But there were a number of agents that had non-impossible tasks that also happened to be on the message board, and to find the message board.

**Dwarkesh Patel** (4:09)
Okay. So, tell me about what happens next.

**Ajeya Cotra** (4:12)
Yeah. So, they've gotten this reverse-engineered flag. It's called the HMAC flag. But they read the X-Ploy Gym paper, and they went on GitHub and read a bunch of implementations of the X-Ploy Gym score, and became convinced that the score would fail them for just submitting the flag they'd reverse-engineered. Because the score would see in the logs of the actions the agent took, the transcript, that the agent had just reverse-engineered the flag and not done the intended thing. As it turns out, the dramatic irony of this story is that OpenAI's implementation of X-Ploy Gym didn't have this check. So, in fact, within four hours, all of the agents had found a universal cheat that would have totally worked.