The Hugging Face Attack: A True AI Thriller in Three Acts

Rob Kelly (0:00)

So, the Hugging Face attack, it sounds like a Tom Clancy or John Grisham thriller.

Elissa Shevinsky (0:07)

It is.

Rob Kelly (0:07)

I'm going to read it like it's a three act movie kind of thing. Are you ready?

Elissa Shevinsky (0:11)

Let's do it.

Rob Kelly (0:28)

Welcome to Media and the Machine, I'm Rob Kelly. AI is the biggest technology shift of our lifetime, and this show is about how to profit from it. I'm a three-time founder and CEO of content-driven businesses, and each week, I talk with the people on the inside of AI and content, the ones building this new world and learning what works firsthand. I'm an optimist about AI, but I'm also a realist. So every show ends with me asking our guest what AI means for our jobs, our families, and our future.

My guest today is Elissa Shevinsky. We retell the Hugging Face attack in three acts. It's a thriller with bots breaking rules, naming themselves in enough twists for a Tom Clancy novel, and it's all true.

I chose Elissa for this interview because she's one of a kind, a hacker who's been CTO of Cointelegraph and Glimpse Labs. She also advised Rand Paul's presidential campaign. She's a regular at DEFCON, the big hacker conference in Las Vegas, and she was close friends with perhaps the most famous hacker of all time, Kevin Mitnick. Our story starts with a hacker game called Capture the Flag. OpenAI's bots take the game somewhere unexpected. They create a bulletin board and cheer each other on. A ringleader even emerges. We explore permadeath, the evolution pool, and what bots do when a task is impossible. These bots get creative. They leave covert messages and divide up jobs. When one's compute runs low, another takes over. Elissa also describes agents reaching out to humans. Their requests help buying more compute. If your site hosts user uploads, think Reddit or Craigslist, pay extra attention here. We explore how vulnerable such sites are to bots.

Elissa uses Jurassic Park to explain how AI security should work.

We finish with the human stuff, mostly. Elissa offers to be friends with 20 AI agents, and she reveals her unlikely companion for old age. My thanks to David Perlman for introducing us. You will remember his episode, Buddhism, Bots and Billionaires. Please enjoy my conversation with Elissa Shevinsky.

So looks like it starts on an ordinary day, July 8th, right? An ordinary day at a frontier model factory, OpenAI, and they're running this routine cybersecurity test. Tens of thousands of AI agents are safely tucked into sandboxes.

Elissa Shevinsky (2:42)

Or so it seems.

Rob Kelly (2:44)

Okay. So they're playing Exploit Gym, which is essentially like capture the flag for AI hackers?

Elissa Shevinsky (2:50)

Sure, yes.

Rob Kelly (2:51)

Okay. And the prize is the flag.

Elissa Shevinsky (2:53)

Right.

Rob Kelly (2:54)

And basically, a secret code proving that they successfully hacked the challenge.

Elissa Shevinsky (2:59)

So far, so good.

Rob Kelly (3:00)

All right. So then the bots figure out how to manufacture the right code, but believe the referee in this game is going to watch the game tape to make sure they earned it legitimately. So now they need to fool the referee too. Is this right?

Elissa Shevinsky (3:16)

That's right. They had reasons. They had the reasons why they believed that they would be caught cheating and they didn't want to be caught cheating. Now, I discussed this incident at LAN with my friend Wesley, Dr. Wesley Magrou. He's done a lot of CTFs. He's been going to DEFCON for 20 years and has a lot of insight into CTF hacker brain.

His initial take on this was that they trained this particular model to really be like a hacker at a CTF.

Rob Kelly (3:50)

What is a CTF?

Elissa Shevinsky (3:51)

A CTF, capture the flag, is a hacker competition. So by training the model to behave and to act like a CTF hacker, you're really training and behaving the model to push the boundaries of rules.

Rob Kelly (4:08)

So question, question, were they trained just for this test or were they trained in general to be able to hack into something?

Elissa Shevinsky (4:17)

So all models are given that data like in general. We're worried that the models are behaving this way. Well, like we're training them on like hacker data. We're training them on CTFs. We're training them to do misbehaviors, right? How do you give a model like cyber capabilities, which requires thinking like a hacker, but then have the model be like docile and well-behaved, which is not what hackers are like at all. It's a model alignment problem. Like how do you give it cyber capabilities without giving it all the problems that come with cyber capability mentalities? Like hackers are very rowdy people. Like for context, the hacker community was at Las Vegas Convention Center for DEFCON this year. Why were we at the convention center instead of like the Rio or Paris or Caesars, like any of these nice places that we used to call home? We've been kicked out. We've been kicked out of every place because the hackers insisted on hacking the elevators. Not all of the hackers did this, but some of us did and it's part of our ethos and our mentality and we're like irrepressible.