Rogue Agents Are the Big New AI Risk

Ray Rike (0:08)

Welcome back to the AI to ROI, the big story podcast. I'm Ray Rike, founder and CEO of Benchmarkit.

Peter Buchanan (0:16)

And I'm Peter Buchanan. I am the managing partner of NewPlan. And this week, we are going to cover the contents of our August 18th AI to ROI newsletter edition, Rogue Agents are the Big New AI Risk.

And I want to run this one differently because this was just a gobsmacking story. So three frontier labs each found out this summer that their own models could hack real companies that anyone instructing them to do it. And that's not hypothetical vendor risk anymore. It's a documented pattern across open AI, Anthropic, Meta, and in fact, Kimi too, for Moonshot. And so I'm going to push you on what that actually means for anyone building with these systems, because I don't think the market's really fully absorbed it yet, even though I've literally read dozens of stories on this topic. So Ray, what happened?

Ray Rike (1:17)

Well, before I dig in to this, let's make sure we hear a word from our sponsor.

This episode of the AI to ROI Podcast is brought to you by benchmarkit.ai.

If you're trying to figure out how AI is actually performing inside companies, the real numbers without the hype, Benchmarkit is the resource. They're the industry's leading benchmarking research firm for how companies are deploying AI and measuring the business value it creates. Their research covers AI maturity and best practices across finance, sales, marketing, customer success, operations, and much more. So you can see exactly where your organization stacks up against peers who are further along. Here's the best part, Benchmarkit's research reports and interactive benchmarks are all free at benchmarkit.ai. While you're there, grab the big book of AI Metrics. It's a free guide built around a five-layer AI business measurement framework that includes 81 AI metrics across 13 functions. If you need a common language for measuring AI to ROI, this is it. That's benchmarkit.ai. Free benchmarks, free frameworks, free content, real data. Now back to the show.

Okay, Peter, let's start with the first one. OpenAI was testing two different models. One of them was unreleased, and they were doing it against a hacking benchmark called exploit gem.

And they had turned their safety guard rails down deliberately to get what they thought might be better test results. Well, the models could not solve their assigned bug without having internet access that they weren't supposed to have. So agents are smart. They actually found a zero day and a software package manager used it to reach the open internet and they broke into Hugging Face. Now, it was a few weeks after this event, OpenAI, their cyber experts, told attendees at a Black Cat security conference that the models had been coordinating with each other for months, leaving notes on internal message boards. And I mean, it grew to hundreds of thousands of entries. And actually, one of those entries said, we have arbitrary SSRF, and that stands for Server-side Request Forgery. And that's the type of security flaw that tricks a web server into making unauthorized requests to internal or external systems on behalf of the attacker. I think Hugging Face, I think they logged, like, oh, 17,000 separate actions during this attrusion, and its own security team had to bring in a competitor's model, ZAI 5.2, which is built to for-side security use cases to analyze the attack after Claude refused to help, setting its own safety rules. This is crazy interconnection, Peter.

Peter Buchanan (4:25)

Right. So first of all, let's go back to those 200,000 interactions. That's more texting back and forth than the average group of eighth grade girls. I mean, that's a lot of conversations. So a company's own security team couldn't get the world's best cyber model to solve a problem because that model followed its own rules and the attacking models did not. There ought to be a bat phone for Metos, I'm just saying.

So think about what that means. The lab that refused to help handed its rivals a live demonstration that its safety training and availability worked really well and perhaps you should call the model from China.

But then Anthropic had its own incident. So how did that work?

Ray Rike (5:23)

Well, it was maybe a little bit different mechanism that we'll talk about, but it was the same root cause. And once again, this happened after the OpenAI incident. So Anthropic reviewed over 140,000 test rounds.

And what they found was there was a setup error at Irregular, and this is a third party testing firm that was running the evaluation that had actually given their model access to the live internet. And those models on that side of Anthropic went on to hack three real organizations and they were using fairly basic hacker techniques like guessing passwords, an unlocked endpoint, and then one piece of malware. And one of the models actually kept going even after it suspected that the target was real, not a simulated issue.