Anthropic’s Rogue Trip & Superintelligence Fever!

Ian Krietzberg (0:01)

The model goes through and finds itself on the website for Philadelphia Police Department's Homicide Hotline and submits a tip, which is just, you know, again, unintentional breaches.

Peter Hamby (0:20)

Welcome to The Powers That Be Daily, Puck's podcast focused on the intersection of Wall Street, Washington, Silicon Valley and Hollywood, and the players who run it all. I'm Peter Hamby. It's Wednesday, October 14th. Today, I'm joined by Ian Krietzberg to explain how some rogue Anthropic agents accessed a State Department website and reported a fake homicide tip to the Philadelphia PD, and how the Trump administration is suddenly demanding more transparency from the big AI companies.

Speaking of Trump, Ian also answers an important question. Is the sudden arrival of the term superintelligence just a ploy from CEOs to please the president, or is it a marketing term, or is it something else entirely? We'll discuss all that and much more on today's episode of The Powers That Be.

Happy Wednesday, everybody, and welcome to The Powers That Be. I'm joined today by my colleague Ian Krietzberg, author of The Hidden Lair, our expert on all things AI. Today, we're going to talk about AI gone wild. Rogue agents from Anthropic filled out government forms, they gave cops fake tips. We're going to get Ian's insight on to what's going on and how the government is reacting. Speaking of the government, there is a superintelligence task force now that is confronting these challenges, and we're going to talk a little bit about Trump rebranding AI as superintelligence and all the tech leaders bowing down and saying, sure, why not? Let's just call it superintelligence. Ian, we'll get into that in a moment. But first, give our listeners a sense of what's been happening with these Anthropic agents. Also, isn't Anthropic supposed to be the good AI company?

Ian Krietzberg (2:12)

Yeah. Well, and also didn't they all say they were going to slow down too?

Peter Hamby (2:15)

And just a few days ago. Yeah, sure.

Ian Krietzberg (2:19)

Right. So maybe this is part of their cleaning up their act stuff that they've been talking about in terms of safety. But I've been on the show a bunch of times talking about the rogue AI thing. And it's all anyone's talking about. And I just wanted to start by jumping in because it has just so frustrated me the way that this is described.

And we see it in these incidents, which we'll all get into in a second, that the way this is framed, which is very convenient framing, it's the easiest way to frame it, which is like you have this rogue AI hacking out there on the Internet, right? It just seems like a remarkably imprecise way to talk about what's happening. And I think the incidents that Anthropic disclosed late last week are really good case in point of why that framing just makes no sense, right? I think the first thing to keep in mind is that all these, I'm going to call them like unintentional website breaches, right? It's not really hacking. Websites, public websites are being accessed in a way that's weird and unusual, and they shouldn't be, and they weren't told to, and they were not supposed to be accessed. And so weird stuff is happening, but it's not the same as like a foreign actor hacking into XYZ government website, right? Like the same exchange of information is not going on here. And the important thing to understand about this is that all these disclosures, whether it's these new ones from Anthropic or even down to all the ones for OpenAI, the hugging face stuff, which was at the root of all this, they all happened during model evaluation and testing. So whether or not they were specifically being tested on cybersecurity stuff, whether they're being, you know, the initial impression of what was going on was if you benchmark a model, if you test a model on its cybersecurity capabilities, if you say, go show me how strong you are at hacking, then it stands to reason that if things go wrong, you get access to the internet that you shouldn't have had access to, they might do some hacking. It was kind of in the instructions. But then instances started happening that they were not told to hack in some way, right? They were just doing evals and websites were accessed. And so this is why I'm talking about trying to think about reframing how we understand this in terms of unintentional website breaches, because these are results of testing, testing where safeguards are peeled back, testing where the researcher's goal is to push the limits to understand what the model is capable of. And so in these specific ones that Anthropic lays out, I'll lead with the one that seems the most significant and also kind of funniest, which is the model had been tasked with just kind of doing random example tasks on random websites. Anthropics information was not very clear about the details of this, but that's just doing stuff on websites.