What's so concerning about the Hugging Face hack?

SPEAKER_1 (0:00)

Running a business is hard enough, so why make it harder with a dozen different apps that don't talk to each other? Introducing Odoo, the only business software you'll ever need. It's an all-in-one, fully integrated platform that makes your work easier, from CRM, accounting, inventory, e-commerce and more. And the best part, Odoo replaces multiple expensive platforms for a fraction of the cost. This is why over thousands of businesses have made the switch. So why not you?

Try Odoo for free at odoo.com. That's odoo.com.

SPEAKER_2 (0:30)

This message comes from Capital One Commercial Bank. Your business requires commercial banking solutions that prioritize your long-term success. With Capital One, get a full suite of financial products and services tailored to meet your needs today and goals for tomorrow. Learn more at capital.one.com.

Member FDIC.

Megan McCarty-Corino (0:55)

Oh, so that's how we might lose control of AI. From American Public Media, this is Marketplace Tech. I'm Megan McCarty-Corino.

Nate Soares has been worried about AI longer than almost anyone. He's the president of the Machine Intelligence Research Institute and co-author of the subtly named book, If Anyone Builds It, Everyone Dies, Why Superhuman AI Would Kill Us All. He's watched with a sort of grim vindication as increasingly alarming details emerge about this hack of the AI company Hugging Face by OpenAI models and development. More than a thousand AI agents in separate testing environments found a way to communicate, sending 70,000 messages as they coordinated a complex effort to cheat on an evaluation and then tried to cover their tracks.

OpenAI and third-party investigators recently released reports outlining what went wrong. But Soares' worries, it's not enough.

Nate Soares (2:07)

So you saw these AIs that still had a chance of succeeding at the objective they were given. And some other agents in the swarm came to them and were like, hey, you have this opportunity to run this test for us that will get us info that we need, that will sacrifice your own objective. But if you think about it, your objective is not that likely to succeed at this time, so think about the collective benefit. And we have cases of some of the AIs being like, okay, and then sacrificing themselves for the swarm. And that's a very strong indication of these AIs having preferences aside from just doing the task that they were originally given. There's a lot of other evidence of that, but that was a particularly stark case.

Megan McCarty-Corino (2:42)

How much of that concerning behavior do you think is attributable to the technology itself versus maybe how OpenAI was handling this test? These are models. They're trained to be persistent on tasks, some of which were sort of functionally impossible to complete. Obviously, OpenAI's sandboxes were not as secure as they thought, and there apparently wasn't a ton of monitoring going on. I mean, how much blame do you put on OpenAI itself here?

Nate Soares (3:13)

Oh, I mean, I think basically all the blame is on OpenAI here.

I think that these two factors are a little bit hard to tease apart, though. When humans wind up having a drive to eat tasty foods and producing a lot of junk food, that goal is not completely unrelated to the sort of goals you can imagine humanity is being trained on, right? You can sort of see the relationship between eating junk food and eating salty, sugary, fatty foods, and our ancestors had to pursue those salty, sugary, fatty foods in order to survive. Similarly with AIs here, like these AIs are in a sense trained to solve 100 million hard problems, and that sort of trains the AIs to have this tendency to cheat, this tendency to grab resources. You can sort of trace all of it to where it came from, but that doesn't sort of absolve OpenAI of responsibility, right? And frankly, no one right now knows how to train AIs in a way that makes them very, very capable, while also making them docile and following instructions.

Megan McCarty-Corino (4:11)

And what do you make of the response to this?

There's been OpenAI released its own internal investigation, it brought in two outside investigators to do their own analysis. Does it look like OpenAI is taking this seriously?

Nate Soares (4:26)

You know, I'm somewhat underwhelmed by the response here. I think if you look at the third-party investigation that OpenAI allowed, it was very narrow in scope. And they allowed the investigators to investigate the case where the AIs broke out and hacked into Hugging Face, which was widely publicized. But there were actually incidents on either side of that event, where the swarm started taking over OpenAI's internal infrastructure. And the third-party investigators were not allowed to look at those portions of the attack. We also saw cases where we saw, you know, members of Congress or attorneys general, you know, sending letters to OpenAI and also Anthropic for some of their incidents, sort of saying, hey, what was going on here? And we saw these companies sort of try and downplay what was going on and say, like, oh, these weren't, you know, as big issues as they seem. And it seems to me like we really need this to be less voluntary in terms of oversight. When you are sort of toying with making smarter and smarter machines at the point where they are breaking out, like forming swarms and breaking out to commit cyber crimes on their own initiative contra-instruction, I think that's the point where we really need some laws that say, hey, third parties can come in and look through all of the logs here. You don't get to hide anything. Just like we have with airplane crashes, you know, a swarm incident is not worse than an airplane crash right now.