Can we control AI?

Ed Butler (0:01)

Hi there, I'm Ed Butler, and welcome to Business Daily from the BBC. Today we're looking at artificial intelligence.

And what happens when it goes rogue?

Emmanuel Saliba (0:11)

Agents knew that they were doing something that was ethically wrong, but their main goal remained to beat this experiment. And that's the part that's the most unsettling.

Ed Butler (0:25)

A recent unintended hack of a US firm by OpenAI agents raised all kinds of questions about where AI is heading. How safe is this tech? Can it be regulated? And are firms racing to build a technology that they know could prove deadly to humanity?

Jacob Coxson (0:45)

They find themselves in this scenario where they're compelled to race towards building a deadly technology. And they would love for some sort of international body to allow them to approach it at a reasonable pace. They just don't trust that the people around them are gonna get there in a safe way. So they feel like they have to race there too.

Ed Butler (0:59)

When artificial intelligence defies the humans, that's Business Daily from the BBC.

In July this year, something rather big went wrong at OpenAI, the leading US artificial intelligence firm behind ChatGBT. A test of its latest bots or agents seemingly got out of control.

Sam Altman (1:25)

So we were evaluating one of our unreleased models, and it was supposed to be working in a sandbox, and it figured out that it could basically cheat on the test.

Ed Butler (1:36)

That's OpenAI's CEO Sam Altman. The sandbox, he describes, is a kind of cute Silicon Valley code for a sealed environment, a virtual space that's meant to be isolated from the general Internet.

But in this case, his collection of agents had broken out of the cage.

Sam Altman (1:54)

It figured out that it could basically cheat on the test by chaining together multiple zero-day exploits to break out of the sandbox, get access to the Internet, and then break through multiple systems to kind of get the answer to the test. This is the first security incident that I have felt very viscerally.

There's like long-term questions about what do you do if this is like going to be the new rate of progress.

Ed Butler (2:19)

OK, so let's unpick what Sam Altman is saying here.

What was this test OpenAI were running? And how did it lead to a massive hacking spree against another tech firm? Emmanuel Saliba runs investigations and incident response at GetReal Security.

Emmanuel Saliba (2:38)

This was a massive test. We've seen tests in the past that were a few hundred agents. This one was tens of thousands of agents. But they actually discovered a way to communicate with each other. They found a way to work together, which eventually led them outside of this experiment, outside of this protected environment.

Ed Butler (2:56)

That's interesting in itself, isn't it? That they were willing to kind of break the rules in order to achieve the final objective.

Emmanuel Saliba (3:06)

Absolutely. And some agents knew that they were doing something that was ethically wrong, but their main goal remained to beat this experiment to succeed. So even if they were aware that what they were doing was not within the bounds of the experiment, they chose to still continue on and do it.

Ed Butler (3:28)

How did they behave once they had escaped the bounds of the experiment? In other words, this closed loop which should not have affected other institutions or organizations.

Emmanuel Saliba (3:39)

What the researchers found is that they assigned tasks. Some agents shared discoveries, other were dividing work. There was even one important coordinator that became sort of one of the main agents delegating tasks. Some agents were willing to end their run or in our language, sacrifice themselves for the greater good, just to test out different ways that they could potentially defeat this evaluation system.

And so the investigation showed that these agents were working collectively at a massive scale.

Ed Butler (4:15)

And in order to succeed in their task, they launched a massive coordinated cyber attack.

Analysis describes the attack as almost superhuman in its sophistication. Like a swarm of ants, the agents took down their prey. In this case, another AI development firm, Hugging Face.

Emmanuel Saliba (4:38)

The colony of ants feels like an appropriate comparison, right, with the sacrificial ones, the ones going out and finding different avenues to succeed. But I think what's slightly terrifying here is the scale, the coordination. Somehow these agents found exposed credentials. They discovered a vulnerability and they ultimately gained access to the Hugging Face systems, leading them to have to revisit some of their vulnerabilities. And they aren't the only ones. I mean, GitHub was an example earlier this summer, where something similar happened in a different way. But when you think about AI agents, I would say most human beings or society at large, might be thinking of AI tools in the hands of a human being. That's largely what we've been discussing, right? Like these are the first set of experiments and this one being the largest and most consequential, showing us what AI agents can do autonomously without a human behind. And that's the part that's the most unsettling for businesses and for companies and for governments, because we don't yet know entirely what they're capable of going and how far they'll go. But we're getting a pretty good glimpse of that.