OpenAI Reports New AI Safety Incidents; AI CEOs Weigh In on AI Debate

SPEAKER_1 (0:02)

Bloomberg Audio Studios. Podcasts, radio, news.

Bloomberg Tech is live from the heart of Silicon Valley with Ed Ludlow in San Francisco.

Ed Ludlow (0:23)

This is Bloomberg Tech. Coming up, OpenAI reveals new cases of its AI models misbehaving and a new plan to track when things go wrong. Plus Databricks CEO, Ali Ghodsi, and machine learning pioneer, Andrew Ng, weigh in on the global AI safety debate. And while AI may pose an existential risk in Silicon Valley, the immediate fear is missing out on the money. That story later this hour.

AI warnings take the global stage as King Charles III sounds the alarm.

King Charles III (0:54)

There seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands and being used in potentially catastrophic ways. Surely then we need sufficient means of control before it is all too late.

Ed Ludlow (1:17)

That was King Charles pressing some of AI's biggest names for reassurance they can keep their AI under control. NVIDIA's Jensen Wong, OpenAI's Sarah Fryer and DeepMind's Demis Asabis were among those gathered in Scotland for the event. I want to get to Bloomberg Tech editor Shona Ghosh over in London with the latest. Let's start with why King Charles pulled this event together, Shona.

Shona Ghosh (1:39)

Yes. I mean, it's not an obvious choice, is it, to convene an AI gathering, but still a monarch still has the sway.

Ultimately, it's against the backdrop of Anthropic CEO, Dario Amadei's blog post over the weekend, saying that he thinks there should be a pause, a slowdown in frontier AI development. That's been accompanied separately by admittedly lower level, but AI researchers quitting some of the top AI companies, including Anthropic, also DeepMind, effectively saying that AI could be lethal for humans. So there's a sense that fear around AI has been a debate within Silicon Valley, but actually now we're seeing those fears spill out beyond Silicon Valley. MPs, lawmakers in the UK and globally are quite concerned. The general public are also quite concerned about what happens if AI isn't adequately controlled. And again, that's all been triggered in turn by events over the summer where we saw AI agents from OpenAI, you know, the phrase is going rogue. I wouldn't say that's quite right. You know, escaping guardrails and attacking other organizations in ways that, you know, OpenAI wasn't necessarily expected.

Ed Ludlow (2:50)

Shona, remind us which leaders of industry were in the room, so to speak, and what King Charles' message to them was, maybe even his instruction to them was.

Shona Ghosh (3:00)

So, gathered some big names. We had NVIDEO CEO Jensen Wong, the former CEO and co-founder of DeepMind, Demis Hesabis, also Sarah Friar, Chief Financial Officer of OpenAI. Anthropic did have a representative in the room, but it was their global policy chief. It wasn't any of the co-founders, it wasn't Dario, so kind of missing a big name there. Also, no one from Meta or X, interestingly. So, you know, good representation, but not everyone.

And the monarch prepared a speech that was sort of released in advance. And essentially, you know, he told these leaders something he said before, actually, which is, you know, if you're going to develop this tech, can you make sure that it's good for humanity, good for the natural world, which is, you know, environmentalism is something King Charles has particularly been interested in, has been for a long time. So he was really pushing for the technology to benefit humanity and to just pause and think about those consequences.

Ed Ludlow (3:57)

Bloomberg Show Negotiation London, thank you very much. Another big story. OpenAI is revealing previously unreported cases of its AI models going off script, fabricating data, bypassing restrictions, sharing files they were told to keep private. Now the company is setting up a new system to track and disclose those incidents. Bloomberg's Shireen Ghaffari, who leads our coverage of the Frontier Labs, is with us. Let's start with the plan that OpenAI has presented on what it's going to do if it finds models misbehaving. What have they said?

Shireen Ghaffari (4:28)

That's right. So they have said that they are now going to essentially have a triage-like system for employees to disclose incidents where they think that the AI models are essentially misbehaving, are eliciting some kind of concerning behavior that goes against what they intend the AI to do.

Ed Ludlow (4:48)

There were some interesting case studies. OpenAI also made it clear that in the 6th they disclosed, there was no hacking of third parties. But some of them are worrying, right? Take us through what OpenAI disclosed this morning.

Shireen Ghaffari (5:01)

Yeah. So even though when we asked, OpenAI said that this did not involve these incidents, the 6th that they have disclosed did not involve reaching third parties, there are still some concerning behaviors that we're seeing here. For example, AI models trying to share files that should have been private with each other, communicating with each other in ways they're not supposed to, telling AI models essentially, learning how to disregard or instructing others to disregard their instructions on how to behave. So these are all incidents where you can see the potential for that to evolve into something even more nefarious, especially if those AIs were successful and going against what the creators of the AI expect these AIs to do.