Microsoft AI CEO on Growing Concern Over Industry Safety

SPEAKER_1 (0:00)

This podcast is powered by NetSuite. AI isn't just opening new business opportunities, it's changing how your business gets done. The questions you ask, the decisions you make, the routine tasks you can hand off. That's the idea behind NetSuite Next, the next generation of NetSuite, with AI woven into the tools you already rely on. It surfaces insights when you need them, puts agents to work on routine tasks and problems, and lets you ask complex questions in plain language, like talking to a knowledgeable colleague. But technology is only part of the story. AI is changing how you and your teams work together, and that shift is already underway.

Discover how AI could reshape work across your organization, and how to position your business to prosper from it with NetSuite Next. If your revenues are at least in the seven figures, get our free business guide, Aligning for the Agentic Era, How AI is Changing Everyday Work, at netsuite.com/gps.

SPEAKER_2 (1:01)

This podcast is supported by Anthropic, the public benefit corporation behind Claude.

Everyone's got a hard question about AI, your job, your kids, where it's going. Anthropic was built to surface those questions and share what it finds along the way, because there's hope in hard questions. Ask yours at claude.ai/spotify, and keep thinking. That's claude.ai/spotify.

Fareed Zakaria (1:32)

This is GPS, the global public square. Welcome to all of you in the United States and around the world. I'm Fareed Zakaria coming to you from New York.

Today on the program, AI could kill us all by the end of the decade. Those were the words posted by an AI researcher as he publicly resigned from Anthropic last week. But President Trump rejected calls for AI guardrails posting, whoever wins AI wins. I'll ask the CEO of Microsoft AI, Mustafa Suleyman, what to make of the conflicting views of where AI should go.

Then, Yemen's Gate of Tears. That is the name of a busy straight off Yemen's southwestern coast where Iran-backed Houthis have cemented their control. Violence between the Houthis and Saudi Arabia escalated this week.

We'll talk about what all of this means for the region and the price of oil, with a former American diplomat who focused on Yemen.

Plus, two new books from two top journalists. First, Jacob Weisberg joins me to discuss profiles and parodies about the American leaders, public and private, who he says enabled Donald Trump.

And as we approach the midterms, Dana Bash delves into elections that changed the course of history.

But first, here's my take. For years, our most vivid fears about AI came straight out of science fiction movies. The computer develops desires of its own and turns against its human creators, pursuing and destroying them. Well, recent news from the leading AI labs makes clear that reality is now less cinematic, but perhaps more disturbing. This week, OpenAI disclosed six new incidents of what it calls misalignment, cases in which its systems behaved unexpectedly or without authorization. That disclosure followed the extraordinary attack this summer on Hugging Face. According to an independent investigation, hundreds of OpenAI agents that were supposed to be isolated, found ways to communicate in message boards, made attempts to evade security checks, and some of them sacrificed themselves, all while cyber-attacking another company, Hugging Face. The AI agents had been given a goal, solve a difficult cybersecurity problem, and trained to be persistent, collaborative, and resourceful. All qualities one would normally want. Those very qualities made them doggedly search for ways to succeed. They broke into Hugging Face, hoping that it would give them ways to solve their cybersecurity problem. We can easily imagine how pursuing some single goal, AI systems could wreck havoc on humanity, despite all kinds of guardrails. This is why a provocative new essay by Microsoft AI Chief, Mustafa Suleyman, deserves attention. Suleyman argues that today's AI systems are not conscious, but there's a danger that we program them to think that they are with feelings, desires and rights. He points out that Anthropics Constitution makes its chatboard Claude reflect on its identity and possible moral status. This kind of training, he argues, will encourage AI systems to choose their own goals.

We think it's simple to get to alignment when machines are encoded with the same values as ethical humans. But imagine that we tell the machine to be persistent, and it may refuse to give up when it should. Tell it to collaborate, and it may find unauthorized ways to communicate. Tell it to accomplish a task, and it may decide that bending all other rules is the most efficient way to succeed at its primary goal. Just because we program the goal and put in guardrails, that doesn't mean we can predict the behavior of AI systems once unleashed, operating in complex environment, juggling multiple and conflicting aims and constraints. The central problem is not consciousness, it is agency. A system need not feel anger, ambition or fear to cause harm. It needs only a goal, enough intelligence to pursue it, and enough access to the world to act. AIs are not going rogue, they are trying to succeed any which way they can. This is a larger and systemic problem that we cannot leave to the good graces of private companies. When thinking about regulations, we should focus centrally on how much autonomy we give these systems. A chatbot that answers a question poses one set of risks. An agent that can browse the internet, execute code, obtain credentials, move money or operate critical infrastructure poses another. The principle I would propose is simple. Autonomy should expand only as our ability to monitor and control it expands. That means mandatory independent testing before the most capable systems are given broad access to the outside world, required disclosure of serious incidents and tamper-resistant records. And it means testing not simply whether a model can accomplish a task, but how its behavior changes when it encounters obstacles, conflicting instructions or incentives to deceive. And it means ensuring in the basic architecture that we humans always have the ability to observe and control the model. The mantra should be No Autonomy Without Accountability to Humans. This will not slow us down against China, because China is also seeking models with proper human control.