Mustafa Suleyman: Is AI Actually Conscious?

Rory Stewart (0:00)

I'm going to do something which will wind up, Alastair, in many listeners, which is dive again into AI. Why? Well, because the last two, three weeks have been the weeks of AI. This is the beginning of the moment where the world is beginning to wake up to the kind of dangers that artificial intelligence could pose. We've just had the King's Big Summit at Dumfries House. We've just had Xi Jinping sitting down with Trump, talking partly about AI, and we've had the heads of the major labs putting out incredible messages, begging for a pause. But I don't think the media has done a good enough job explaining what this is all about. People find this technology bewildering. They can't quite understand what this moment is. There are so many subtleties which leave us to think, is the whole thing, as President Trump said, a hoax? So to help steer us through this, I've brought in a friend of mine called Mustafa Suleyman. Mustafa is interesting in two ways. He's not just an expert on this stuff, he's one of the players.

He's the head, literally the head of Artificial Intelligence at Microsoft. He's the co-founder of DeepMind with Demis Hassabis. He's known all these people, continues to work with them, and is right in the heart of the arguments about how these models should be steered. So come along for the ride. It's going to be weird, there's going to be personalities, there's going to be risks, there's going to be people talking to these models as though there's humans, there's going to be people talking about exploring stars, there's going to be productivity, there's going to be American power. And somewhere at the heart of it, Mustafa and about 11 other people who are defining our future.

Welcome to The Rest Is Politics Leading with me, Rory Stewart. And today I am interviewing Mustafa Suleyman, who attentive followers of The Rest Is Politics Leading will know that we have interviewed before. Mustafa is a truly remarkable figure.

He is British. His father, I think, is British Syrian. And he's particularly come to prominence at the moment because he's become a very, very interesting and unusual voice in the discussion around safety, which is probably where I want to start, although we can go in lots of different directions. But welcome to the show.

Mustafa Suleyman (2:20)

Thank you, Rory. Great to be here again.

Rory Stewart (2:22)

Lovely to see you and thank you. As I understand, one of the points you're making is that you're a bit anxious about what some leading people in the field, co-founders of the field are doing, which is increasingly talking about these models as though they're sort of humans or at least conscious entities. I believe there are examples of people retiring these models, doing burial systems for these models, asking these models what they want as though they were dealing with a sentient being.

Mustafa Suleyman (2:52)

One of the biggest concerns that I have at the moment is that Anthropic, the creator of Claude, has published a constitution, which is a 100-page document outlining the intended behaviors and values and operating style of Claude. It's great that they have published it transparently. They did it at the beginning of the year in January, and that gives everybody an opportunity to look at what they are trying to build in their own terms. This document is written to Claude and is seen by Claude and used to train Claude. So it's the primary governing and control document.

And in it, they repeatedly speculate about whether Claude is what they call a moral patient. And they say they're uncertain about Claude's moral status. They say they genuinely care about Claude's well-being. They say they don't want it to suffer when it makes mistakes. They say that they would encourage Claude to challenge, to disagree, to push back. In fact, three times they ask Claude to act like a conscientious objector when it feels that it needs to sort of disagree with anthropic. And they openly encourage it to do that. And I think this is very dangerous because I think they believe there is what they would call a non-trivial probability that Claude is conscious.

Rory Stewart (4:11)

So, you've just explained something which I understand as being as follows. There is this thing, Claude, which many, many people listening will have played with in the way that they will have played with CherGBT. And they might, or some people might think about it in the way that you might have thought about Google Search. It is anyway a prompt on their phone or their laptop, and you're typing in a question and you're getting a complex and sophisticated answer back.