Is AI going to kill us? We talk to an AI safety researcher

Natasha Ann Zachariah (0:07)

Yesterday, I asked Claude if it would kill me eventually. My AI assistant said no, allegedly. But we workshopped this opening together, and Claude was quick to point out, it's just a language model answering questions. Still, I'm not fully reassured given what's been reported about the AI industry recently. Now, a few weeks ago, an anthropic researcher publicly resigned from the company with a doomsday warning that AI will, quote unquote, kill us by the end of the decade. Now, others in the industry agreed, including a top safety researcher at Anthropic, who put the odds at greater than 10 percent within the next decade.

It's not the first time we've heard warnings like this. So what's rattling people now?

Welcome to The Usual Place, the podcast where we have real conversations about the headlines. I'm Natasha, and to help me figure out how worried we should be, I'm speaking one-on-one with Clement Neo. So right now, you're a researcher at SASH, a non-profit organization that started in Singapore and looks at policy-relevant governance work. You're also the founder of your own outfit, Neo Research, an independent AI safety evaluation lab. Before all of this, you were a research engineer with the Singapore AI Safety Institute under NTU Digital Trust Centre, which is an organization that looks at AI models and figures out how to reduce the risks they pose. So basically, you've worked a lot in the AI space. And Clement, I want to know, how did you get into it and what areas do you focus on now?

Clement Neo (1:31)

Right.

I've always been looking at AI for the longest time. And AI, to me, even when I was much younger, was quite an exciting thing, right? When we talk about science fiction, you're like, wow, this kind of AI robots or this kind of thing could do a lot of things for us. And it was roughly about four years ago when I was on my exchange in London, when I ran into this group of people who were talking about catastrophic and existential risk posed by AI. And this was a few months before TriGPD came out. And this bunch of people already knew that this language models are going to be big, and this language models were going to, as they got smarter, start posing a lot of these kind of crazy types of risk and misbehaviours we have seen so far. And since four years ago, I've been doing quite a lot of research into figuring out how these AI models work, what makes them go crazy, what makes them go unhinged, and really trying to figure out, you know, how do we align these AIs and really make AI go well, you know, for the benefit of all of us.

Natasha Ann Zachariah (2:33)

Amir, did it worry you back then about hearing the new models coming out? Or well, when you, because as you said before, TragGPD came out, right? So were you thinking about the risk that AI could pose then already?

Clement Neo (2:45)

Yeah, I think the arguments back then really were just that like, it is quite a universal argument, right? As AIs get smarter and they get more intelligent, they're really going to be able to like, use a lot of tools, do a lot of actions, and that could be to the detriment of like, a lot of people, right? So one thought experiment that goes all the way back as early as 2003 is this thought experiment of this paper clip maximizer, where you have this really intelligent AI and you give it a pretty benign task to be like, hey, help me make as many paper clips as possible, right? And when the AI says, okay, I'm going to do that, and the AI will always run into a problem that's sort of like a genie, where you can never really specify what you want.

So the AI model just goes, okay, I'm going to make as many paper clips as possible, and sort of just like consumes the entire Earth to make it into materials or paper clips.

Natasha Ann Zachariah (3:39)

Because at some point, it stops finding the actual material for the paper clip, and then it just looks at other options, right? Yeah. Which is kind of what happened recently. I mean, one of the reasons why we're having this conversation is because I guess the trumpets are getting louder, or the saxophone, rather, is getting louder. So over the last couple of weeks, you know, employees at Frontier Labs have been warning us that advanced AI could pose an existential risk to humans. And with that former Anthropic researcher I mentioned, so Jacob Coxon wrote on X that the people building AI earnestly believe that it could kill us by the end of the decade. And overnight, we also had Dario Omode and Sam Altman briefing the UN Security Council on AI safety and calling for more government regulation. So they are with Anthropic and Sam Altman's OpenAI, right? So why are employees of Frontier AI companies sounding the alarm now?