How to Parse Escalating Warnings About AI

SPEAKER_2 (0:03)

Support for KQED Podcasts comes from Stanford Live. Featuring violin duo, Simone Porter and Blake Pouliot's California inspired program. Plus, Rob Capolo with What Makes It Great. Tickets on sale at live.stanford.edu.

SPEAKER_3 (0:23)

A dark night, a quiet house, but inside danger was lurking.

SPEAKER_4 (0:27)

Not really.

SPEAKER_2 (0:28)

We have Xfinity Shield, so we're not worried. What if protecting your home and devices could be drama free? Xfinity.

SPEAKER_3 (0:35)

Imagine that. Restrictions apply, not available in all areas.

Mina Kim (0:39)

Welcome to Forum. I'm Mina Kim. With anxieties about AI, at a fever pitch after the CEOs of Anthropic and OpenAI over the weekend called for a slowdown of AI development, both progressive Senator Bernie Sanders and far-right pundit Steve Bannon yesterday attended the same pro-human rally, reflecting that concerns about the tech are across the political spectrum.

But what set off this latest eruption of AI fears was when 27-year-old AI researcher Jacob Coxen resigned from Anthropic and posted a viral tweet claiming AI companies are gambling with our lives. Here's what Coxen told CNN last week.

Jacob Coxen (1:17)

Right now, there's no risk of extinction. The current models, the worst they can do is maybe hack into something potentially cause a lot of damages in infrastructure, but they're not intelligent enough to outsmart us at the level that would lead to extinction. I think what's just crazy is to look at the rate of progress, and there is a very real possibility that in the immediate future, these years, like next year, the year after, recursive self-improvement will happen and will enter the phase of Evan's post where he argues that there's a chance we could all die. That is coming soon.

Mina Kim (1:52)

Joining me first is Maxwell Zeff, who spoke with Jacob Coxen for Wired. Maxwell, thank you for joining us.

Maxwell Zeff (1:59)

Yeah. Thanks for having me.

Mina Kim (2:01)

So tell me, what does Coxen foresee? How could we all die? How could AI kill us all?

Maxwell Zeff (2:07)

Yeah. So I think it's kind of important to lay out just first of all that these are predictions from an AI researcher who was very close to the technology at Anthropic and OpenAI, but they are predictions.

In my interview with Coxen, he told me that he's worried, like a lot of AI researchers, that these models could get much better and much more ingrained in society, kind of enter the physical world through robots and get involved with data centers and more embedded into kind of critical infrastructure. And I think the problem he's pointing to is the fact that recent incidents like OpenAI's hack of Hugging Face show that we don't really know how to control AI systems today. And as we embed AI models further into society and they get more and more capable, we just don't really know how to make sure they're going to do what we want them to do. And I think that's really his main concern.

Mina Kim (3:08)

Yeah. And things that he's concerned they could do, it sounded like, were things like, you know, wage biological warfare, maybe release some kind of terrible virus, as you say, hacking essential infrastructure, the other being use of autonomous weapons, taking them over?

Maxwell Zeff (3:27)

Yeah. Those are the kind of scenarios that safety advocates like Cox and, you know, Point 2 as, you know, the ways that AI could pose a threat to humanity. I mean, to be clear, like, these are projections and I should also note, these are not uncommon ones from people in the AI industry. There's safety advocates at OpenAI and, you know, Anthropic and safety researchers who say that, you know, they've been saying this kind of thing for years, that AI systems could pose a threat to humanity if we don't rein them in.

Mina Kim (4:01)

Right. But they didn't spark the kind of response from the CEOs of the company saying, yes, we do in fact need to slow down. So talk to me a little bit about why his raising this fear now seems to be breaking through. You mentioned Hugging Face and other, I think, safety incidents occurred recently. Can you just talk about what's been happening?

Maxwell Zeff (4:21)

Sure. So yeah, like you just said, I mean, Silicon Valley has been warning about the risks of AI for decades now. So I think to people in the Valley, this kind of national wake up to these risks was a bit confusing in the last week. Why now? But it also makes sense because, like I said, yeah. So there was an incident earlier this summer in which OpenAI was training some AI systems internally, and they ended up without the company's knowledge, hacking into another company's production infrastructure. This company was called Hugging Face. And this has been widely cited as the most serious AI security incident ever.