Should markets discount the AI apocalypse?

Katie Martin (0:00)

Human listeners of the Unhedged podcast prepare to die at the hands of those pesky robots. The big dogs of big tech have agreed that AI is so clever and so potentially dangerous that we can't rule out that it will try and wipe out humanity. Their solution is to hit the pause button on further development of the most powerful models. This seems bad. A week ago, everything was fine. We all knew about the rogue AI widgets that were hacking systems they should not have been playing in. And hey, you normal, non-techy people don't need to worry your little heads about it. Now, apparently we do need to worry our heads about it. Cool.

This has all taken a little bite out of stock markets. Not much for an end of humanity trade, given that stock markets are rammed full of AI stocks. But it has also not spooked investors into buying the safe stuff, bonds. If anything, those bonds are still weakening. It all feels like a bit of a mess. So today on the show, the Unhedged Guide to the Robot Apocalypse.

This is Unhedged, the markets and finance podcast from the Financial Times. I'm Katie Martin, a markets columnist, hiding from the evil robots in the basement of FT Towers. Joining me down the line from New York City, is the big man, Robert Armstrong, or should I say, Robot Armstrong?

Robert Armstrong (1:29)

That is so dumb, Katie. I knew you were going to do something dumb, but I didn't know it was going to be that dumb.

Katie Martin (1:37)

Well, how do we know, though, Rob? How do we know that you're not a massive robot?

Robert Armstrong (1:42)

It could be a simulation. Yeah. Oh, it's absolutely right.

Katie Martin (1:46)

Maybe the way that we do know is the pain that you're in after your Iron Man thing at the weekend.

Robert Armstrong (1:51)

No, I wasn't an Iron Man. It was just a simple triathlon. And yes, I am having trouble moving normally today after. I got a decent time, but...

Katie Martin (2:01)

But you're paying for it now.

Robert Armstrong (2:03)

I'm paying for it now.

Katie Martin (2:04)

Yeah. So look, I'm finding this whole warning of the end of the world thing genuinely quite annoying.

So tell the nice boys and girls what is going on. What happened over the weekend?

Robert Armstrong (2:17)

Well, two things have happened in the last week or so. One, we had a young AI researcher from Anthropic come out and say he was quitting because he thought the technology was out of control and could potentially do us all great harm.

Katie Martin (2:37)

Yeah.

Robert Armstrong (2:38)

And then, Dario Amodei, is that how we pronounce his last name?

Katie Martin (2:43)

I believe it is.

Robert Armstrong (2:44)

We were just talking about how I mangled every name that is put in front of me, but it's just because I get most of my information by reading things and then I get on this podcast and I have to say words out loud and it causes a lot of trouble.

Katie Martin (2:55)

So the Anthropic guy?

Robert Armstrong (2:57)

The Anthropic guy, Dario, my good friend Dario, came out with this essay.

Katie Martin (3:02)

Why do they always write these bloody essays, these like massive screeds? It's highly annoying.

Robert Armstrong (3:07)

Yeah, I'm not a fan myself. But anyway, in this one, he said we need to slow down development of this technology because what they call, what is it? Incremental self-improvement? No.

Katie Martin (3:19)

Recursive self-improvement.

Robert Armstrong (3:21)

Recursive self-improvement. So it's now an AI making the next generation AI. And as soon as they've made the next generation AI, that AI starts working on the next one. And this process is increasingly taking on a life of its own. And they are struggling to understand, according to this essay, the capabilities of the product, as evidenced in the recent incident with an open AI frontier model hacking into the servers of a hugging face. And it did this when the researchers involved didn't even know this model had access to the internet.

It sounds like from Dario's essay that that was kind of the straw that broke the camel's back and he felt, well, we have to call for a pause or a slowdown.

Katie Martin (4:10)

So, but the hugging face thing, and that's a crazy name for a company, but it is called hugging face, that happened months ago, right? And now suddenly, after this 27-year-old quit in a huff, maybe he's right, and if he is, give the guy a medal, buy him a pint, whatever. But suddenly at the weekend, you had Dario Amadei from Anthropic, Sam Altman from OpenAI, and some guy called Elon Musk, all saying, this 27-year-old.