Nate Soares on How AI Really Could Kill Us All

Joe Weisenthal (0:02)

Bloomberg Audio Studios, Podcasts, Radio, News. Hello, and welcome to another episode of the Odd Lots Podcast. I'm Joe Weisenthal.

Tracy Alloway (0:23)

And I'm Tracy Alloway.

Joe Weisenthal (0:25)

Tracy, I'm very tired today because I flew in on the red eye last night from San Francisco. I was out at this very interesting conference called Curve in Berkeley.

A lot of like, you know, I guess a lot of people very concerned about AI, so to speak. Interesting mix of folks. But yes, people are concerned about what the future is going to look like as we develop stronger and stronger AI models.

Tracy Alloway (0:53)

Just from a sociological or anthropological standpoint, I think living in San Francisco and mingling with, you know, AI bros at the moment must be one of the most interesting things because there must be so much cognitive dissonance in their heads, right? Where on the one hand, they'll say like, oh, they're building this new ultra-powerful model. And on the other hand, sort of almost as an afterthought, they'll be like, oh yeah. And I'm a little bit worried about the possibility that kills us all. How do you square that?

Joe Weisenthal (1:22)

It's super interesting and you hear it like, so at this conference, people are working for various safety type orgs, people who want to regulate AI in some way. And of course, people from the labs themselves, some may be working on policy stuff, but some actually like the actual builders of the technology. And they're all deeply anxious about the very thing that they're building. But they just keep building it. And so they have all these worries about, at a minimum, how disruptive it could be or how dangerous it could be. And of course, we all now know that there are dangers, particularly in the wake of the hugging face attack, so people can see that it's very possible for models to go rogue and commit crimes and stuff like that.

And yet, they keep building it.

Tracy Alloway (2:18)

And yet.

Joe Weisenthal (2:19)

And they would say, well, if someone's going to build it, so why not me? Because I can do it well, maybe is how they think about it, or there's some game theory aspect where it doesn't make sense for me to slow down or pause what I'm doing because someone, either my direct competitor in town or maybe someone in China will build it. So not building it isn't going to do anything. But yeah, it's very strange, it's very odd.

Tracy Alloway (2:47)

We've used the term runaway train before to describe how all of this feels, which it almost feels at this point like there's no stopping it, right? Because if one lab doesn't do it, then another labs can do it and that's how they justify it. If one country doesn't develop the next big model, then they're worried about another country building the next big model. So you have this, I guess, collective action problem for humanity, and a lot of it feels inevitable. And I have to say, today's guest predicted this inevitability problem in a very salient way. And one thing I'll just say is, I read the book over the weekend. For those who don't know, the book talks about AI killing us all, and it talks about one hypothetical scenario in which an agent could go rogue and kill us all. I read it over the weekend, woke up on Sunday to the headlines, the headlines about an escaped plague from a Russian lab.

And I thought, wow, I couldn't have chose better time to read this book. Now, to be fair, the escaped plague was in Siberia. And so if there is a rogue...

Joe Weisenthal (3:50)

Is it been proven yet, by the way?

Tracy Alloway (3:52)

Well, supposedly, according to reports. So if an escaped AI agent was trying to kill us all or kill a select a few individuals through a lab...

Joe Weisenthal (4:02)

Oh, I see, through a biotech, yeah.

Tracy Alloway (4:04)

It might not be in Siberia. That's the one thing. That's the one thing that made me feel a little better this weekend. But the fact that we're thinking about these scenarios at all, the fact that when I see a headline about a lab leak, I think, oh, God, I hope it's not a rogue AI agent because I just read this book.

It's a weird time.

Joe Weisenthal (4:24)

It is a really weird time.

So obviously, something I've been thinking about is like, the pace of AI development itself is really extraordinary, and then the pace of what people are worried about. It's like the conversation is changing constantly. So for example, six months ago, if you're like, what are people concerned about? Maybe a year ago, it's like, what are people concerned about with AI? There's like a job loss or something like that. And then-