#257 - GPT 6 Astra, AI Extinction, Security Incidents

Andrey Kurenkov (0:10)

Hello, and welcome to the Last Week in AI podcast, where you can hear a chat about what's going on with AI.

As usual in this episode, we will summarize and discuss some of last weeks, and also the week before, because let's say we are behind on this one, AI news. And I am your regular host, Andrey Kurenkov. I studied AI in grad school and now work at the AI startup AstroKate.

Jeremie Harris (0:33)

And hey, everybody, I'm your other regular co-host, Jeremie Harris. I'm at Gladstone AI, AI National Security and stuff. And the reason that we didn't do an episode last week, so I apologize. It's been a bit of a mess the last few weeks. As you might imagine, all these rogue AI incidents and the upcoming Trump-She summit and all this stuff means there's a lot of interest in DC and kind of like unpacking a lot of this stuff. And I ended up doing a 24-hour turn and burn in DC last week. And then another one this week for about three days. The length of this most recent trip was unexpectedly long. And then the entire trip last week was totally unexpected. So sort of responding in real time to a bunch of things that are happening. And we're also prepping to launch something public. We've been doing a lot of work behind the scenes for the last few months. I think like, I don't know, like for at least for me, it feels like pretty ambitious work to try to understand sort of the AI endgame for US, China, and we'll be coming up with some of that stuff pretty soon as well. So we'll be sharing that over over Substack if anybody's interested. So that should be probably actually at the time that this goes live, that should be out with our first post about sort of like the diplomatic realities ahead of the Trump, Xi summit and is it plausible to get a US, China deal on AI? What would it look like? And so on and so forth. So that's kind of been part of my work. And for that reason, I usually spend about five or six hours preparing for each of these episodes. I have not been able to do that this time. So you're going to be getting my very like unfiltered, sort of unstudied perspective on a lot of these stories. Not all because a lot of them obviously were things that have been working on in the last two weeks too. But where we go off the beaten path, I'll be a little bit more cagey than usual in terms of airing my opinions because I haven't developed them in a lot of these stories yet.

Andrey Kurenkov (2:19)

So as usual, apologies for not being last week. And as much of what we'll be covering, a lot of developments from last week in this episode. Jeremie, sounds like you've been in a lot of flights and a lot of meetings. Back when we started hosting together in like March of 2020 free being an AI national security expert must have been a pretty niche role. And I would expect it's still a little not a common thing to be technical and into policy, right?

Jeremie Harris (2:49)

Yeah, I mean, I think that's actually one of the biggest problems in the space. There's kind of like the national security side of the house that doesn't speak AI policy. And then you've got like a million different silos that don't talk to each other. And just like one data point, right? There are a lot of companies that are doing right now AI security work or yeah, I'll just say broadly AI security work, trying to either prevent nation states from stealing AI models or develop AI treaty verification technologies so that we can do trust but verify.

And often, usually when you go to those companies and you're like, hey, have you like spoken to a single person in the US intelligence community to have them evaluate the technology you're spending millions or tens of millions of dollars on developing? And the answer is no.

This is horrible. It's not their fault. It's just like these things happen in silos. And even within the US intelligence community, one person who's super, super expert on this particular way of breaking into a building will talk to another person who is an expert on a different way of breaking into a building. And if you talk to one of them, you'll think, oh, yeah, I'm covered down on nation state physical security issues. But then you talk to the other one, you're like, well, now I'm starting from scratch. And so the siloing of things like this happens at all levels. Obviously, we have a technical background originally in AI and alignment research. And then in AI security, what's happened now is like, we're trying to help bridge some of those silos as fast as we can. So a weird amount of it, it feels weird because not writing nearly as much code as I'm used to. A lot of it is like relationship stuff like, you need to talk to you right now, and that sort of thing. So anyway, it's been a weird couple of weeks for that.