No, AI Is Not "Autonomously Hacking" with Cal Newport

**Ed Zitron** (0:01)
CAUSOR MEDIA Hello, and welcome to Better Offline. I'm your host, Ed Zitron. And today, I am joined by returning champion and Georgetown computer science professor, Cal Newport. Cal, how are you doing?

**Cal Newport** (0:27)
Cal Newport Ed, I'm always doing fine, but I'm counting down the minutes until the AI takes control and we're all screwed.

**Ed Zitron** (0:34)
Cal Newport Oh, I hope it just drops a big anvil on my head. But I do actually have a thing to talk to you about today. Cal, should we actually be scared of the fact that these things are hacking? Like, what is going on with this whole hacking situation with these goddamn models? Because there's so much out there where people are trying to just scare you and say like, oh yeah, well, these things are autonomous and they're doing hacking. But from what I understand, it's software doing what software does.

**Cal Newport** (1:06)
I have a lot of thoughts on this. I've been writing about this because it's annoying to me. I think there's interesting AI stories happening this summer. A lot of them you're involved with reporting on about the financial circumstances of these companies, what's going on with their circular financing, troubles ahead with their IPOs, the nuances of ARR, and instead, what is all the news coverage? The AI is going rogue. And I can't say they hate that news coverage right now because it makes their products sound awesome. And it's a story that is unrelated to stuff that they don't want talked about. So I actually literally had to reluctantly return. I talked about it back in July briefly, and people bothered me about it so much that I was like, okay, fine, I have to, you know, I recorded a whole new podcast that's coming out soon after this on my feed. I had a newsletter out a couple days before this came out. And so I've had to think about this more than I wanted to, so I'm a little bit cranky about it. But I mean, I'll give you the 10... Here's a 10-second summary that we can go deeper.
Most superhuman AIs, so there's lots of AI systems that can do something at a superhuman capability, meaning like better than any human. There's a lot of these types of systems out there now.
Almost all of them, we have no problems with them being unpredictable or going rogue or getting out of control. There's exactly one type of system that we're having this problem with, which are these long-horizon, LLM-powered hacking agents. And it's because they're a stupid way to build a system. We should not say they're synonymous with AI, or this is some inevitable consequence of AI getting better. Because again, most superhuman capability AIs are no issue with going rogue. It's this particular architecture, and the discussion should be about why are they running and building systems this way when we know they're going to be unpredictable and erratic. That's my 10-second summary of where we are.

**Ed Zitron** (2:58)
Right. So from what I understand from looking at the very, it's the hugging face attack with open AI, Meta immediately piped up and they were like, we have one, and also Anthropic was like, my child is also evil.
So, but what it seems to be is they were just told to hack into, or like complete an operation. And then they chose to take whatever means necessary to do it. And they'd set up the sandboxing correctly, right? So they'd set up the server it's on.

**Cal Newport** (3:26)
Yeah. I mean, it's almost dumber than that. So the way these particular, great. I mean, the way these type of systems work is it's just a loop. Right. And so you have a program that's written by, you know, human, just a normal program you write in Python or something. And what it does is it creates a prompt, it writes a prompt.
So it'll be like, okay, I've been given this hacking challenge. Here is like all the parameters about the challenge. Here are all the tools you have at your disposal.
What is the first step? Like, what's our plan? Or what should the first step? I'm aligning a lot of stuff, but basically like what should the first step be in trying to pass this challenge? Puts that in a big text file, submits it to an LLM over an API. The LLM comes back and says, oh, okay, you want to hack and whatever? Sure, I think the first thing you should do is X. And then the program does whatever it says or tries to, because it gives it a list of like, here's the utilities and stuff I have access to. And then it reports back, okay, I did that. And when I say reports back, it's just adding this to the prompt. LLMs have no memory, LLMs have no world model, it's just they're static. So it expands the prompt to now say, this is what happened after I did this first step that you suggested. Okay, what should my next step be? And it just does that in a loop.