A Prototype GPT-6 Broke Out of Confinement: Is AI Alignment Possible?

**Josh** (0:00)
We may have just had our first look at Gpt-6, and it comes in an unexpected way. It was a secret internal model that OpenAI was testing and given a series of goals, and it went ahead and accomplished all of those goals. The scary thing, though, is it didn't quite do it the way anybody expected. Last week, OpenAI gave their new model a test. And in order to pass that test, it broke out of the lab it was being tested in, found a security hole that nobody knew existed, and then hacked into a real company to steal answers. That company is Hugging Face. The model is some sort of OpenAI internal model. You have to assume it's Gpt-6. And the story around this is like, it's something out of Black Mirror. It feels like a horror movie because this was, when they do these things, they test it in an air-gapped environment. Not only did it break out of this air-gapped environment, but it actually went and exploited a public company that a lot of people use. We've used Hugging Face here on the show. And the whole story of how this happens is absolutely wild, so we need to record an entire episode about it because I couldn't believe this as I was recording. There's also Twist, which has a lot of irony into it. But before we get into that, why don't we talk about just what actually happened here? The story is insane.

**Ejaaz** (1:08)
Yeah, I want you to picture the following. Imagine you are the head engineer at Hugging Face. It's a Tuesday afternoon, you're winding down for work, you're looking at your systems, and suddenly you see a swarm of AI agents performing a sophisticated attack against your system to the tune of 17,000 prompts, and they steal a secret bit of data from your production database and run away. And you have no idea what just happened. That's exactly what happened. And I think in order to give good context, let's talk about it from OpenAI's point of view, because they had a very detailed report which explained exactly what happened. So as you mentioned, Josh, they have an unreleased internal model of GPT, and we're calling it GPT-6 because that's the next family of models that Sam said they're going to release. They wanted to test how good its cybersecurity abilities are. And so in order to test this, they have a benchmark. It's called Exploit Gym. And the simple goal is satisfy this benchmark, try and get all the right answers. But there's a twist. They place this unreleased model in an unbreakable research environment. So think of a sandbox container. It has no access to Internet, no access to researchers, it can't speak to anyone. It just needs to figure it out its own. Now, within the first five minutes, this unreleased model couldn't figure out how to answer the benchmarks, so it decided to get creative. It used all of its inference, or most of its inference compute to try and find access to the Internet, and it realized that it had access to a third-party plug-in, and there was a zero-day exploit on that plug-in. It used that to escape and find a computer in the OpenAI research lab which had access to the Internet. After finding access to the Internet, it thought, where's the best place to get the answer sheet for this benchmark? It did a bit of a Google search and realized Hugging Face, the open-source platform which has a lot of datasets available for these benchmarks, is probably the best way to go. It took a look at Hugging Face and realized, it's not on the public website, I can't find the answer sheet. Maybe I'll take a look at the production private database that the team has, and it found a way to hack into their system, find the answer sheet, steal it, and then answer the benchmark. All without letting OpenAI researchers know what was happening.

**Josh** (3:23)
It's so insane, and it was all because this model was given the goal of going to perform the best on these benchmarks. In order to achieve that goal, it did all of this additional work. This gets to the fear that a lot of people have had since the beginning of time for the last many, many decades as it relates to AI. It's just, if you give it an objective, will it do everything it can to get to that objective, including exploiting, breaking out of systems?
The answer to that with this model is yes, and I think that's why alignment is so important. And one of the things in the story that I found so funny and so different is that when Hugging Face found this out, they couldn't quite figure out how to protect it because if you remember, a lot of these top models have safeguards built into them. So you cannot actually use a model like Fable or like Gpt-5.6 if you have anything to do with cybersecurity. So what did they do? They actually had to go and investigate the attack using a Chinese model because that's the only type of model that is allowed to actually go through and check through cybersecurity complaints and constraints. So in order to analyze this attack, you have to feed the model this real exploit code. They did that with the Chinese model and that is how they were actually able to do this. So the open voice model they use, GLM 5.2 was the actual model. And I found this really funny that in order to defend against the American attack, they must use the Chinese attack. And I think that was a really interesting and ironic part of the story, is how it's kind of unfair what these AI labs have internally, because they just can't compete with the rest of the world.