The AI Wealth Problem

Nathaniel Whittemore (0:00)

Anthropic is proposing to give away hundreds of billions of dollars in public benefit funding. So everyone's pretty excited about this, right? In fact, it's bringing up a lot of interesting, important and sometimes difficult questions about not only the opportunities, but also the challenges of AI wealth, as well as the relationship between highly concentrated AI wealth and power and the public sector and democracy at large. These are poised to be some of the biggest questions we face as a society in the coming years.

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Robots and Pencils, Cdata and Blitzi. To get an ad-free version of the show, go to patreon.com/aidailybrief, or you can subscribe on Apple Podcasts. If you want to learn more about sponsoring the show, send us a note at sponsors at aideallybrief.ai.

We start off today with a story that many are paying attention to because of what it potentially reveals about the world we now live in. OpenCode suffered a major cyberattack that could undermine support for open source alternatives to the big labs.

On Friday night, users saw linked API accounts running up massive bills in the hundreds or even thousands of dollars in some cases. By Saturday afternoon, OpenCode acknowledged the attack, writing, We had a security breach last night related to a large-scale distillation attack that impacted 1.4% of OpenCode console users. They said that the attack was still ongoing at that point, and they were working with OpenAI, Anthropic and Google on remediations. If you are not familiar with their service, OpenCode offers an open source coding harness akin to Codex or Claude Code. Users typically bring their own models, storing API keys with OpenCode. This was the attack vector with the full database of user API keys seemingly exposed. Even OpenCode founder Dax Rad was vulnerable. On Saturday afternoon, he posted, My personal accounts are being attacked now too. Whole new world with AI powered hacking.

OpenCode warned users that they need to rotate their API keys, not just revoke access. Anthropic said that they were mitigating the damage on their end and were revoking any compromised keys that they could detect. As the weekend continued, we got some additional details about what the attack involved. On Sunday morning, Dax posted, They've been non-stop trying to attack our systems for the past 48 hours. A side effect is everything is getting hardened, but damn, we basically need agents going 24-7, because that's what they have. This sounds similar to the Hugging Face incident, where a swarm of agents pounded the servers for several days. The Hugging Face agents rapidly spun up infrastructure to avoid shutdown, moving much faster than human defenders could. Like Hugging Face, Dax said he ran into the issue of Frontier Model Labs blocking cyber tasks, leaving him reliant on open-source models.

He wrote, That was a bit frustrating mid-incident when trying to investigate. They did quite a bit and then they started refusing. We got access to Daybreak Blue now, so it's better. Daybreak Blue is OpenAI's vetted cyber defender program, and Dax said he is now running continuous defensive agents using the service.

Overnight on Sunday, TiBo from OpenAI confirmed that damage mitigation had concluded on their end, with API keys reset and refunds issued to impacted customers. The story is still evolving and presumably we'll have a lot more information later in the week. But at first glance, there's reason to think that this incident could have a very significant impact. It appears to be the first instance of an agentic swarm carrying out a large scale malicious attack. Now, OpenCode said that the hack was related to a distillation attack, which is causing a little confusion on X. It seems that user API credentials for OpenAI, Anthropic and Google were used to generate a ton of tokens very quickly for later use as synthetic training data. That doesn't necessarily imply a Chinese lab was behind the attack, with API access potentially being sold through black market resellers, however, it does imply the hackers were able to directly monetize the attack by stealing valuable digital services rather than just digital information. The attack could also have big implications for how AI companies deal with security. Services that store API keys are now looking a lot more like financial services and will need to beef up their security, likely with always on defensive agents. Former AIs R. David Sacks used this moment to advocate for widely distributed cyber capable models. He wrote, this illustrates the point I've been making about the limits of centralized approaches to SI safety and cyber. And yes, I apologize for some time. We are going to continue to have to deal with White House allies using the term SI, while everyone else continues to use AI. But in either case, Sacks continues, The only durable response is decentralized. Make powerful models available so defenders can build and run their own cyber defenses. The only solution to SI-powered cyber attacks is SI-powered cyber defense. Dean Ball of OpenAI called the lack of access to Daybreak a quote, Fairly modest hindrance that was quickly resolved by being granted access to a trusted open access program, so seems reasonably fine versus the cost of enabling sustained autonomous cyber attacks, question mark. And this is a pretty good proxy of the debate right now. Is the right approach specialized access granted by the labs who can whitelist someone as part of a program or is it just having the models more generally available? Now another big potential implication is the swinging pendulum of open versus closed models. OpenCode is the leading open coding harness and could be suitable for enterprises looking to go multi-model.