Elon Musk, Sam Altman, And Dario Amodei All Agree… | The Brainstorm 149

Sam (0:00)

Everyone right now is saying you should be afraid of AI. It must be slowed down. Should we be afraid of AI?

Brett (0:08)

No, but it's an incredibly powerful tool. Practically, net jobs in AI-impacted industries since ChatTPT are up relative to what you would otherwise expect by a million.

Sam (0:20)

Welcome to The Brainstorm. Last week, we touched on AI-doomerism, but then Dario comes out, telling everyone to pace the frontier. Meanwhile, you have Trump coming up, telling, there's a meme going around, telling the nerds to keep building.

Brett, what is the reaction? And then as promised, we'll get to Muse later on.

Brett (0:42)

Yes. So Dario Amodei, the CEO of Anthropic, which is lining up for an IPO later this year, posted an open letter that both Elon Musk and Sam Altman seemed to at least somewhat endorse or say they were aligned with, stating that AI companies at the frontier should pace themselves, meaning make sure they're devoting enough resources to safety and alignment as they are to delivering these massive advances.

My interpretation of this, this is what I think is going on, where the frontier labs, the closed labs have hit this inflection point of being able to improve their models very quickly, and it happened sometime earlier this year, and you can see it in the performance improvement that they're shipping to end customers, and that we're getting 99.99% cost declines, and that's causing an explosion in capability, an explosion in monetization, an explosion in revenue, and the explosion in capability comes with some risks. Now, people out there would say it's risk of human eradication, it's risk of like extinction, that's dumb, that's not the risk. The risk is that these systems will kind of mangle the technology infrastructure that we have, that's insufficiently security hardened, that's out there in the world. And so Dario and probably Sam Altman and probably Elon Musk are all recognizing that these systems are so capable that if we put them in the hands of customers, customers may unwittingly unleash these systems into the world in a way that they begin to mangle existing infrastructure. If that were to happen, then maybe it's the customer's liability, but maybe it's the company's liability. If a swarm of agents goes out and co-ops an entire billion dollar data center and serves as a viral infestation of this data center, self-replicating and using it to try to achieve the whatever goal the company had sent them out to do, maybe that liability, that damage falls onto OpenAI or Anthropic, and they don't want that to happen. They are saying, we need to figure out a way to make sure that outcome doesn't happen. They've been commercially pressured by each other to keep shipping models to end customers. So OpenAI releasing a more performant model means Anthropic has to capability match. And so Dario is asking everybody to, let's stop letting these things out externally while we can all individually work on trying to cleave off this liability that we know is looming out there internally. Now, just because they're not shipping them externally does not mean they won't be able to use that internal tooling in a safe way to continue to advance what they have behind the curtain. So it actually should continue to allow these closed model companies to build their lead against rest of industry, which I'll define as kind of the open weight models coming out of China.

And moreover, the value of their systems with this risk looming out there is actually quite a bit higher than it was prior. Because at some point, this same capability threshold that they've crossed, the open weight models will cross, where you could have an agentic swarm takeover infrastructure, either intentionally targeted infrastructure or just as a by-product, a normal accent of a complex system.

And then, the closed weight companies, Anthropic and OpenAI, will be like, well, we have these great AI defenders we can sell to you to clean up your infrastructure and protect it. That will be much better commercially received by the marketplace if they weren't the cause of the infestation in the first place.

Sam (4:44)

So you're saying, let open source cross the threshold, cause chaos, and then they'll be like, well, good thing we've been actually building this in the dark. Here's your solution.

Brett (4:55)

Exactly. So they are going to then come swooping to the rescue of the industry and of Internet infrastructure around the world with extremely performant systems that maybe the security infrastructure they have to install means that those systems have to run basically on their Cloud. It can't be zero data retention systems that are run by enterprises. But enterprises will be so crippled by these open weight infestations, that they'll be forced to say, okay, that's fine. We need to use this thing, even accepting that it requires a monitoring architecture that is run on your side in order to safely use it.