AI Cannot Reliably Run a Company for Six Months | Richard Ngo
MTS
November 3, 2026
Former OpenAI governance and DeepMind safety researcher Richard Ngo analyzes why rapid recursive self-improvement timelines are overhyped and breaks down how frontier labs justify race dynamics.
Richard Ngo
The sort of thing that feels more difficult is running a company for long periods of time.
And when you're running a company, especially a larger company, you just have this enormous attack surface where one mistake can be exploited by a bunch of adversaries, like hackers, your rivals. You can mess up and then be in trouble with the legal system in a way that shuts down the entire company. So all of those things prioritize a certain level of robustness that the models don't currently have. And my sense is that it could be pretty difficult to get the models actually reliable on a task like run a company for six months and make sure it's profitable and doesn't end up in legal trouble.
SPEAKER_2
And we are back. We are live with Richard Ngo, who is an independent AI researcher who worked on AGI safety previously at Google DeepMind and on governance at OpenAI before leaving to become independent. Richard, welcome to MTS.
Richard Ngo
Thanks for having me.
SPEAKER_2
Absolutely. So why don't we talk about, start with, there seem to be a lot of people now in the mainstream EA world recommending resigning from the labs, making things more difficult for the labs.
Like Robert Wiblin, for example, from the 80,000 Hours podcast just posted this long video today, actually, about how people have been able to get the labs to change behavior, basically through activism. And you've been talking about this for some time. You've been recommending people resign from labs. Do you think that this is a positive change from the mainstream of EA?
Richard Ngo
That's right. I think that, you know, EA historically has been very conciliatory towards the labs.
And not, I think, for a particularly good reason, just because people were somewhat conflict-averse and scared of challenging, you know, prestigious, powerful companies. So at the very least, being willing to do that is a pretty good sign. I do think we're still not quite at the point where people are willing to actually analyze the power dynamics involved in labs and try to be honest about what's actually driving them. But at the very least, being willing to speak out against some of the racing that they're doing seems pretty seems like a pretty good step, right?
SPEAKER_2
And similarly, Scott Alexander just wrote a post called was called against. I'm forgetting the acronym.
SPEAKER_3
Sura. Sura.
SPEAKER_2
Yes, yes. Against Sura, which is the strategy of stop unilaterally and raise awareness. A lot of people were saying this is basically a, you know, self deceptive defense of anthropic where anthropic is, you know, against Sura itself. It is not stopping unilaterally and raising awareness. It is continuing to push the frontier to some extent.
Do you think that this is a good thing for anthropic to be doing?
Richard Ngo
Yeah. So, interestingly, a lot of the time when people discuss this stuff, they slip into this very almost basic moral reasoning of like good guys versus bad guys. And sometimes the good guys, you know, OpenAI cast themselves as the good guys against DeepMind early on, and then anthropic cast themselves as the good guys versus OpenAI, and possibly versus China as well.
My sense is basically that I think there are a bunch of people who are trying quite hard to be serious about this stuff in many of the labs, and somehow there's just this very, yeah, very basic, in some ways, almost childish reasoning about trying to beat your opponents, trying to get there first. I'm not saying it's fully wrong, but it's very interesting that it's the kind of trap that we see throughout history, and I think people have reasoned their way into thinking that this case is different, that in this case, you actually need to race ahead and beat your opponents. This time, everything's up for grabs because AGI is such a powerful technology. So, that's my rough sense of what's going on, that there's some kind of... They hit the point of the standard tribal reasoning that people get into, and then they sort of stop, and it's hard to get beyond that point.
SPEAKER_2
Right. You wrote this, or you're currently writing this, this sequence of posts on LessWrong, as a retrospective against the failures of the AI alignment movement over the last decade or so.
Do you think that the community has responded at all to this? Do you think they have... Have you seen any signs of change towards what you would want them to be doing?
Richard Ngo
Not a huge amount. My sense is that there are a bunch of people who are reasonably thoughtful and open-minded but tend to be in a pretty scared and panicky mindset because of the rate of AI progress. And so it's very difficult because almost everyone throughout the last, over the last decade or two, has underestimated the rate of AI progress. And really a lot of the forecasts, a lot of the capabilities we see right now are pretty sci-fi compared with what people expected five or 10 years ago. And so it does feel very hard to rule out the possibility that people are still dramatically underestimating the rate of progress.
Read the other 20 minutes of this episode here.
$15 gets you this episode and 99 more. Credits never expire. After checkout, one link brings you back here to read it.
Not what you expected? Full refund within 14 days.
Thousands of transcripts fetched by people building searchable podcast archives
Fetch transcripts with code
The same credits work from a script or an AI agent. The demo key returns a sample episode in full, no card needed:
curl -H "x-api-key: pt_demo" \
https://spoken.md/transcripts/1000651996090Markdown with the speakers named, for your notes, your knowledge base, or anything that makes HTTP calls.
From $0.10 per transcript. No subscription. Credits never expire. Prices exclude VAT, added at checkout for EU customers. Not what you expected? Email us within 14 days with 20 or fewer credits used and we refund the pack in full.
Using your own key:
curl -H "x-api-key: YOUR_KEY" \
https://spoken.md/transcripts/1000799816892