Why Rapid AI Doomsday Scenarios Are Flawed | Sarah Constantin artwork

Why Rapid AI Doomsday Scenarios Are Flawed | Sarah Constantin

MTS

October 30, 2026

Sarah Constantin of Principles of Intelligence explores the blind spots in AI agent evaluation and why fast doom scenarios misjudge physical science.

0:00

Sarah Constantin

I have skepticism about very fast end-of-the-world scenarios.

Things like skipping the need to do experiments, because you can just see into the nature of physics and create a super weapon. It's like, that seems to be a thing that is more common to believe among people who have no experience or interest in the natural sciences, because in real life, things don't turn out the way you think a priori almost ever, even if you're very smart. That's just not how engineering works. That's not how experimental science works.

0:38

SPEAKER_2

And we are back. We are live with Sarah Constantin, who builds research programs at Principles of Intelligence, which is a nonprofit that brings scientists from other fields into AI safety. She also writes Rough Diamonds on Substack. Sarah, welcome.

0:52

Sarah Constantin

Great to see you.

0:54

SPEAKER_2

So tell us about how we can know when AI agents fail. How do we know when we can trust their output?

1:04

Sarah Constantin

Oh, I see. I read a thread about this. That was sort of thinking out loud. So this is something that I think it's an unsolved problem. There are situations where you can test code or you can try to verify BAS. And for a lot of things, you're looking at a human user saying, do I buy this? Do I like this?

How do you evaluate it? And people are often using these in very quick succession. So there's a danger that you're like, seems fine, seems fine, seems fine, and you're falling it off a cliff.

So I do think that there are better and worse ways to do this. I think that, for example, you can have an adversarial system. If you're not, if you don't trust just a straightforward human evaluation of AI output, you can have somebody's trying to do a thing with AI, and somebody maybe also AI assisted is trying to disprove it. You can look at track records. You can say, I'm going to try something that the AI said, and I'm like, see how it turned out in the future, months later or something like that. AI are apparently, AI forecasters are pretty good, but this is something that you could say when people are thinking about, this is a research direction I'm going at, am I going to later consider this to have been fruitful?

And it's just helpful for people to share what works for them, what doesn't. I think there's a big deficit of people. It seems so personal and it doesn't seem like an important thing to share, but hey, I tried this with my agent, it didn't work. It gives you a sense of what sorts of failure modes to expect and where you never see it and you can gloss over it.

2:56

SPEAKER_2

What works and doesn't work for you?

2:59

Sarah Constantin

What works and doesn't work for me?

Writing is bad. Generally, I don't see hallucinations. I don't see the most overt. It cites a text and it references something that is not in the text at all. Or it cites a fact and the fact is untrue. Generally, I don't see syntax errors in code.

Often, I do see bad ideas if I'm trying to idea generate. Often, I do see things where if there's a taste issue, I don't like the results. I've been struggling to do some personal projects where I'm trying to make a font and it's terrible. And yeah, there's a lot of things where it's very comfortable but faster to human output, web search tasks. And there's obviously certain kinds of bugs and failure modes in software projects that if you don't put in safeguards against them, the VibeCoder websites tend to have a bunch of client side bugs where you reload, it goes away, but only a little bit has been tested and hasn't tested all the different possible user paths.

4:10

SPEAKER_2

Yeah, the idea generation in particular with current models is just so bad.

I don't know if you've seen, I've been VibeCoding a bunch of maps recently over the last few months. Yeah, I VibeCoded a map of all the trains in the world, of flight routes in the world, of like TFR over time in different parts of the world and so on and so on. And I asked all the best models for more ideas in a similar vein. Basically a useful aggregated map of something valuable that doesn't exist yet in some easy format. And all of its ideas were terrible. Yeah. Weird.

4:48

Sarah Constantin

I write next, picture my next project. This is a unanimous thing I hear is what's the next research direction? What's the next project? They give you stuff you wouldn't want to do.

Read the other 11 minutes of this episode here.

$15 gets you this episode and 99 more. Credits never expire. After checkout, one link brings you back here to read it.

Not what you expected? Full refund within 14 days.

Thousands of transcripts fetched by people building searchable podcast archives

Fetch transcripts with code

The same credits work from a script or an AI agent. The demo key returns a sample episode in full, no card needed:

request
curl -H "x-api-key: pt_demo" \
  https://spoken.md/transcripts/1000651996090

Markdown with the speakers named, for your notes, your knowledge base, or anything that makes HTTP calls.

From $0.10 per transcript. No subscription. Credits never expire. Prices exclude VAT, added at checkout for EU customers. Not what you expected? Email us within 14 days with 20 or fewer credits used and we refund the pack in full.

Using your own key:

request
curl -H "x-api-key: YOUR_KEY" \
  https://spoken.md/transcripts/1000798897250