Fully Automated AI Researchers Coming in Next Year | Nick Beckstead artwork

Fully Automated AI Researchers Coming in Next Year | Nick Beckstead

MTS

October 29, 2026

Secure AI Project co-founder Nick Beckstead breaks down the push for frontier AI safety laws, arguing why government limits on training run compute are necessary to prevent catastrophic loss of control.

0:00

Nick Beckstead

The solutions that might be a good idea in January, 2027, might be different from the solutions that are great in January, 2028

And you could also just have things move even faster with recursive self-improvement. I think a lot of folks are talking about having a fully automated AI researcher in the next year. And things might move several fold faster than they are now. And I'm not sure that the government will be able to either keep up or kind of put in place all the scalpel-like rules.

0:38

SPEAKER_2

And we are back. We are live with Nick Beckstead, who is the co-founder and CEO of the Secure AI Project, which is an AI policy and safety org pushing for stronger frontier AI safety rules in California and elsewhere. Nick, welcome to MTS. People are talking about the potential for a special session of the legislature.

What do you think? How likely is that?

1:02

Nick Beckstead

You know, I'm hearing that a special session is somewhat unlikely. You know, it requires some amount of agreement between the governor and the legislative leadership.

But you know, I'm supportive of, you know, Governor Newsom calling for that executive order. So he put out this executive order stating, requiring a group of academics to come together and assess embedded third-party risk assessments being required, the kind of thing that Dario and Sam Altman and other CEOs have kind of expressed support for. And they called it a kill switch. I don't know if the kill switch is the ideal word for it, but you know, setting up some mechanism to say, have a means to stop particularly dangerous things from happening. And I think that there's some very interesting ideas that are in there. And I'm excited to see what happens when the report comes out on November 16th.

2:10

SPEAKER_2

Why is kill switch a bad name?

2:15

Nick Beckstead

You know, I think the phrase kill switch has caused a lot of confusion. I think in AI safety, going back to SB 1047, and in cyber security, going back to legislation that was proposed a decade ago, I think there's just two ideas that we should distinguish. And I think one of them is good and one of them is not good. I think the not good idea is a big red button that the government can press that turns off all the AI models.

I think the good idea is more like an operational plan. Think about an evacuation plan for a building. Say, well, at our company, there's a man named Bob. He sets our evacuation plan. If we need to leave the building, these are the exits that will go out. We have a fire drill for that. I think we want something similar for the AI companies and it's achievable for, if it looks like your models are going rogue, can you turn them off? Can you figure out which models are affected? And do you have a procedure for that?

I just don't think it should be like a literal mechanical switch that you have to install in the CEO's office or with the government.

3:23

SPEAKER_2

I see. Do you think that there are labs currently that don't have the means to shut off their products or internal or external deployments if they deemed it necessary to do so?

3:37

Nick Beckstead

Well, I think that with the exception of open weight models, I believe people do have that ability, and it's good to make sure that they have it and that they continue to have it.

And I think if you look at the hugging face incident, it wasn't like, oh, we noticed that these models were doing all of these things that we didn't like, and then it took us a month to turn it off, and we had to do a bunch of R&D to figure out how to do it. I think people largely do know how to do it, and I think you should be exempting the open weight models from that kind of thing. My org published something on its website, actually. It kind of outlines what we think should be in an ideal solution to address kind of catastrophic risks around cyberattacks, bioweapons and loss of control. I could go through that if that would be a good launching point for us. Or I could tell you a little bit about, like, hey, these are the laws that are that exist right now. And these are these are how I see the gaps in the field.

4:40

SPEAKER_2

Yeah, I like the second angle. Yeah. What are the laws that exist right now? And what are the gaps in the field?

Read the other 15 minutes of this episode here.

$15 gets you this episode and 99 more. Credits never expire. After checkout, one link brings you back here to read it.

Not what you expected? Full refund within 14 days.

Thousands of transcripts fetched by people building searchable podcast archives

Fetch transcripts with code

The same credits work from a script or an AI agent. The demo key returns a sample episode in full, no card needed:

request
curl -H "x-api-key: pt_demo" \
  https://spoken.md/transcripts/1000651996090

Markdown with the speakers named, for your notes, your knowledge base, or anything that makes HTTP calls.

From $0.10 per transcript. No subscription. Credits never expire. Prices exclude VAT, added at checkout for EU customers. Not what you expected? Email us within 14 days with 20 or fewer credits used and we refund the pack in full.

Using your own key:

request
curl -H "x-api-key: YOUR_KEY" \
  https://spoken.md/transcripts/1000798711579