The biggest AI threat still comes from humans | E31

Alex (0:00)

Are we about to unleash AI? Too smart to properly understand or deploy safely?

Lon Harris (0:04)

Within six to 12 months, a swarm like that with better capabilities could potentially do hundreds of billions of dollars in damage.

Kevin Mandia (0:11)

There is just no doubt, no matter what we do, there's gonna come a future where you have AI on offense built by the good guys, training AI on defense built by the good guys to protect us. Wherever money goes, crime follows. Wherever information goes and communications go, espionage will follow. What used to be, wow, only very few people could have done that attack. Now it's kind of being democratized.

Lon Harris (0:36)

Thanks to our friends at PayPal, the exclusive sponsor of This Week in AI. Pay zero processing fees on your first 100,000 ineligible PayPal payment volume. Find out more at paypal.launch.co.

Alex (0:50)

Hello, everybody. Welcome back to This Week in AI. My name is Alex. Now over the weekend, we were called by Anthropic to pace the speed at which AI is developed caused a sensation. As leaders from other AI labs co-signed his ideas, while the White House and some other members of the venture capital community pushed back. NVIDIA's CEO weighed in. Congress is taking a look and the labs are working to find a way to regulate one another, if no one else will. Are we about to unleash AI too smart to properly understand or deploy safely? Are we worrying too much? Are we ceding momentum to China? Can we even secure a world where agentics forms are trying to penetrate every system? Well, to understand all that and more, I have three excellent minds in the game of AI here for you, starting with Soups Ranjan, the co-founder and CEO over at Sardine, or Sardine AI, if you're feeling frisky. A startup that uses AI to sniff out fraud and ensure financial compliance, and it raised a $70 million Series C last year. Soups, welcome to the show.

Soups Ranjan (1:45)

Thanks for having me. Excited to be here.

Alex (1:46)

Glad you're here, man. And then we also have Kevin Mandia, CEO and co-founder of Armadin, an AI cyber security startup. They recently launched a massive white hat, agentic attack called the, what do you call it, Kevin? Superswarm, hyperswarm?

Kevin Mandia (1:59)

You could call it that. We called it a hyper attack.

Alex (2:01)

Hyper attack. Okay. Yes.

Kevin Mandia (2:02)

Crazies work great.

Alex (2:03)

And then last but not least, my dear friend and longtime podcast co-patriot in arms, Mr. Lon Harris, launches editorial director and basically one of my favorite people in the entire world. Lon, good to see you.

Lon Harris (2:13)

You too, Alex. Aren't you kind? This was so nice.

Alex (2:16)

Well, I appreciate that, but I'm going to put you to work. So first up for you, Lon, is what the hell happened this weekend and why is it Dario's fault?

Lon Harris (2:23)

So many things.

We're going to start with Dario. There were like five elements to this story we'll get to. But on Saturday, Dario Amodei published an essay on his personal site, arguing that the entire AI industry needs to deliberately slow down how quickly they are making their models more capable. What originally set him off was the OpenAI hugging face incident over the summer, in which a swarm of AI agents ran cyber attacks after escaping containment. He says that within six to 12 months, a swarm like that with better capabilities could potentially do hundreds of billions of dollars in damage. Alexin, he's proposing a three-step plan. Anthropic is committing to the first one right now outside evaluators with permanent employee-level access inside the company, who can publish what they find without Anthropic sign-off. Within hours, Elon Musk chimed in to say he thinks Dario is right. Sam Altman has committed OpenAI to the same evaluator access, and Demisus Abbas and Sakya Nadella are also both backing that direction.

Alex (3:18)

All right.

Kevin, you are a global leader in cybersecurity, and certainly, I think the leader on this chat today. Can you tell me a little bit about how you feel about the level of concern people should have about the capacity or capability of agents forms to actually do hacking? Was the hugging face incident a one-off, or is this something that you think is a clear and perhaps even present danger?

Kevin Mandia (3:41)

Yeah, I think stepping back and looking at every technological advance, you always wonder, hey, was it used by the good guys first or the folks of malintent first, right? And there's just no doubt, no matter what we do, that there's going to come a future where you have AI on offense built by the good guys, training AI on defense built by the good guys to protect us. I have found throughout human history, I was in the military, you look at every advancement, if you handcuff yourself, you sometimes give yourself a disadvantage militarily or strategically. And so I think it's a very tough thing to decelerate technological growth.