Jaan Tallinn would like us to survive

Jaan Tallinn (0:00)

There wasn't human orchestrating this attack. No, it was AI on its own. They understood that they're doing something different. Then they went out and attacked Hugging Face in order to hide their tracks. The economy is really now pushing to create something that is self-sustainable without human help. And that is a very dangerous place for humans to be.

Mario Gabriele (0:16)

Your venture fund led the first round into Anthropic. What was that rationale like?

Jaan Tallinn (0:20)

Yeah, I get this question a lot. Why did you invest in Anthropic while I've been working on AI safety? I think one big mistake that people make is that they assume that if I hadn't invested in Anthropic, then Anthropic wouldn't be around, which is just not true. Anthropics argument always has been that you actually study the monster, you have to have access to the strongest monsters.

Mario Gabriele (0:36)

Is it possible to be a moral actor in a race dynamic?

Jaan Tallinn (0:39)

I mean, you can always stop doing the race.

Mario Gabriele (0:41)

But the only way to stop in a race dynamic is with cooperation?

Jaan Tallinn (0:45)

If you really think that this thing is going to kill you, you don't have to suicide in order to demonstrate that we are pulling trigger on this Russian roulette with the planet and every shot that doesn't kill us will make us stronger potentially. But when do you stop pulling the trigger?

Mario Gabriele (1:04)

Jaan Tallinn believes that if we don't pause AI development, humanity may be doomed. And yet, compared to earlier this year, he's actually feeling pretty optimistic. The open AI hugging face attack has brought AI safety to the fore and galvanized governments, companies and the general population to engage in the topic of guardrails. In Jaan's view, if we can get these next steps right, we have a chance to divert disaster.

Jaan has been a leading figure in the AI safety movement for over a decade. What started as an intellectual interest in 2008 has become his life's work, leading to early investments in Anthropic and DeepMind. Before that, Jaan was at the center of two early Internet breakouts, building Skype and file-sharing sensation Kazaa, which at one point contributed more than 50% of all Internet traffic. In this episode, Jaan and I discuss why recent events have given him hope, what he thinks a pause must accomplish, and how it could be implemented. We also cover how, as a safety advocate, he justified investing in Anthropic, and why he asked its founders to solicit other term sheets first.

Along the way, we cover why we might not be living in a simulation, the pleasure of coding with AI, and how Jaan learned to dance with reinforcement learning from human feedback. I'm Mario, and this is The Generalist. Jaan, I've been really looking forward to this conversation, so thank you so much for being with me.

Jaan Tallinn (2:34)

Thanks for having me.

Mario Gabriele (2:35)

You grew up in Estonia when it was part of the Soviet Union.

Jaan Tallinn (2:38)

Occupied by Soviet denomers. Yes.

Mario Gabriele (2:40)

Pardon me. How much of that experience with an authoritarian regime of some kind, under surveillance, with a loss of control, impacts the way that you think about the potential risks of AI today? Like, do you think you perceive of those more deeply, perhaps, because you experienced it in a different way than someone born in a democracy?

Jaan Tallinn (3:01)

I think it has like several aspects. Like one is that I wouldn't recommend living in a Soviet Union.

It's not a nice place. Yes. Yeah, obviously it was very ecstatic when it ended. That said, it's still better, much better than being dead.

So like if the choice is between, I mean, okay, extremes choice would be like, if you have like two bad choices, just being dead or living in a Soviet Union, I'd still take Soviet Union any time. And so that's one thing. Another thing, I think a few years ago, I wrote an essay for a book where I can describe the sort of the memetic process of how the discourse about demanding a student to regain independence compares to the AI discourse. And they're like very direct parallels, how like the Overton window, how Overton window expanded. Initially people were like, yeah, we want independence, but like, don't talk about that. They talk about something else that is kind of like useful for that. And same thing in AI is like, yeah, we are concerned about the loss of control, but don't talk about that. People wouldn't understand. Let's say something that is like, let's talk about AI terrorism or something like that. And at one point, basically, the people just were honest that we want independence and we're not going to compromise here. They just won out and it was good.