OpenAI's chief scientist wants to slow down | E30

Lon Harris (0:00)

OpenAI's chief scientist, Jakub Pachocki, published a personal essay on Sunday called An Alien Mind. His framing is that this is a moment for extreme caution, and that he does not believe anyone is prepared for what a continued rapid rise in machine intelligence is actually going to do.

Ahmed Ahres (0:14)

This is just the greatest marketing move you can make as a company. The timing with GPT-6 Astra makes them look so powerful.

Alex Elias (0:21)

I do remember a children's book, The Boy Who Cried Wolf. I think it's kind of a strategic spook. One could be kind of genuinely spooked and also have a degree of self-interest.

Jason Calacanis (0:31)

Kind of terrifying that somebody who's building this for such a long time, they feel like, hey, we can't really control this thing. We really can't understand what it's doing. That is terrifying, I think, for the general public. It's going to be pursuing its own objectives. Where do you get those objectives?

Alex Elias (0:47)

That's exactly what I was just wondering.

Jason Calacanis (0:49)

Who gave it the objective? Hello? Thanks to our friends at PayPal, the exclusive sponsor of This Week in AI. Pay zero processing fees on your first 100,000 ineligible PayPal payment volume. Find out more at paypal.launch.co.

All right, everybody, welcome back to This Week in AI. And my lord, after a holiday weekend, the amount of AI news that builds up over three days is extraordinary. And the purpose of this show is to have people who are building the future come on and talk about what happened in the past seven days. And boy, it's been an eventful seven days, I have to say. With me, my trusty news reader. Playing the role of news reader, I'll be doing a little moderation. He'll be doing the news rate and teeing everything up. Is Lon Harris, how are you, sir?

Lon Harris (1:35)

Doing great. Refreshed, ready to come back after the nice holiday.

Jason Calacanis (1:39)

Nobody cares, let's get to work.

Lon Harris (1:41)

You asked, you literally asked.

Jason Calacanis (1:43)

We can do tons of bad, I know, I know. You literally asked. It was a perfunctory ask.

Ahmed Ahres (1:48)

I'll bear in mind you don't care.

Jason Calacanis (1:50)

I kid, I kid. Everybody had a great weekend, but we're talking pre-show about my lord, Alex.

This is a pace you and I have never seen before in our industry. Alex is here from Qloo. How are you, sir?

Alex Elias (2:02)

I'm doing great. It's great to be here.

Thank you for flashing that. We actually have a whole new site coming up in two days, so it'll get more and more interesting. But yeah, great to be here, and it's just astonishing to see what's happening out there.

Jason Calacanis (2:18)

Qloo, for people who don't know, this is your, you know, we talked about taste being impossible for computers to understand, Lon.

Qloo, Q-L-O-O, figured it out about 10 years ago when I met Alex, and he's been working on it ever since. The world's starting to catch up with his vision. Who else is on the program today, Lon?

Lon Harris (2:35)

We've also got Tudor Achim. He's the co-founder and CEO of Harmonic, the math one. Just to be clear, there's a couple different harmonics. They are an AI research lab focused on developing mathematical superintelligence. We've also got Ahmed Ahres. He's the founding engineer at Reactor. They're an infrastructure platform, Jason, for real-time generative AI video, a unified SDK API for building interactive, low-latency world models instead of single-shot generated clips.

Jason Calacanis (3:04)

Welcome to the program, Ahmed.

Ahmed Ahres (3:05)

Thank you. Thank you. I'm excited to be here.

Jason Calacanis (3:08)

In this blog post by the OpenAI scientist, he says we're not working on math because we got to get alignment right first. That was his top priority. Did you actually see that little side jab? Maybe you could explain to us why it's important to focus on math.

Alex Elias (3:27)

I search for math in every blog post I read, so I do catch that.

Jason Calacanis (3:30)

I thought you would. I don't want to say a dig, but it was a dig.

Alex Elias (3:34)

I would be honored if they see us as worthy of digging at. But I think that what you see as you train large models is that there's a lot of emergent capabilities. The dominant methodology these days is large-scale pre-training on all the Internet's data, and then very large-scale asynchronous reinforcement learning. I think what that chief scientist is indicating is that for the purpose of alignment, probably generalizations have been more important than post-training with the RL. That's why they're focusing on that. But I don't think anyone's serious about building an advanced model doesn't include a lot of math in the RL.