When AI Research Starts Moving Faster Than Human Research - Zhengyao Jiang

SPEAKER_1 (0:00)

College football is back, so Hilton called on me, the Superstition Concierge, to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey? Oh, that smells lucky.

Hilton's unmatched hospitality can keep up with any superstition. Even a marching band wake up call at 555 in 55 seconds. Hit it! When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for the stay.

SPEAKER_3 (0:30)

This episode is brought to you by Paypal. You know how a mom's bag has everything? Sunscreen, snacks, a stapler?

The new Paypal app is like that, but for your money. Shop, pay, manage your account and earn rewards all in one place. And with purchase protection on eligible items, biometric security and pass keys, you're protected at every step. Download the new Paypal app to get started. See paypal.com/protectionterms.

Zhengyao Jiang (0:59)

To me, harness engineering is more of a cheap way and efficient way to adapt the intelligence to a specific task.

Tim Scarfe (1:09)

So back in July, a new AI company based in London posted a very viral tweet on Twitter. It got about 1.8 million views claiming to demonstrate the first evidence of recursive self-improvement.

The thing is, not everybody bought it though.

Zhengyao Jiang (1:26)

I'm Zhengyao Jiang, co-founder and CEO of Weco AI, where we build self-improving agents.

Tim Scarfe (1:32)

You did actually say auto-researching the auto-research agent for eight days.

Zhengyao Jiang (1:37)

Where we run auto-research on auto-research harness, they managed to discover a better auto-research harness than we keep tuning in the last two years.

Tim Scarfe (1:49)

Now, when most people think about recursively self-improving intelligence, what they conjure up in their mind is a system that can rewire its own brain in a loop. But what if the system was only changing the code around the brain?

Would that still count?

Zhengyao Jiang (2:06)

We care about the outcome. Like, does it actually improve the capability of the system, not a specific layer?

Tim Scarfe (2:14)

And also, what about this small matter of Goodhart's law, which is that when a measure becomes a target, it ceases to be a good measure.

Zhengyao Jiang (2:23)

And more interestingly, we find that the design mechanisms prevent itself from reward hacking.

Tim Scarfe (2:30)

What is the biggest misconception people have about RSI?

Zhengyao Jiang (2:34)

If you arrive RSI, there's going to be a technical singularity or like intelligence explosion. I don't think that will be the case.

Tim Scarfe (2:45)

So there's a diaspora of folks who are building startups that are focusing on recursively improving super intelligence.

Zhengyao Jiang (2:53)

This project actually started about one and a half years ago, where we get interested in the concept of a recursive self-improvement. I mean, historically, all of the research have diminishing returns. There is an idea, though, what if you can allow the researcher improve its own efficiency? Historically, it's possible by methodology improvement, better tools, but human researchers, like human researchers' brain, for example, is always the key bottleneck. If we now have an autonomous research system, can we point the research topic to itself so that it can improve its research efficiency? And the hope is one day, this improvement of efficiency can overcome the diminishing return.

So that, yeah, the curve of the effort output turn from concave to convex.

Of course, that's like the grand goal, a bit like AGI, no one knows how far we can go, but improving the efficiency of self-referential research is definitely already on the roadmap.

Tim Scarfe (4:24)

So let me set this up properly, right? So Weco, they created this agent called Aide, and it is a research harness, right? And it's very, very simple. You give it a metric, you give it a problem, and it will hill climb towards that problem. But what they did was they set it up recursively. So they actually like hill climbed towards improving the agent itself. And what that means is improving the harness. So, you know, the prompts and the tooling and, you know, all of the skill surface, all of the things that the harness can do. And they just left it running. They just left it running and they saw what happened.

Zhengyao Jiang (4:57)

There's an interesting thing we observe on self-evolved auto-research harness.

We see the code it generated. Exactly. It's like an alien code. It's spagetic.

But for some reason, it generalizes really well. Of course, we have a set of benchmarks we're trying to hear climb on. After that, we will test it on holdout benchmarks. Those are public benchmarks, like MLE Bench Light, ALE Bench Light. ALE Bench is like algorithmic discovery benchmark from Sakana. MLE Bench is the OpenAI Machine Learning engineering benchmark.