Why Coding Agents Keep Making Your Codebase Worse

SPEAKER_1 (0:00)

College football is back, so Hilton called on me, the superstition concierge, to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey? Oh, that smells lucky.

Hilton's unmatched hospitality can keep up with any superstition. Even a marching band wake up call at 555 in 55 seconds. Hit it! When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for the stay.

SPEAKER_2 (0:30)

This episode is brought to you by Paypal. You know how a mom's bag has everything? Sunscreen, snacks, a stapler?

The new Paypal app is like that, but for your money. Shop, pay, manage your account and earn rewards all in one place. And with purchase protection on eligible items, biometric security and pass keys, you're protected at every step. Download the new Paypal app to get started. See paypal.com/protectionterms.

Dex Horthy (0:59)

Oh my God. I ran out six codec subscriptions in a week. Isn't that amazing? What did you ship, dude? A Sloth code bench? Oh, that's incredible. So have you seen this? Do not mortgage your code base in the hopes that in a year, the model will come out that's smart enough to fix all your problems. I'm never letting anyone merge anything with this pattern in it again. Am I 99th percentile? Am I being left behind? Do less things in parallel? The same thing that was always true about engineering is still true and all work.

Patrick Akil (1:26)

Today, I'm joined by Dex Horthy, CEO of HumanLayer, the man who coined the term context engineering. We talk about most of what you see online for agents is completely fake. Why token maxing doesn't work in the real world, how benchmarks work and which one should look at, how much faster you should actually aim to be with agents. And it's not 100X. And at the end, he shares his agentic engineering workflow. What is automated? What is a software factory? And where does he still spend his time?

Dex Horthy (1:53)

I saw a bunch of people commenting as well that like Astra got much better at computer use. It got much better at like blenders and like game slop and this kind of stuff. But it didn't actually get that much better at like writing maintainable code or like controlling complexity or like removing things or like changing abstractions or whatever it is.

And I think that was my feeling a lot and like a lot of people had the vibe. And so one of the things we're talking about tomorrow is sort of this benchmark I discovered called Slop Code Bench. It was made by the lab of Gabe Orlansky. He's like a researcher at the University of Wisconsin. Have you seen this?

Patrick Akil (2:33)

No.

Dex Horthy (2:34)

A Slop Code Bench? Oh, that's incredible. It's an entire benchmark designed around proving like models make code bases worse over time. Really? Yeah. So up until that point, there have been like, I would say, like three flavors of benchmarks. There's like the core, like suibench benchmark, which is like, here's a tiny PR from a real open source repo.

We'll rewind the repo to before the PR was made. We'll like pull the tests out of the PR, basically say, hey, model, here's the issue that was filed 10 years ago on this random Ruby project or whatever it is. Go like fix the issue. And then we would take basically any change that it made to the test, we would throw them out, and we would drop the tests in from the benchmark case. And then we would run those tests against the models code. And then if it passed, it would get a reward. And if it failed, then it also has to do it without breaking any other tests. But it's very, very small targeted, can you get the test to pass?

Obviously, these are benchmarks that are different from verifiers. But it's like all the benchmark, oh, this model score is 95% on sui bench. Coding is solved. And it's like, that's not what real coding looks like. So you had this other chapter of things where it's like, OK, cool. Instead of solving this one little PR, it's like rebuild all of Microsoft Excel from scratch with every single feature. Or rebuild all of Redis from the specs with every single feature. So you have these marathon benchmarks. Those two are from one called sui marathon, actually. It was like four to 400 hour tasks.

And then Cognition introduced this idea of like, yes, OK, we're going to use much bigger tasks. And also we are going to have a judge model run afterwards and contribute to the score on the exercise of like, did this thing break any of our code based rules or conventions or patterns or whatever it is? So it's sort of like telling it like, hey, look, in C++, we can't do logging like this, it has to be like this.