[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

**SPEAKER_1** (0:12)
We're here at NeurIPS with John Yang, SWE-bench and many other things. But welcome.

**John Yang** (0:17)
Thanks so much for having me. Yeah, really happy to be here.

**SPEAKER_1** (0:20)
Last year, I talked to Ofir, and I think Carlos as well, one of your co-authors. How SWE-bench doing? Like just generally, the project is like one and a half years old? Yeah, yeah.

**John Yang** (0:32)
I think one and a half years old in terms of when it was actually useful. We put it out October 23, and then people didn't really touch it too much. Then of course, Cognition came on the scene, and Devon was an amazing release, and I think after that, it kicked off the arms race.

**SPEAKER_1** (0:48)
Did they tell you beforehand? They just showed up.

**John Yang** (0:51)
I got an email about two weeks ago. I think it was from Walden. He was like, hey, we have a good number on it. I was like, wow, congrats. Thanks for using it. Then the release was mind-blowing. I was like, wow, these guys did an excellent job.

**SPEAKER_1** (1:06)
Amazing. And then, SWE-bench verified was like maybe last year.

**John Yang** (1:09)
That's right.

**SPEAKER_1** (1:11)
Catch us up this year. You have other languages.
There's like a whole bunch of varieties of SWE-bench now. So what should people know?

**John Yang** (1:19)
Yeah, for sure. I think there's a couple extensions that are happening. One is like more SWE-benches, SWE-bench Pro, SWE-bench Live.

**SPEAKER_1** (1:27)
Oh, SWE-bench Pro. Was that with you guys? Because it looks independent. It's like different authors.

**John Yang** (1:31)
It's completely independent.

**SPEAKER_1** (1:32)
So they just call themselves SWE-bench Pro without your blessing?

**John Yang** (1:37)
I think we're okay with it. When we came out, we were like, oh, cool. Interesting. It would have been fun to be part of it. But I mean, congrats to them. That's a great benchmark. Yeah. All right.

**SPEAKER_1** (1:47)
But yeah, Multimodal.

**John Yang** (1:48)
Yeah. We did Multimodal and Multilingual. And I think those have Multilingual seems to be...

**SPEAKER_1** (1:54)
Is it like JavaScript?

**John Yang** (1:56)
What else? Yeah. Multilingual, it's like nine languages. It's like 40 repos. But yeah, you got them. Like JavaScript, Rust, Java, C, you know, Ruby. Yeah. Yeah. You got them. Yeah.

**SPEAKER_1** (2:07)
And then Coursor, you mentioned itself. A lot of people, like they talk about the Django focus.

**John Yang** (2:12)
Yes.

**SPEAKER_1** (2:15)
Is there like... I don't know. How do we move past Django?

**John Yang** (2:18)
Yeah, for sure. I mean, it's cool to see a lot of the newer benchmarks like really try to diversify the repos. Like in the two follow ups we did with Multimodal and Multilingual, we made it a point to do that. So I think...

**SPEAKER_1** (2:30)
But you can also just put out SWE-bench 2025 and just...

**John Yang** (2:33)
That is true. And do a new distribution. Yeah. Yeah. So it's been cool to see the follow ups. I think quietly, and it's an open question for me, I'm excited to see how people curate the next sets. Like it's kind of interesting to see in the literature or in their blog posts, like how they're justifying why they're creating their separate split. The easier ones were like, oh, more languages, more repos. And then I think now people are like, well, ours is more difficult because of this curation technique. And I'm excited to see how long that lasts and where we're going to guide the evaluation towards. Yeah.

**SPEAKER_1** (3:08)
And more recently, you're working on CodeClash.

**John Yang** (3:10)
Yes, that's right.

**SPEAKER_1** (3:11)
So let's get people, you've already done other podcast about it. I will refer people to that with your chat with Andy. But just give people like a one, two sentence.

**John Yang** (3:20)
Yeah. No, happy to do it, especially on your podcast. It's an honor. Yeah. So basically the idea is, I don't like unit tests as a form of verification. And I also think there's the issue with SWE-bench where all of the task instances are independent of each other. So the moment you have the model kind of submit it, it's done, and that's the end of the story, end of the episode, you know? So with CodeClash, what we're thinking is, let's try to really evaluate like long horizon development, and development on a codebase that is consequential and condition upon what a model did before to that codebase. And so the general idea is you have two or more language models, and they play a programming tournament. And what that means is each model maintains their own codebase, and each round of the tournament, first they get to like edit and improve their codebase, however they see fit, very self-determined. And then in the competition phase, those two codebases are pitted against each other. So the codebases are run, and there's generally an arena, we have a lot of diverse arenas, but the arenas determine like codebase A is better than codebase B. And then you kind of repeat that across multiple.