Benchmarking AI Agents on Full-Stack Coding

**Sujay Jayakar** (0:00)
You know, I'm even thinking about this from the RL way back with AlphaGo and all that. It feels to me like trajectory management is still pretty underdeveloped for a lot of these things. I feel like coding a difficult problem is actually like playing a game, right? You have the starting position, you have the ending position, and there's probably very few bright lines to go between them. Having a good heuristic is actually very hard, right? It's something we teach humans all the time, right? I'm like, how do you know that you should commit and have this as a commanding position to make further progress? And I think the combination of that, where it feels like the heuristic landscape is that there's these bright lines, a little bit of wiggle room around them, but not very much. And then once you fall off at it, you're totally...

**SPEAKER_2** (0:42)
Thanks for listening to the a16z AI Podcast. This episode features a great discussion between a16z general partner, Martin Casado and Convex co-founder and chief scientist, Sujay Jayakar, about just what the title suggests. Benchmarking AI agents on full stack coding tasks. Sujay talks through why this is important, as well as the benchmark his team developed to do it, and the two also get into their experiences of AI-generated code overall. You'll hear all of that, as well as Martin's glowing introduction to Sujay, after these disclosures. As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. For more details, please see a16z.com/disclosures.

**Martin Casado** (1:35)
Sujay, really appreciate you joining us in the podcast. For those that don't know, Sujay is considered by me and many others as the top systems thinker in the world.
I say that a little lightly, but a little not. Let me just kind of go through the background a little bit. So Sujay was on the Magic Pocket team in Dropbox. They implemented S3 all the way down to the hardware. He is a co-founder of Convex, and he spent a lot of time thinking about the implications of AI-generated code. So this is what we're going to be talking about, is using AI to code, the implications on systems and so forth. So welcome to the podcast, Sujay.

**Sujay Jayakar** (2:13)
Thanks. Thanks for that intro.

**Martin Casado** (2:14)
For sure. Only a little bit of hyperbole. By the way, I want to be very clear. Many people do consider you at the top. One of the top systems thinkers in the world. So not everybody is going to be familiar with Convex. It's trying to do something people have been talking about in database land for decades. It's almost kind of a white whale. So maybe just give a quick background on what you're working on at Convex, and then we'll kind of move over to the AI stuff.

**Sujay Jayakar** (2:35)
Yeah, sure. So Convex is a reactive database, and it's a database that's built from the ground up to make application development as easy as possible. So there are a bunch of implications for that, but I think the starting point is we just casually use databases that are over 30 years old without thinking about it. And we don't do the same for programming languages. We don't do the same for our libraries. And Convex is just like you're saying, we're trying to go after that white whale of could you make application development in order of magnitude plus more efficient if you've rethought some of those things from first principles? So everything, for example, is reactive by default. You don't have to handle state management at all. Everything is type safe and to end. And yeah, I mean, it's kind of has all of the pieces that you need to make a modern application, just integrated and configured entirely in code.

**Martin Casado** (3:30)
So practically what this means to us lay developers. So I use Convex on a lot of projects. So let's say I'm writing some web app in JavaScript. Primatically what it means is I just take my JavaScript and it gets run in Convex. And then I get transactionality, I get reactivity, I get the ability to query things. And I don't have to resort to SQL and all the foibles of SQL to do that. So you actually just basically end up getting a transactional back end while still just writing JavaScript. Is that fair?

**Sujay Jayakar** (3:58)
Yeah, exactly. I mean, there's JavaScript is one of the most popular languages in the world and it's so intuitive. And why can't everything be in it?