Harvey Co-Founder Gabe Pereyra on the Token Pricing Reckoning Coming for AI

**Gabe Pereyra** (0:00)
The models are getting more expensive and they're getting better. We're just seeing this huge explosion of usage, cost, tokens, everything.

**Molly O'Shea** (0:05)
We need to talk about tokens. Winston mentioned you guys got up to 13 trillion.

**Gabe Pereyra** (0:10)
We are consuming a ton of tokens for just serving all this stuff. The big misconception right now is I don't think people realize how expensive this is going to get. And they're gonna be like, what did my agent do that cost me $10 billion?

**Niko Grupen** (0:23)
We launched Lab, which is our legal agent benchmark. It's a benchmark for measuring the performance of agents on real world legal tasks. We posted our first sort of initial results of how closed source frontier models, like those from OpenAI, Anthropic, DeepMind perform on the benchmark.

**Molly O'Shea** (0:50)
Gabe, welcome to Sourcery.

**Gabe Pereyra** (0:52)
Thanks so much for having me.

**Molly O'Shea** (0:53)
Thanks for having us here. We're in the secret speakeasy back room.

**Gabe Pereyra** (0:56)
Exactly, yeah, our secret.

**Molly O'Shea** (0:58)
So today, we're gonna walk through the latest release of your benchmarks, which is such a fun topic. Winston was so excited to talk about it too.
So I guess to start, let's break it down. What are you measuring with your benchmarks, and where did we get to today?

**Gabe Pereyra** (1:13)
Yeah, so the thing we open sourced was our legal agent benchmark. And I think the thing that we're super excited about is, we have done a bunch of benchmarking in the past. So before Legal Agent Bench, we had Big Law Bench, which was more of a chat-based benchmark, which was kind of a QA dataset. And we shared some of that. We worked with kind of the labs and the providers. But at the time, we were a smaller company and we didn't have the bandwidth to fully open source it and support it. And so what we wanted to do with Legal Agent Bench was fully open sourced this so anyone working on agents could run their agents and kind of work with us in the community to evaluate these things. And the way I would think of this dataset is we tried to really mimic what the coding agent benchmarks did.
And the way to think of the coding agent benchmarks is with things like sui bench, terminal bench, you have a GitHub repo, which is kind of all the code for a project. You have an issue, which is someone said, hey, this doesn't work, can you fix it? And then you have a bunch of unit tests that will tell you, you know, if you write a bunch of code and run it, the unit test will tell you, you know, you broke this thing or this thing passed. And the way the agent gets scored is, did it write all this code that passed all the unit tests? And I think there's a bit of a misconception that, oh, legal subjective, so you can't do this. And I think the thing for especially big law is a lot of the work you can actually quantify. And this is what senior associates and partners are doing when their junior associates do a bunch of this work.
And so as maybe an example of a task in the legal agent bench, it would be, can you do diligence? Which is, you get a data room, so you're trying to buy a company, and you'll get all of that company's contracts. And what the law firm needs to do is go through all the contracts and make sure that there's nothing questionable or that might blow up the deal in there. And so the way we set up this data set, like one of these tasks is, you have a data room, that's all the contracts. You have a partner request, which is a very high level and set of instructions you would get from a partner if you're an associate. And usually for something like diligence, it would be, hey, we've got the data room, can you look at it? Right? And you would need to infer, I know what I need to do based on that. And then I think part of the novelty here is we use both of those to create essentially legal unit tests. And so what are all the things that a partner would be looking for when they're trying to assess an associate's work product? And so you would take that data room and you would generate a diligence memo, which is what the law firm gives to their client to basically say, here's all our findings on this diligence. And the things you would look for, so one section is going to analyze all the change of control provisions, which is basically, if I'm buying a company, there's going to be a change in control of the company. And many contracts have things that can be triggered. And most of them don't matter, but there can be ones where this important vendor contract will change in a material way and you need to flag this. And so that would be like a unit test of like you want to check that.