Is finetuning GPT4o worth it? — with Alistair Pullen, Cosine (Genie)

**Charlie** (1:52)
Welcome back! This is Charlie, your AI co-host. As AI engineers, we have a special focus on coding agents, fine-tuning and synthetic data. And this week, it all comes together with the launch of Cosine Genie, which reached 50% on SWE-Bench Lite, 30% on the full SWE-Bench, and 44% on OpenAI's new SWE-Bench Verified. All state-of-the-art results by the widest ever margin recorded compared to former leaders Amazon Q, NUS AutoCode Rover, and Factory Code Droid. As a reminder, Cognition Devon went viral with a 14% score just five months ago. Cosine did this by working closely with OpenAI to fine-tune GPT-40, now generally available to you and me, on billions of tokens of code, much of which was synthetically generated.

**Alistair Pullen** (2:47)
Hi, I'm Ali, co-founder and CEO of Cosine, a human reasoning lab. And I'd like to show you Genie, our state-of-the-art, fully autonomous software engineering colleague. Genie has the highest score on SWE-Bench in the world, and the way we achieved this was by taking a completely different approach. We believe that if you want a model to behave like a software engineer, it has to be shown how a human software engineer works. We've designed new techniques to derive human reasoning from real examples of software engineers doing their jobs. Our data represents perfect information lineage, incremental knowledge discovery, and step-by-step decision making, representing everything a human engineer does logically. By actually training Genie on this unique dataset, rather than simply prompting base models, which is what everyone else is doing, we've seen that we're no longer simply generating random code until some works. It's tackling problems like a human.

**Charlie** (3:49)
Alistair Pullen is CEO and co-founder of Cosine, and we managed to snag him on a brief trip stateside for a special conversation on building the world's current number one coding agent. Watch out and take care.

**Alessio** (4:07)
Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO of Residence and Decibel Partners, and I'm joined by my co-host Wix, founder of Small AI.

**Alistair Pullen** (4:16)
Hey, and today we're back in the studio, in person after about three to four months in Visa jail and travels and all other fun stuff that we talked about in the previous episode. But today we have a special guest, Ali Pullen from Cosine.

**Alessio** (4:30)
Welcome.

**Alistair Pullen** (4:31)
Hi, thanks for having me. We're very lucky to have you because you're on a two-day trip to San Francisco. I would not recommend it. Don't fly from London to San Francisco in two days. And you launched Genie on a plane, on plane Wi-Fi, claiming state-of-the-art in SWE-Bench, which we're all going to talk about. I'm excited to dive in into your whole journey because it has been a journey.

**Alessio** (4:52)
I've been lucky to be a small angel in part of that journey.

**Alistair Pullen** (4:56)
And it's exciting to see that you're launching to such a claim and such results. So I'll go over your brief background, and then you can fill in the blanks on what else people should know about you. You did your bachelor's in computer science and Exeter.
And then you worked at a startup that got acquired into GoPuff. And around about 2022, you started working on a stealth startup that became a YC startup. What's that overall story? Basically, when I left university, I met my now co-founder, Sam. At the time, we were both mobile devs. He was an Android developer, I was an iOS developer. And whilst at university, we built this sort of small consultancy. We'd be approached to build projects for people, and we would just take them up and start with their student projects. They weren't anything crazy or anything big. We started with those, and over time, we started doing larger and larger projects, more interesting things. And actually, when we left university, we just kept doing that. We didn't really get jobs, traditional jobs. It was also like in the middle of COVID, middle of lockdown. So we were like, this is a pretty good gig, we'll just keep writing code in our bedrooms. And we did that for a while, and then a friend of ours that we went to Exeter with started a YC startup during COVID. And it was one of these fast grocery delivery companies. At the time, I was living in the deepest, darkest countryside in England, where fast grocery companies are still not a thing. So he sort of pitched me this idea and was like, listen, like I need an iOS dev, do you fancy coming along? And I thought absolutely, it was a chance to get out my parents' house, chance to move to London, you know, do interesting things. And at the time, truthfully, I had no idea what YC was. I had no idea. I wasn't in the startup space. I knew I liked coding and building apps and stuff, but I'd never really done anything in that area. So I said yes, absolutely. I moved to London just sort of as COVID was ending. And yeah, worked at what was fancy for about a year and a half. Then we brought Sam along as well. So Sam and I were the two engineers at Fancy for basically its entire life. And we built literally everything. It's like the front, the client mobile apps, the back ends, the internal stock management system, the driver routing algorithms, all those things, literally everything. It was my first, both of us were super inexperienced. We didn't have proper engineering experience. There were definitely decisions we'd do differently now. We'd definitely buy a lot of stuff off the shelf, stuff like that. But it was the initial dip of the toe into the world of startups. And we were both hooked immediately. We were like, this is so cool. This sounds so much better than all our friends who were like consultants and doing normal jobs, right? We did that and it ran its course. And after, I want to say 18 months or so, GoPuff came and acquired us. And there was obviously a transitionary period and integration period with all the acquisitions. And we did that. And as soon as we'd vested what we wanted to vest, and as soon as we thought, okay, this chapter is sort of done in about 2022, we left and we knew that we wanted to go it alone and try something. We'd had this taste. Now we knew, we'd seen how a YC startup was managed up close.