The RL Fine-Tuning Playbook: CoreWeave's Kyle Corbitt on GRPO, Rubrics, Environments, Reward Hacking

**Erik Torenberg** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, my guest is Kyle Corbitt, founder of the reinforcement learning and custom fine tuning company, OpenPipe, which CoreWeave acquired last year. I opened this conversation with a bit of a confession. I've done a lot of supervised fine tuning work over the last few years, both for Waymark in the early days of getting GPT-3 to write decent video scripts, and for research projects such as the emergent misalignment paper. But I've done essentially no hands-on RL work, both because my perception has been that front-tier models are probably my best option in any case, and because I'm afraid, perhaps irrationally, of reward hacking.
Kyle says that while it may or may not be worth the extra work and slower iteration time, he does believe that using RL on an open source model probably would deliver me better performance, and would certainly reduce both latency and inference cost dramatically. With that motivation in mind, Kyle proceeds to offer a master class on all things RL, which repeatedly challenged my premises and in multiple instances updated my understanding. He explains how RL differs from SFT in terms of the weight updates it makes to the models. How this difference makes RL fine-tuning less likely to cause catastrophic forgetting. What distinguished the DeepSeq GRPO algorithm from its predecessors, and what additional improvements on GRPO people are using in industry today. We talk about the distillation strategies that Chinese labs are using to fast-follow American frontier models, and he argues that their use of LLMs as judge in the context of RL post-training is a bigger deal than supervised fine-tuning. He also explains why he thinks that compute is the primary constraint preventing Chinese companies from catching up, and why he believes that we're already in a recursive self-improvement loop. He describes the cottage industry of reinforcement learning environment companies that sprung up to serve frontier labs, and why though it is a good business to be in for now, he's declined to invest in any of them. He surveys the use cases that are most commonly deployed by coreweave customers, and he offers a lot of advice on how to run RL in practice, including how to develop and iterate on evaluation rubrics, whether to train N models for N tasks, or a single model to perform multiple tasks, how the flagrant nature of reward hacking makes it relatively easy to deal with, at least when you're focused on specific narrow tasks, and how coreweave's use of LoRa adapters drives efficiency and convenience for their customers.
Kyle is both a technical expert and successful commercial practitioner, and from start to finish, this is a super high signal conversation on a classic training technique that has become an industry unto itself. And so, I hope you learn as much as I did from coreweave's RL fine-tuning guru, Kyle Corbitt.
Kyle Corbitt, founder of OpenPipe, now after an acquisition leading the serverless training team at coreweave, welcome to the Cognitive Revolution.

**Kyle Corbitt** (3:00)
I am super excited to be here.

**Erik Torenberg** (3:03)
Thank you. I'm excited to have you. This has been a long time coming since we've met almost a year ago now, and I'm glad to finally be doing it. So, and that's all on me, by the way, if it's just so everybody knows. So, you are a specialist in reinforcement learning. What I want to do in the next hour and a half or so is get basically a comprehensive survey, crash course, and rundown of what is going on in reinforcement learning, how we should understand it, and like what the techniques are looking like, who's using it, and where, and for what purposes, and who's having success, and not, and you know, what makes a difference in all those things. And so, I guess I was just going to start by telling you my story very briefly, and then allowing you to react to that, and tell me if I'm like way off base or not. My story in short is, I've done a lot of model fine-tuning over time, mostly on managed platforms, not so much like on open weights models, just a little bit of that, more so on like the OpenAI platform. But almost entirely supervised fine-tuning, not really much at all reinforcement learning fine-tuning.
And the story I'm telling myself, which you're invited to pick apart, is, for one thing, increasingly these days, like I can just use base models with a few-shot prompting, and that's getting me a lot of what I need. But even before that was possible, the problems that I was working on in the context of my company, Waymark, are sort of taste-driven problems, where we always kind of felt like we'd be better off going to our creative team and say, hey, give us 100 great examples. We'll fine-tune on that and hope that the AI can follow your lead, rather than try to go through some sort of seemingly more complicated, maybe more powerful, but kind of harder to wrap our heads around notion of like, well, if we get the ad, do it, and then we compare and we score, maybe there's an LLM as a judge. We're kind of like, I don't know.