Should You Train Your Own Model?

Andy Lyu (0:00)

Training is not something we would recommend until you've exhausted every other option to improve your agent. That includes things like prompt optimization, having better workflows, having evals. If you don't have any of these set up, quite frankly, you are not ready for reinforcement learning. You're not even ready for SFT. You are still in that builder stage.

Shane Thomas (0:21)

We're going to be doing something that we like to call Builders Learn ML, because if you know anything about Abhi and I, you know that we're not formally trained in machine learning. But we've picked up quite a bit over the last few years. And some of the reasons is we've made friends like Professor Andy, who has helped us along.

Abhi Aiyer (0:40)

Yeah, I'm super stoked. It's been a while since Professor Andy has graced us with his knowledge.

Shane Thomas (0:45)

Yeah. And so for some of you who have been around some of the OGs, you might have seen him, I don't know, it's probably been almost a year since he was on the show. But for those that haven't, you're in for a treat.

Hey, guys.

Andy Lyu (0:58)

It's been a while.

Abhi Aiyer (0:59)

For the new viewers, why don't you introduce yourself?

Andy Lyu (1:01)

Sure.

Abhi Aiyer (1:02)

Yeah.

Andy Lyu (1:02)

So I'm Andy. I'm the co-founder and CTO of a company called Osmosis. The high level, what we do is we help companies train their own AI models. You know, many companies come to us and say that, hey, we're spending a ton of money on OpenAI and Anthropic every month, but our agent use case is just a routing use case or a customer support use case. So what we do is we help them take their data, take their infrastructure and their harness, and then create a model that is more predictable, malleable, and as well as more performant than if they were to use off-the-shelf models.

Abhi Aiyer (1:31)

That's very cool. Let's start from the very basics again. When people say train your model, what does that even mean?

Andy Lyu (1:37)

So right now, there are two common ways to train your model. The first one might be more easy to understand. It's a technique called supervised fine-tuning. The idea is you want to gather a bunch of traces or gather a bunch of agents doing work that you know is good, and then take those trace and teach another model, which may be weaker, but is smaller and cheaper, to do the same work. So that is supervised fine-tuning. You're essentially asking a model to mimic a more powerful teacher. The other thing that we like to talk about in the industry is called reinforcement learning. This has the potential to really excel at what the frontier is currently capable of. So I think in a new story later, you guys are gonna talk about OxAlpha and related models. These models are mainly trained with reinforcement learning. And in fact, most of modern-day LLemps, like ChadGBT 5.6, Claude Opus 5, these are also improved with reinforcement learning. The TRDR is inspired from nature.

When you touch a really hot stove, or when you do something that's not the best of judgment, your body will tell you, hey, don't do that. If you touch a hot stove, your hand will pull back. That's instinctive because there's a penalty from the environment around you. And similarly, when you eat food, when you go to bed, these have a reward, right? Like you feel better after doing so. If you're really hungry and you have a burger right in front of you, you're going to feel good eating that, right? So to the machine, this is kind of the same thing.

We want to reward the model for doing the correct action in an environment. And similarly, if they do something that we don't want them to do, we penalize them. We scale them up to hundreds of thousands of these similar interactions. And over time, the model learns a good idea or a good strategy on how to solve really complicated questions like coding or office work.

Shane Thomas (3:21)

Has the process, I guess, changed over the last year on how you do this reinforcement learning? I know typically people think, you know, I'm just taking a step back from the outside looking in, this sounds really challenging. When would I actually want to train my model? How would I make the decision of when to do supervised fine-tuning or reinforcement learning? How much data do I need? When does it start to make sense? Because obviously there's a cost barrier to, you know, you might be sending a ton of tokens to a larger model, but there's also a cost and a time to actually do this training process.