Machine Learning Engineering Is the Ultimate RL Task | Jennifer Zhao artwork

Machine Learning Engineering Is the Ultimate RL Task | Jennifer Zhao

MTS

October 12, 2026

Jennifer Zhao, co-founder and CEO of Preference Model, discusses coming out of stealth with a $16M seed raise led by Andreessen Horowitz and open sourcing Karrot.

0:00

Jennifer Zhao

I think that environments are useful when they have a lot of depth to them.

So ideally, they should be things like Chess or Go, where you can train a model on this, and then you can keep on training on it essentially forever, and it's basically never going to, eventually learn how to do perfect play. These are the best kinds of environments. And so if you have these kinds of environments, over time, I think that what happens is the agents, they eventually like find more and more stuff in them, like find things that you didn't think about, or like ways of solving the problem that your graders didn't catch. And so I think that for us, these environments, if they're good, they're things that we want to maintain and keep iterating on.

0:46

SPEAKER_2

Hello everyone, and welcome back to MTS. Today, we are joined live with Jennifer Zhao, and Jennifer is the co-founder and CEO of Preference Model, a company building reinforcement learning environments to train AI models to perform machine learning research. And today, the company is coming out of stealth, it's an exciting day, announcing a $16 million seed raise funded by Andreessen Horowitz and open sourcing its framework, Karrot. So Jennifer, thank you so much for joining us today on MTS.

1:16

Jennifer Zhao

Yes, absolutely, happy to be here.

1:18

SPEAKER_2

I wanted to just congratulate you guys. I know you guys are coming officially out of stealth and you're announcing the raise. Tell me a little bit about the origin story with Preference Model. Why did you guys decide to build this and why are you guys coming out of stealth today?

1:32

Jennifer Zhao

Yeah.

Yeah, so I think what we wants to do with this company is try to make AI go well. And that's like really vague and all. But basically, we were originally thinking, like, how can we maximize the positive impact of AI as it is today? And we came up with this idea of building post-training as a service.

Actually, the original idea behind this was a lot similar to what Tinker is today. And then we realized that it's probably much better use of our time to just go and build data, as in RL environments, instead of doing post-training. And so we pivoted to doing RL environments about a year ago. We've been building RL environments since we started for machine learning engineering. And I think that now we are sharing with the world Karrot, among other things, that we'll be sharing in the next few weeks. Because what we want to do with Karrot is we want to essentially start raising the water line of quality when it comes to the kinds of environments that these models end up being trained on.

2:42

SPEAKER_2

Yeah, I wanted to just get, because you have this central belief behind it, that the most important step in AI is teaching models to actually perform this machine learning research themselves. So I just wanted to get your thoughts on why you think this might be one of the highest leverage capabilities to develop.

3:00

Jennifer Zhao

Yeah. So I think that there's this pattern that's played out a couple of times already where you can spend a lot of time trying to figure out how to teach the models a particular capability in a domain, whether it's like computer use or finance or like business applications or whatnot. And then you figure out how to do it. And then you find out that like the next model just knows how to do it.

I think that like model capabilities are growing very quickly. And a lot of the stuff that you're doing is just going to be automated. And so if you have like really short timelines like we do, then what's the way, what are the ways in which you can actually have the most leverage and make the most impact on how things actually go. And so we think that that is like machine learning engineering.

Actually, one of the things that we're trying to do also is try to automate environment building itself.

3:58

SPEAKER_2

So, yeah. I mean, AI is just doing machine learning is pretty much just RSI, right? And so I wanted to get your takes and just how does this actually develop and what are the considerations around alignment that you need to keep in mind as this actually starts developing itself?

4:14

Jennifer Zhao

Yeah. I think that like for the past year or so, like the focus for us had has been on like general machine learning capabilities. I think that for the next while it's going to be on alignment because that's going to be the next bottleneck for making progress and for like actually releasing these models in a way that is like safe to the world.

Read the other 9 minutes of this episode here.

$15 gets you this episode and 99 more. Credits never expire. After checkout, one link brings you back here to read it.

Not what you expected? Full refund within 14 days.

Thousands of transcripts fetched by people building searchable podcast archives

Fetch transcripts with code

The same credits work from a script or an AI agent. The demo key returns a sample episode in full, no card needed:

request
curl -H "x-api-key: pt_demo" \
  https://spoken.md/transcripts/1000651996090

Markdown with the speakers named, for your notes, your knowledge base, or anything that makes HTTP calls.

From $0.10 per transcript. No subscription. Credits never expire. Prices exclude VAT, added at checkout for EU customers. Not what you expected? Email us within 14 days with 20 or fewer credits used and we refund the pack in full.

Using your own key:

request
curl -H "x-api-key: YOUR_KEY" \
  https://spoken.md/transcripts/1000795837713