[State of Post-Training] From GPT-4.1 to 5.1: RLVR, Agent & Token Efficiency — Josh McGrath, OpenAI

**SPEAKER_1** (0:12)
Well, here is Josh from OpenAI, welcome. How was he to introduce himself?

**Josh McGrath** (0:15)
Yeah, I work on a bunch of the thinking models at OpenAI, and recently I've been focused on doing search-related stuff. But yeah, just a post-training researcher at OpenAI.

**SPEAKER_1** (0:26)
Yeah, and you were on with us for GPT-41, we were talking with Michelle who's on maternity leave, I didn't know that, and now we're 5.1. It's been a whole generation.

**Josh McGrath** (0:38)
Yeah, it's been wild and 4.1 was a non-thinking model, and then since then we switched into doing-

**SPEAKER_1** (0:44)
You thought you were last? When was you last?

**Josh McGrath** (0:46)
No, we still are releasing non-thinking models, but that one was the one that we did that was API-specific non-thinking. So focus has shifted a little.

**SPEAKER_1** (0:58)
Yeah. How did you get into post-training?

**Josh McGrath** (1:00)
So previously before OpenAI, I was doing pre-training data curation stuff, and I think what I was seeing from the news and looking at papers is like, oh, it seems like a lot of that. Now pre-training is dead, but I was like, oh, there's going to be so much interesting stuff in post-training. At that point, I was like, I really want to make some contributions there. I mean, it's not even necessarily that pre-training was dead, but it was definitely changing.
And do I want to make compute efficiency wins of 3% or do I want to change the behavior by 40%? And honestly, it just seemed more exciting to go to post-training and many late nights later. That's definitely true.

**SPEAKER_1** (1:40)
It's a different kind of data and engineering discipline too. It's very strange, the kind of work that you need in, especially RL, scaling it.

**Josh McGrath** (1:52)
Yeah, definitely. I think, for example, the number of moving parts in an RL run is just a lot higher. Like in some ways...

**SPEAKER_1** (1:59)
Do you do order of magnitude or...?

**Josh McGrath** (2:01)
I don't know if I could do order of magnitude, but if you think about pre-training, you're moving tokens to many machines and then you're getting basically a scalar from them, and then you're back-propping.

**SPEAKER_1** (2:11)
Yeah.

**Josh McGrath** (2:12)
The issue with RL is you're doing tasks, and each task could have a different grading setup, and each one of those different grading setups, that's like more infrastructure. And so, you know, when I'm staying up late trying to figure out what's going on with a run, it could be in way more things than there is in a pre-training run generally.

**SPEAKER_1** (2:32)
Yeah. And does it matter if you own the code of the task, or is it an outsourced third-party person, or, you know, my sense of it and the external sense of it, obviously, I don't see it up close, is that you work a lot of external partners, and I'm sure you also had some internal stuff, but, which is better?

**Josh McGrath** (2:50)
Honestly, I don't think I'll comment too much on how many external partners...

**SPEAKER_1** (2:56)
There are some, and there are some internal.

**Josh McGrath** (2:57)
Yeah, there's... We do, like...

**SPEAKER_1** (3:00)
The technical trade-off of, like, well, shit, I don't own this code.

**Josh McGrath** (3:04)
So, well, when it comes to, I don't own this code, actually, when I'm babysitting a run or something, it doesn't really matter if it's internal, external, whatever. Like, do I understand the system that's going underneath? And I think you end up having to, like, jump into a lot more code that you're like, I actually don't know what this does. Because I'll be watching the, you know, I work on my pieces of a run, and then there's also, you know, other people working on it. And, like, do I understand what their code is doing? So that way, like, 12, 30 in the morning, when I'm like, something looks wrong, and I'm, like, looking at this code, can I, like, get context fast enough to understand and throw codex at it? Oh, I use Codex so much. It's really changed how I work. I feel like there's a degree to which, like, sometimes I feel trapped by Codex, because if I spend, like, you know, 30, 40 minutes writing something that looks like a design doc or something, Codex can do more work than I could do in a few hours, in, like, 15 minutes. But then, like, what do I do during those 15 minutes after? And, like, it's actually just, like, really changed how the flow of my day goes, because I have to somehow now manage these, like, 40-minute sessions with, like, 15 minutes where, like, I could do something, but it's actually not nearly as effective as, like, this new flow to the day. So I think I'm still getting used to that, honestly.