How OpenAI Built Its Coding Agent

**Anjney Midha** (0:00)
What happens when AI stops helping you autocomplete code and starts acting like a real teammate? Today, on AI plus a16z, we're exploring Codex, OpenAI's coding agent. Anjney Midha is joined in studio by Alexander Embiricos who leads product for Codex at OpenAI. They discuss the origin story, why reasoning models plus tools unlock agents, how developers are actually using Codex in the wild, and what all this means for the future of software engineering, from debugging and prototyping to how CS students should think about their careers. Let's get into it.

**Anjney Midha** (0:38)
Hey, Alex.

**Alexander Embiricos** (0:39)
Hey, how's it going?

**Anjney Midha** (0:40)
Good, thanks for coming.

**Alexander Embiricos** (0:41)
Yeah, good to see you again.

**Anjney Midha** (0:42)
You are one of the folks working on product for Codex, which is probably one of the most exciting launches to come out of the OpenAI team, for me at least, in a while. So, for a lot of people, though, it was confusing.

**Alexander Embiricos** (0:58)
For sure.

**Anjney Midha** (0:58)
Because it was the fifth Codex release from OpenAI. But of course, it's completely new and different from the previous Codexes. So, let's just start with the origin story. What is the backstory on how the current version of Codex came to be?

**Alexander Embiricos** (1:12)
Yeah. And man, our naming is so fun at OpenAI. I'm excited for the naming to make more sense over time with Codex, as we bring this all together. But yeah, let's go back way back to the beginning. The first Codex product was actually released, I think it was in 2021, I might get the year wrong. But actually, it was like a code completion model that powered GitHub Copilot. And so recently, we were basically talking about a whole bunch of coding stuff we want to do, like models, but like models in product. We were thinking about what to call it, and we just felt like the Codex name was really cool, and so we wanted to go back to it. So how did this Codex product come about? Basically, we've been thinking a lot about agents, as everyone has, and before that, we've been thinking about reasoning models. And basically, in our minds, one way you could think about an agent is you take a reasoning model and then you give that reasoning model access to the tools that some agent would want to use or some human and given function would want to use, and an environment that tool works with, take side effects of it, and then from there, you come up with what kind of tasks would this person do. So basically, you have this model, you give it tools, and then you make sure that the model is really good at doing the specific tasks that some function would do.
And the task bit is actually super important, because if you think of, like, there's a difference between, like, writing and journalism, and similarly, there's a difference between, like, coding and, like, software engineering. So we've been doing a lot of this tinkering with reasoning models internally, getting them to write code, and so the first tool we were given was like terminals. And we've been, like, poking at this for a while, and just starting, it was, like, actually one of the first, like, real, like, fieldy AGI moments for me was when someone showed me a website editing itself by being prompted itself, because we had this, like, reasoning model, like, very hackily connected to a terminal. And then, you know, it was editing this terminal.

**Anjney Midha** (2:57)
It was just editing the DOM basically directly as a CLI.

**Alexander Embiricos** (3:00)
Yeah, exactly.

**Anjney Midha** (3:00)
Okay.

**Alexander Embiricos** (3:01)
Well, it wasn't the DOM directly, it was React, but, like, whatever, you know? And it was, like, super cool.

**Anjney Midha** (3:04)
How was it parsing the visual? Did you give it access to a browser?

**Alexander Embiricos** (3:08)
No, it was, like, I like to use this term, like, site reading. It was just, like, site reading the code. So it wasn't, like, taking screenshots of itself or any of this, like, stuff that now, like, people are building.

**Anjney Midha** (3:16)
Okay.

**Alexander Embiricos** (3:16)
It was just, like, editing React. And so we had this prototype, like, a while ago, and just people internally, like, really loved it. So we were starting to, like, write more and more code. And then we were starting to think about, like, okay, well, you know, what is the right form factor for this thing? When it's editing code, it's, like, pretty great. Like, on my computer, it's pretty great, but it's, like, you know, quite annoying to only have it able to work on one thing at a time. It's also, like, a giant safety and security question if you just have this, like, agent, like, unleashed entirely on your computer. And so around this time, we started exploring, like, a lot of different places to put this reasoning model that has access to a terminal. And so we had a prototype that, like, ran in CI when your, like, tests failed. We had a prototype that, like, through some crazy hack, like, automatically fixed your, like, linear issues. But that was actually running in CI. We had this prototype that was, like, running on your computer. And so basically, the Codex product we launched was, like, a distillation of that, where we thought, OK, well, what is the most powerful incarnation of this? And we figured, you know, like, if you think about, like, what an agentic teammate will be like in the future, you'll, like, hire them, you'll tell them what their job is, give them some compute or a laptop, and give them some permissions, and then they'll, like, go off and do work. And so we figured, OK, this is going to be kind of, like, a strange, like, unwieldy research preview, but let's, like, put all our, or, like, the vast majority of our effort into this form factor of an agent working remotely, and, like, kind of see what happens. And so that led to the Codex product that released, which is, like, a cloud agent that can, you know, basically answer questions and write PRs in the background.