From API to AGI: Structured Outputs, OpenAI API platform and O1 Q&A — with Michelle Pokrass & OpenAI Devrel + Strawberry team

**Charlie** (0:40)
welcome back. This is Charlie, your AI co-host. Michelle Pokrass built massively scalable platforms at Google, Stripe, Coinbase, and Clubhouse, and now leads the API platform at Open AI. She joins us today to talk about why Structured Output is such an important modality for AI engineers that Open AI has now trained and engineered a Structured Output mode with 100% reliable JSON Schema Adherence. To understand why this is important, we have to go all the way back to June of last year when Open AI first added a function calling capability to Gpt-4-0613 and GPT 3.5 Turbo 0613, which was then followed by November's Dev Day, where the team shipped JSON Mode, a simpler Schema-less JSON Output mode that nevertheless became more popular because function calling often failed to match the JSON Schema given by developers. Meanwhile, in open source, many solutions arose, including Instructor and Langchain, from our former guests Jason Liu and Harrison Chase, who by the way is returning to co-host an Agents episode soon, and Outlines from World's Fair speaker Remi Louf, and Llama.cpp's constrained grammar sampling using the GGML extension of the Bacchus-Nauer form or BNF syntax. Fast forward to April of 2024, Open AI started implementing constrained sampling with a new tool choice required parameter in the API, and finally in August closed the loop by releasing the new Structured Output mode, which extends constrained sampling with specific post-training to improve the performance of complex JSON schema following, especially with the strict true flag.
The other big labs seem to be following suit with Gemini Shipping Structured Outputs and Enum mode in this past month. We sat down with Michelle to talk through every part of the process, as well as quizzing her for updates on everything else the API team has shipped in the past year. From the Assistance API to Prompt Caching, Gpt-4. Vision, Whisper, the upcoming Advanced Voice Mode API, Open AI Enterprise features, and why every Waterloo grad seems to be a cracked engineer. In Latent Space Community News, if you're in Germany, the first meetup for AI engineers is happening in Cologne in two weeks, and the second AI engineer summit, the curated invite-only conference run by the AI engineer World's Fair team, is now moving to January in New York City. See the show notes for details. Watch out and take care.

**Alessio** (3:34)
Hey, everyone. welcome to the Latent Space Podcast. This is Alessio, Partner and CTO in Residence and Decibel Partners, and I'm joined by my co-host, Swix, Founder of SmallAI.

**Swix** (3:43)
Hey, and today we're excited to be in the in-person studio with Michelle. welcome.

**Michelle Pokrass** (3:48)
Thanks. Thanks for having me. Very excited to be here.

**Swix** (3:50)
This has been a long time coming. I've been following your work on the API platform for a little bit, and I'm finally glad that we could make this happen after you ship Structured Outputs. How does that feel?

**Michelle Pokrass** (4:01)
Yeah, it feels great. We've been working on it for quite a while, so I'm very excited to have it out there and have people using it.

**Swix** (4:07)
We'll tell the stories soon, but I want to give people a little intro to your backgrounds. So you've interned and always worked at Google, Stripe, Coinbase, Clubhouse, and obviously OpenAI. What was that journey like? The one that has the most appeal to me is Clubhouse because that was a very, very hot company for a while. Basically, you seem to join companies when they're about to scale up really a lot, and obviously OpenAI has been the latest. What are your learnings and your history going into all these notable companies?

**Michelle Pokrass** (4:35)
Yeah, totally. For a bit of my background, I'm Canadian. I went to the University of Waterloo, and there you do like six internships as part of your degree. So I started, actually my first job was really rough. I worked at a bank, and I learned visual basic, and I like animated bond yield curves, and it was not-

**Swix** (4:51)
Me too.

**Michelle Pokrass** (4:52)
Oh, really?

**Swix** (4:52)
Yeah, I was a derivative trader. Interest rate swaps, that kind of stuff.

**Michelle Pokrass** (4:56)
Yeah, so I liked having a job, but I didn't love that job. And then my next internship was Google, and I learned so much there. It was tremendous. But I had a bunch of friends that were into startups more, and, you know, Waterloo is like a big startup culture, and one of my friends interned at Stripe, and he said it was super cool. So that was kind of my- I also was a little bit into crypto at the time, then I got into it on Hacker News, and so Coinbase was on my radar. And so that was like my first real startup opportunity, was Coinbase. I think I've never learned more in my life than in the four-month period when I was interning at Coinbase. They actually put me on call.