From Poetry to Programming: The Evolution of Prompt Engineering with Riley Goodside of Scale AI

**Erik Torenberg** (0:00)
Hi, everyone. Excited to announce a new podcast that just launched from Turpentine, Complex Systems, with Patrick McKenzie. Patrick, who is better known as Patio11 on the internet, thinks a lot about systems, software, financial infrastructure, and so on. If you're tired of hearing that everything is broken, this podcast is for you.
Patrick surfaces conversations with experts who actually built and understand the complicated but not unknowable systems we rely on. You might be surprised at how quickly Patrick and his guests can put you in the top 1% of understanding for stock trading, tech hiring, and more. Subscribe to Complex Systems with Patrick McKenzie everywhere you get your podcasts or at the link in the description.

**Nathan Labenz** (0:41)
Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg.
This episode is brought to you by WorkOS. If you're building a B2B SaaS application, at some point your customers will start asking for enterprise features like SAML authentication, skim provisioning, role-based access control, and audit trails. That's where WorkOS comes in, with easy to use and flexible APIs that help you ship enterprise features on day one without slowing down your core product development. Today, some of the hottest startups in the world are already powered by WorkOS, including ones you probably know like Perplexity, Vercel, Jasper, and Webflow. WorkOS also provides a generous free tier of up to 1 million monthly active users for user management, making it the perfect authentication and authorization solution for growing companies. It comes standard with rich features like bot protection, MFA, roles and permissions, and more. If you're currently looking to build SSO for your first enterprise customer, you should consider using WorkOS. Integrate in minutes and start shipping enterprise plans today.
Hello, and welcome back to The Cognitive Revolution. This is Nathan's AI voice clone, powered by 11 Labs. The real Nathan is in Brazil this week to give a presentation on AI automation at the Adapta Summit in Sao Paulo, and today's episode, featuring returning guest Riley Goodside, the world's first staff prompt engineer at Scale AI, turned out to be a perfect prelude to that presentation, which we'll share here soon as well. Going back to 2022 and into early 2023, Riley became famous in the AI community for coming up with one clever prompting trick after another. He was one of the first to put large language models into a read, evaluate, print loop, or REPL, which most of us would now recognize as a precursor to AI agents. And I think you could make a strong case that he had demonstrated the deepest, most practically useful understanding of language models' idiosyncrasies, and outright weirdness of anyone in the world.
Today, our conversation reflects how much LLMs have continued to progress, even since GPT-4. As models have undergone more and more post-training, the need for quirky tricks has declined. And, as Riley puts it, prompt engineering has become less like poetry and more like programming. Meanwhile, in the enterprise context that Scale AI serves, companies are starting to move beyond ad hoc chatbot interactions, and instead using the full range of best practices, including curating gold standard examples, capturing reasoning traces, all sorts of retrieval, augmented generation, fine-tuning, and anything else they can come up with.
All to push language models to their performance limits, with the goal of achieving human-level performance or better, and ultimately saving serious time and money on routine tasks.
It was a treat for me to dig into these topics with Riley, and I was glad to find that our approaches are mostly in sync. For anyone building AI systems, this episode is packed with practical value. And of course, I couldn't help myself from sneaking in some questions about jailbreaking, the prospect for superhuman intelligence, AI safety, and more along the way as well. As always, if you're finding value in the show, we'd love it if you'd share it with a friend. Your support helps us continue bringing you conversations with the leading minds in AI. The real Nathan will be back soon, but for now, he hopes you enjoy this deep dive into the evolution of prompt engineering with Riley Goodside of Scale AI. Riley Goodside, the world's first staff prompt engineer of Scale AI. Welcome back to The Cognitive Revolution.

**Riley Goodside** (4:34)
Thank you. It's great to be back.

**Nathan Labenz** (4:36)
It's been a minute, and obviously a lot has happened.
In preparing for this, I went back to your Twitter feed, which was, of course, your original claim to fame in the AI space with lots of super interesting examples coming out of the Tex DaVinci 2 era. And I found you've been a little quiet on Twitter recently. I think the AI community has certainly grown. There's lots of people doing that kind of stuff these days. But what have you been up to that's had you quiet online in recent months?