Delving into The Prompt Report, with Sander Schulhoff of LearnPrompting.org

**Erik Torenberg** (0:00)
Hey everyone, Erik here.
At Turpentine, we're building the first media outlet for tech people by tech people. We produce the show you're listening to right now. We're also building a network product that is bringing together the best founders and executives in the game from all stages.
The Turpentine Network is a high trust space where exceptional people can talk. We already have over 400 members and growing. The community benefits include tactical advice, tech stack recs, hiring referrals, in-person events, an investor database, and exclusive perks. If you want to join, you can apply at the link in the description.

**Nathan Labenz** (0:35)
Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to welcome back Sander Schulhoff, creator of learnprompting.org and organizer of the Hack-a-Prompt contest that we discussed when Sander was last on the show back in January. This time, Sander's here to talk about the recently released Prompt Report, a mammoth 78-page survey paper covering the current state of prompting techniques for large language models.
As AI capabilities have exploded over the last couple of years, so too have the number of papers exploring how to get the most out of these models through clever prompting techniques. Sander and his team took on the Herculean challenge of reviewing and categorizing this fast literature to create a comprehensive taxonomy and guide to the field. In today's conversation, we delve deep into some of the key findings and insights from this work. We discussed best practices for few-shot prompting, the challenges of ensembling and evaluation for open-ended tasks, multilingual and multimodal techniques, the current state of prompting for AI agents, and even how automated prompt optimization systems like DSPy can outperform human prompting experts like Sander. A result that I personally can't help but see as a sign of things to come.
Beyond the technical details, Sander also shares his experience leading a large research team and offers reflections on trust, testing, and project management that I think will be valuable for anyone embarking on a similarly complex technical research project.
As always, if you're finding value in the show, we'd appreciate a review on Apple Podcasts or Spotify, or you can just share it online with your friends. If you have any feedback or questions, you can always reach us via our website cognitiverevolution.ai or you can DM me on your favorite social network. Finally, for now, a quick update on our jobs board. I'm glad to say that I have made the first of what I hope will be many very high quality connections between someone looking for a new role and a very interesting startup.
In this case, the startup happens to be in stealth mode for a little longer yet, but I hope to tell you more about that soon. And in the meantime, I hope that encourages you to submit your resume if you're interested in a new opportunity and haven't already done so. Now, I hope you enjoy this wide ranging discussion on all things prompt engineering with Sander Schulhoff of learnprompting.org. Sander Schulhoff, creator of learnprompting.org, previously here to talk about the Hack-a-Prompt contest and paper and now returning to talk about The Prompt Report. Welcome back to The Cognitive Revolution.

**Sander Schulhoff** (3:24)
Thank you very much.

**Nathan Labenz** (3:26)
So, you are prolific. I wanted to start off by just talking a little bit about what you do and how you're doing it all. You've got the LearnPrompting, which originally a project, last we talked, had kind of evolved into a startup. And then you've put together a couple like mega papers along the way as well. So, what does the Sander Schulhoff portfolio look like these days? What's the status of learnprompting.org as a business?
And how are you finding time to bring these teams together and do this deep analytical work at the same time?

**Sander Schulhoff** (4:00)
Yeah. So, starting with LearnPrompting as a business, we are currently building out our generative AI course offerings and building up the team, actually looking for a first full-time first hire at the moment, something of generalist to help with a bit of video creation, a bit on the business side. And recently, I've been experimenting a lot with automating content creation for like information teaching. So, there's a lot of YouTubers out there, there's a lot of content on Coursera, all these other platforms, talking to some C-suite execs at like consulting firms.