Scaling Forecasting: AI Forecasting Tournaments & the Road to Epistemic Security, with Deger Turan, CEO of Metaculus

**SPEAKER_1** (0:02)
Hey, everyone. I'm excited to tell you about our newest show on the Turpentine Network called 1 to 100 1 to 100 is a weekly deep dive with founders of notable hyper growth companies that could be your next career move. On the upcoming season, you'll get the inside scoop on breakout companies like Modal Labs, Clay, and Mercor. You'll hear directly from the founders about how they built their products, shaped their company cultures, and scaled their businesses from 1 to 100 and beyond. If you're considering a career change or just want to stay ahead of the curve in tech, 1 to 100 will give you an edge. A link is in the description. Listen wherever you get your podcasts.

**Nathan Labenz** (0:40)
Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. This episode is brought to you by Work OS. If you're building a B2B SaaS application, at some point, your customers will start asking for enterprise features like SAML authentication, SCIM provisioning, role-based access control, and audit trails. That's where Work OS comes in. With easy to use and flexible APIs that help you ship enterprise features on day one without slowing down your core product development. Today, some of the hottest startups in the world are already powered by Work OS, including ones you probably know like Proplexity, Vercel, Jasper, and Webflow. Work OS also provides a generous free tier of up to one million monthly active users for user management, making it the perfect authentication and authorization solution for growing companies. It comes standard with rich features like bot protection, MFA, roles and permissions, and more. If you're currently looking to build SSO for your first enterprise customer, you should consider using Work OS. Integrate in minutes and start shipping enterprise plans today.
Hello, and welcome back to The Cognitive Revolution. Today, my guest is Dere Turan, CEO of Metaculus, a leading forecasting platform that aims to bring epistemic security to how we understand the future. Forecasting has long fascinated me. I participated in one of Philip Tetlock's original super forecasting tournaments some 15 years ago, and I actually did reasonably well, though I was not ultimately named a super forecaster. Since then, I've often visited forecasting sites like Metaculus to take a pulse on future events, and I've always intuitively believed that forecasting could really help most organizations to make better decisions. And yet, despite its promise, forecasting has not become a mainstream technology. That's in part because it is a ton of pain in the brainstaking work to do forecasting well, and thus really hard to scale to the point where forecasts become consistently relevant to decision makers as they operate in highly contextual situations. As a case in point, Metaculus has hosted some of the most referenced AGI timeline forecasts for years, and still today, fewer than 1,500 people have registered their own individual forecasts. If you've listened to this show for any length of time, you can probably guess where this is going next. What if we had AIs do the forecasting? It turns out that with some real effort required, and some important remaining caveats, this already works. Multiple recent papers, including some by Tetlock and others, show that AI systems forecasts are competitive with the best individual humans, though they do still fall somewhat short of the wisdom of human crowds. Der and I first review this literature together, and then go on to discuss Metaculus' reasons for gathering more complex probability distribution style predictions, rather than simply seeking to discover market prices. We also review the community's track record on major world events, and we get into Metaculus' ambitious new AI Forecasting Benchmark Tournament. This is a year-long competition that aims to advance the state-of-the-art in AI Forecasting capabilities, with free API credits available and meaningful prizes to be awarded quarterly. Beyond that, we explore Deger's vision for the future of forecasting, not just as a way to predict outcomes, but as a tool for building better world models and improving collective decision-making. His ideas on using forecasting for resource allocation, policy development, and even enhancing democracy are both exciting and thought-provoking. And another example, not only of how AIs already superhuman strengths, in this case scalability, have the potential to change the world even if the AIs perform no better than humans on a given task, but also how AIs are meaningfully generalizing beyond their training data. After all, the future is in some sense definitionally out of distribution.