Philip Tetlock on the Art of Forecasting

**SPEAKER_1** (0:10)
Get ready to enter a brave new world with your host, Vasant Dhar.

**Vasant Dhar** (0:23)
Hello, and welcome to Brave New World. I want to dedicate this episode to Sunil Thomas, a close friend of mine who passed away last week after a long fight against prostate cancer. As Yogi Berra famously remarked, it's tough to make predictions, especially about the future.
Sunil displayed an amazingly consistent skill at prediction, which he used to generate an annualized average return of over 25% over the last 30 years in the US equity market.
If you do the math, the numbers are staggering. They make Warren Buffett's performance pale in comparison. Maybe he just got lucky, but based on my many regular conversations with him at the Bank Cafe in New York over the years, I felt otherwise. I just couldn't tell what made him unusual until I read Phil Tetlock's work on super forecasters.
Phil has studied forecasting more systematically than anyone. By studied forecasting, I don't mean theorizing about it, but actually measuring the prediction accuracy of thousands of people across thousands of cases where the truth eventually revealed itself. The question is, how do different styles of reasoning translate into accuracy? Do some people just quote unquote have it? Or is it what they do that makes them good? What's even more intriguing is the possibility of using such people who tend to be curious, well-informed and facile with numbers to forecast consequences of actions that have no objectively verifiable answer. Fascinating, isn't it? I certainly think so. So listen carefully. If you're an intellectually curious person, you might learn something from our conversation that could change the way you make predictions, especially about the future. Phil, welcome to Brave New World. I am delighted to have you on the show.

**Philip Tetlock** (2:15)
Oh, thank you.

**Vasant Dhar** (2:17)
So, you know, you've been studying forecasting for a long, long time.
What are your super forecasters telling you about when they expect COVID to be over? That's a half serious question, but I'm curious.

**Philip Tetlock** (2:34)
I suppose it hinges in part on exactly what we mean by COVID, but of all the possible mutations of COVID, I think they'll be floating around for many years.
Have we gone through the worst mortality spikes due to mutations of COVID? I actually don't know what the most recent forecast of the super forecasters are on that. People think that I'm a forecaster or that I'm a super forecaster and I'm not. I mean, I study how people think and the super forecasters are one of my favorite research populations, but I actually, I don't keep a really close track of all their predictions. I mean, that's something that, I mean, they make predictions in finance, they make predictions about U.S.-Russia, U.S.-China relations, they make predictions about all sorts of things. And I'm interested in accuracy, and I'm interested in how styles of reasoning translate into accuracy. But I shouldn't be confused with an epidemiologist or a Soviet or a Russian expert or what used to be called a Sovietologist.
I'm not a subject matter expert in that sense. I'm a cognitive researcher who is very curious about how people think and how styles, which styles of reasoning serve us better or worse in which types of environments.

**Vasant Dhar** (3:48)
Tell us a little bit about how you got started, because you've been studying human decision making for a long time. Yeah, just curious, like what really got you started in this domain?

**Philip Tetlock** (4:00)
Simple answer, exasperation.
I was a relatively young professor in the mid 1980s when Mikhail Gorbachev was coming to power, and I was serving on a couple of committees staffed with experts who were making, who were actually, we're making predictions of sorts about where events were going.
And it turned out that virtually none of the experts whom I knew, there were a few who did get it right, but virtually none of the experts in my initial population anticipated Gorbachev. The more liberal observers expected that Reagan was driving the Soviet Union into an ever harder line stance, and he was basically fueling neo-Stalinism. And it'd be, the successor to Chernenko would be tougher. And the conservatives thought that, you know, Soviet Union really is never gonna change. It's a kind of a self-reproducing totalitarian system, so we should expect more of the same. We'll eventually wait them out, but don't expect any big changes. So Gorbachev was essentially a phenomenon that neither liberals nor conservatives anticipated in advance, but within about six to 12 months, when it became apparent that Gorbachev really was pursuing a rather radical liberalization agenda in the Soviet Union, both liberals and conservatives thought they could explain it perfectly. The liberals thought, well, you know, of course this is happening. It didn't happen because of Reagan. It happened because the Soviet economy was collapsing and they realized they couldn't keep going on that way. And the conservatives said, yeah, of course it's happening, but it's due to Reagan. You know, we ratcheted up the pressure in the arms race and they're blinking.