Wrong numbers and why they survive, with Aaron Brown

**Patrick McKenzie** (0:02)
Welcome to Complex Systems, where we discuss the technical, organizational and human factors underpinning why the world works the way it does.
Hi, everybody. My name is Patrick McKenzie, better known as patio11 on the Internet. And I'm here with Aaron Brown, who is the author of Wrong Number and the adjacent series of videos on reason about various anecdotes of misuse of statistics in publishing and other places. Aaron, thanks very much for coming on the program today.

**Aaron Brown** (0:36)
Thank you for having me, Patrick.

**Patrick McKenzie** (0:38)
So I usually don't take pitches for this. I find my own guests out in the universe, but you had an extremely interesting motivational anecdote in my inbox about how two organizations that you would expect to understand the demand curve for diesel fuel, both equally confidently advanced to very, very different hypothesis of what that demand curve looked like.
Can you let the rest of the world know about that? Because it's one of those things where it's like, okay, clearly one of the two can't be right.

**Aaron Brown** (1:11)
Yeah, and I use this in the book because it was sort of my foundational. It set me on the path so that 50 years later, I end up writing this book. So I'm a freshman and I get introduced to Fisher Black, who in those days, most of your listeners probably know of him as the famous finance professor of Black-Scholes model, but I had no idea at that time who he was. And he wasn't knowing it for that time for anything famous.
But I was the closest thing he ever had to a student. He was not a nurturing, mentoring kind of guy. And so I ended up having a long career working with him. But anyway, so I'm a freshman and I need a summer job.
And he recommended me for the National Standby Gas Rationing Project. So this is 1975 And it's a real possibility that we're going to have to ration gasoline. And so the federal government wanted a standby plan that they could put in for us. And I will say Fisher was totally skeptical that this plan would ever do anything good. And so him recommending me for it probably said more about what he thought about the plan than what he thought about me. But I did get this job. I was actually supposed to go to a senior graduate student, but he figured, why waste? Why waste somebody like that on it? So anyway, I get on this project, and I am assigned worrying about fuel for agriculture. If we're going to ration gasoline, we want to make sure that the crops get harvested and delivered.
And I'm a city boy. I have no idea where tractors run on gasoline or diesel fuel. So this is the first thing I have to figure out. So I call the Department of Agriculture and get hazed around. Yeah, here's a guy, he's just finished a big model about it, but he's out of the office right now. So we leave a message for him to call me back. And I call Ford because Ford makes most of the tractors and I get passed around again. A guy and a guy get to a guy and he said, yeah, I just finished a comprehensive economic model and 75% of the tractors in the country are diesel. Great, thank you. You know, that's what I need. Then I should talk to him a bit and got a lot of detail. How do you know this? Where are they? What's the life cycle and so on? Then I get a call back from the ad guy that afternoon and I want to be polite. I don't want to just tell him I already got it. You don't have to bother. So he says to me, yeah, I just finished my model and 75% of the tractors in the country are gasoline. I said, oh, you need diesel, don't you? He said, no, no, no, gasoline. I said, well, the Ford guy told me diesel. Oh, he said that he's talking about new tractors being produced. He's not talking about the existing stock and that's just not true, by the way. I went with the Ford guy. The Ford guy later said, oh, the agriculture department, they're talking about tractors rusting in barns or shipped to Mexico decades ago. You know, they don't, also not true. You know, both of these people had data and whatever. So what hit me about it was just they were disagreed about such a basic thing, but that both of them wanted to teach. Neither one of them was interested in learning. They didn't want to reconcile their number. They didn't want to, you know, subject it to skepticism. They just wanted to be an expert. But that's not the end of the story. Believe it or not, that's not the worst part. The worst part is I told this story to the senior economists and universally everyone had the same reaction. They laughed and they topped it with an even more outrageous story. So I would say, well, how do we have any idea that our project is going to be, if we can't establish whether tractors are diesel or gasoline, how are we going to design a plan for the whole country? And none of them, I don't know, it just didn't get through to them. The question just sort of wished off. And I said, we should talk to some of these Soviet Union, Gazplan people because they've got 50 years experience actually doing this. Maybe they should tell us something. Then no, no. Communists are idiots. They don't understand economics. We can't learn from them. And I said, okay, well, let's talk to the people who did rationing in World War II. Oh no, they weren't economists. They were just amateurs. They don't know anything that could help us. So, so the plan got written and I, you know, I am very confident the plan would have been a total disaster had it ever been implemented. So I want to focus on the thing about the book is, what I'm really interested in is not just why people make mistakes. How did the Ford guy get 75 and how did the ad guy get his number and which one was right or neither or whatever. It's there's an entire institutional structure that ignores this problem. That, okay, I understand why a researcher makes a mistake and gets wedded to a number. What I don't understand is why nobody cares.