There are a lot of smart, knowledgeable people in the world. The problem with humans isn’t a lack of knowledge. It’s that we always think we know far more than we do! Choosing a good path forward requires making predictions about the future. Here’s today’s reminder about how homo-sapiens perform at predictions.
This task was so much easier than everything else we ask our experts. They were predicting an event less than two weeks out. Almost every question in politics involves making some very difficult predictions about how the future will play out in the years or decades ahead. But this one? This is a softball question compared to almost everything else. Since the timeline is so short, we can actually check and see how they did.
But I’m actually not upset that the experts were off by a factor of five – that’s not the problem. Prediction is hard. The scary part is just how laughably overconfident almost every single one of them was. For most of them, their “high-end estimates” were off by just as much as their best guess. Yikes! In hindsight it’s clear that these people had no idea what to expect, so why did they try to thread the needle through such tiny confidence intervals? If you’re not sure what’s going to happen, that’s fine – the future always has lots of uncertainty! Thus you should increase your error-bars to make sure reality falls somewhere within your prediction range.
See the one guy third from the top who allowed for lots of uncertainty and used really wide error-bars and estimated that the number of cases would be anywhere up to half a million? He was one of the best predictors of the group, and yet people probably laughed at him when he made that prediction. They probably laughed at how imprecise his prediction was. But he very correctly concluded that when there’s a lot of uncertainty involved, it’s appropriate to include that uncertainty in your forecast. Otherwise, you’ll be just another overconfident “expert” who is wrong about everything.
Unfortunately, it gets even worse. You know what the REALLY scary part is? Not only are humans the type of creature that would laugh at one of our best forecasters, they would laugh at him even after we see the graph where he’s proven right.
NO NO NO NO NO NO NO!!!!!!!! GOOD LORD NO!!!!!!!!!!! Of all the people to make fun of in that graph, he’s just about the last one that deserves it. His prediction included information, which is more than most of them can say.
Let me put it like this:
Imagine you’re in charge of a manhunt. You call in one of your detectives to ask him where the suspect is. The detective says, “He’s at 14 Avacado Drive in Austin, Texas. He’s currently on the second floor, lifting weights while watching Jurassic Park on tv.” Then you ask a second detective, who says, “All I can say for sure is that’s he’s somewhere on the Eastern Seaboard.” An hour later the suspect is found in Maine. Which detective gave the best info?
The next time you see someone confidently explaining what’s going to happen a few years in the future, remember this graph of experts predicting two weeks out. When you’re trying to decide which sources to trust, try to discern which ones are more like the top three on this chart, acknowledging uncertainty when appropriate.
Superforecasters Michael Story and Tom Liptay wrote a piece defending these experts a bit. I hope I wasn’t too critical of the experts themselves. The point I most wanted to make is that as a society we need to get better at handling uncertainty, and at the very least, we need to stop making fun of people who get things right. That seems like a low bar.