You’re pissed off. Our long national nightmare of peace and prosperity is over. Donald Trump is the President-elect. It’s time for a witch hunt. Who better to go after but the person who gave you so much comfort in the last election, “calling” 51 out of 51 and causing a Karl Rove melt down? “Nate Silver, how could you screw this one up so badly?”
People are bad at interpreting probabilities
Nate Silver and the FiveThirtyEight team give probability estimates of events — in this case, the outcome of the 2016 Presidential election. The way to understand that probability is: if you take the information we currently have and hold an election 100 times, Candidate A will win the election X times and Candidate B will win the election 100-X times. It’s the same concept behind die rolling or coin flipping: if you flip a fair coin 100 times, the number of heads you get will be close to 50. It might be 48, or 52, or even 60, but it is vastly more likely to be 49 than it is to be 1. (It’s definitely possible to get 1, it’s just very, very unlikely.) And if you weight each of those outcomes by the frequency of its occurrence, your expected outcome is 50 heads out of 100.
The difficulty with elections is: while it’s easy to flip a coin multiple times, you only get one shot at an election. Real world events, for which Nate Silver and FiveThirtyEight give probabilities, resolve into discrete outcomes. Nate Silver doesn’t get 100 elections to demonstrate that his model is accurate, he gets one every four years.
The upshot of this is that the lay interpretation of probability often reduces to: a greater than 50% probability of a binary (i.e. yes/no) event translates to “I think this event is going to happen,” while a less than 50% probability translates to “I think this event is not going to happen.” In the 2016 Presidential election, a lot of public commentary reflected this thinking: that a 70% probability of Hillary winning is the same as “Hillary is going to win.” And if Hillary doesn’t win, that means the probability was wrong. Beware this thinking — it is seductive, but wrong.
In FiveThirtyEight’s final pre-election “polls-plus” forecast, they assigned a 71.8% probability to Clinton winning vs. a 28.2% probability for Trump. As someone smarter than me pointed out, a VC would consider 30% a high probability. And a .300 batting average in baseball is really, really good. Just because a specific outcome doesn’t fall in line with the >50% predicted outcome doesn’t mean a model is necessarily wrong, or bad, or inaccurate. It just means we live in the timeline where the less likely outcome occurred.
It means that if we have 9 more events with the same probability split (and if those probabilities are accurate), in roughly 2 more of those instances the less likely outcome will occur — and in 7, the more likely one will.
So how do you evaluate a forecaster?
This raises the question: how do you know if a forecaster is any good? Since we only have one outcome, and it was the outcome deemed less likely, how do we know if the probabilities FiveThirtyEight assigned were correct on any level? Isn’t someone who just said “Trump will win” more correct here?
No.
Being a good forecaster is about a reliable, repeatable process more than the perceived accuracy of any one particular outcome. Anyone can be lucky and right once. What takes skill is having a process that can repeatedly get close to the truth.
Here’s how you evaluate a forecaster: take a set of 100 events we would like to have a forecaster predict. The forecaster makes a prediction for each, assigning each event a probability of occurring. To simplify this illustration, let’s assume they assign a 80% probability to each event occurring. If 80 of those 100 events actually happen, then the forecaster is good.
The problem is that this doesn’t fit our mental model of “80% = yes and 20% = no.” But the forecaster was, in fact, correct in their evaluations, and certainly more correct than someone who said that every event was definitely going to happen. Only through repeated testing of a forecaster’s predictions can you understand whether they are any good at the task.
This is why indexing on one specific event as an evaluation of a forecaster’s skill is dangerous and misleading. When Nate Silver “called” the 2012 election “correctly” in all 50 states, he was hailed as an oracle. But in reality, he was just lucky that the states to which he assigned a >50% probability to a specific party went to that same party. And that’s a very specific kind of luck: it’s not that his forecast was actually any better, it’s that people perceived his forecast as being better because the results aligned with the binary way people tend (incorrectly) to interpret probabilities.
The need for transparency
Nassim Taleb made an insightful point about high variance in probabilities implying the forecaster essentially has no clue about the future. It boils down to: if, on one day, a forecaster predicts an 80% probability of an event happening, and after a couple of days they learn new information that causes them to revise their prediction to an 80% probability of the same event not happening, they probably have no idea whether the event is going to happen and should just assign it 50% and go home.
His overall point is that he believes forecasters — even some of the best — are overconfident. How do we distinguish charlatans from forecasters who should be taken seriously?
Forecasters — or some third party — should analyze and publish their track records. Why don’t we treat forecasters the same way we treat any other skill-based occupation with clearly defined outcomes, like athletes? We know who won the World Series and we have metrics to understand who the best pitchers are based on their real-world effectiveness. Forecasters have discrete events that showcase their abilities, so we should compile their track records to expose the frauds and promote the real prognosticators.
What about that AI that predicted a Trump win?
This is where things get really interesting. Although MogIA’s approach appears, on the surface, somewhat reductive (looking primarily at engagement stats, e.g. status updates, video views, etc.), its track record so far has been good. But therein lies the rub: so far. And it’s still unclear what exactly MogIA actually predicted: was it a state-by-state electoral vote model, or was the team just estimating popular vote and inferring who would win the Presidency from that? The distinction matters. The USC/LA Times poll got the “correct” ultimate outcome, but their methodology is clearly not repeatable. While they predicted Trump winning the popular vote by 3%, Clinton is likely to win the popular vote by nearly 2%. So if MogIA was also predicting the popular vote and using that as a proxy for who would win the election, it’s unlikely that its methodology is robust. But without further detail on the team’s modeling, it’s tough to say if it is likely to have continued success.
That specific implementation aside, the beauty of computers (and machine learning) is they can scale previously human-dependent processes beyond imagination. Imagine if you could incorporate essentially every data set available — every bit of social media, every poll, every market indicator (both hyper-regional and global/macro), and beyond — and use that to train your state-by-state (or maybe even district-by-district) model. The limiting factor will be the amount of data you can gather, not the amount of data you can process. More data and more signals will provide truly superhuman capabilities in forecasting. Computers will inevitably surpass Nate Silver’s predictive capabilities (if they haven’t already) by automating the “create model, test model, iterate model” process at a far greater scale than humans can ever achieve. Soon, we should be looking for the Kasparov moment in forecasting.
This, by the way, is exactly the same reason why computers will be better at driving cars than people. They can incorporate and process more data sources more rapidly than humans. They won’t have to check their blind spots; they will be checking them all the time at higher resolution than any human ever could even if completely focused on that one task. The same logic applies to forecasting: computers will be continuously re-evaluating and updating their models about the future. Nate Silver’s days are numbered not because data is wrong, but because data is right and computers are orders of magnitude better at processing data than humans are.
In data we trust
Many real-world tasks can be reduced to “gather data, analyze data, make a decision and take action”. Over millions of years, living creatures have evolved biological systems that allow them to do some of these tasks well because they helped us survive (e.g. identifying and avoiding a hungry predator or spotting and hunting down your next meal) while other capabilities fell by the wayside (e.g. calculating the cube root of 2,384,243 to 7 decimal places). For the latter set of tasks, we created tools to allow us to address them: language, numbers, math, and — perhaps most important of all — the scientific method.
All of this technology has allowed us to live longer, healthier, happier lives. We no longer pray for rain because we have weather forecasting, irrigation, and advanced agronomy. We no longer practice bloodletting because we have antibiotics, advanced medical imaging technology, and the like. Some of the advances that have pushed humankind forward have been serendipitous (e.g. penicillin), but many more others have been the result of structured inquiry: someone forms a hypothesis, designs a test for it, gathers data, analyzes it, and learns from the results. Until recently, computers were merely a tool to facilitate that process. Machine learning and AI represent the potential for computers to take over the whole loop. We can automate the scientific method. Speed of discovery is limited by speed of iteration, and computers can increase the velocity of hypothesis testing by several orders of magnitude.
The enduring appeal of punditry is its ability to tell stories. We prefer to have narratives connecting events in an attempt to understand the world around us. We want to know why an event happened; it’s not enough to just know that it did. Why did Trump do so well in the Rust Belt? Is it because of anger at coastal elites, or is it economic hardship, or are people just really racist and sexist? Ultimately, these questions are all hypotheses, likely answerable with data — just as, hundreds of years ago, heliocentrism was a hypothesis about how our solar system behaved. We just may not have the right data or the right measurement techniques to answer every last question yet.
Nate Silver will be one of the last of his kind: he is political forecasting’s John Henry. While we will continue to appreciate the human element (“artisanal”, “small batch”, “hand-made”, etc.) in fashion, food, and beyond, there is little nostalgia for the era when people routinely died of quotidian bacterial infections. And in the future, few will long for the time when millions of people died every year in car crashes.
Punditry and geopolitical forecasting will follow the trajectory of medicine and driving, and we will be better for it.