We are drowning in data. In this age of analyzing everything from exit, phone and online polls to the sentimentality of Tweets and the Like, we have managed to turn the electoral process into a chess match of data. For the data wonks in the room, we have died and gone to algorithmic heaven. For the rest of the population, this has resulted in an epidemic of extraneous “fact dropping” behavior — my poll is better than your poll, my data is better than your data or, in the most extreme case, the disregard of data because facts should not, after all, get in the way of politics, campaigning or governing.
Now, if you don’t know who Nate Silver is, keep on reading. (If you do know, keep on reading as well.) Nate is “a statistician and political forecaster at The New York Times who became a national sensation in the United States when his predictions during the 2008 presidential election trumped most mainstream polls.” Nate called the election in ’08 based on a set of models and data that take into account elements of the market most everyone else ignores – including calculating for “house effects” (the tendency for polling firms to favor either the Democratic or Republican candidate) and what he calls the prediction paradox – the more confidence one has in the data, the less uncertainty, the more favorable a prediction. The blog, FiveThirtyEight, is a treasure trove of data AND analysis, something that most of the mainstream media completely and utterly lacks. Reports, posts and sound bites about the “latest” poll do nothing but decrease the signal to noise ratio.
As we enter the last days of this Presidential race, challenge yourself to take a step back and look beyond the current data, apply the analysis, and look into the future using the past. Make sense? A single data point is useless. What you should be looking for are trend lines, patterns, elements of symmetry in the data that can reasonably predict the future based on the past. History is the best teacher and a reliable lens into the future. The questions remain: Is the average American prepared to do the work? Ask the hard questions? Look at data outside their comfort zone? (For an excellent exploration of how we seek facts and data that fit our worldview and the damage this can cause, refer to Michael Specter’s Denialism: How Irrational Thinking Hinders Scientific Progress, Harms the Planet, and Threatens Our Lives.) If the answer is yes, then you are on your way to making the right decision based on all the data. Congratulations. If the answer is no, then will all due respect, keep your data to yourself.
For those who are really interested in getting under the hood of statistics and probably Khan Academy is an excellent resource. You can begin here and here.