The Politics of Prediction

Too Much Data

Too Much Data

We are drowning in data. In this age of analyzing everything from exit, phone and online polls to the sentimentality of Tweets and the Like, we have managed to turn the electoral process into a chess match of data. For the data wonks in the room, we have died and gone to algorithmic heaven. For the rest of the population, this has resulted in an epidemic of extraneous “fact dropping” behavior — my poll is better than your poll, my data is better than your data or, in the most extreme case, the disregard of data because facts should not, after all, get in the way of politics, campaigning or governing.

Now, if you don’t know who Nate Silver is, keep on reading. (If you do know, keep on reading as well.) Nate is “a statistician and political forecaster at The New York Times who became a national sensation in the United States when his predictions during the 2008 presidential election trumped most mainstream polls.” Nate called the election in ’08 based on a set of models and data that take into account elements of the market most everyone else ignores – including calculating for “house effects” (the tendency for polling firms to favor either the Democratic or Republican candidate) and what he calls the prediction paradox – the more confidence one has in the data, the less uncertainty, the more favorable a prediction. The blog, FiveThirtyEight, is a treasure trove of data AND analysis, something that most of the mainstream media completely and utterly lacks. Reports, posts and sound bites about the “latest” poll do nothing but decrease the signal to noise ratio (see my earlier post and The Signal and Noise).

As we enter the last days of this Presidential race, challenge yourself to take a step back and look beyond the current data, apply the analysis, and look into the future using the past. Make sense? A single data point is useless. What you should be looking for are trend lines, patterns, elements of symmetry in the data that can reasonably predict the future based on the past. History is the best teacher and a reliable lens into the future. The questions remain: Is the average American prepared to do the work? Ask the hard questions? Look at data outside their comfort zone? (For an excellent exploration of how we seek facts and data that fit our worldview and the damage this can cause, refer to Michael Specter’s Denialism: How Irrational Thinking Hinders Scientific Progress, Harms the Planet, and Threatens Our Lives.) If the answer is yes, then you are on your way to making the right decision based on all the data. Congratulations. If the answer is no, then will all due respect, keep your data to yourself.

For those who are really interested in getting under the hood of statistics and probably Khan Academy is an excellent resource. You can begin here and here.

Author:

Lydia Loizides

Lydia Loizides is serial entrepreneur, technology provocateur and relentless challenger of the status quo. She spends her days as Founder & CEO of Talentedly, a technology company on a mission to help people grow from good to great at work (technology + people = amazing results). The rest of her waking moments are spent running, reading, learning, and trying to prove that the answer to the ultimate question of life, the universe and everything is 42. You can follow Lydia @lydiaNYC @GetTalentedly, on LinkedIn and the Huffington Post.

  • Jaffer

    Lydia, I take issue with the statement: ” History is the best teacher and a reliable lens into the future.”. History MAY be reliable…but not REFLEXIVELY a reliable lens into the future. If it was reliable, we would be able to predict Arab Spring…and a host of other events.
    If the road ahead resembles the road traveled, then looking into teh rear view mirror is reliable. But I believe we can list hundreds of examples of history being a poor lens into the future. I give a serious nod to Taleb here…

  • Paula Lynn

    So glad to see you again, Lydia, so to speak. The Tulip crisis in Holland that crashed the European market of the day during the 16th (or 17th) century predicted the credit default system’s default. The “experts” just don’t like it and deny.