Insightology

You can read my previous two election posts here and here. In this post I will take a look at who got the election right and what factors to look for in making that evaluation. Those factors include single polls versus (simple and complex) poll aggregations, use of combination forecasting, the use of cell phone only households in surveys and the astonishing performance of quantitative models that accurately predicted the final results almost a year back. Keep in mind that as of this writing, the final results are still not in both in terms of vote share and in terms of states (Missouri). That said, the results are close enough that we can get a good idea of what went right.

The first thing to note is that this election was not as close as the last two presidential elections. Generally speaking pollsters are going to perform better when the race is not close because there are fewer opportunities for states to flip and cause embarrassment. So as expected, many national polls did well in coming close to the actual vote share (52.5% to 46.2% in favor of Obama at this writing). But as has been pointed out by others, there is also a tendency for national polls to "converge" right at the end and some people attribute this to a tendency on the part of pollsters to adjust their final projections by looking at other polls. Given that I've argued in previous posts that we shouldn't be looking at single polls, it is enough to say that they generally got it right, rather than try to figure out who came closest. 

The polling aggregation sites such as RCP, fivethirtyeight, and pollster.com  also did very well. As I explained in previous posts, aggregating polls reduces error and provides more stable estimates than looking at single polls. The Wall Street Journal's Carl Bialik has a good write-up on how various polls and poll aggregators performed.

But not all poll aggregators are created equal. Most, like RCP, use a simple average of the various polls. A more complex method was used by Nate Silver (a sabermetrician who has turned his baseball stats analysis skills to politics) at fivethirtyeight.com who weights the different polls in his aggregation based on past performance of those polls. He also runs a simulation of the election 10,000 times every time new information becomes available in order to get a probabilistic assessment of the election outcome. This is similar to the kind of analysis sabermetricians perform to figure out the playoff odds of various baseball teams. He came very close in terms of vote share (52.3/46.2) and electoral college votes (349 for Obama).

Do cell phone only households (currently estimated at about 15% of the country) have an impact on polling results? National polls that included cell phone only households were predicting a bigger margin for Obama than those that didn't. Here is a comparison of those with a nice chart. It looks like there was no advantage to including cell phone only households in the surveys.

What happened to undecided voters? Did they break disproportionately for one candidate? The two cell phone charts seem to show that McCain's support was underestimated more than Obama's support, perhaps implying that undecided voters broke more for McCain. But to get a better idea across a range of polls consider this interesting trend chart.  It shows the results of many polls over time. Notice how the trend line for Obama keeps rising from mid-September indicating undecided voters may have begun to break for him then. McCain's trend line is more interesting, indicating that he started losing support around mid-September and by the time of the final polls had only managed to gain back the support he lost (52/44). But the actual vote count shows that he got around 46% indicating that most of the election day deciders went for him. So while it is quite likely that the election day undecideds went disproportionately for McCain, over the final month or so they had been breaking for Obama.    

Finally, how did combination forecasting perform in this election? Recall that this method combines the predictions from different methods (survey aggregation, election markets, expert surveys and quantitative models) to provide a forecast. Their final estimate of the two party vote was 53/47 which is a hair away from the actual number of 53.2/46.8. None of their individual components produced drastically different results either. In terms of electoral college votes, their final estimate of 354 is quite close to the final outcome (365 for Obama with Missouri outstanding).        

Now, a word about quantitative models. These are usually not based on polls and tend to use macro economic indicators and factors like party of the incumbent president to identify who is going to win the election. The actual candidate and tactics are not considered. As shown here the average score of these models have been consistently telling us since January 2008 that the Republican nominee was going to lose the election. That's right. Since well before it was known who the Democratic and Republican nominees were, these models have been consistently arguing that the Republican nominee was going to get only about 47%-48% of the two party vote. How remarkable is that, considering everything that has happened over the last few months and that the final Republican vote share is about 46.2%.     

Overall we can say that this election was kind to pollsters and has witnessed the emerging prominence of poll aggregators and new methods of developing projections. But, since survey research still forms the basis for nearly everything (with quantitative modeling a prominent exception)) and it is becoming increasingly difficult to execute, it is an open question as to how good the predictions will be in 2012.