We’re now on the virtual eve of the Iowa caucuses and the busy season for the early contests in the Presidential nominating process in 2012. You’ll be seeing a lot of polls over the coming days and weeks and many will give you wildly different results.
Just in the last week we’ve seen polls either showing Ron Paul winning in Iowa (Public Policy Polling and Insider Advantage) or in fourth place (Rasmussen Reports). Nationally this month we’ve seen polls conducted in overlapping time frames showing Newt Gingrich getting as much as 40% of the Republican vote and as little as 28%. All of those cannot be right.
And that’s just this year. In 2008, much to the chagrin of many pollsters, the average of all polls taken in New Hampshire in the week before the primary there predicted an eight point Obama victory—Hillary Clinton won by nearly three points.
So what should the reader, as an informed consumer of polling data, be looking for as you read pre-election polls from Iowa and later states? Here are four pieces of advice.
1) Sample selection: This recent article highlights work by Greenberg Quinlan Rosner Research and a Professor at Harvard’s Kennedy school that sums up the issue facing a lot of polling that you’ll see in the next few weeks—by using an obsolete approach to sampling and taking voters at their word about their likelihood of participating, many newspaper pollsters wind up interviewing a lot of people who aren’t going to vote.
That article is part of the reason that many pollsters, including us, have pointed out that the newspaper polls and some political pollsters who still rely on random digit dialing (RDD) samples rather than voter file based samples are producing biased results.
But that article primarily addresses general election votes. Primary voting, and particularly caucus participation (which is much more time consuming and demanding), are subject to even bigger biases.
Optimally, you should look for primary and caucus surveys based on voter lists where actual past participation is being used to predict participation this year. This won’t guarantee accurate results, but it avoids one source of bias that is almost certain to make many newspaper and academic polls inaccurate.
2) Interview mode: The choice between live interviewers and interactive voice response (IVR; recorded voice) polls has been an issue of some contention among pollsters for some time.
The American Association for Public Opinion Research (AAPOR) analysis in the aftermath of the 2008 primaries found that the choice of interview mode did not significantly affect the accuracy of immediate pre-primary polls. But, others have observed that there tends to be much more variation in IVR samples than live interviewer samples farther out from the election.
One hypothesis to explain the high variation in IVR samples far out, but parity in accuracy closer in, is that IVR polls do a good job of measuring opinion once minds are made up but a relatively poor job of measuring voting intentions that are less certain because of the lack of personal interaction to probe and extract a meaningful but less strongly committed response.
3) Looking ahead can be dangerous: In every primary season, the results of earlier states change the course of later states in ways both large and small. Earlier state results provide important context to the decisions of voters in later states by validating the appeal of a winning candidate or casting new doubts on the viability of others.
Candidates may either factually or effectively drop out of the race after one or more early primary losses, thus changing the decision of those who were supporting them. The media narrative will also begin to evolve toward a two-person race as the earliest primaries are completed, providing a critical frame to voter evaluations.
For all of these reasons, it is impossible to accurately predict a later primary state before the previous states vote. At best, a pollster can assess what voters think currently and provide probabilities of certain changes in respond to likely outcomes in previous primaries. But an informed observer will treat any conclusions reached about New Hampshire, for example, as speculative at least until Iowa votes and South Carolina as speculative until both Iowa and New Hampshire have had their say.
4) Ignore the national horserace: National data is dominated by data from a handful of large states that don’t vote until Super Tuesday or later. It is also dominated by voters who are significantly less engaged than the voters of Iowa, New Hampshire, or other early states.
While the national trends can tell us something about the waxing and waning fortunes of the candidates, they are only loosely related to the results in any single early state or even to the results of the collection of early states.
My recommendation is to view the national polls as objects of interest, but to never mistake them for real predictors of the nomination or the results in the early states.
‘Tis the Season (for lots of polling)
Posted by Chris Wilson on Wednesday, December 21st, 2011 at 12:26 PM
We’re now on the virtual eve of the Iowa caucuses and the busy season for the early contests in the Presidential nominating process in 2012. You’ll be seeing a lot of polls over the coming days and weeks and many will give you wildly different results.
Just in the last week we’ve seen polls either showing Ron Paul winning in Iowa (Public Policy Polling and Insider Advantage) or in fourth place (Rasmussen Reports). Nationally this month we’ve seen polls conducted in overlapping time frames showing Newt Gingrich getting as much as 40% of the Republican vote and as little as 28%. All of those cannot be right.
And that’s just this year. In 2008, much to the chagrin of many pollsters, the average of all polls taken in New Hampshire in the week before the primary there predicted an eight point Obama victory—Hillary Clinton won by nearly three points.
So what should the reader, as an informed consumer of polling data, be looking for as you read pre-election polls from Iowa and later states? Here are four pieces of advice.
1) Sample selection: This recent article highlights work by Greenberg Quinlan Rosner Research and a Professor at Harvard’s Kennedy school that sums up the issue facing a lot of polling that you’ll see in the next few weeks—by using an obsolete approach to sampling and taking voters at their word about their likelihood of participating, many newspaper pollsters wind up interviewing a lot of people who aren’t going to vote.
That article is part of the reason that many pollsters, including us, have pointed out that the newspaper polls and some political pollsters who still rely on random digit dialing (RDD) samples rather than voter file based samples are producing biased results.
But that article primarily addresses general election votes. Primary voting, and particularly caucus participation (which is much more time consuming and demanding), are subject to even bigger biases.
Optimally, you should look for primary and caucus surveys based on voter lists where actual past participation is being used to predict participation this year. This won’t guarantee accurate results, but it avoids one source of bias that is almost certain to make many newspaper and academic polls inaccurate.
2) Interview mode: The choice between live interviewers and interactive voice response (IVR; recorded voice) polls has been an issue of some contention among pollsters for some time.
The American Association for Public Opinion Research (AAPOR) analysis in the aftermath of the 2008 primaries found that the choice of interview mode did not significantly affect the accuracy of immediate pre-primary polls. But, others have observed that there tends to be much more variation in IVR samples than live interviewer samples farther out from the election.
One hypothesis to explain the high variation in IVR samples far out, but parity in accuracy closer in, is that IVR polls do a good job of measuring opinion once minds are made up but a relatively poor job of measuring voting intentions that are less certain because of the lack of personal interaction to probe and extract a meaningful but less strongly committed response.
3) Looking ahead can be dangerous: In every primary season, the results of earlier states change the course of later states in ways both large and small. Earlier state results provide important context to the decisions of voters in later states by validating the appeal of a winning candidate or casting new doubts on the viability of others.
Candidates may either factually or effectively drop out of the race after one or more early primary losses, thus changing the decision of those who were supporting them. The media narrative will also begin to evolve toward a two-person race as the earliest primaries are completed, providing a critical frame to voter evaluations.
For all of these reasons, it is impossible to accurately predict a later primary state before the previous states vote. At best, a pollster can assess what voters think currently and provide probabilities of certain changes in respond to likely outcomes in previous primaries. But an informed observer will treat any conclusions reached about New Hampshire, for example, as speculative at least until Iowa votes and South Carolina as speculative until both Iowa and New Hampshire have had their say.
4) Ignore the national horserace: National data is dominated by data from a handful of large states that don’t vote until Super Tuesday or later. It is also dominated by voters who are significantly less engaged than the voters of Iowa, New Hampshire, or other early states.
While the national trends can tell us something about the waxing and waning fortunes of the candidates, they are only loosely related to the results in any single early state or even to the results of the collection of early states.
My recommendation is to view the national polls as objects of interest, but to never mistake them for real predictors of the nomination or the results in the early states.