Presidential elections present a unique opportunity for polling firms to embarrass themselves, and in 2012 it was Gallup’s turn. Gallup’s last pre-election estimate of the 2012 popular vote for the Presidential election predicted a race that was too close to call, with Romney at 49% of the vote and Obama at 48%, both with a margin of sampling error of plus or minus 2 percentage points. The final result was Romney 47%, Obama 51%, at the edge of the margin of error.

In a review of its methodology announced today, Gallup concluded that 4 factors accounted for most of the measurement error.
  1. Gallup’s likely voter estimating exaggerated support for Romney. Pre-election polling measures “a population that doesn’t exist yet,” that of voters in the upcoming election. Likely voter estimation is an attempt to identify this population, and is based on scoring respondents for voting behavior, attitudes and intended action. In this case, much of the skew towards Romney came from Gallup’s question, “How much thought have you given to the upcoming election for president — quite a lot, some, or only a little?”
  2. Gallup’s regional representation missed Obama voters. The organization weighted to U.S. Census regions, but underrepresented the Eastern time zone within the Midwest and South and underrepresented the Pacific time zone within the West.
  3. Gallup’s race and ethnicity questions were nonstandard. Gallup weighted its data according to race estimates from the Current Population Survey of the U.S. Census Bureau but used its own question battery (designed for telephone) rather than the CPS questions (which were designed for mail). The result was disproportionate respondent identification as multiracial, American Indian and Alaska Native.
  4. Gallup’s shift to listed landlines instead of RDD sampling. By relying on listed numbers instead of a probability sample using Random Digital Dialing, Gallup skewed its results to older, more Republican voters.

Too much of the discussion of survey error cursorily ascribes it to sampling error, when there are many types of error at play. Kudos to Gallup for its candid review of 13 possible sources of error in its work.

Author Notes:

Jeffrey Henning

Gravatar Image
Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.