Jon Krosnick, Professor of Communication, Political Science, and Psychology, Stanford University spoke at the NYAAPOR meeting on May 15th. On the topic of polling, Jon started by pointing out the crisis in the public image of polling, citing Nate Silver’s comment that polls are failing.

Jon explained that we as an industry can proceed in one of two directions. One path, he described at ‘the good path’ contrasting with the other path leading to ‘the death of surveys’.

I was able to catch up with Scott McDonald, President of Nomos Research, after the presentation for his commentary on three key themes.

Question Design

Jon posited that question wording matters for the purpose of reliability in polls. He gave a number of examples of different polling organizations and how some of their question wording was ‘leading’, even ‘disrespectful’. Verbatim, open-ended questions were offered as one solution to avoid these issues. Jon opined that an organization like AAPOR could develop and promote a set of standards on question design to enhance reliability and polling rigor.

Scott commented that Jon assumes that all of these polling organizations are motivated by competitive differentiation in their selection of wording, but since the public doesn’t usually perceive quality differences based on this dimension, I doubt that it is effective way to brand those organizations. However the pollsters do like to be cited in other media – and sometimes the best way to do that is to generate a surprising statistic.

Survey Error

Jon discussed sources of survey error, showing slides from research comparing probability studies (both phone and online) to opt-in studies. He referenced the ‘heroin dealers’ in the polling industry, pointing out that opt-in studies showed tremendous variance (e.g. 80% of Americans reported that they visited a movie theater in the last thirty days). This survey error can be a compromising factor in poll reliability.

The contention that the randomness of the sample design that matters is widely accepted (if not as accepted as it used to be), and I think Jon is also right that no amount of post-sample weighting can correct for shortcomings in sample design – despite the various claims made in the industry to the contrary. The problem is that real random surveys are usually very expensive and most of the business questions being addressed have narrow audiences that would be much too cost prohibitive to target through random sampling. One of our clients has a total addressable market of just 16,000 organizations: it has to be targeted nonrandomly.

Real-World Implementation

I posed the question to Jon on how to communicate probability sampling and representativeness in layman’s terms. For context, margin of error and sample size depicted in a table is relatively digestible and clients adjust budgets accordingly. Jon agreed that probability/random sampling is a topic that is less accessible to a broad audience.

Many of the things we try to measure are so small and fragmented that they would require gigantic (and very expensive) random samples. What’s more, many of these events (say, for example, a visit to a website) are transient enough and low enough in salience as to be quickly forgotten. Hence the vast difference between passive measurement of things like radio stations (which picks up the fragmentation) and self-reports (which tend to favor the bigger brands). When a survey organization needs to measure highly fragmented phenomena, they are pushed to super-large samples — and through that door to samples that don’t meet the standards of random sampling.

After the presentation, Jon took the stage with his band, Charged Particles, along with special guest, Paul McCandless. Certainly one of the most improbable juxtapositions of any talk I’ve attended! I left the event with a signed ‘Sparks’ CD and new insight into percussion—and probability.

Author Notes:

Jeffrey Henning

Gravatar Image
Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.