The Gold Standard: Probability Sampling

While probability sampling is rarely used with online research, it helps to keep probability sampling in mind because many of the techniques to improve the representativeness of online surveys mimic or emulate the key characteristics of probability sampling.

Probability sampling has two key requirements:

1. COVERAGE – A known and non-zero chance of selecting any member of the target population. (Roger Sapsford and others argue it has to be an equal chance of selection.)
2. EXTERNAL SELECTION – Random selection of members of the population to participate in the survey.

The requirement for external selection is frequently forgotten, yet it is vital. We have selected someone to participate in this survey, and we are going to keep reminding them of that fact until they finally participate. To maximize participation among those selected, surveys of mail probability samples often have five rounds of notification (an invitation and four reminders); telephone surveys may call a landline number dozens of times using automatic dialers; online surveys may follow-up an invitation with one or more reminder emails.

Failure to follow through on external selection may lead to sampling bias. For instance, back in 1993, I worked with Bill Ablondi of Strategy Analytics on a landmark mobile professional study. The protocol was to call the professionals over a dozen times on their office numbers (this was before the age of affordable cell phones): otherwise our sample would have been skewed to those professionals who were in the office more (i.e., less mobile).

Some critics argue that today’s probability sampling, with its low response rates, no longer qualifies as a scientific sample.

Yet this view is controversial, and repeated studies show that probability sampling is remarkably robust – even as response rates have dramatically declined, and the use of weighting has increased to compensate, the results have maintained similar accuracy.

It is possible to conduct online probability sampling – you just need a complete list of the target market being studied. This is surprisingly easy for e-commerce companies: Amazon, Ebay, and JetBlue can all readily conduct probability surveys by randomly selecting email addresses from their databases and then sending an invitation and a series of reminders. This has the greatest chance of minimizing sampling bias.

Most organizations are not so lucky. And even Amazon, Ebay, and JetBlue can’t conduct online probability sampling on their own if they want to survey book readers, bargain hunters, or travelers in general.

For online surveys representative of wider markets, you will have to rely on other methods, which may or may not approximate some of the characteristics of probability samples.

This is an excerpt from the free Researchscape white paper, “Improving the Representativeness of Online Surveys”. Download your own copy now.

Author Notes:

Jeffrey Henning

Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.