Unfortunately for market researchers, the science of sampling error is often counterintuitive. For instance, the public considers convenience samples with tens of thousands of responses to be more accurate than random samples with 400 responses, not realizing that the random sample is far more accurate. Surveys of small populations are similarly counterintuitive, as some of my clients have bumped up against this year.

Three of my business-to-business clients had an average of approximately 100 customers that they surveyed. All overestimated the precision of the results based on the number of responses they received. One client had only 20 responses, which has a margin of error of 20%. In other words, a response of 50% to a question (e.g., 50% are in favor of feature X) has an actual range of 30% to 70% for the entire population of 100 clients, 19 times out of 20 (95% confidence). Another client had a 50% response rate to its census of customers – but this still provides a margin of error of 10% for a population of 100.

Margin of Error Declines Rapidly As Sample Size Increases

What makes this particularly counterintuitive is that 400 responses to a probability survey of the U.S. adult population of 235 million (a response from less than 0.0002% of the population) has a margin of error of 5%. To achieve that margin of error with a population of 100, you need to survey 80 people!

The key takeaway for B2B surveys is that much of the emphasis during fielding needs to be on increasing the response rate. You should implement multiple rounds of reminders, using multiple modes of contact (online and telephone at least) to drive that response rate higher.

Otherwise your survey results will only be qualitative, or directional, in nature.

Author Notes:

Jeffrey Henning

Gravatar Image
Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.