Unfortunately for market researchers, the science of sampling error is often counterintuitive. For instance, the public considers convenience samples with tens of thousands of responses to be more accurate than random samples with 400 responses, not realizing that the random sample is far more accurate. Surveys of small populations are similarly counterintuitive, as some of my clients have bumped up against this year.
Three of my business-to-business clients had an average of approximately 100 customers that they surveyed. All overestimated the precision of the results based on the number of responses they received. One client had only 20 responses, which has a margin of error of 20%. In other words, a response of 50% to a question (e.g., 50% are in favor of feature X) has an actual range of 30% to 70% for the entire population of 100 clients, 19 times out of 20 (95% confidence). Another client had a 50% response rate to its census of customers – but this still provides a margin of error of 10% for a population of 100.
What makes this particularly counterintuitive is that 400 responses to a probability survey of the U.S. adult population of 235 million (a response from less than 0.0002% of the population) has a margin of error of 5%. To achieve that margin of error with a population of 100, you need to survey 80 people!
The key takeaway for B2B surveys is that much of the emphasis during fielding needs to be on increasing the response rate. You should implement multiple rounds of reminders, using multiple modes of contact (online and telephone at least) to drive that response rate higher.
Otherwise your survey results will only be qualitative, or directional, in nature.
Author Notes:
Jeffrey Henning
