To improve the representativeness of our online surveys, we – like most survey researchers – use quota sampling to closely mirror the U.S. population. In our case, we may go a little overboard. We allocate 32 quota cells for each combination of age range, gender, and region, and stop accepting results after we’ve achieved the quota limit for each cell. For instance, in a survey of 2,000 U.S. adults aged from 18- to 71-years old, the quota cell for 18- to 22-year-old males in the Northeast will require 19 completes, and the quota cell for 36- to 55-year-old females in the South (a wider age range in a more populous region) will require 158 completes. Once the target for a quota cell is achieved, subsequent respondents of that demographic are screened out of the survey.
If we didn’t use quota cells, our samples would over-represent middle-aged adults (and conversely under-represent 18-22 year olds and 56-71 year olds) and would over-represent women (and conversely under-represent men).
Because the demographics of online panels don’t exactly mimic the U.S. population, early incoming results to live surveys exhibit exactly the bias we are seeking to correct for! As a result, early dashboards can be misleading about final results, in those cases where attitudes and behaviors prove to differ by age or by gender.
Here are how our quota cells typically fill up, from first to last, in order of hitting the target number of completes:
- 36-55, Female, South
- 23-35, Female, South
- 36-55, Female, Northeast
- 36-55, Female, Midwest
- 36-55, Female, West
- 23-35, Female, West
- 56-71, Female, South
- 23-35, Female, Northeast
- 23-35, Female, Midwest
- 36-55, Male, South
- 23-35, Male, South
- 23-35, Male, West
- 23-35, Male, Northeast
- 36-55, Male, West
- 56-71, Female, Midwest
- 36-55, Male, Northeast
- 56-71, Female, West
- 36-55, Male, Midwest
- 56-71, Female, Northeast
- 23-35, Male, Midwest
- 56-71, Male, South
- 18-22, Female, South
- 56-71, Male, Midwest
- 56-71, Male, West
- 56-71, Male, Northeast
- 18-22, Female, West
- 18-22, Male, South
- 18-22, Female, Midwest
- 18-22, Female, Northeast
- 18-22, Male, West
- 18-22, Male, Northeast
- 18-22, Male, Midwest
This means surveys can get 80% of their results right away and then slowly march on trying to fill up the final quota cells.
But, by eliminating a known source of bias, the wait ends up being worth it.
For more on quota sampling and other techniques for improving the quality of online research, see the free Researchscape white paper, “Improving the Representativeness of Online Surveys”. Download your own copy now.
Author Notes:
Jeffrey Henning
