River Sampling

People who are willing to be members of panels differ in many ways from people who aren’t. This too lessens the representativeness of panel research.

For instance, in a recent CASRO webinar, NPD Group reported that 70% of their panel members are introverted, compared to approximately 50% of the U.S. population. Panel members most likely differ in other key ways regarding their personalities; Inna Burdein, direct of panel analytics for NPD, reported that diligent panelists are “high on introversion, have a high need for cognition, enjoy thinking, and prefer complex to simple problems, and they like surveys – they find surveys worthwhile.” 

A particular type of individual keeps completing surveys – someone who actually enjoys the introspection of the process. And the experience of staying in the panel changes their answers over time. On the positive side, the practice effect sees them improve the accuracy of their responses and they become more self-aware; on the negative side, panel conditioning exposes them to more studies, making them less like the general population being researched. Given these findings, and given that panel tenure can fluctuate as new panel sources are added, many organizations weight their data by panel tenure to keep findings consistent over time. Weighting in this case not to improve representativeness but to improve replicability.

To offset practice effects and panel conditioning, river sampling is meant to provide fresh meat – fresh eyes – to surveys. These respondents are intercepted by banner ads and popups as they surf the web and are recruited to take a survey. Such respondents take many fewer surveys than panelists, with a high proportion reporting this is the only survey they’ve taken. Such people are hard to build a panel on (they don’t come back for subsequent surveys). They are often integrated into surveys through routers that try to balance the proportion of new respondents with established panelists.

This is an excerpt from the free Researchscape white paper, “Improving the Representativeness of Online Surveys”. Download your own copy now.

Author Notes:

Jeffrey Henning

Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.