People who are willing to be members of panels differ in many ways from people who aren’t. This too lessens the representativeness of panel research.

For instance, in a recent CASRO webinar, NPD Group reported that 70% of their panel members are introverted, compared to approximately 50% of the U.S. population. Panel members most likely differ in other key ways regarding their personalities; Inna Burdein, direct of panel analytics for NPD, reported that diligent panelists are “high on introversion, have a high need for cognition, enjoy thinking, and prefer complex to simple problems, and they like surveys – they find surveys worthwhile.” 

A particular type of individual keeps completing surveys – someone who actually enjoys the introspection of the process. And the experience of staying in the panel changes their answers over time. On the positive side, the practice effect sees them improve the accuracy of their responses and they become more self-aware; on the negative side, panel conditioning exposes them to more studies, making them less like the general population being researched. Given these findings, and given that panel tenure can fluctuate as new panel sources are added, many organizations weight their data by panel tenure to keep findings consistent over time. Weighting in this case not to improve representativeness but to improve replicability.

To offset practice effects and panel conditioning, river sampling is meant to provide fresh meat – fresh eyes – to surveys. These respondents are intercepted by banner ads and popups as they surf the web and are recruited to take a survey. Such respondents take many fewer surveys than panelists, with a high proportion reporting this is the only survey they’ve taken. Such people are hard to build a panel on (they don’t come back for subsequent surveys). They are often integrated into surveys through routers that try to balance the proportion of new respondents with established panelists.

This is an excerpt from the free Researchscape white paper, “Improving the Representativeness of Online Surveys”. Download your own copy now.