Major changes are transforming online surveys, making traditional panel research less representative, severely impacting tracking studies.
- Increasing Mobile Participation – First, over 40% of panel surveys are now taken on mobile devices, with this expected to exceed 50% this year, according to GreenBook. Our clients who have created desktop-first surveys have seen their respondent pool skew older and to lower incomes, with aggregate results that are no longer representative of the wider market they are seeking to understand.
- Rising Abandonment Rates – Second, abandonment rates are climbing, especially on mobile, with a Kinesis study showing that only 7% of mobile respondents (nonpanel) complete web surveys; a Ypulse meta study revealed that younger respondents are abandoning surveys sooner than older respondents did when their age.
- Growing Panelist Fraud – Third, panel populations are increasingly stretched thin and screener fraud is on the rise, with some panelists lying to qualify for a survey, typically after “screening out” of a number of earlier surveys for no incentive. While for general population studies fraud typically accounts for 2% of responses, for low-incidence studies such fraud can account for over half the collected responses.
Given these challenges, a break from the past is warranted in order to “future proof” a methodology for long-term stability. Such an approach does require some radical changes to restructure the approach to address panel-research challenges:
- Mobile First – For most of our clients’ ad hoc studies, we program the survey into an online tool using Responsive Web Design to ensure that it is optimized for completing on desktop, laptop and mobile devices, in order to maximize the representativeness of respondents. For a long-term tracker, however, we believe a mobile-first design is essential, since mobile will be the first choice of response within a year. This typically means a one-question-per-screen approach, with shorter, even terse, question text and choice text.
- Micro-Surveys – Rising abandonment rates and the impending majority use of mobile mean that questionnaires have to be succinct. We have found that 10 to 15 question surveys work best in the mobile environment. Keep in mind that many mobile surveys are taken to kill time while waiting in line, in between tasks, or when looking for a short diversion; in such use cases, it is no surprise that long surveys are abandoned. For Pella, we propose having separate surveys for each product category (windows, front entry door, patio doors, etc.). A respondent who has answered a profile survey specifying product purchases will then immediately see in their panelist portal a short survey for each purchased product category. This means that a panelist will experience the current survey as 2 or more short surveys typically spread out over a week. We find an 82% re-contact rate using this approach, higher than completion rates for traditional tracking studies. Results of multiple surveys will be joined by panelist ID and presented to Pella as if the panelist had taken a single survey.
- Panelist-Friendly Incentives – Traditional panels see high fraud because their panelist experience is broken: panelists take a series of screeners in order to qualify for a 20- to 40-minute study that will pay a large incentive; they are not compensated for the screener surveys (some of which run to 10 or 20 questions themselves) all to earn $2 to $5 in points for the longer study. No wonder some panelists lie on screeners! In one panelist trust study we conducted of screener fraud, fraud jumped from 0% at $1 for the completed survey to 20% at $2. Instead, for Pella we will compensate every single respondent (40,000 per time period) for taking the screener survey, which itself will be a micro-survey. This removes all incentive to cheat. Additionally, since all panel surveys are kept to no more than 15 questions, panelists have been trained to expect shorter surveys for regular rewards, redeemable in an e-currency that can be used for a variety of purchases.
By redesigning the tracker to address the rise of mobile, shorter attention spans, and panelist behavioral incentives, Researchscape can offer Pella the most reliable tracker for today’s environment and for years to come.
Author Notes:
Jeffrey Henning
