Major changes are transforming online tracking studies, as traditional panel research becomes less representative:
- Increasing Mobile Participation – First, over 40% of panel surveys are now taken on mobile devices, with this expected to exceed 50% this year according to GreenBook. Our clients who have created desktop-first surveys have seen their respondent pool skew older and towards lower incomes, with aggregate results that are no longer representative of the wider market they are seeking to understand.
- Rising Abandonment Rates – Second, abandonment rates are climbing, especially on mobile, with a Kinesis study showing that only 7% of mobile respondents (nonpanel) complete web surveys; a Ypulse meta study revealed that younger respondents are abandoning surveys earlier in the questionnaire than older respondents did when their age.
- Growing Panelist Fraud – Third, panel populations are increasingly stretched thin and screener fraud is on the rise, with some panelists lying to qualify for a survey, typically after “screening out” of a number of earlier surveys without earning any incentives. While for general-population studies, fraud typically accounts for 2% of responses, for low-incidence studies such fraud can account for over half the collected responses.
Given these challenges, a break from the past is warranted in order to “future proof” a tracker for long-term stability. Such an approach does require some radical changes to address panel-research challenges:
- Mobile First – For most of our clients’ ad-hoc studies, we program the survey into an online tool using Responsive Web Design to ensure that it is optimized for completing on desktop, laptop, and mobile devices, in order to maximize the representativeness of respondents. For a long-term tracker, however, we believe a mobile-first design is essential, since mobile will soon be the first choice of response. This typically means a one-question-per-screen approach, with shorter, even terse, question text and choice text.
- Micro-Surveys – Rising abandonment rates and the impending majority use of mobile mean that questionnaires have to be succinct. We have found that 10- to 20-question surveys work best in the mobile environment. Keep in mind that many mobile surveys are taken to kill time while waiting in line, in between tasks, or when looking for a short diversion. In such use cases, it is no surprise that long surveys are abandoned. For our clients implementing new trackers, we propose breaking up the old questionnaire into multiple surveys, such as by purchased product category. A respondent who has answered a profile survey specifying product purchases will then immediately see in their panelist portal a short survey for each purchased product category. This means that a panelist will experience the survey as 2 or more short surveys typically spread out over a week. We find an 82% re-contact rate using this approach, higher than completion rates for traditional tracking studies. Results of multiple surveys are joined by panelist ID and presented to our client as if the panelist had taken a single survey.
- Panelist-Friendly Incentives – Traditional panels see high fraud because their panelist experience is broken: panelists take a series of screeners in order to qualify for a 20- to 40-minute study that will pay a large incentive; they are usually not compensated for the screener surveys (some of which run to 10 or 20 questions themselves), all to earn $2 to $5 in points for the longer study. No wonder some panelists lie on screeners! In one panelist trust study we conducted of screener fraud, fraud jumped from 0% at $1 for the completed survey to 20% at $2. Instead, for these tracking studies, we compensate every single respondent for taking the screener survey, which itself will be a micro-survey. This removes all incentive to cheat. Additionally, since all panel surveys are kept to no more than 15 questions, panelists have been trained to expect shorter surveys for regular rewards, redeemable in an e-currency that can be used for a variety of purchases.
By redesigning the tracker to address the rise of mobile, the advent of shorter attention spans, and panelist behavioral incentives, you can field a more reliable tracker for today’s environment and for years to come.
Author Notes:
Jeffrey Henning
