Our standard practice for business-to-business surveys and surveys of our clients’ own house lists is to include incomplete responses. Depending on the topic and the length of the survey, 10% to 30% of respondents may not complete the entire questionnaire. A common reason that respondents abandon surveys is because of topic salience: they simply find the subject of a survey to be uninteresting to them. Including their answers for those questions to which they did respond improves the representativeness of results, which would otherwise skew towards those with a higher engagement with the topic of the study.
Of course, another reason that respondents abandon surveys is simply that they’ve been interrupted while answering. We will often see examples of this in the completion data; for instance, in one survey we data-cleansed, a respondent was recorded as having taken 28 hours to complete a 7-minute questionnaire. Clearly, they were interrupted, then returned to that tab in their browser and finished the survey the next day.
Now that said, some partial responses are too trivial to be included. For instance, responses from those who screened into a survey by answering the up-front demographic questions but who then abandoned it before answering the first topic-specific question. Those partial responses might provide some clues as to how nonresponse bias varied demographically, and could be analyzed in the context of screen outs, but aren’t worth including in the main analysis.
The standard practice for many organizations is to remove all partial responses, but this has always struck me as disrespectful of the time that respondents spent answering as much of the survey as they did. (For surveys with incentives, respondents only earn the incentive if they complete the survey, so there’s no added cost to including partial responses.) And given that weighting is designed to compensate for different rates of representation, for addressing the fundamental issue that not every group takes surveys at the same rate, dropping partial responses seems counterproductive.
For nationally representative surveys – which use quota sampling to approximate the proportions of the population by age, gender, region, ethnicity, education, or other attributes – partial responses may skew the results away from this representativeness. For instance, respondents with only a high school degree have higher abandonment rates than college-educated respondents. Because of the problems with partial responses when using quota sampling, for such surveys we typically only include complete responses.
In summary:
- Pros of including partial responses
- Respectful of all respondents’ contributions
- Increases the sample size for early questions
- Increases the reportable sample size for newsmaker surveys
- Reduces the bias of topic salience
- Cons
- Incompatible with quota sampling
- Confusing to readers when the sample size declines from question to question
First published on May 21, 2018, but updated to point to more recent links.
Author Notes:
Jeffrey Henning
