Your client says that questionnaire you’re writing should ask your customers “What is your overall satisfaction with our product?” and should use a five-point scale. Sounds simple, you think to yourself, then go to write the question. Which format do you use?
Like many other tactical issues regarding questionnaire design, scale formats have been carefully studied. According to the summary of available research by Jon Krosnick and Leandre Fabrigar in “Designing rating scales for effective measurement in surveys”:
- Respondents prefer rating scales with more verbal labels.
- Respondents believe such scales provide more valid measurement.
- Choosing a labeled choice is a more natural mental activity (not to mention more conversational) than selecting a number within a range.
- Longitudinal reliability is greater when using fully labeled scales instead of partially-labeled scales.
- Validity, especially inter-rater validity, is greater using fully labeled scales.
- Using fully labeled scales provides greater reliability and greater validity from respondents with low to moderate education.
- Because numeric values can confuse respondents and affect the choices they make, it is better to omit numeric labels altogether.
Fully labeling scales extends even to avoiding unqualified labels: if you are using a modifier on some choices, you shouldn’t present that text without a modifier as one of the choices. For instance, instead of “Satisfied” as a midpoint use “Somewhat satisfied.” (See Saris 1988.)
Scales often get presented in list order (vertically rather horizontally) on smartphone screens:
With this in mind, it’s important to know how participants evaluate such screens. Roger Tourangeau, Mick P. Couper, and Frederick G. Conrad write, “Participants may assign meanings to visual cues, such as screen position, that are not actually intended to convey any meaning.” (Public Opinion Quarterly, Volume 77, SI, January 2013.) In this case, they often use the heuristic “Up means good” to favor an item at the top of a list. (See “Failing to Listen to the Research… on Research.”)
Given the importance of fully labeling a rating scale, choose an existing rating scale where possible, rather than writing your own scale.
Originally published 2017-07-11. Updated with a discussion of unqualified labels and “Up is good.”
Author Notes:
Jeffrey Henning
