Your client says that questionnaire you’re writing should ask your customers “What is your overall satisfaction with our product?” and should use a five-point scale. Sounds simple, you think to yourself, then go to write the question. Which format do you use?

examples-of-question-scale

Like many other tactical issues regarding questionnaire design, scale formats have been carefully studied. According to the summary of available research by Jon Krosnick and Leandre Fabrigar in “Designing rating scales for effective measurement in surveys”:

  • Respondents prefer rating scales with more verbal labels.
  • Respondents believe such scales provide more valid measurement.
  • Choosing a labeled choice is a more natural mental activity (not to mention more conversational) than selecting a number within a range.
  • Longitudinal reliability is greater when using fully labeled scales instead of partially-labeled scales.
  • Validity, especially inter-rater validity, is greater using fully labeled scales.
  • Using fully labeled scales provides greater reliability and greater validity from respondents with low to moderate education.
  • Because numeric values can confuse respondents and affect the choices they make, it is better to omit numeric labels altogether.

Fully labeling scales extends even to not having any unqualified labels: e.g., instead of “Satisfied” as a midpoint use “Somewhat satisfied.” (See Saris 1988.)

Scales often get presented in list order (vertically rather horizontally) on smartphone screens:

satisfaction-scale-on-smartphone

With this in mind, it’s important to know how participants evaluate such screens. Roger Tourangeau, Mick P. Couper, and Frederick G. Conrad write, “Participants may assign meanings to visual cues, such as screen position, that are not actually intended to convey any meaning.” (Public Opinion Quarterly, Volume 77, SI, January 2013.) In this case, they often use the heuristic “Up means good” to favor an item at the top of a list. (See “Failing to Listen to the Research… on Research.”)

Given the importance of fully labeling a rating scale, choose an existing rating scale where possible, rather than writing your own scale.

Originally published 2017-07-11. Updated with a discussion of unqualified labels and “Up is good.”

Author Notes:

Jeffrey Henning

Gravatar Image
Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.