At the AAPOR Annual Conference last month, Marcus Berger of the Center for Behavioral Science Methods at the U.S. Census Bureau presented “Leveraging the Linguistic Conventions that Govern Discourse to Improve Questionnaire Design.” I wasn’t able to attend, but Marcus was kind enough to share the slides with me.
P. Grice’s Cooperative Principle is a theory that explains how people communicate and cooperate with each other in conversation. Grice wrote, “Our talk exchanges do not normally consist of a succession of disconnected remarks, and would not be rational if they did. They are characteristically, to some degree at least, cooperative efforts; and each participant recognizes in them, to some extent, a common purpose or set of purposes, or at least a mutually accepted direction.”
Respondents apply this conversational frame, and these four maxims, to their understanding of answering survey questions. (These frames are important. Robert Groves, in his 1989 book, Survey Errors and Survey Costs, argues that each survey mode puts respondents into a different frame of mind or mental script: “Face-to-face surveys prompt a guest script. Respondents are more likely to treat face-to-face interviewers graciously and hospitably, leading them to be more agreeable… Phone interviews prompt a solicitor script. Respondents are more likely to treat phone interviews the way they treat calls from telemarketers, making them more likely to satisfice.”)
Grice’s principle is further divided into four maxims that people follow to make their communication more effective and efficient: maxims of Quantity, of Quality, of Relation, and of Manner. As Marcus put it, “These assumptions shape how each person interprets the meaning behind the other’s words. When these assumptions are violated, misunderstandings often occur.”
- The Maxim of Relevance states that people should only say things that are relevant to the conversation. This maxim most affects using open-ended questions vs. closed-ended questions. Respondents are unlikely to report information that seems irrelevant to them, either because it is self-evident to them or seems out of place. As a result, such items are underreported when compared to answers collected in a closed-ended question. By listing the choices, the researcher implies that these choices are in fact relevant. To follow this maxim, Marcus advises: “Offer only content that is relevant for the respondent and their specific situation when possible. Consider the respondent’s … previous responses to determine relevance.”
- The Maxim of Quantity states that people should provide as much information as is necessary for the conversation, but not more. Researchers should avoid giving too much information or too little information, as both can be unhelpful for the conversation. For instance, the 2013 American Community Survey had a question about “self-employment income from own nonfarm businesses or farm businesses”, which provides more information than necessary. By not just saying “own businesses,” this question may confuse respondents. Marcus recommends: “Provide sufficient information for the respondent to understand concepts and intentions… Do not provide excessive information.”
- The Maxim of Quality states that people should only say things that they believe to be true and have evidence to support. Marcus extends this to mean that you shouldn’t imply falsehoods. He gives the example of a question asking why parents might not get their children a COVID-19 vaccine, “Don’t believe children need a COVID-19 vaccine,” which may be interpreted as implying that children don’t need a vaccine (they do). Marcus advises, “Take care not to say or imply false information.”
- The Maxim of Manner states that people should communicate in a clear and concise manner. Avoid using ambiguous language, being overly complex, or using hard-to-understand language. Marcus cites another example from a Census survey, this one of being overly complex: the question asked people to include as household members “everyone living or staying here for more than 2 months” and anyone “staying for 2 months or less.” To adhere to the Maxim of Manner, Marcus recommends, “Avoid ambiguous grammar… Avoid double-barreled questions.”
Applying the four maxims of the Cooperative Principle to survey design can improve the quality of the results by ensuring that questions are clear, concise, and relevant to the respondents.
For further reading, Marcus suggests especially the writings of Norbert Schwarz.
Author Notes:
Jeffrey Henning
