Margin of error as popularly understood overstates the reliability of research results in at least three key ways.

First, those interpreting margin of error forget an important caveat. The results are estimates and typically vary within a narrow range around the actual value that would be calculated by completing a census of everyone in a population. On occasion (1 out of 20 times, at a 95% confidence level), the results from a particular question may be completely outside the interval of error. So in a 20-question survey, one question might have no relationship to the true value, according to probability sampling theory, and such an outlier is to be expected.

Second, many other types of error occur: generally categorized as non-sampling error, these include mistakes in how the question was asked (e.g., leading questions, incomplete choice lists) or interpreted (e.g., being misread or misheard by the respondent), among many others. Total Survey Error recognizes that multiple sources of error can reduce the validity of survey research: besides sampling error, the five types of non-sampling error include specification error, frame error, nonresponse error, measurement error, and processing error. All of which means that the results for many questions may be outside the interval of the margin of error.

Third, many survey questions are about intent rather than attitude or past behavior. Such questions require interpretation through the development of voting models and purchase-likelihood models, which introduce their own sources of error.

Continued on the blog of the Market Research Institute International.