Pretesting Questionnaires: Content Analyses of Respondents' Concurrent Verbal Protocols

Conventional questionnaire pretesting methods focus on directly identifying question defects, such as an ambiguous question. This paper proposes a new method that identities respondents' cognitive difficulties as they form answers to survey questions. It entails a content analysts of concurrent verbal protocols elicited during pretest interviews. The effectiveness of the methodology is illustrated with pretests of multiple versions of the same survey. The results are used to illustrate how this method yields diagnostic information about questionnaire problems and improvements. Then, the results are compared with the results of observational monitoring by managers. The findings indicate that a questionnaire pretesting methodology that quantifies respondents' cognitive difficulties is a useful enhancement for identifying and “improving” defective questions.