Questionnaire Pretesting Methods: A Comparison of Cognitive Interviewing and Respondent Debriefing Vis-à-vis the Study of the Adoption of Decision Support Systems by Knowledge Workers

10.6702/ijbi.201806_13(2).0001

Emmanuel Matthew Ikart

Cognitive interviewing ； respondent debriefing ； EIS ； CBIS ； pretesting

International Journal of Business and Information

13卷2期（2018 / 06 / 01）

119 - 154

英文

As a vehicle for data collection, the questionnaire is a vital component in achieving higher quality in research studies. A frequent difficulty with questionnaire design is that respondents often misinterpret the questions. This difficulty has been recognized in the literature. In recent years, there has been increased emphasis on building quality into questionnaire design through pretesting. A pretest aims to ensure quality by subjecting the questionnaire to some types of evaluations to assess its ability to collect the desired data. Whether constructing a new scale or revising an existing scale, through pretesting, researchers may be able to confirm whether the scale uses clear and appropriate language, has no obvious errors or omissions, and has adequate psychometric properties. Questionnaire pretesting, therefore, is a crucial element of a good study design. Prior to the use of pretests, questionnaire problems were diagnosed mainly by the questionnaire designers (through their expertise in the subject) and the interviewers (through their experience in administering the questionnaire). Thus, detailed reports of appropriate methods to undertake pretesting are currently underrepresented. As the first of its kind, this paper compares the results of two pretesting methods - namely, cognitive interviewing and respondent debriefing - used to evaluate a series of questions from the questionnaire developed for the study of executive information systems (EIS) adoption factors by knowledge workers. We believe the protocols contribute to reducing shortfalls in pretesting guidance for researchers and practitioners. This paper also aims to inform others about the research and to encourage discussions on the topic.

1. Babonea, A.,Voicu, M.(2011).Questionnaires pretesting in marketing research.Challenges of the Knowledge Society,1,1323-1330.
2. Beatty, P.(2004).The dynamic of cognitive interviewing.Methods for Testing & Evaluating Survey Questionnaires,New York:
3. Beatty, P.C.,Willis, G.B.(2007).Research synthesis: The practice of cognitive interviewing.Public Opinion Quarterly,71,287-311.
4. Belson, W.(1981).Design and Understanding of Survey Questions.Aldershot, England:Gower Publishers.
5. Cannell, C.F.,Oksenberg, L.,Kalton, G.(1991).New techniques for pretesting survey questions.Journal of Official Statistics,7,349-365.
6. Collins, D.(2003).Pretesting survey instruments: An overview of cognitive methods.Quarterly of Life Research,12,229-238.
7. DeMaio, T.(ed.).(1983).Approaches to Developing Questionnaire.Washington, DC:Office of Management & Budget.
8. DeMaio, T.J.,Rothgeb, J.,Hess, J.(1998).,Washington, DC.:U.S. Bureau of the Census.
9. Drennan, J.(2003).Cognitive interviewing: Verbal data in the design and pretesting of questionnaires.Journal of Advanced Nursing,42,57-63.
10. Ericsson, K.A.,Simon, H.A.(1984).Protocol Analysis.Massachusetts:The MIT Press.
11. Esposito, J.L.,Campanelli, P.C.,Rothgeb, J.,Polivka, A.E.(1991).Determining which questions are best: Methodologies for evaluating survey questions.Proceedings of the Section on Survey Research Methods
12. Foddy, W.(1993).Constructing Questions, Interviews, and Questionnaires: Theory and Practice in Social Research.London:Cambridge University Press.
13. Gerber, E.(1996).Hidden assumptions: The use of vignettes in cognitive interviewing.Proceedings of the Survey Research Methods Section
14. Greco, L.D.,Walop, W.(1987).Questionnaire development: The pretest.Canadian Medical Association Journal,136(10),1025-1026.
15. Haeger, H.,Lambert, A.D.,Kinzie, J.,Gieser, J.(2012).Using cognitive interviews to improve survey instruments.Proceedings of the Annual Forum of the Association for Institutional Research,New Orleans, Louisiana:
16. Hess, J.,Singer, E.(1995).The role of respondent debriefing questions in questionnaire development.Proceedings of the American Statistical Association (Survey Research Methods Section),Alexandria, VA:
17. Hilton, C. E.(2015).The importance of pretesting questionnaires: A field research example of cognitive pretesting the exercise referral quality of life scale (ER-QLS).International Journal of Social Research Methodology,1-14.
18. Hughes, K.A.(2004).Comparing pretesting methods: Cognitive interviews, respondent debriefing, and behavior coding.Proceedings of the Annual Meeting of the Federal Committee on Statistical Division Methodology,Arlington, VA:
19. Market Research Guy. (2017). Survey Design Best Practices: How To Write a Good Questionnaire. Retrieved from: http://www.mymarketresearchmethods.com/survey-design-best-practices/
20. Oksenberg, L.,Cannell, C.,Kalton, G.(1991).New strategies for pretesting survey questions.Journal of Official Statistics,7(3),349-365.
21. Peter, P.(ed.),Margrit, P.(ed.),Jackon, F.F., Jr.(ed.),Methoden, Z.(ed.),Zuma, A.(ed.)(2003).Questionnaire Evaluation Standards.Mannheim, Germany:ZUMA.
22. Presser, S.,Blair, J.(1994).Survey pretesting: Do different methods produce different results?.Sociological Methodology,Beverly Hills, CA:
23. Presser, S.,Couper, M P.,Lessler, J.T.,Martin, J.,Rothgeb, J.M.,Singer, E. v(2004).Methods for testing and evaluating survey questions.Public Opinion Quarterly,68(1),109-130.
24. Schwartz, N.(ed.),Sudman, S.(ed.)(1996).Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Rsearch.San Francisco:Jossey-Bass Publishers.
25. Subar, A.F.,Thompson, F.E.,Smith, A.F.,Jobe, J.B.,Ziegler, R.G.,Potischman, N.,Harlan, L.C.(1999).Improving food frequency questionnaires: A qualitative approach using cognitive interviewing.Journal of the American Dietetic Associatio,95(7),781-788.
26. Synodinos, N.(2003).The ‘art’ of questionnaire construction: Some important considerations for manufacturing studies.Integrated Manufacturing Systems,14(3),221-237.
27. Tanur, J.M.(ed.)(1992).Question about Questions.New York:Russell Sage Foundation.
28. Turner, C.,Martin, E.(1984).Survey Subjective Phenomena.New York:Russell Sage Publishers.
29. Willis, G.,Schechter S.,Whitaker K.(1999).A comparison of cognitive interviewing, expert review, and behavior coding: What do they tell us?.Proceedings of the American Statistical Association (Survey Research Methods Section),Alexandria, VA:
30. Willis, G.B.(2005).Cognitive Interviewing. A Tool for Improving Questionnaire Design.London:Sage Publishing.