Recording Response Codability Live: An Investigation of Its Usefulness and Possible Interference

訪員現場記錄回答可過錄性：紀錄之正確性，對訪問造成的影響，及作為研究資源的可靠性之評估

10.7014/SRMA.2023100001

Meng-Li Yang(楊孟麗)

codable response ； survey paradata ； accuracy of interviewers' records ； interviewer experience ； index of response processes ； 可過錄的回答 ； 調查周邊資料 ； 訪員記錄正確度 ； 訪員經驗 ； 回答歷程的指標

調查研究-方法與應用

51期（2023 / 10 / 01）

9 - 56

英文;繁體中文

A codable response is an initial response which matches one of the response options. Among the various interviewer's and respondent's behaviors that survey researchers collect, the information of whether an initial response is codable (response codability, RC) is the most useful in evaluating survey questionnaires and data quality. Yet, because RC is always additionally coded after surveys, along with other behaviors, the information is rarely used in social science research. Instead, social scientists rely on a closely related index of the response behavior, response latency (RL, the length of time taken to answer a question), to tap respondents' underlying mental processes when answering a question. The use of RL brings challenges, however, because respondents may answer before the question is finished, give an extremely lengthy response, or engage in conversations unrelated to the question. Researchers' strategies of dealing with the challenges in turn bring uncertainty to their findings. Such challenges are not likely to emerge when using RC, because RC considers only whether the response is codable or not. This becomes an advantage for applying RC to social science research. This paper suggests that useful RC information can be collected during survey interviews. It posits that interviewers can accurately record RC live without interfering with the interviewing, because they are used to evaluating the adequacy of every answer in each question-answer sequence. In addition, to show the usefulness of RC, it argues that RC is associated with response processes in ways parallel to RL. The propositions are assessed in three studies, using two surveys conducted in 2017 and 2018. Interviewers recorded RC live in both surveys. RC information for the 2017 survey was additionally independently collected for verification of the interviewers' RC records. Study 1 examines hypotheses concerning accuracy of the interviewers' RC records for the 2017 survey. The accuracy rate was 93.4%. Results of three-level logistic regression models further support the hypotheses by showing that the accuracy increased as interviewers recorded for more interviews. Study 2 assesses the usefulness of the interviewers' records for social science research by using the interviewers' RC records and the verification records. Hypotheses on associations of RC with two response processes (attitude accessibility and acquiescence) were formed based on research using RL. That is, a codable response is associated with higher likelihoods of both strong attitudes and acquiescence. Analytical results of the interviewers' records were similar to those of accurate information, and both were able to predict the two processes. Study 3 roughly assesses whether the RC recording task interfered with interviewing, by using incidences of inappropriate probing in the 2018 survey. A cross-tabulation suggests that frequencies of inappropriate probing were related to question types, but not related to the task. In conclusion, this paper contributes to social science research by showing that reliable RC information can be made available by interviewers recording it live. The paper ends with discussions of the consistency of findings concerning the two processes with prior research, of guidelines for interviewer training, and of limitations.

可過錄的回答，是指受訪者第一次就從訪員提供的選項擇一而答。回答的可過錄性（RC）常用以評量問卷與訪問品質。但傳統上RC必須在訪問結束後額外蒐集。本文認為，RC應可由訪員在訪問時正確記錄，且不會因此干擾訪問。本研究於2017與2018年分別進行一次電話調查，訪員於訪問時記錄RC；2017年訪問的正確RC資料，則於調查結束後由專人獨立蒐集。本文利用這些資料從三方面探討這項主張。研究一分析2017年的資料，發現訪員記錄正確率達93.4％；而且記錄的正確性，隨訪員的記錄經驗增多而提高。研究二探討訪員RC記錄的可用性，結果發現訪員的RC記錄，如同正確的RC資料一般，都可有效預測明確態度與默許兩種回答歷程。研究三利用2018年調查的訪員評量資料，藉以觀察：多做這項記錄的工作，是否影響訪問的進行。分析結果顯示，訪員不當追問的比例，跟是否做記錄無關，故訪問時記錄RC並不影響訪問程序。最後，本文就如何訓練訪員於現場記錄RC提出建議。

1. American Association for Public Opinion Research(2016).,未出版
2. Bassili, J. N.,Krosnick, Jon A.(2000).Do Strength-Related Attitude Properties Determine Susceptibility to Response Effects? New Evidence from Response Latency, Attitude Extremity, and Aggregate Indices.Political Psychology,21(1),107-132.
3. Bassili, John N.(1993).Response Latency Versus Certainty as Indexes of the Strength of Voting Intentions in a CATI Survey.Public Opinion Quarterly,57(1),54-61.
4. Bassili, John N.(1996).Meta-Judgmental Versus Operative Indexes of Psychological Attributes: The Case of Measures of Attitude Strength.Journal of Personality and Social Psychology,71(4),637-653.
5. Bassili, John N.,Fletcher, Joseph F.(1991).Response-time Measurement in Survey Research: A Method for CATI and a New Look at Nonattitudes.Public Opinion Quarterly,55(3),331-346.
6. Baumgartner, Hans,Steenkamp, Jan-Benedict EM(2001).Response Styles in Marketing Research: A Cross-National Investigation.Journal of Marketing Research,38(2),143-156.
7. Baumgartner, Hans,Weijters, Bert,Pieters, Rik(2018).Misresponse to Survey Questions: A Conceptual Framework and Empirical Test of the Effects of Reversals, Negations, and Polar Opposite Core Concepts.Journal of Marketing Research,55(6),869-883.
8. Cannell, Charles F.,Fowler, Floyd Jackson, Jr,Marquis, Kent H.(1968).The Influence of Interviewer and Respondent Psychological and Behavioral Variables on the Reporting in Household Interviews.Vital Health Stat 2,26,1-65.
9. Cannell, Charles F.,Oksenberg, Lois(1988).Observation of Behaviour in Telephone Interviewers.Telephone Survey Methodology,New York:
10. Cohen, Jacob(1960).A Coefficient of Agreement for Nominal Scales.Educational and Psychological Measurement,20(1),37-46.
11. Conner, Mark,Sparks, Paul(2002).Ambivalence and Attitudes.European Review of Social Psychology,12(1),37-70.
12. Curtin, Richard,Presser, Stanley,Singer, Eleanor(2005).Changes in Telephone Survey Nonresponse over the Past Quarter Century.Public Opinion Quarterly,69(1),87-98.
13. Dahlhamer, James A.,Maitland, Aaron,Ridolfo, Heather,Allen, Antuane,Brooks, Dynesha(2020).Exploring the Associations Between Question Characteristics, Respondent Characteristics, Interviewer Performance Measures, and Survey Data Quality.Advances in Questionnaire Design, Development, Evaluation and Testing,Hoboken, NJ:
14. Draisma, Stasja,Dijkstra, Wil(2004).Response Latency and (Para) Linguistic Expression as Indicators of Response Error.Methods for Testing and Evaluating Survey Questionnaires,New York:
15. Dykema, Jennifer,Garbarski, Dana,Wal, Ian,Edwards, Dorothy Farrar(2019).Measuring Trust in Medical Researchers: Adding Insights from Cognitive Interviews to Examine Agree-Disagree and Construct-Specific Survey Questions.Journal of Official Statistics,35,353-386.
16. Dykema, Jennifer,Lepkowski, James M.,Blixt, Steven(1997).The Effect of Interviewer and Respondent Behavior on Data Quality: Analysis of Interaction Coding in a Validation Study.Survey Measurement and Process Quality,New York:
17. Dykema, Jennifer,Schaeffer, Nora Cate,Garbarski, Dana,Hout, Michael(2020).The Role of Question Characteristics in Designing and Evaluating Survey Questions.Advances in Questionnaire Design, Development, Evaluation and Testing,Hoboken, NJ:
18. Fazio, Russell H.(1989).On the Power and Functionality of Attitudes: The Role of Attitude Accessibility.Attitude Structure and Function,Mahwah:
19. Fazio, Russell H.(1990).Multiple Processes by which Attitudes Guide Behavior: The MODE Model as an Integrative Framework.Advances in Experimental Social Psychology,23,75-109.
20. Fazio, Russell H.(1995).Attitudes as Object-Evaluation Associations: Determinants, Consequences, and Correlates of Attitude Accessibility.Attitude Strength: Antecedents and Consequences,Mahwah:
21. Fleiss, Joseph L.(1981).Statistical Methods for Rates and Proportions.New York:John Wiley.
22. Fowler, Floyd J.(2011).Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions.Question Evaluation Methods: Contributing to the Science of Data Quality,New York:
23. Fowler, Floyd J.,Mangione, Thomas W.(1990).Standardized Survey Interviewing: Minimizing Interviewer Related Error.Newbury Park, CA:Sage.
24. Garbarski, Dana,Schaeffer, Nora Cate,Dykema, Jennifer(2011).Are Interactional Behaviors Exhibited When the Self-Reported Health Question Is Asked Associated with Health Status?.Social Science Research,40(4),1025-1036.
25. Garbarski, Dana,Schaeffer, Nora Cate,Dykema, Jennifer(2016).Interviewing Practices, Conversational Practices, and Rapport: Responsiveness and Engagement in the Standardized Survey Interview.Sociological Methodology,46(1),1-38.
26. Gilbert, Daniel T.(1991).How Mental Systems Believe.American Psychologist,46(2),107-119.
27. Hanley, Charles(1965).Personality Item Difficulty and Acquiescence.Journal of Applied Psychology,49(3),205.
28. Hess, Jennifer,Singer, Eleanor,Bushery, John(1999).Predicting Test-Retest Reliability from Behavior Coding.International Journal of Public Opinion Research,11(4),346-360.
29. Hicks, Wendy D.,Edwards, Brad,Tourangeau, Karen,McBride, Brett,Harris-Kojetin, Lauren D.,Moss, Abigail J.(2010).Using CARI Tools to Understand Measurement Error.Public Opinion Quarterly,74(5),985-1003.
30. Holbrook, Allyson L.,Johnson, Timothy P.,Cho, Young Ik,Shavitt, Sharon,Chavez, Noel,Weiner, Saul(2016).Do Interviewer Errors Help Explain the Impact of Question Characteristics on Respondent Difficulties?.Survey Practice,9(2),1-11.
31. Jackman, Mary(1973).Education and Prejudice or Education and Response Set?.American Sociological Review,38(3),327-339.
32. Javeline, Debra(1999).Response Effects in Polite Cultures: A Test of Acquiescence in Kazakhstan.Public Opinion Quarterly,63(1),1-28.
33. Kalton, Graham,Flores-Cervantes, Ismael(2003).Weighting Methods.Journal of Official Statistics,19(2),81.
34. Keeter, Scott,Hatley, Nick,Kennedy, Courtney,Lau, Arnold(2017).What Low Response Rates Mean for Telephone Surveys.Pew Research Center,15(1),1-39.
35. Knowles, Eric S.,Condon, Christopher A.(1999).Why People Say “Yes”: A Dual-Process Theory of Acquiescence.Journal of Personality and Social Psychology,77(2),379-386.
36. Kreuter, Frauke(2013).Improving Surveys with Paradata: Analytic Uses of Process Information.New York:John Wiley and Sons.
37. Krosnick, Jon A.(1999).Survey Research.Annual Review of Psychology,50(1),537-567.
38. Krosnick, Jon A.(1991).Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.Applied Cognitive Psychology,5(3),213-236.
39. Krosnick, Jon A.,Alwin, Duane F.(1987).An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement.Public Opinion Quarterly,51(2),201-219.
40. Krosnick, Jon A.,Petty, Richard E.(1995).Attitude Strength: An Overview.Attitude Strength: Antecedents and Consequences,Mahwah:
41. Mangione, Thomas W.,Fowler, Floyd J.,Louis, Thomas A.(1992).Question Characteristics and Interviewer Effects.Journal of Official Statistics,8(3),293.
42. Mayerl, Jochen(2013).Response Latency Measurement in Surveys: Detecting Strong Attitudes and Response Effects.Survey Methods: Insights from the Field,1-27.
43. McBride, Loren,Moran, Gary(1967).Double Agreement as a Function of Item Ambiguity and Susceptibility to Demand Implications of the Psychological Situation.Journal of Personality and Social Psychology,6(1),115-118.
44. McClendon, McKee J.(1991).Acquiescence and Recency Response-Order Effects in Interview Surveys.Sociological Methods & Research,20(1),60-103.
45. McClendon, McKee J.(1991).Acquiescence: Tests of the Cognitive Limitations and Question Ambiguity Hypotheses.Journal of Official Statistics,7(2),153-166.
46. Miller, Joanne M.,Peterson, David A. M.(2004).Theoretical and Empirical Implications of Attitude Strength.The Journal of Politics,66(3),847-867.
47. Miller, Peter V.(1984).Alternative Question Forms for Attitude Scale Questions in Telephone Interviews.Public Opinion Quarterly,48(4),766-778.
48. Mulligan, Kenneth,Grant, J. Tobin,Mockabee, Stephen T.,Monson, Joseph Quin(2003).Response Latency Methodology for Survey Research: Measurement and Modeling Strategies.Political Analysis,11(3),289-301.
49. Olson, Kristen,Peytchev, Andy(2007).Effect of Interviewer Experience on Interview Pace and Interviewer Attitudes.Public Opinion Quarterly,71(2),273-286.
50. Olson, Kristen,Smyth, Jolene D.(2020).What Do Interviewers Learn? Changes in Interview Length and Interviewer Behaviors Over the Field Period.Interviewer Effects from a Total Survey Error Perspective,Boca Raton:
51. Ongena, Yfke P.,Dijkstra, Wil(2006).Methods of Behavior Coding of Survey Interviews.Journal of Official Statistics,22(3),419-451.
52. Ongena, Yfke,Unger, Sanne(2020).The Effects of Task Difficulty and Conversational Cueing on Answer Formatting Problems in Surveys.Advances in Questionnaire Design, Development, Evaluation and Testing,New York:
53. Peabody, Dean(1961).Attitude Content and Agreement Set in Scales of Authoritarianism, Dogmatism, Anti-Semitism, and Economic Conservatism.The Journal of Abnormal and Social Psychology,63(1),1-11.
54. Peabody, Dean(1966).Authoritarianism Scales and Response Bias.Psychological Bulletin,65(1),11-23.
55. Petty, Richard E.,Krosnick, Jon A.(1995).Attitude Strength: Antecedents and Consequences.Mahwah:Erlbaum.
56. Peytchev, Andy(2013).Consequences of Survey Nonresponse.The Annals of the American Academy of Political and Social Science,645(1),88-111.
57. Presser, Stanley(ed.),Rothgeb, Jennifer M.(ed.),Couper, Mick P.(ed.),Lessler, Judith T.(ed.),Martin, Elizabeth(ed.),Martin, Jean(ed.),Singer, Eleanor(ed.)(2004).Methods for Testing and Evaluating Survey Questions.Hoboken, NJ:John Wiley and Sons.
58. Rabe-Hesketh, Sophia,Skrondal, Anders(2006).Multilevel Modelling of Complex Survey Data.Journal of the Royal Statistical Society: Series A (Statistics in Society),169(4),805-827.
59. Ray, John J.(1983).Reviving the Problem of Acquiescent Response Bias.Journal of Social Psychology,121(1),81-96.
60. Ray, John J.(1979).Is the Acquiescent Response Style Problem Not So Mythical After All? Some Results from a Successful Balanced F Scale.Journal of Personality Assessment,43(6),638-643.
61. Redlawsk, David P.(2002).Hot Cognition or Cool Consideration? Testing the Effects of Motivated Reasoning on Political Decision Making.The Journal of Politics,64(4),1021-1044.
62. Schaeffer, Nora Cate,Maynard, Douglas W.(2008).The Contemporary Standardized Survey Interview for Social Research.Envisioning the Survey Interview of the Future,New York:
63. Schaeffer, Nora Cate,Maynard, Douglas W.(1996).From Paradigm to Prototype and Back Again: Interactive Aspects of Cognitive Processing in Survey Interviews.Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research,San Francisco:
64. Schuman, Howard,Presser, Stanley(1981).Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context.New York:Academic Press.
65. Snyder, Mark,Gangestad, Steve(1981).Hypothesis-testing processes.New Directions in Attribution Research, vol. 3,Hillsdale, NJ:
66. Swain, Scott D.,Weathers, Danny,Niedrich, Ronald W.(2008).Assessing Three Sources of Misresponse to Reversed Likert Items.Journal of Marketing Research,45(1),116-131.
67. Sykes, W.,Collins, M.(1988).Effects of Mode of Interview: Experiments in the UK.Telephone Survey Methodology,New York:
68. Tourangeau, Roger,Rips, Lance J.,Rasinski, Kenneth(2000).The Psychology of Survey Response.Cambridge:Cambridge University Press.
69. Van Vaerenbergh, Yves,Thomas, Troy D.(2013).Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies.International Journal of Public Opinion Research,25(2),195-217.
70. West, Brady T.,Blom, Annelies G.(2017).Explaining Interviewer Effects: A Research Synthesis.Journal of Survey Statistics and Methodology,5(2),175-211.
71. West, Brady T.,Sinibaldi, Jennifer(2013).The Quality of Paradata: A Literature Review.Improving Surveys with Paradata: Analytic Uses of Process Information,New York:
72. Wetzel, Eunike,Böhnke, Jan R.,Brown, Anna(2016).Response Biases.The ITC International Handbook of Testing and Assessment,New York:
73. Yang, Meng-Li(2019).Using Respondents’ Response Codability as an Indicator of Attitude Strength.International Journal of Public Opinion Research,31(4),738-752.