题名

追蹤調查中問卷回答品質的混合模式效應

并列篇名

Mode Effects on Response Behaviors in a Mixed-mode Survey

DOI

10.7014/SRMA.2023100002

作者

杜素豪(Su-hao Tu)；于若蓉(Ruoh-Rong Yu)；洪琮瑋(Tsung-Wei Hung)

关键词

追蹤調查 ； 模式效應 ； 題項無回應 ； 默許 ； 極端回答 ； 傾向分數配對法 ； panel survey ； mode effect ； item nonresponse ； acquiescence ； extreme response ； propensity score matching

期刊名称

調查研究-方法與應用

卷期/出版年月

51期（2023 / 10 / 01）

页次

57 - 117

内容语文

繁體中文;英文
中文摘要

為了提高母體涵蓋率與問卷回收率，降低調查執行成本，混合模式（mixed mode）已普遍應用於大型追蹤調查。若不同調查模式之間的樣本非隨機分配，除了存在因不同模式產生的測量效應（即模式效應），同時還會有樣本以何種模式完訪的選樣效應。若擬探討不同模式對受訪者答題行為的影響，首要的挑戰是如何釐清並存的兩種效應。本研究採用可因應樣本非隨機分派的傾向分數配對法，探討大型追蹤調查答題行為的模式效應，比較總體題項無回應、拒答、「不知道」回答，以及態度量表的默許、極端回答風格在面訪模式與網路自填問卷模式之間是否有明顯的差異。分析資料來自「家庭動態調查」在2018年以面訪、網路自填問卷模式同步蒐集到的問卷資料。由於面訪模式與網路自填問卷模式之間的樣本數差異大，為了提高傾向分數配對法的有效配對數，擴大估計的涵蓋範圍，以及降低估計誤差，本研究採用以擴大樣本（oversampling）與配對替代（replacement）為原則的兩種配對方法，分別為半徑配對（radius matching）與核函數配對。模式效應的估計結果顯示，網路自填問卷模式發生題項無回應與「不知道」回答的可能性明顯較面訪為高，與既有文獻的發現相近。在平衡型態度量表題組中，面訪的默許風格明顯較網路自填問卷模式嚴重，但網路自填問卷模式的極端回答風格明顯較面訪嚴重。其中，默許風格的發現如同預期，但極端風格的發現則不同於既有研究。儘管本文的研究課題限於題項無回應、態度量表回答風格兩類答題行為，而傾向分數配對法的應用上仍可能存在改善的空間，在調查方法研究、分析方法應用與調查實務上均具參考價值。
英文摘要

In order to improve coverage and response rates, and to reduce survey cost, mixed modes have been commonly used in large-scale panel surveys. If the assignment of a survey mode is not random, the respondents' answers to a mixed-mode survey might be subject to two kinds of biases. One is measurement bias (i.e., mode effects) evoked by the modes themselves, and the other is sample selection bias, which results from the respondents' non-random assignment to different modes. How to disentangle these two biases is a crucial challenge for the estimation of mode effects. This study adopts propensity score matching, an analytical method which can deal with non-random sample assignment, to examine mode effects on response behaviors in a panel survey with a mixed-mode design of face-to-face and self-administered online modes. The outcome variables analyzed in this study include overall item nonresponse, "refusal" and "don't know" answers, and two response styles in balanced attitude scales, namely the acquiescence and extreme response styles. Data analyzed in this study are from the Panel Study of Family Dynamics survey conducted in 2018, in which the sample was pre-assigned to face-to-face and self-administered online modes based on whether they provided an email address and finished an online questionnaire previously sent with a festival greeting card. There is a large difference in the numbers of complete questionnaires between the face-to-face and self-administered online modes. In order to improve the estimates for measurement effects, this study uses two matching methods, including the radius and kernel matching methods. These two matching methods are based on an oversampling strategy and matching with replacement. The results of the two matching methods indicate that the probabilities of occurrence of item nonresponse and "don't know" answers in the self-administered online mode were significantly higher than those in the face-to-face mode, consistent with previous studies. It is also consistent with previous studies in that, regarding responses to the balanced attitudinal scales, respondents who finished the questionnaires by face-to-face interviews were significantly more likely to provide acquiescent responses than those who filled out online questionnaires by themselves. However, different from previous studies, our findings indicate that respondents who completed self-administered online questionnaires were more likely to provide extreme responses to items on the balanced attitudinal scales than face-to-face interviewees. One other finding worth mentioning is that this study did not find significant mode effects for "refusal" answers. This study contributes to research on mode effects, applications of propensity score matching, and survey practices. Our findings suggest that, to mitigate mode effects, a mixed-mode survey including face-to-face and self-administered modes should adopt the same design for "don't know" and "refusal" options between modes, and respondents in the face-to-face mode should be allowed to enter answers to questions with social desirability concerns by themselves. Despite these academic and practical contributions, our study still has its limitations. One is that the response behaviors explored in this study are confined to overall item-nonresponse, "don't know" and "refusal" answers, and the style of responses to attitude scales. In addition to extending the investigation of mode effects to a broader range of survey questions, future research should endeavor to increase the application of propensity score matching methods in order to disentangle the mode effects and selections effects in mixed-mode surveys. Future directions include but are not confined to the selection of covariates for the logistic model used to predict propensity scores, the methods of imputing missing values, and other matching strategies.
主题分类 社會科學 > 社會科學綜合
参考文献
  1. 于若蓉, Ruoh-Rong,黃奕嘉, I-Chia(2018)。家庭動態調查：樣本結構、問卷內容、資料外釋與應用。中國統計學報，56(4)，98-115。
    連結：
  2. 侯佩君, Pei-Chun,杜素豪, Su-Hao,廖培珊, Pei-Shan,洪永泰, Yung-Tai,章英華, Ying-Wha(2008)。台灣鄉鎮市區類型之研究：『台灣社會變遷基本調查』第五期計畫之抽樣分層效果分析。調查研究—方法與應用，23，7-32。
    連結：
  3. Aakvik, Arild(2001).Bounding a matching estimator: the case of a Norwegian training program.Oxford Bulletin of Economics and Statistics,63(1),115-143.
  4. Austin, Perter C.(2013).The Performance of Different Propensity Score Methods for Estimating Marginal Hazard Ratios.Statistics in Medicine,32(16),2837-2849.
  5. Austin, Perter C.(2014).A Comparison of 12 Algorithms for Matching on the Propensity Score.Statistics in Medicine,33(6),1057-1069.
  6. Austin, Perter C.(2011).An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies.Multivariate Behavioral Research,46(3),399-424.
  7. Austin, Perter C.(2009).Some Methods of Propensity-Score Matching had Superior Performance to Others: Results of an Empirical Investigation and Monte Carlo Simulations.Biometrical Journal,59(1),171-184.
  8. Baser, Onur(2006).Too Much Ado about Propensity Score Models? Comparing Methods of Propensity Score Matching.Value in Health,9(6),377-385.
  9. Becker, Sascha O.,Caliendo, Marco(2007).Sensitivity Analysis for Average Treatment Effects.The Stata Journal,7(1),71-83.
  10. Berhman, J. R.,Cheng, Y.,Todd, P. E.(2004).Evaluating Preschool Programs When Length of Exposure to the Program Varies: A Nonparametric Approach.Review of Economics and Statistics,86(1),108-132.
  11. Biemer, Paul P.(2001).Nonresponse Bias and Measurement Bias in a Comparison of Face to Face and Telephone Interviewing.Journal of Official Statistics,17(2),295-320.
  12. Biemer, Paul P.,Lyberg, Lars E.(2003).Introduction to Survey Quality.NJ:John Wiley & Sons.
  13. Börkan, Bengü(2010).The Mode Effect in Mixed-mode Surveys: Mail and Web Surveys.Social Science Computer Review,28(3),371-380.
  14. Bottigliengo, D.,Bald, I.,Lanera, C.,Lorenzoni, G.,Bejko, J.,Bottio, T.,Tarzia, V.,Carrozzini, M.,Gerosa, G.,Berchialla, P.,Gregori, D.(2021).Oversampling and Replacement Strategies in Propensity Score Matching: A Critical Review Focused on Small Sample Size in Clinical Settings.BMC Medical Research Methodology,21(256),1-16.
  15. Bottigliengo, D.,Lorenzoni, G.,Ocagli, H.,Martinato, M.,Berchialla, P.,Gregori, D.(2021).Propensity Score Analysis with Partially Observed Baseline Covariates: A Practical Comparison of Methods for Handling Missing Data.International Journal of Environmental Research and Public Health,18(6694),1-17.
  16. Caliendo, Macro,Kopeinig, Sabine(2008).Some Practical Guidance for the Implementation of Propensity Score Matching.Journal of Economic Surveys,22(1),31-72.
  17. Cernat, Alexandru,Revilla, Melanie(2021).Moving from Face-to-face to a Web Panel: Impacts on Measurement Quality.Journal of Survey Statistics and Methodology,9(4),745-763.
  18. Chen, Chuansheng,Lee, Shin-ying,Stevenson, Harold W.(1995).Response Style and Cross- cultural Comparisons of Rating Scales among East Asian and North American Students.Psychological Science,6(3),170-175.
  19. Choi, J.,Dekkers, O. M.,Cessie, S. L.(2019).A Comparison of Different Methods to Handle Missing Data in the Context of Propensity Score Analysis.European Journal of Epidemiology,34,23-36.
  20. Cochran, W. G.,Rubin, D. B.(1973).Controlling Bias in Observational Studies: A Review.Sankhya Series A,35,417-446.
  21. Datta, P.,Walsh, K.,Terrell, D.(2002).The Impact Demographics on Choice of Survey Modes.Communications of the Association for Information Systems,9(13),223-240.
  22. De Leeuw, Edith D.(2005).To Mix or Not to Mix Data Collection Modes in Surveys.Journal of Official Statistics,21(5),233-255.
  23. De Leeuw, Edith D.(2018).Mixed-mode: Past, Present, and Future.Survey Research Methods,12(2),75-89.
  24. De Leeuw, Edith D.,Dillman, Don A.,Hox, Joop J.(2008).Mixed Mode Surveys: When and Why.International Handbook of Survey Methodology,NY:
  25. Díaz de Rada, Vidal(2022).Concurrent Mixed Modes: Response Quality, Speed, and Cost.Field Methods,34(3),191-205.
  26. Dillman, Don A.,Smyth, Jolene D.,Christian, Leah Melani(2009).Internet, Mail, and Mixed- Mode Surveys: The Tailored Design Method.Hoboken, NJ:John Wiley & Sons.
  27. Everitt, Brian S.(ed.),Howell, David C.(ed.)(2005).Encyclopedia of Statistics in Behavioral Science.NJ:John Wiley & Sons.
  28. Fowler, Floyd Jackson,Gallagher, P. M.,Stringfellow, V. L.,Zaslavsky, A. M.,Thompson, J. W.,Cleary, P. D.(2002).Using Telephone Interviews to Reduce Nonresponse Bias to Mail Surveys of Health Plan Members.Medical Care,40(3),190-200.
  29. Fricker, Scott,Galesic, Mirta,Tourangeau, Roger,Yan, Ting(2005).An Experimental Comparison of Web and Telephone Surveys.Public Opinion Quarterly,69(3),370-392.
  30. Galdo, Jose C.,Smith, Jeffrey,Black, Dan(2008).Bandwidth Selection and the Estimation of Treatment Effects with Unbalanced Data.Annales d’Économie et de Statistique,91(92),189-216.
  31. Garrido, M. M.,Kelley, A. S.,Paris, J.,Roza, K.,Meier, D. E.,Morrison, R. S.,Aldridge, M. D.(2014).Methods for Constructing and Assessing Propensity Scores.Health Research and Educational Trust,49(5),1701-1718.
  32. Guo, Shenyang,Fraser, Mark W.(2015).Propensity Score Analysis.LA:Sage.
  33. Harris, Heather,Horst, S. Jeanne(2016).A Brief Guide to Decisions at Each Step of the Propensity Score Matching Process.Practical Assessment, Research, and Evaluation,21(4),1-11.
  34. Hazlett, Chad(2020).Kernel Balancing: A Flexible Non-parametric Weighting Procedure for Estimating Causal Effects.Statistica Sinica,30,1155-1189.
  35. Heerwegh, Dirk(2009).Mode Differences Between Face-to-face and Web Surveys: An Experimental Investigation of Data Quality and Social Desirability Effects.International Journal of Public Opinion Research,21(1),111-121.
  36. Heerwegh, Dirk,Loosveldt, Geert(2008).Face-to-face versus Web Surveying in a High-internet-coverage Population: Differences in Response Quality.Public Opinion Quarterly,72(5),836-846.
  37. Heerwegh, Dirk,Loosveldt, Geert(2011).Assessing Mode Effects in a National Crime Victimization Survey Using Structural Equation Models: Social Desirability Bias and Acquiescence.Journal of Official Statistics,27(1),49-63.
  38. Hibbing, Matthew V.,Cawvey, Matthew,Deol, Raman,Bloeser., Andrew J.,Mondak, Jeffery J.(2019).The Relationship between Personality and Response Patterns on Public Opinion Surveys: The Big Five, Extreme Response Style, and Acquiescence Response Style.International Journal of Public Opinion Research,31(1),161-177.
  39. Holbrook, Allyson L.,Green, Melanie C.,Krosnick, Jon A.(2003).Telephone versus Face-to-face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias.Public Opinion Quarterly,67(1),79-125.
  40. Holford, Augus J.,Pudney, Stephen(2015).IZA Discussion PapersIZA Discussion Papers,Bonn:Institute for the Study of Labor (IZA).
  41. Huber, M.,Lechner, M.,Steinmayr, A.(2015).Radius Matching on the Propensity Score with Bias Adjustment: Tuning Parameters and Finite Sample Behavior.Empirical Economics,49,1-31.
  42. Jäckle, Annette,Roberts, Caroline,Lynn, Peter(2010).Assessing the Effect of Data Collection Mode on Measurement.International Statistical Review,78(1),3-20.
  43. Jiang, W.,Ha, L.,Abuljadail, M.,Alsulaiman, S. A.(2017).Item Non-response of Different Question Types and Formats in Mixed-mode Surveys: A Case Study of a Public Broadcasting TV Station’s Members.Journal of Communication and Media Research,9(1),173-184.
  44. Jones, M. K.,Calzavara, L.,Allman, D.,Worthington, C. A.,Tyndall, M.,Iveniuk, J.(2016).A Comparison of Web and Telephone Responses from a National HIV and AIDS Survey.JMIR Public Health Surveill,2(2),1-15.
  45. King, Gary,Nielson, Richard(2019).Why Propensity Scores Should Not Be Used for Matching.Political Analysis,27(4),435-454.
  46. Klausch, L. Thomas,Schouten, Barry,Hox, Joop J.(2014).,Statistics Netherlands.
  47. Klausch, L. Thomas,Schouten, Barry,Hox, Joop J.(2017).Evaluating Bias of Sequential Mixed-mode Designs Against Benchmark Surveys.Sociological Methods & Research,46(3),456-489.
  48. Kohler, U.,Karlson, K. B.,Holm, A.(2011).Comparing Coefficients of Nested Nonlinear Probability Models.The Stata Journal,11(3),420-438.
  49. Kreuter, Frauke,Müller, Gerrit,Trappmann, Mark(2010).Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data.Public Opinion Quarterly,74(5),880-906.
  50. Kreuter, Frauke,Presser, Stanley,Tourangeau, Roger(2008).Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity.Public Opinion Quarterly,72(5),847-865.
  51. Krosnick, Jon A.(1991).Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.Applied Cognitive Psychology,5(3),213-236.
  52. Krosnick, Jon A.,Alwin, Duane F.(1987).An Evaluation of a Cognitive Theory of Response-order Effects in Survey Measurement.Public Opinion Quarterly,51(2),201-219.
  53. Link, Michael W.,Mokdad, Ali(2006).Can Web and Mail Survey Modes Improve Participation in an RDD-based National Health Surveillance?.Journal of Official Statistics,22(2),293-312.
  54. Liu, Mingnan,Conrad, Frederick G.,Lee, Sunghee(2017).Comparing Acquiescent and Extreme Response Styles in Face-to-face and Web Surveys.Quality & Quantity,51,941-958.
  55. Lugtig, Peter,Lensvelt-Mulders, Gerty J. L. M.,Frerichs, Remco,Greven, Assyn(2011).Estimating Nonresponse Bias and Mode Effects in a Mixed-mode Survey.International Journal of Market Research,53,669-686.
  56. Lynn, Peter(2013).Alternative Sequential Mixed-mode Designs: Effects on Attrition Rates, Attrition Bias, and Costs.Journal of Survey Statistics and Methodology,1(2),183-205.
  57. Mariano, Louis T.,Elliott, Marc N.(2017).An Item Response Theory Approach to Estimating Survey Mode Effects: Analysis of Data from a Randomized Mode Experiment.Journal of Survey Statistics and Methodology,5(2),233-253.
  58. Moors, Guy(2008).Exploring the Effect of a Middle Response Category on Response Style in Attitude Measurement.Quality & Quantity,42,779-794.
  59. Pearl, Judea(1995).Causal Diagrams for Empirical Research.Biometrika,82,669-688.
  60. Pearl, Judea(2009).Causality: Models, Reasoning and Inference.NY:Cambridge University Press.
  61. Piccitto, G.,Liefbroer, A. C.,Emery, T.(2022).Does the Survey Mode Affect the Association between Subjective Well-being and its Determinants? An Experimental Comparison Between Face-to-Face and Web Mode.Journal of Happiness Studies,23,3441-3461.
  62. Ripollone, J. E.,Huybrechts, K. F.,Rothman, K. J.,Ferguson, R. E.,Franklin, J. M.(2018).Implications of the Propensity Score Matching Paradox in Pharmacoepidemiology.American Journal of Epidemiology,187(9),1951-1961.
  63. Rosenbaum, Paul R.(2002).Observational Studies.NY:Springer.
  64. Rosenbaum, Paul R.,Rubin, Donald B.(1983).The Central Role of the Propensity Score in Observational Studies for Causal Effects.Biometrika,70(1),41-55.
  65. Rosenbaum, Paul R.,Rubin, Donald B.(1985).Constructing a Control Group Using Multivariate Matched Sampling Methods that Incorporate the Propensity Score.The American Statistician,39(1),33-38.
  66. Rubin, Donald B.(2001).Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation.Health Services & Outcomes Research Methodology,2(3),169-188.
  67. Rubin, Donald B.(1974).Estimating Causal Effects of Treatments in Randomized and Non-randomized Studies.Journal of Educational Psychology,66,688-701.
  68. Sakshaug, Joseph W.,Yan, Ting,Tourangeau, Roger(2010).Nonresponse Error, Measurement Error, and Mode of Data Collection: Tradeoffs in a Multi-mode Survey of Sensitive and Non-sensitive Items.Public Opinion Quarterly,74(5),907-933.
  69. Schouten, Barry,van den Brakel, Jan,Buelens, Bart,van der Laan, Jan,Klausch, Thomas(2013).Disentangling Mode-specific Selection and Measurement Bias in Social Surveys.Social Science Research,42(6),1555-1570.
  70. Valentino, Nicholas A.,Zhirkov, Kirill,Hillygus, D. Sunshine,Guay, Brian(2020).The Consequences of Personality Biases in Online Panels for Measuring Public Opinion.Public Opinion Quarterly,84(2),446-468.
  71. Vannieuwenhuyze, Jorre T. A.,Loosveldt, Geert(2013).Evaluating Relative Mode Effects in Mixed-mode Surveys: Three Methods to Disentangle Selection and Measurement Effects.Sociological Methods & Research,42(1),82-104.
  72. Vannieuwenhuyze, Jorre T. A.,Loosveldt, Geert,Molenberghs, Geert(2014).Evaluating Mode Effects in Mixed-mode Survey Data Using Covariate Adjustment Models.Journal of Official Statistics,30(1),1-21.
  73. Vannieuwenhuyze, Jorre T. A.,Loosveldt, Geert,Molenberghs, Geert(2012).A Method to Evaluate Mode Effects on the Mean and Variance of a Continuous Variable in Mixed-Mode Surveys.International Statistical Review,80(2),306-322.
  74. Vannieuwenhuyze, Jorre T. A.,Loosveldt, Geert,Molenberghs, Geert(2010).A Method for Evaluating Mode Effects in Mixed-mode Surveys.Public Opinion Quarterly,74(5),1027-1045.
  75. Voogt, Robert J. J.,Saris, Willem E.(2005).Mixed Mode Designs: Finding the Balance between Nonresponse Bias and Mode Effects.Journal of Official Statistics,21(3),367-387.
  76. Weijters, B.,Schillewaert, N.,Geuens, M.(2008).Assessing Response Styles across Modes of Data Collection.Journal of the Academy of Marketing Science,36,409-422.
  77. Yanovitsky, I.,Zanutto, E. L.,Hornik, R.(2005).Estimating Causal Effects of Public Health Education Campaigns Using Propensity Score Methodology.Evaluation and Program Planning,28(2),209-220.
  78. Zhang, Chan,Conrad, Frederick(2014).Speeding in Web Surveys: The Tendency to Answer Very Fast and Its Association with Straightling.Survey Research Methods,8(2),127-135.
  79. 于若蓉，2019，家庭動態資料庫的建立：第十七年計畫（RR2018）（C00333_1），取自中央研究院人文社會科學研究中心調查研究專題中心學術調查研究資料庫。（Yu Ruoh-Rong, 2019, Panel Study of Family Dynamics: RR2018 (C00333_1) [data file]. Available from Survey Research Data Archive, Academia Sinica. https://doi.org/10.6141/TW-SRDA-C00333_1-1）
print cancel