Rater Effects and Corresponding Statistics for Performance Assessment

10.6145/jme201403

Shih-Chieh Liao

performance assessment (PA) ； rating inconsistency ； rater effects ； inter-rater reliability and validity

Journal of Medical Education

18卷1期（2014 / 03 / 01）

15 - 22

英文

For the past two decades remarkable changes have been implemented in medical education performance assessment (PA) reform, bringing into focus knowledge and clinical skill, but also the development of attitude, values, and interpersonal skills. Due to the emphasis on evaluating the comprehensive competencies of medical students by multiple raters, rating inconsistency inevitably occurs, with rater effects thus becoming the most serious drawback causing low inter-rater reliability and validity. This article attempts to provide a conceptual framework for and a concise update on commonly man-made errors/noises by analyzing the cause of each rater effect using a four-phase inter-rater reliability model. The author gives quantitative evidence that confirms the existence of rater effects. Four recommendations for minimizing rater effects and improving rating accuracy are offered as well.

1. Afflerbach, P,Kapinus, B,Winograd, P(1994).Developing alternative assessments: six problems worth solving.Read Teach,47(5),420-23.
2. Arter, J(2005).Teaching about performance assessment.Educ Meas: Issues and Practice,18,30-44.
3. Aschbacher, PR(1991).Performance assessment: state activity, interest, and concerns.Appl Meas Educ,4,275-88.
4. Bernardin, HJ,Walter, CS(1977).Effects of rater training and diary-keeping on psychometric error in ratings.J Appl Psychol,62,64-9.
5. Clauser, BE,Clyman, SG,Swanson, DB(1999).Components of rater error in a complex performance assessment.J Educ Mea,36,29-45.
6. Cooke, M,Irby, DM,Sullivan, W(2006).American medical education 100 years after the Flexner report.N Engl J Med,355,1339-44.
7. Cusimano, MD,Cohen, R,Tucker, W(1994).A comparative analysis of the costs of administration of an OSCE.Acad Med,69,571-5.
8. Darling-Hammond, L,Adamson, F(2010).,Stanford, CA:Stanford University Press.
9. Dunbar, SB,Koretz, D,Hoover, HD(1991).Quality control in development and use of performance assessments.Applied Measurement in Education,4,289-304.
10. Epstein, RM(2007).Assessment in medical education.N Engl J Med,356,387-96.
11. Harnisch, DL(1994).Performance assessment in review: new directions for assessing student understanding.Int J Educ Res,2,341-50.
12. Herman, JL,Aschbacher, PR,Winters, L(1992).A Practical Guide to Alternative Assessment.Alexandria, VA:Association for Supervision.
13. Holzbach, RL(1978).Rater bias in performance ratings: superior, self-, and peer ratings.J Appl Psychol,63,579-88.
14. Howley, LD(2004).Performance assessment in medical education: where we've been and where we're going.Eval Health Prof,27,285-303.
15. Iramaneerat, C,Yudkowsky, R(2007).Rater errors in a clinical skills assessment of medical students.Eval Health Prof,30,266-83.
16. Kane, MB,Mitchell, R(1996).Implementing Performance Assessment: Promises, Problems, and Challenges.Mahwah, NJ:Lawrence Erlbaum Associates Press.
17. Landy, FJ,Farr, JL(1980).Performance rating.Psychol Bull,87,72-107.
18. Liao, SC,Hunt, EA,Chen, W(2010).Comparison between inter-rater reliability and inter-rater agreement in performance assessment.Ann Acad Med Singap,29,613-8.
19. Linn, R,Baker, E,Dunbar, S(1991).Complex, performance-based assessment: expectations and validation criteria.Educational Researcher,20,16-21.
20. Madaus, GF,O'Dwyer, LM(1999).A short history of performance assessment: lessons learned.Phi Delta Kappa,80(9),688-95.
21. Maurer, SD,Lee, TW(2000).Accuracy of the situational interview in rating multiple job candidates.J Bus Psychol,15,73-96.
22. Miller, GE(1990).The assessment of clinical skills/competence/performance.Acad Med,65,563-67.
23. Muraki, E,Hombo, CM,Lee, YW(2000).Equating and linking of performance assessments.Appl Psych Meas,4,325-37.
24. Norcini, JJ,McKinley, DW(2007).Assessment methods in medical education.Teaching and Teacher Education,23,239-50.
25. Raymond, MR,Houston, WM(1990).Detecting and Correcting for Rater Effects in Performance Assessment.Boston, MA:American College Testing Press.
26. Raymond, MR,Webb, LC,Houston, W.M.(1991).Correcting Performance-rating errors in oral examinations.Eval Health Prof,14(1),100-22.
27. Rowe, PM(1967).Order effects in assessment decisions.J Appl Psychol,51,170-3.
28. Swanson, DB,Norman, GR,Linn, RL(1995).Performance-based assessment: lessons from the health professions.Educational Researcher,24(5),5-11.
29. Tsui, AS,Barry, B(1986).Research notes: interpersonal affect and rating errors.Acad Manage J,29,586-99.