Variable Results of Using Explicit Behavioral Criteria to Improve Validity of Students' Self-Evaluation in Problem-based Learning

以測量明確的行為來提昇醫學生問題導向學習自我評估的準確度

10.6145/jme.200812_12(4).0003

梁繼權(Kai-Kuen Leung)；王維典(Wei-Dean Wang)

醫學教育 ； 問題導向學習 ； 學習評估 ； 自我評估 ； medical education ； problem-based learning ； evaluation ； self-assessment

Journal of Medical Education

12卷4期（2008 / 12 / 01）

216 - 224

英文

Purpose: To test the hypothesis that using explicit behavioral criteria for evaluation can improve the accuracy of students' self-evaluation in problem-based learning (PBL). Methods: Second to fourth-year medical students of a seven-year curriculum were studied. PBL performance of students was assessed by students using a student version of the Tutotest (S-Tutotest(superscript C)) for self-evaluation, and by facilitating tutors using the tutor version of the Tutotest (T-Tutotest(superscript C)) and a global rating score for evaluation and feedback. A multiple-choice achievement test was taken by students at the end of the PBL course. Results: 315 students' evaluations were obtained with a 78.9% completion rate. Students' self-ratings were poorly correlated to tutors' ratings. Tutors' Tutotest ratings were moderately correlated to their global ratings. Students' achievement test scores were poorly correlated to tutors' Tutotest ratings and global ratings, and uncorrelated with students' self-ratings. Students of poor achievers tended to overestimate and students of good achievers tended to underestimate their performance. Conclusions: Our study did not fully support Arthur's argument that explicit criteria of performance are important for the validity improvement of self-evaluation. Results of student evaluation varied with methods of evaluation. Explicit behavior criteria are useful for average grade students to use self-evaluation for PBL learning but not so for students with poor or good academic achievement.

Purpose: To test the hypothesis that using explicit behavioral criteria for evaluation can improve the accuracy of students' self-evaluation in problem-based learning (PBL). Methods: Second to fourth-year medical students of a seven-year curriculum were studied. PBL performance of students was assessed by students using a student version of the Tutotest (S-Tutotest(superscript C)) for self-evaluation, and by facilitating tutors using the tutor version of the Tutotest (T-Tutotest(superscript C)) and a global rating score for evaluation and feedback. A multiple-choice achievement test was taken by students at the end of the PBL course. Results: 315 students' evaluations were obtained with a 78.9% completion rate. Students' self-ratings were poorly correlated to tutors' ratings. Tutors' Tutotest ratings were moderately correlated to their global ratings. Students' achievement test scores were poorly correlated to tutors' Tutotest ratings and global ratings, and uncorrelated with students' self-ratings. Students of poor achievers tended to overestimate and students of good achievers tended to underestimate their performance. Conclusions: Our study did not fully support Arthur's argument that explicit criteria of performance are important for the validity improvement of self-evaluation. Results of student evaluation varied with methods of evaluation. Explicit behavior criteria are useful for average grade students to use self-evaluation for PBL learning but not so for students with poor or good academic achievement.

1. Arthur H(1995).Student self-evaluations: how useful? How valid?.Int J Nurs Stud,32(3),271-276.
2. Barnsley L,Lyon PM,Ralston SJ(2004).Clinical skills in junior medical officers: a comparison of self-reported confidence and observed competence.Med Educ,38,358-367.
3. Boud D,Falchikov N(1989).Quantitative studies of student self-evaluation in higher: a critical higher education: a critical analysis of finding.J Higher Educ,18,529-549.
4. Chur-Hansen A(2000).Medical students'essay-writing skills: criteria-based self- and tutor-evaluation and the role of language background.Med Educ,34,194-198.
5. Das M,Mopfu D,Dunn E(1998).Self and tutor evaluations in problem-based learning tutorials: is there a relationship?.Med Educ,32,411-418.
6. Farnill D,Hayes SC,Todisco J(1997).Interviewing skills: self-evaluation by medical students.Med Educ,31,122-127.
7. Fitzgerald JT,Gruppen LD,White CB(2000).The influence of task formats on the accuracy of medical students'self-evaluations.Acad Med,75,737-741.
8. Fitzgerald JT,White CB,Gruppen LD(2003).A longitudinal study of self-evaluation accuracy.Med Educ,37,645-649.
9. Gordon MJ(1991).A review of the validity and accuracy of self-evaluations in health professions training.Acad Med,66,762-769.
10. Hébert R,Bravo G(1996).Development and validation of an evaluation instrument for medical students in tutorials.Acad Med,71,488-494.
11. Herbert WN,McGaghie WC,Droegemueller W(1990).Student evaluation in obstetrics and gynecology: self- versus departmental evaluation.Obstet Gynecol,76,458-461.
12. Magarian GJ,Mazur DJ(1990).Evaluation of students in medicine clerkships.Acad Med,65,341-345.
13. Mattheos N,Nattestad A,Falk-Nilsson E(2004).The interactive examination: assessing students'self-evaluation ability.Med Educ,38,378-389.
14. Methany WP(1991).Limitations of physician ratings in the evaluation of student clinical performance in an obstetrics and gynecology clerkship.Obstet Gynecol,78,136-141.
15. Murphy KR,Balzer WK(1989).Rater errors and rating accuracy.J Appl Psychol,74,619-624.
16. Reiter HI,Eva KW,Hatala RM(2002).Self and peer evaluation in tutorials: application of a relative-ranking model.Acad Med,77,1134-1139.
17. Rezler AG(1989).Self-evaluation in problem-based groups.Med Teach,11(2),151-156.
18. Sullivan ME,Hitchcock MA,Dunnington GL(1999).Peer and self-evaluation during problem-based tutorials.Am J Surg,177,266-269.
19. Torsignant M,DesMarchais JE(2002).Accuracy of student self-evaluation ability compared to their own performance in a problem-based learning medical program: a correlation study.Adv Health Sci Educ Theory Pract,7(1),19-27.
20. Ward M,Gruppen L,Regehr G(2002).Measuring selfevaluation: current state of the art.Adv Health Sci Educ Theory Pract,7,63-80.
21. Woolliscroft JO,TenHaken J,Calhoun JG.(1993).Medical students'clinical self-evaluations: comparisons with external measures of performance and the students'self-evaluations of overall performance and effort.Acad Med,68,285-294.