An Innovative Approach to Support the Reliabilities of Case-based Discussion Assessment

10.6145/jme.202012_24(4).0003

Heng-Hui Lien；Tien-Shang Huang；Yu-Chun Chiu；Huey-Ling Chen；Ming-Been Lee；Yen-Yuan Chen

case-based discussion ； reliability

Journal of Medical Education

24卷4期（2020 / 12 / 01）

173 - 182

英文

Purpose: To convincingly examine the reliabilities of Case-based Discussion (CbD), we developed three videotaped clinical scenarios played by standardized patients and standardized residents. The objective of this study was to examine the test-retest reliability, inter-rater reliability and internal consistency reliability of CbD to assess medical trainees' clinical reasoning and decision-making. Methods: We developed three clinical scenarios and videotaped as CbD I, CbD II, and CbD III for guaranteeing that all assessors rated exactly the same scenario. We purposefully recruited the assessors with fruitful experiences of assessing medical trainees' clinical reasoning and decision-making from four governmental and private university-affiliated medical centers, and two Veterans Affairs Council-affiliated medical centers. Each assessor was required: (1) to watch the videotaped discussions in consecutive order starting from CbD I in a single rating session; and (2) to assess each resident's clinical reasoning and decision-making in each discussion. We estimated the test-retest reliability, interrater reliability, and internal consistency reliability of CbD using Pearson's correlation coefficients (PCC), intra-class correlation coefficient (ICC), and internal consistency reliability (Cronbach's alpha), respectively. Results: We found that: (1) an assessor gave similar rating results in face of the same case-based discussion as indicated by acceptable test-retest reliability (PCC = .70~.78, P ＜ .01); (2) assessors gave similar rating results to a resident's clinical reasoning and decision-making as indicated by a good to excellent interrater reliability (PCC = .73~.88, P ＜ .01); and (3) all the items of CbD reflected uni-dimensionally a construct-a resident's clinical reasoning and decision-making as supported by a much better internal consistency reliability (Cronbach's alpha = .92~.95). Conclusions: The study results convincingly showed that CbD is a reliable assessment tool for assessing a medical trainee's or medical learner's clinical reasoning and decision-making, and can be widely used in clinical encounters to assess clinical reasoning and decision-making.

1. Awad, Z,Hayden, L,Muthuswamy, K(2014).Does direct observation of procedural skills refl ect trainee's progress in otolaryngology?.Clin Otolaryngol,39(3),169-173.
2. Birden, H,Glass, N,Wilson, I(2014).Defining professionalism in medical education: A systematic review.Med Teach,36(1),47-61.
3. Bodgener, S,Denney, M,Howard, J(2017).Consistency and reliability of judgements by assessors of case based discussions in general practice specialty training programmes in the United Kingdom.Educ Prim Care,28(1),45-9.
4. Brown, N,Holsgrove, G,Teeluckdharry, S(2011).Casebased discussion.Adv Psych Treat,17(2),85-90.
5. Bullock, A,Barnes, E,Ryan, B(2014).Case-based discussion supporting learning and practice in optometry.Ophthalmic Physiol Opt,34(5),614-621.
6. Chavez, AK,Swygert, KA,Peitzman, SJ(2013).Within-session score gains for repeat examinees on a standardized patient examination.Acad Med,88(5),688-692.
7. Chen, CC,Green, IC,Colbert-Getz, JM(2013).Warm-up on a simulator improves residents’ performance in laparoscopic surgery: A randomized trial.Int Urogynecol J,24(10),1615-1622.
8. Couto, TB,Farhat, SC,Geis, GL(2015).Highfidelity simulation versus case-based discussion for teaching medical students in Brazil about pediatric emergencies.Clinics (Sao Paulo),70(6),393-399.
9. Mehta, F,Brown, J,Shaw, NJ(2013).Do trainees value feedback in case-based discussion assessments?.Med Teach,35(5),e1166-e1172.
10. Miller, A,Archer, J(2010).Impact of workplace based assessment on doctors’ education and performance: A systematic review.BMJ,2010(341),c5064.
11. Mohanaruban, A,Flanders, L,Rees, H(2018).Case-based discussion: Perceptions of feedback.Clin Teach,15(2),126-131.
12. Moran-Atkin, E,Abdalla, G,Chen, G(2015).Preoperative warm-up the key to improved resident technique: A randomized study.Surg Endosc,29(5),1057-1063.
13. Swick, HM(2000).Toward a normative definition of medical professionalism.Acad Med,75(6),612-616.
14. Van Der Vleuten, CP(1996).The assessment of professional competence: Developments, research and practical implications.Adv Health Sci Educ Theory Pract,1(1),41-67.
15. Wikipedia: Internal consistency. Retrieved from: https://en.wikipedia.org/wiki/Internal_consistency Accessed 18 December 2020.
16. Wikipedia: Intraclass correlation. Retrieved from: https://en.wikipedia.org/wiki/Intraclass_correlation  Accessed 18 December 2020.
17. Wikipedia: Test retest. Retrieved from: https://en.wikipedia.org/wiki/Repeatability Accessed 18 December 2020.