Developing Assessment Criteria and Rubrics for Narrative Medicine Writing

10.6145/jme201710

Hung-Chang Liao；Ya-huei Wang

assessment ； narrative medicine ； writing scoring rubric ； validity ； reliability

Journal of Medical Education

21卷3期（2017 / 09 / 01）

87 - 100

英文

Purpose: Narrative medicine writing can help health care students and professionals make empathetic connections with their patients in order to make patient-centered decisions. Nonetheless, though the use of narrative medicine writing exercises within medical-care education and on-the-job training is expanding, the assessment tools to determine narrative medicine writing levels are lacking. Therefore, there is a need for developing assessment criteria and rubrics for evaluating health care students' and professionals' narrative medicine writing. Methods: Extensive literature review and expert panel discussion were used to develop an analytic narrative medicine writing scoring rubric (ANMWSR). Once developed, the scoring rubric was validated through iterative cycles of examination, including item objective congruence (IOC), inter-rater reliabilities using percent exact and adjacent agreement, and inter-rater and intra-rater reliabilities using Pearson's correlation coefficients. Results: The results from the study demonstrated that the content validity of the ANMWSR was appropriate; 17 out of 20 qualitative scoring items reached the acceptable IOC indexes, ranging from 0.5 to 1.0. Three items that did not reach the criteria were modified and verified. The consensus estimates of inter-rater reliability with percent exact and adjacent agreements were between 92% and 100%. The consistent estimates of inter-rater reliabilities were between 0.81 and 0.88. Intra-rater reliability was 0.92 and 0.99. Conclusion: Upon testing, the iterative verification of content validity, inter-rater reliability, and intra-reliability, with derived values all higher than the reported estimates in most studies, proved the developed ANMWSR rubric a valid and reliable evaluation instrument to evaluate health care students' and professionals' narrative medicine writing skills.

1. TSA: Research ethics framework of society institute in Taiwan. Available at: http://proj3.sinica.edu.tw/~tsa/modules/tadnews/index.php?nsn=43. Accessed 23 November 2015.
2. AACSB: AACSB Assurance of learning: An interpretation. 2013. Available at: http://www.aacsb.edu/~/media/AACSB/Publications/whitepapers/wp-assurance-of-learning-standards.ashx. Accessed 08 Aug 2014.
3. Allen, S,Knight, J(2009).A method for collaboratively developing and validating a rubric.International Journal for the Scholarship of Teaching and Learning. IJSOTL,3,Article 10.
4. Batistatou, A,Doulis, EA,Tiniakos, D(2010).The introduction of medical humanities in the undergraduate curriculum of Greek medical schools: Challenge and necessity.Hippokratia,14,241-3.
5. Brown, GTL,Glasswell, K,Harland, D(2004).Accuracy in the scoring of writing: Studies of reliability and validity using a new Zealand writing assessment system.Assessing Writing,9,105-21.
6. Brown, JD(1996).Testing in language programs.NJ:Prentice Hall Regents.
7. Charon, R(2006).Narrative medicine: Honoring the stories of illness.Oxford:Oxford University Press.
8. DasGupta, S,Charon, R(2004).Personal illness narratives: Using reflective writing to teach empathy.Acad Med,79,351-6.
9. Greenhalgh, T,Hurwitz, B(1998).Narrative based medicine.London:BMJ Books.
10. Gull, SE(2005).Embedding the humanities into medical education.Med Edu,39,235-6.
11. Hammer, R,Dingel, M,Ostergren(2012).The experience of addiction as told by the addicted: incorporating biological understandings into selfstory.Cult Med Psychiatry,36,712-34.
12. Hannah, W(2013).Narrative and a Christian bioethics.Ethics & Medicine,29,111-26.
13. Harrison, A(1983).A language testing handbook.Hong Kong:Macmillan.
14. Hojat, M,Gonnella, JS,Mangione, S(2002).Empathy in medical students as related to academic performance, clinical competence and gender.Med Educ,36,522-7.
15. Jonsson, A,Svingby, G(2007).The use of scoring rubrics: Reliability, validity and educational consequences.Edu Res Rev,2,130-44.
16. Kalitzkus, V,Matthiessen, PF(2009).Narrative-based medicine: Potential, pitfalls, and practice.Perm J,13,80-6.
17. McNamara, F(1996).Measuring second language performance.London:Longman.
18. Moskal, BM(2000).Scoring rubrics: what, when and how?.Practical Assessment, Research & Evaluation,7(3)
19. Moskal, BM,Leydens, JA(2000).Scoring rubric development: Validity and reliability.Prac Assess Res Eval,7,71-81.
20. Musial, JL,Rubinfeld, IS,Parker, AO(2007).Developing a scoring rubric for resident research presentations: A pilot study.J Surg Res,142,304-7.
21. Reddy, MY(2011).Design and development of rubrics to improve assessment outcomes: A pilot study in a Master's level business program in India.Qlty Assur Edu,19,84-104.
22. Remen, RN(1996).Kitchen table wisdom: Stories that heal.New York:Riverhead Books.
23. Smith, LJ(2008).Grading written projects: What approaches do students find most helpful.JEB,83,325-30.
24. Solomon, M(2016).On ways of knowing in medicine.CMAJ,188,289-90.
25. Stemler, SE(2004).A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability.Practical Assessment, Research & Evaluation,9(4)
26. Stevens, DD,Levi, A(2005).Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning.Sterling, VA:Stylus Publishing.
27. Wald, HS,Borkan, M,Taylor, SJ(2012).Fostering and evaluating reflective capacity in medical education: Developing the REFLECT rubric for assessing reflective writing.Acad Med,87,41-50.