Construction and Validation of an Analytic Reflective Writing Scoring Rubric for Healthcare Students and Providers

10.6145/jme.202006_24(2).0002

Ya-Huei Wang；Hung-Chang Liao

healthcare ； medical education ； reflective thinking ； reflective writing ； writing assessment

Journal of Medical Education

24卷2期（2020 / 06 / 01）

53 - 72

英文

Purpose: This paper reports results from a study on constructing and validating a rubric for systematic writing assessment in the context of medical education and practice. More specifically, the rubric was designed for the assessment of reflective writing by Taiwanese healthcare students and providers. Methods: Literature related to reflective thinking and rubric development was reviewed and discussed by expert panels in order to set up a theoretical framework for the proposed Analytic Reflective Writing Scoring Rubric for Healthcare Students and Providers (ARWSR-HSP). Once the rubric was initially drafted, three experts, using the index of Item Objective Congruence (IOC), evaluated the congruence between the qualitative scoring items included in the rubric and the objectives of the ARWSRHSP. Further, the experts calculated inter-rater and intra-rater reliabilities for consistency estimates, using Spearman correlation coefficients. Weighted kappas were also calculated to confirm the inter-rater consensus estimates. Results: The IOC results revealed that among 30 qualitative criteria statements, 24 statements fell within the acceptable IOC range of 0.5 to 1.0. The remaining 6 items falling outside this range were reviewed individually in order to confirm content validity. The inter-rater reliabilities using Spearman correlation coefficients were between 0.757 and 0.946; the inter-rater reliabilities using weighed kappas were between 0.706 and 0.884. The intra-rater constancy estimates were between 0.825 and 0.930. Conclusions: The verification of content validity, IOC values, inter-rater reliabilities, and intra-rater reliabilities demonstrated that the ARWSR-HSP provides a rigorous-indeed, objective-scoring rubric for the assessment of reflective writing.

1. Broad, B(2003).What We Really Value: Beyond Rubrics in Teaching and Assessing Writing.Logan: Utah:Utah State University Press.
2. Brown, GT,Glasswell, K,Harland, D(2004).Accuracy in the scoring of writing: Studies of reliability and validity using a New Zealand writing assessment system.Assessing Writing,9(2),105-121.
3. Chen, I,Forbes, C(2014).Reflective writing and its impact on empathy in medical education: Systematic review.J Educ Eval Health Prof,11(20)
4. Cohen, J(1968).Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit.Psychological Bulletin,70(4),213-220.
5. Cortina, JM(1993).What is coefficient alpha? An examination of theory and applications.J Appl Psychol,78(1),98-104.
6. Cronbach, LJ(1951).Coefficient alpha and the internal structure of tests.Psychometrika,16(3),297-334.
7. Dewey J: How We Think: A Restatement of the Relation of Refl ective Thinking to the Educative Process. Boston: D.C. Heath & Co Publishers, 1933.
8. Dornisch, MM,McLoughlin, AS(2006).Limitations of web-based rubric resources: Addressing the challenges.Practical Assessment, Research and Evaluation,11(3),1-8.
9. Dymoke, S,Harrison, J(2008).Reflective Teaching and Learning: A Guide to Professional Issues for Beginning Secondary Teachers.London:SAGE Publications.
10. Fasko, D(ed.)(2003).Critical Thinking and Reasoning : Current Research, Theory, and Practice.Cresskill: NJ:Hampton Press.
11. Gallagher, K,Patridge, C,Tran, HT(2017).Nursing & parental perceptions of neonatal care in Central Vietnam: A longitudinal qualitative study.BMC Pediatr,17(1),161.
12. Gill, GS(2014).The nature of reflective practice and emotional intelligence in tutorial settings.Journal of Education and Learning,3(1),86-100.
13. Gronlund, NE(1993).How to Make Achievement Tests and Assessments.Boston:Allyn & Bacon.
14. Gwet, KL(2012).Handbook of Inter-rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Multiple Raters.Gaithersburg: MD:Advanced Analytics.
15. Hamp-Lyons, L(ed.)(1991).Assessing Second Language Writing in Academic Contexts.Norwood: New Jersey:Multilingual.
16. Harrison, A(1983).A Language Testing Handbook.Hong Kong:Macmillan Education.
17. Husu, J,Toom, A,Patrikainen, S(2008).Guided refl ection as a means to demonstrate and develop student teachers’ reflective competencies.Reflective Practice,9(1),37-51.
18. Jarvis, P(2001).Learning in Later Life: An Introduction for Educators and Careers.London:Kogan Page.
19. Jeffries, PR(ed.)(2007).Simulation in Nursing: From Conceptualization to Evaluation.New York:National League for Nursing.
20. Jonsson, A,Svingby, G(2007).The use of scoring rubrics: Reliability, validity and educational consequences.Educ Res Rev,2(2),130-144.
21. Kember, D(2001).Reflective Teaching and Learning in the Health Professions.England:Blackwell Science.
22. King, PM,Kitchener, KS(2004).Reflective judgment: Theory and research on the development of epistemic assumptions through adulthood.Educ Psychol,39(1),5-18.
23. Kızılkaya, G,Askar, P(2009).Problem cozmeye yonelik yansıtıcı dusunme becerisi olceginin gelistirilmesi.Egitim ve Bilim,34(154),82-92.
24. Langendyk, V,Mason, G,Wang, S(2016).How do medical educators design a curriculum that facilitates student learning about professionalism?.Int J Med Educ,7,32-43.
25. Lumley, T,McNamara, TF(1995).Rater characteristics and rater bias: Implications for training.Language Testing,12(1),54-71.
26. Mamede, S,Schmidt, HG,Penaforte, JC(2008).Effects of reflective practice on the accuracy of medical diagnoses.Med Educ,42(5),468-475.
27. Marusteri, M,Bacarea, V(2010).Comparing groups for statistical differences: How to choose the right tatistical test?.Biochem Med,20(1),15-32.
28. McGuire, L,Lay, K,Peters, J(2009).Pedagogy of reflective writing in professional education.Journal of the Scholarship of Teaching and Learning,9(1),93-107.
29. Mertler, CA(2001).Designing scoring rubrics for your classroom.Practical Assessment, Research and Evaluation,7(25),1-10.
30. Mezirow, J(1990).Fostering Critical Reflection in Adulthood: A Guide to Transformative and Emancipatory Learning.San Francisco:Jossey-Bass Publishers.
31. Montgomery, K(2002).Authentic tasks and rubrics: Going beyond traditional assessments in college teaching.College Teaching,50(1),34-39.
32. Moskal, BM,Leydens, JA(2000).Scoring rubric development: Validity and reliability.Practical Assessment, Research & Evaluation,7(10),71-81.
33. O’Connor, A,Hyde, A,Treacey, M(2003).Nurse teachers’ constructions of refl ection and refl ective ractice.Reflective Practice,4(2),107-119.
34. Parke, CS(2001).An approach that examines sources of misfit to improve performance assessment items and rubrics.Educational Assessment,7(3),201-225.
35. Reddy, YM,Andrade, H(2010).A review of rubric use in higher education.Assessment & Evaluation in Higher Education,35(4),435-448.
36. Ruland, JP,Ahern, NR(2007).Transforming student perspectives through reflective writing.Nurse Educ,32(2),81-88.
37. Sanford, PG(2010).Simulation in nursing education: A review of the research.The Qualitative Report,15(4),1006-1011.
38. Schon, DA(1995).The Reflective Practitioner: How Professionals Think in Action.London:Arena.
39. Shapiro, J,Kasman. D,Shafer, A(2006).Words and wards: A model of reflective writing and its uses in medical education.J Med Humanit,27(4),231-244.
40. Shipman, D,Roa, M,Hooten, J(2012).Using the analytic rubric as an evaluation tool in nursing education: The positive and the negative.Nurse Education Today,32(3),246-249.
41. Si, J(2018).An analysis of medical students’ reflective essays in problem-based learning.Korean Journal of Medical Education,30(1),57-64.
42. Taylor, EW(2014).Empathy: The stepchild of critical refl ection and transformative learning.Educational Refl ective Practices,4(2),5-24.
43. Thompson, N,Pascal, J(2012).Developing critically refl ective practice.Refl ective Practice,13(2),311-325.
44. Tsingos, C,Bosnic-Anticevich, S,Lonie, JM(2015).A model for assessing reflective practices in pharmacy education.Am J Pharm Educ,79(8),124.
45. Wald, HS,Borkan, JM,Taylor, JS(2012).Fostering and evaluating reflective capacity in medical education: Developing the REFLECT rubric for assessing reflective writing.Acad Med,87(1),41-50.
46. Wald, HS,Reis, SP,Borkan, JS(2009).Reflection rubric development: Evaluating medical students’ refl ective writing.Med Educ,43(11),1110-1111.
47. Wang, W(2014).Students’ perceptions of rubricreferenced peer feedback on EFL writing: A longitudinal inquiry.Assessing Writing,19,80-96.
48. Warrens, MJ(2011).Cohen’s linearly weighted kappa is a weighted average of 2×2 kappas.Psychometrika,76(3),471-486.
49. Wear, D,Zarconi, J,Garden, R(2012).Reflection in/and writing: Pedagogy and practice in medical education.Acad Med,87(5),603-609.
50. Weigle, SC(2002).Assessing Writing.New York:Cambridge University Press.
51. Woerkom, M(2010).Critical refl ection as a rationalistic ideal.Adult Education Quarterly,60(4),339-356.