Construction and Validation of an Analytic Critical Thinking Scoring Rubric for Healthcare Students and Providers

10.6145/jme201631

Pan-Fu Kao；Ya-Huei Wang；Hung-Chang Liao

critical thinking ； medical education ； scoring rubric ； validity ； reliability

Journal of Medical Education

20卷4期（2016 / 12 / 01）

308 - 320

英文

Purpose: The study attempted to construct and validate a systemized evaluation for critical thinking performance in the Taiwanese medical cultural context. Methods: Literature related to critical thinking and rubric development was reviewed and discussed in expert panels in order to set up a theoretical framework for the proposed Analytic Critical Thinking Scoring Rubric for Healthcare Students and Providers (ACTSR‒HSP). Three experts used the index of Item Objective Congruence (IOC) for the critical thinking and scoring rubric to evaluate the congruence between the qualitative scoring items and the objectives. Percent exact agreement and percent adjacent agreement were applied to examine the inter-rater consensus estimates. Pearson correlation coefficients for inter-rater and intra-rater reliabilities were also calculated for consistency estimates. Results: The IOC results showed that among 18 qualitative criteria statements, 14 statements fell in the acceptable IOC range of 0.5 to 1.0; 4 items not meeting the criteria were revised and reviewed to confirm the content validity. The inter-rater consensus estimates were all above 70%, between 90% and 100%. The inter-rater consistency estimates were between 0.83 and 0.89; the intra-rater constancy estimates were between 0.95 and 0.98. Conclusion: Results of content validity, IOC, inter-rater reliabilities, and intra-reliabilities proved that the ACTSR‒HSP could be a valid and systemized evaluation tool for critical thinking performance in the Taiwanese medical cultural context.

1. Scriven N, Paul R: Defining critical thinking. Available at [http://www.criticalthinking.org/page.cfm?PageID=410&CategoryID=51.] Accessed on: August 12, 2014.
2. AACSB.AACSB Assurance of Learning Standards: An Interpretation.
3. Adams, M,Whitlow, J,Stover, L(1996).Critical thinking as an educational outcome: an evaluation of current tools of measurement.Nurse Educ,21(3),23-32.
4. Alfaro-LeFevre, R(1999).Critical Thinking in Nursing: A Practical Approach.Philadelphia:Saunders.
5. Allen, S,Knight, J(2009).A method for collaboratively developing and validating a rubric.International Journal for the Scholarship of Teaching and Learning,3(2)
6. Andrade, H,Du, Y(2005).Student perspectives on rubricreferenced assessment.Practical Assessment, Research & Evaluation,19(5),1-11.
7. Andrade, H,Valtcheva, A(2009).Promoting learning and achievement through self-assessment.Theory into Practice,48(1),12-19.
8. Beyer, BK(1995).Critical Thinking.Bloomington:Phi Delta Kappa Educational Foundation.
9. Beyreli, L,Arı, G(2009).The use of analytic rubric in the assessment of writing performance: inter-rater concordance study.Educational Sciences: Theory & Practice,9(1),105-125.
10. Biggs, J(1996).Enhancing teaching through constructive alignment.Higher Education,32,1-18.
11. Brown, GT,Glasswell, K,Harland, D(2004).Accuracy in the scoring of writing: studies of reliability and validity using a new Zealand writing assessment system.Assessing Writing,9,105-121.
12. Case, R(2002).Partnering to promote critical thinking.School Libraries in Canada,22(1),11-12.
13. Harrison, A(1983).A Language Testing Handbook.Hong Kong:Macmillan.
14. Jonsson, A,Svingby, G(2007).The use of scoring rubrics: reliability, validity and educational consequences.Educational Research Review,2,130-144.
15. Kurt, AA,Izmirli, S(2009).The views of teacher candidates about the use of a scoring rubric for the evaluation of their products in the course of instructional technologies and material development.Procedia Social and Behavioral Sciences,1,988-992.
16. Laurian, S,Fitzgerald, CJ(2013).Effects of using rubrics in a university academic level Romanian literature class.Procedia - Social and Behavioral Sciences,76,431-440.
17. McNamara, F(1996).Measuring Second Language Performance.London:Longman.
18. Moskal, BM,Leydens, JA(2000).Scoring rubric development: validity and reliability.Practical Assessment, Research & Evaluation,7,71-81.
19. Nelson, NW,Van Meter, AM(2007).Measuring written language ability in narrative samples.Reading & Writing Quarterly,23,287-309.
20. Panadero, E,Alonso-Tapia, J,Reche, E(2013).Comparison of rubrics and self-assessment scripts effect on self-regulation, performance and self-efficacy in university students.Studies in Educational Evaluation,39(3),125-132.
21. Panadero, E,Jonsson, A(2013).The use of scoring rubrics for formative assessment purposes revisited: a review.Educational Research Review,9,129-144.
22. Paul, R,Elder, L(2001).The Miniature Guide to Critical Thinking: Concept and Tools.Dillon Beach, CA:The Foundation for Critical Thinking.
23. Popham, JW(1997).What's wrong-and what's right-with rubrics.Educational Leadership,55(2),72-75.
24. Reddy, MY(2011).Design and development of rubrics to improve assessment outcomes: a pilot study in a master's level business program in India.Quality Assurance in Education,19(1),84-104.
25. Reddy, YM,Andrade, H(2010).A review of rubric use in higher education.Assessment & Evaluation in Higher Education,35(4),435-448.
26. Smith, LJ(2008).Grading written projects: what approaches do students find most helpful.Journal of Education for Business,83(6),325-330.
27. Stellmack, MA,Konheim-Kalkstein, YL,Manor, JE(2009).An assessment of reliability and validity of a rubric for grading APA-style introductions.Teaching of Psychology,36(2),102-107.
28. Stemler, SE(2009).A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability.Practical Assessment, Research & Evaluation,9(4)
29. Stevens, DD,Levi, A(2005).Introduction to Rubrics: An Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote Student Learning.Sterling, VA:Stylus Publishing.