A Comparison of Standard-Setting Methods in Taiwan's Nationwide Objective Structured Clinical Examination

10.6145/jme201303

S. Shih-Li Tsai；Yen-Yuan Chen；Tzong-Shinn Chu；Hou-Chang Chiu；Ming-Chien Kao；Pan-Chyr Yang

objective structured clinical examination ； standard-setting method ； Angoff ； borderline-group regression

Journal of Medical Education

17卷1期（2013 / 03 / 01）

21 - 29

英文

Background: For the past five decades, in medical licensure examinations in Taiwan, theoretical knowledge has been assessed by written tests administered by the National Medical License Examination Board of Taiwan. However, patient-centered clinical service has gained recognition in recent ten years. With the government's support, the objective structured clinical examination (OSCE) was applied to the medical education system in Taiwan to assess the clinical competence of medical graduates and to fulfill the societal needs of high-quality medical care, improved physician-patient communication, and medical professionalism. The aim of the United OSCE 2012 was to test which standard-setting method would best reflect society's needs and serve as a point of reference for medical education reform. Purpose: The purpose of this study was to investigate the results of 2 standard-setting methods during the United OSCE 2012 in Taiwan, and to present the results as the basis for medical education training reform. Methods: The United OSCE is a 12-station OSCE consisting of 4 evaluation domains: history-taking (Hx), physical examination (PE), communication and health education (Com), and procedural skills (PS). A passing standard was determined using the Angoff method when every case was developed before the examination. A total of 1173 participants at 19 OSCE centers were enrolled over a 6-day period in April and May of 2012. After the 6-day examination period, the other passing standard was defined using the borderline-group method with regression (BGR). Results: The pass rate was 90.3% using the Angoff method and 95.7% under the BGR. Mean total scores of the fail group using the Angoff method differed significantly from those under the BGR (Angoff 693.51 versus BGR 639.40; p ＜ .001). In logistic regression analyses, the odds ratios for those who passed the examination were found to be similar in the Com domain, when either the Angoff method or the BGR was applied (Angoff 1.086 versus BGR 1.088). The highest odds ratio (Angoff 1.043 versus BGR 1.218) was noted in the PE domain and a lower ratio (Angoff 1.192 versus BGR 1.54) was found in the PS domain, when applying the BGR. The mean score of the Angoff pass but BGR fail group was significantly higher than the Angoff-fail but BGR-pass group (66.52 ± 7.83 versus 61.86 ± 8.43) in the PS domain. Reversed significance was found for the evaluation domain on Hx and PE. However, no significant difference existed between these 2 groups with respect to the Com domain. Conclusion: The United OSCE 2012 in Taiwan reflected the needs of Taiwanese society. Whether under the Angoff method or under the BGR, similar Com performance-level participants were screened out as the fail group. The BGR screened out poorer Hx and PE performance participants as the fail group than did the Angoff method. The Angoff method selected less satisfactory PS performance participants as the fail group. Distinct standard-setting methods help determine learners' performance distributions from an alternate perspective, provide faculty members with a reference for balancing their expectations in various educational domains, and improve educational training goals and strategies in Taiwan.

1. Boursicot, K,Roberts, T(2006).Setting standards in a professional higher education course: Defining the concept of the minimally competent student in performance based assessment at the level of graduation from medical school.High Educ Quart,60,74-90.
2. Derek, R(1987).Assessing Students: How Shall We Know Them?.London:Harper & Row.
3. Downing, SM,Tekian, A,Yudkowsky, R(2006).Procedures for establishing defensible absolute passing scores on performance examinations in health professions education.Teach Learn Med,18,50-7.
4. George, S,Haque, MS,Oyebode, F(2006).Standard setting: comparison of two methods.BMC Med Educ,6,46.
5. Godfrey, P,Richard, F,Matthew, H(2010).How to measure the quality of the OSCE: a review of metrics-AMEE guide no. 49.Med Teach,32,802-11.
6. Impara, JC(1997).,Chicago:the Midwestern Educational Research Association.
7. Impara, JC,Plake, BS(1998).Teachers' ability to estimate item difficulty: A test of the assumptions in the Angoff standard setting method.J of Educ Measu,35,69-81.
8. Kao, MC(2012).Some key issues concerning the introduction of clinical skill test as a subject in the national medical licensing examination.Journal of Taiwan Ministry of Examination: National Elite,6,108-26.
9. Kaufman, DM,Mann, KV,Muijtjens, AM(2000).A comparison of standard setting procedure for an OSCE in undergraduate medical education.Acad Med,75,267-71.
10. Kilminster, S,Roberts, T(2004).Standard setting for OSCEs: trial of borderline approach.Adv Health Sci Educ Theory Pract,9,201-9.
11. Laine, C,Davidoff, F(1996).Patient-centered medicine: a professional evolution.JAMA,275,152-6.
12. Newble, D(2004).Techniques for measuring clinical competence: objective structured clinical examinations.Med Educ,38,199-203.
13. Norcini, JJ(2003).Setting standards on education tests.Med Educ,37,464-9.
14. Pelligrino, JW(ed.),Jones, LR(ed.),Mitchellw, KJ(ed.)(1999).Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress.Washington, DC:National Academy Press.
15. Regehr, G,MacRae, H,Reznick, RK(1998).Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination.Acad Med,73,993-7.
16. Shumway, JM,Harden, RM(2003).AMEE guide no. 25: the assessment of learning outcomes for the competent and reflective physician.Med Teach,25,569-84.
17. Verhoeven, BH,van der Steeg, AF,Scherpbier, AJ(1999).Reliability and credibility of an angoff standard setting procedure in progress testing using recent graduates as judges.Med Educ,33,832-7.
18. Verhoeven, BH,Verwijnen, GM,Muijtjens, AM(2002).Panel expertise for an Angoff standard setting procedure in progress testing: item writers compared to recently graduated students.Med Educ,36,860-7.