Examining whether Direct Observation of Procedural Skill (DOPS) is a Reliable Assessment Tool for Assessing the Performance of Procedural Skills

10.6145/jme.202109_25(3).0004

Yu-Ting Tsao；Tien-Shang Huang；Huey-Ling Chen；Tzong-Shinn Chu；Ming-Been Lee；Yen-Yuan Chen

Direct Observation of Procedural Skill ； reliability

Journal of Medical Education

25卷3期（2021 / 09 / 01）

103 - 111

英文

Purpose: The reliability of Direct Observation of Procedural Skills (DOPS) has been of concerns. For convincingly examining the reliabilities of DOPS, we built up three video-recorded clinical scenarios played by standardized patients/family members and standardized house officers. The objective of this study was to investigate the test-retest reliability, interrater reliability and internal consistency reliability of DOPS to assess house officers' capability of performing procedural skills. Methods: We developed Scenario I, Scenario II, and Scenario III for guaranteeing that all raters assessed exactly the same procedural skills performed by standardized house officers. We recruited the raters, using purposive sampling, with plentiful experiences of rating house officers' performance of procedural skills from one governmental university-affiliated, and three medical university-affiliated medical centers, and two Veterans Affairs Council-affiliated medical centers. Each rater was required: (1) to watch the video-recorded performance in a consecutive order starting from Scenario I in a single rating session; and (2) to rate each house officer's performance of procedural skills in each scenario. We examined the test-retest reliability, interrater reliability, and internal consistency reliability of DOPS using Pearson's correlation coefficients (PCC), intra-class correlation coefficient (ICC), and internal consistency reliability (Cronbach's alpha), respectively. Results: We found that: (1) a rater gave similar rating results in face of exactly the same procedural skill as indicated by acceptable or more than acceptable test-retest reliability (PCC = 0.75 ~ 0.81, P ＜ .01); (2) the raters gave the similar rating results to a house officer's performance of procedural skills as indicated by an excellent interrater reliability (PCC = 0.93 ~ 0.96, P ＜ .01); and (3) all the items of DOPS reflected uni-dimensionally a construct-a house officer's performance of procedural skills as supported by a much better internal consistency reliability (Cronbach's alpha = 0.84 ~ 0.91). Conclusions: Our study result convincingly reported that DOPS is a reliable assessment tool for assessing a house officer's performance of procedural skills, and can be widely used in clinical encounter to rate the performance of procedural skills.

1. Miller GE: The assessment of clinical skills/competence/performance. Acad Med 1990; 65(9 Suppl): S63-7.
   連結：
2. Van Der Vleuten CPM: The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ 1996; 1:41-67.
   連結：
3. Hengameh H, Afsaneh R, Morteza K, Hosein M, Marjan SM, Ebadi A: The Effect of Applying Direct Observation of Procedural Skills (DOPS) on Nursing Students' Clinical Skills: A Randomized Clinical Trial. Glob J Health Sci 2015; 7(7): 17-21.
   連結：
4. Cobb KA, Brown G, Jaarsma D, Hammond RA: The educational impact of assessment: A comparison of DOPS and MCQs. Med Teach 2013; 35(11): e1598-e1607.
   連結：
5. Bansal M: Introduction of Directly Observed Procedural Skills (DOPS) as a Part of Competency-Based Medical Education in Otorhinolaryngology. Indian J Otolaryngol Head Neck Surg 2019; 71(2): 161-166.
   連結：
6. Abdalla G, Moran-Atkin E, Chen G, Schweitzer MA, Magnuson TH, Steele KE: The effect of warm-up on surgical performance: a systematic review. Surg Endosc 2015; 29: 1259-1269.
   連結：
7. Chavez AK, Swygert KA, Peitzman SJ, et al.: Within-session score gains for repeat examinees on a standardized patient examination. Acad Med. 2013;88(5):688-92.
   連結：
8. Chen CC, Green IC, Colbert-Getz JM, et al.: Warm-up on a simulator improves residents' performance in laparoscopic surgery: a randomized trial. Int Urogynecol J. 2013;24(10):1615-22.
   連結：
9. Habibi H, Khaghanizade M, Mahmoodi H, Ebadi A, Seyedmazhari M: Comparison of the effects of modern assessment methods (DOPS and Mini-CEX) with traditional method on nursing students’ clinical skills: A randomized trial. Iranian Journal of Medical Education 2013; 13(5): 364-372.
10. Majid A, Reza MS: Direct observation of procedural skills (DOPS) in restorative dentistry: Advantages and Disadvantages in student’s point of view. Iranian J Med Educ 2013; 13(3): 212-220.
11. Eggleton K, Goodyear-Smith F, Paton L, et al. Reliability of Mini-CEX Assessment of Medical Students in General Practice Clinical Attachments. Fam Med 2016; 48: 624-30.