對情緒行為障礙障生觀察評量之評分者間信度研究

The inter-rater reliability of observation assessment in students with emotional and behavioral disorders

10.6502/SEF.201812_(25).0003

陳智修(Chih-Hsiu Chen)

情緒行為障礙學生 ； 評分者間信度 ； 觀察評量 ； students with emotional and behavioral disorders ； inter-rater reliability ； observation assessment

特教論壇

25期（2018 / 12 / 01）

46 - 62

繁體中文

觀察評量廣泛地被家長與教師使用於情障生的鑑定與介入成效評估，故評分者間信度對其格外重要。但過去研究所用的信度指標過於單一，且也較少探討影響評分者間信度的相關因素。因此，本研究旨在分析家長、導師、資源教師在特質定義與評分標準的一致性，並探討影響這些信度指標的因素。本研究以七年級資源班情障生的家長、導師與資源教師為樣本，共計312人。經Kappa一致性係數、McNemar考驗與相關係數分析後，本研究發現：就特質定義而言，導師與資源教師在各評量面向皆有一致性，但家長與導師對行為問題的定義並不一致，且家長與資源教師對認知學習、注意力與過動問題的定義也不一致。從評分標準來說，導師與資源教師在各評量面向也皆有一致性，但家長與兩類教師對人際關係、注意力的評分標準皆不相同。在影響信度的因素部分，「評量情境與評分者配對」可解釋為何家長與教師在特質定義與評分標準的一致性較低；「評量方法或内容」可說明本研究特質定義一致性相對較低的原因；溝通類評分者因素能改善親師在特定評量面向的特質定義或評分標準歧異；知能類評分者因素則近乎無法改善親師間的不一致性。最後，本研究亦提出評量實務與後續研究的相關建議。

Inter-rater reliability is an important issue as observation assessments are widely utilized to identify the students with emotional and behavioral disorders and evaluate their progress by parents and teachers. However, previous studies often tend to adopt constricted reliability index and seldom explore the factors that may affect the reliability. The purpose of this study was to analyze the trait and category definition consistency among parents, tutors and resource teachers, and to explore the factors affecting the reliability indexes. Subjects were 312 parents, tutors and resource teachers whose child or pupil was the seventh grade resource classroom student with emotional and behavioral disorders. Kappa coefficient, McNemar test and correlational research were conducted. The results revealed that: In the trait definitions, though tutors agreed with resource teachers on every assessment dimension, parents and tutors were not in agreement on behavior problem. And parents and resource teachers were not in agreement on cognitive learning, attention and hyperactivity problems. In the category definitions, tutors still agreed with resource teachers on every assessment dimension. But parents disagreed with the two kinds of teachers on interpersonal relationship and attention problems. As for the factors affecting reliability indexes, assessment situation and rater pair could explain why there was less consistency between parents and teachers. The lower trait definition agreement in this study might be due to assessment method and item content. On specific assessment dimensions, raters' communication could improve the trait or category definition discrepancy between parents and teachers. But rater's knowledge could hardly improve the discrepancy. Finally, some suggestions about assessment practice and further research were offered.

1. 蔡明富(2011)。特殊教育中有無伴隨品行疾患之注意力缺陷過動症學生的學校與家庭適應研究。應用心理研究，49，31-63。
   連結：
2. Adamson, R. M.,Wachsmuth, S. T.(2014).A review of direct observation research within the past decade in the field of emotional and behavioral disorders.Behavioral Disorders,39(4),181-189.
3. Barth, J.,de Boer, W. E. L.,Busse, J. W.,Hoving, J. L.,Kedzia, S.,Couban, R.,Kunz, R.(2017).Inter-rater agreement in evaluation of disability: Systematic review of reproducibility studies.British Medical Journal,356
4. Brown, J. D.,Wissow, L. S.,Gadomski, A.,Zachary, C.,Bartlett, E.,Horn, I.(2006).Parent and teacher mental health ratings of children using primary-care services: Interrater agreement and implications for mental health screening.Ambulatory Pediatrics,6(6),347-351.
5. Craighead, W. E.(Ed.),Nemeroff, C. B.(Ed.)(2001).The Corsini encyclopedia of psychology and behavioral science.New York, NY:Wiley.
6. Dart, E. H.,Radley, K. C.,Briesch, A. M.,Furlow, C. M.,Cavell, H. J.(2016).Assessing the accuracy of classwide direct observation methods: Two analyses using simulated and naturalistic data.Behavioral Disorders,41(3),148-160.
7. Epstein, M. H.,Cullinan, D.,Harniss, M. K.,Ryser, G.(1999).The scale for assessing emotional disturbance: Test-retest and interrater reliability.Behavioral Disorders,24(3),222-230.
8. Epstein, M. H.,Harniss, M. K.,Pearson, N.,Ryser, G.(1999).The Behavioral and Emotional Rating Scale: Test-retest and inter-rater reliability.Journal of Child and Family Studies,8(3),319-327.
9. Fernández-Ballesteros, R.(Ed.)(2003).Encyclopedia of psychological assessment.London, England:SAGE Publications.
10. Fernández-Ballesteros, R.(Ed.)(2003).Encyclopedia of psychological assessment.London, England:SAGE Publications.
11. Gage, N. A.,Scott, T. M.(2014).Advancing the science of direct observation in emotional and/or behavioral disorders research: Reliability and unified validity.Behavioral Disorders,39(4),177-180.
12. Hoyt, W. T.,Kerns, M. D.(1999).Magnitude and moderators of bias in observer ratings: A meta-analysis.Psychological Methods,4(4),403-424.
13. Ilgen, J. S.,Ma, I. W. Y.,Hatala, R.,Cook, D. A.(2015).A systematic review of validity evidence for checklists versus global rating scales in simulation - based assessment.Medical Education,49(2),161-173.
14. January, S. A. A.,Lambert, M. C.,Epstein, M. H.,Walrath, C. M.,Gebreselassie, T.(2015).Cross-informant agreement of the Behavioral and Emotional Rating Scale for youth in community mental health settings.Children and Youth Services Review,53,34-38.
15. John Bernardin, H.,Thomason, S.,Ronald Buckley, M.,Kane, J. S.(2016).Rater rating - level bias and accuracy in performance appraisals: The impact of rater personality, performance management competence, and rater accountability.Human Resource Management,55(2),321-340.
16. Kritikos, E. P.(2010).Special education assessment: Issues and strategies affecting today's classrooms.Upper Saddle River, NJ:Merrill.
17. Lewis, T. J.,Scott, T. M.,Wehby, J. H.,Wills, H. P.(2014).Direct observation of teacher and student behavior in school settings: Trends, issues and future directions.Behavioral Disorders,39(4),190-200.
18. Mayes, S. D.,Calhoun, S. L.,Murray, M. J.,Morrow, J. D.,Yurich, K. K. L.,Mahr, F.,Petersen, C.(2009).Comparison of scores on the Checklist for Autism Spectrum Disorder, Childhood Autism Rating Scale, and Gilliam Asperger's Disorder Scale for children with low functioning autism, high functioning autism, Asperger's disorder, ADHD, and typical development.Journal of Autism and Developmental Disorders,39(12),1682-1693.
19. Minkkinen, J.,Lindfors, P.,Kinnunen, J.,Finell, E.,Vainikainen, M. P.,Karvonen, S.,Rimpela, A.(2017).Health as a predictor of students' academic achievement: A 3- level longitudinal study of Finnish adolescents.Journal of School Health,87(12),902-910.
20. Pekrun, R.(Ed.),Linnenbrink-Garcia, L.(Ed.)(2014).International handbook of emotions in education.New York, NY:Routledge.
21. Reynolds, C. R.(Ed.),Vannest, K. J.(Ed.),Fletcher-Janzen, E.(Ed.)(2014).Encyclopedia of special education: A reference for the education of children, adolescents, and adults with disabilities and other exceptional individuals.Hoboken, NJ:Wiley.
22. Reynolds, C. R.(Ed.),Vannest, K. J.(Ed.),Fletcher-Janzen, E.(Ed.)(2014).Encyclopedia of special education: A reference for the education of children, adolescents, and adults with disabilities and other exceptional individuals.Hoboken, NJ:Wiley.
23. Reynolds, C. R.(Ed.),Vannest, K. J.(Ed.),Fletcher-Janzen, E.(Ed.)(2014).Encyclopedia of special education: A reference for the education of children, adolescents, and adults with disabilities and other exceptional individuals.Hoboken, NJ:Wiley.
24. Salvia, J.,Ysseldyke, J. E.,Bolt, S.(2013).Assessment in special and inclusive education.Belmont, CA:Wadsworth Cencage Learning.
25. Stolarova, M.,Wolf, C.,Rinker, T.,Brielmann, A.(2014).How to assess and compare inter-rater reliability, agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs.Frontiers in Psychology,5,509.
26. Stratis, E. A.,Lecavalier, L.(2015).Informant agreement for youth with autism spectrum disorder or intellectual disability: A meta-analysis.Journal of Autism and Developmental Disorders,45(4),1026-1041.
27. Tavakol, M.,Pinner, G.(2018).Enhancing objective structured clinical examinations through visualisation of checklist scores and global rating scale.International Journal of Medical Education,9,132-136.
28. Uebersax, J. (2015). Statistical methods for diagnostic agreement. 2018.05.13 Retrieved from http://www.john-uebersax.com/stat/agree.htm#recs
29. Van Der Ende, J.,Verhulst, F. C.,Tiemeier, H.(2012).Agreement of informants on emotional and behavioral problems from childhood to adulthood.Psychological Assessment,24(2),293-300.
30. Walker, H. M.(Ed.),Gresham, F. M.(Ed.)(2014).Handbook of evidence-based practices for emotional and behavioral disorders: Applications in schools.New York, NY:The Guilford Press.
31. 王天苗（2014b）。特殊教育長期追蹤資料庫：100學年度資料使用手冊（電子檔）。桃園市：中原大學特殊教育學系。
32. 王天苗（2014a）。特殊教育長期追蹤資料庫：100學年度國中家長、普通班教師、資源班教師問卷與資料（電子檔）。桃園市：中原大學特殊教育學系。
33. 台灣精神醫學會譯、American Psychiatric Association(2014)。DSM-5精神疾病診斷準則手冊。臺北市:合記圖書。
34. 余民寧(2011)。教育測驗與評量：成就測驗與教學評量。臺北市:心理。
35. 李姿瑩譯、Kauffman, J. M.、Landrum, T. J.(2013)。兒童與青少年情緒及行為障礙。臺北市:華騰文化。
36. 張世彗、藍瑋琛(2014)。特殊教育學生評量。臺北市:心理。
37. 張正芬編(2014)。身心障礙及資賦優異學生鑑定辦法說明手冊。臺北市:教育部。
38. 教育部（2013）。身心障礙及資賦優異學生鑑定辦法。臺北市：教育部