Generalizability of the Writing Performance Assessment

使用類推理論分析寫作實作評量的信度

10.7108/PT.200406.0029

黃瓊蓉(Chiung-Jung Huang)

實用評量 ； 類推理論 ； performance assessment ； generalizability theory

測驗學刊

51卷1期（2004 / 06 / 01）

29 - 44

英文

本研究的主要目的在籍由使用類推理論，瞭解誤差影響作文分數的類推度。研究樣本是50位大一學生，每位學生必須撰寫1作文2短文；評分的標準採分項評分，有內容、修辭、結構三個評分向度。根據單變數類推理論的分析，題目的變異組成份是大的，並且學生與題目交互作用的變異組成份亦無法忽略，因此，大學聯考讓學生得以自由選擇短文作答，可能是不適當的。 根據多變數類推理論的分析，學生的共變組成份是小的，顯示內容、修辭、結構是異質性的能力；題目的共變組成份是大的，顯示題目間的相關性；而評分者的共變組成份是正的，支持「月暈效應」的存在，因此評分者應有更多的訓練，以提升其評分技巧。

The major purpose of this study was to model the major sources of error that might affect a performance, such as raters, tasks and their interaction by the use of generalizability theory. The subjects were a convenience sample of 50 college freshmen. Univariate generalizability analysis was conducted for each scoring dimension: substance, rhetoric, and structure. Consistent with previous research, the variance component for task is large. Moreover, the variance component for student and task interaction is not negligible. Based on multivariate analyses, the low covariance component for student suggests that the ability to write can be viewed as a heterogeneous attribute. The large components of covariance for tasks suggest intercorrelation among items. The positive components of covariance for raters support the existence of ‘halo effect’. More training should be conducted to refine raters’ scoring skills.

1. Brennan, R. L.,Gao, X.,Colton, D.(1995).Generalizability analyses of work keys Listening and writing tests.Educational and Psychological Measurement,55
2. Clauser, B. E.,Swanson, D. B.,Clyman, S. G.(1999).A comparison of the generalizability of scores produced by expert raters and automated scoring systems.Applied Measurement in Education,12
3. Gao, X.,Shavelson, R. J.,Baxter, G. P(1994).Generalizability of large-scale performance assessments in science: Promises and problems.Applied Measurement in Education,7
4. Koretz, D.,Stecher, B.,Klein, S.,McCaffrey(1994).The Vermont portfolio assessment program: Findings and implications.Educational Measurement: Issues and Practices,13(3)
5. Linn, R. L.,Burton, E.(1994).Performance-based assessment: Implications of task specificity.Educational Measurement: Issues and Practices,13(1)
6. SAS Institute(1991).SAS user's guide.
7. Shavelon, R. J.,Baxter, G. P.,Gao, X.(1993).Sampling variability of performance assessment.Journal of Educational Measurement,30
8. Shavelson, R. J.,Mayberry, P. W.,Li, W.,Webb, N. M.(1990).Generalizability of job performance measurements: Marine Corps rifleman.Military Psychology,2
9. Shavelson, R. J.,Webb, N. M.(1981).Generalizability theory: 1973-1980.British Journal of Mathematical and Statistical Psychology,34
10. Webb, N. M.,Shavelson, R. J.(1981).Multivariate generalizability of general educational development ratings.Journal of Educational Measurement,18
11. Webb, N. M.,Shavelson, R. J.,Maddahian, E.(1983).New Directions for testing and measurement: Generalizability theory: Inferences and practical consideration.

1. 姚漢禱、呂玉華、吳佳儒(2010)。利用概化理論分析太極劍比賽評分。國立體育學院論叢，20(1)，99-108。