從多層面Rasch模式來檢視不同的評分者等化連結設計對參數估計的影響

謝名娟

doi:10.6251/BEP.202012_52(2).0008

山西大学，您好！

感谢采购，人社期刊库｜人社论文库，期间：2024/01/01~2024/12/31

感谢试用，科学期刊库｜科学论文库，期间：2024/10/20~2025/01/20

浏览高级检索

关闭
检索检索高级检索出版物检索
浏览期刊主题浏览核心期刊浏览期刊拼音浏览

关闭

查询符号(半角)	查询符号意义说明
空格	表示为「AND」两个查询词之交集
双引号 ( " " )	片语以双引号标示开始及结束，而且只寻找出现顺序相同的字词，例： " image process "
?	表示一个字母切截，输入两个?表两个字母，依此类推，例：输入「Appl?」，查得结果应为appl e , appl y … ( 常用于英文字查询 )
*	表示不限字母切截，由0~n. 例：输入「appl*」，查得结果应为appl e , appl es , appl y , appl ied , appl ication … ( 常用于英文字查询 )
AND、OR、NOT	布尔逻辑组合关键字，用来扩大或缩小查询范围的技巧。 (1) AND ：缩小查询范围 (2) OR ：扩大查询范围(3) NOT：排除不相关的范围

DOI 是数位物件识别码（ D igital O bject I dentifier ）的简称，
为物件在网路上的唯一识别码，可用于永久连结并引用目标物件。

使用DOI作为永久连结

每个DOI号前面加上「 http://dx.doi.org/ 」便成为永久网址。
如以DOI号为 10.5297/ser.1201.002 的文献为例，此文献的永久连结便是： http://dx.doi.org/ 10.5297/ser.1201.002 。
日后不论出版单位如何更动此文献位置，永久连结所指向的位置皆会即时更新，不再错失重要的研究。

引用含有DOI的文献

有DOI的文献在引用时皆应同时引用DOI。若使用APA、Chicago以外未规范DOI的引用格式，可引用DOI永久连结。

DOI可强化引用精确性、增强学术圈连结，并给予使用者跨平台的良好使用经验，目前在全世界已有超过五千万个对象申请DOI。如想对DOI的使用与概念有进一步了解，请参考（）。

ACI:

数据来源：Academic Citation Index，简称ACI

台湾最大的引用文献资料库，目前收录台湾与港澳地区所出版的人文学、社会学领域学术期刊之书目资料与参考文献，总期刊量超过690种，每年定期公布收录期刊的影响指数(Impact Factor)等指标给大众，并提供专家学者与学术单位实用的计量与分析功能。

五年影响指数（5-Year Impact Factor）：某一期刊前五年所出版的文章在当年度的平均被引用次数。

公式：（前五年发表论文在统计年的被引用次数）÷（前五年论文产出论文总篇数）

例如：A期刊2017年之五年影响指数

(A期刊2012-2016年发表论文在2017年的被引总次数)/(A期刊2012-2016年发表的论文总数)

什么是预刊文章？

为提供读者最前线之学术资讯，于期刊文献获同意刊登后、纸本印制完成前，率先于网路线上发表之文章即为预刊文章。预刊文章尚未有卷期、页次及出版日期资讯，但可藉由DOI号识别。 DOI号是文献的数位身份证字号，不论预刊或正式出版皆不会改变，读者可点击DOI连结，或于DOI号前面加上「 http://dx.doi.org/ 」连结到文献目前最新版本。

如何引用预刊文章？

请使用预刊文章的线上发表日期及DOI号来引用该篇文献。

引用范例(视不同引文格式规范可能有所差异)：

作者姓名。文章篇名。期刊名称。 YYYY/MM/DD线上预先发表。

doi:DOI 号

来源数据

P415 - 436

社会科学 > 心理学
社会科学 > 教育学

DOI： 10.6251/BEP

书目管理工具

相关连结

问题回报

全文下载

从多层面Rasch模式来检视不同的评分者等化连结设计对参数估计的影响

Investigating the Effects of Rater Equating Designs on Parameter Estimates in the Context of Preservice Principal Oral Performance

谢名娟(Ming-Chuan Hsieh)

教育心理学报； 52 卷 2 期 (2020 / 12 / 01) ， P415 - 436

繁体中文 DOI： 10.6251/BEP.202012_52(2).0008

多层面Rasch模式（MFRM）；校长口语评量；等化连结设计；评分严厉度； Many-Facet Rasch model (MFRM) ； preservice principal oral performance ； rater equating design ； rater severity

0 个人觉得这篇文章赞

摘要 │ 参考文献 (55) │ 被引用文献 (3) │ 国际计量

摘要〈TOP〉

并列摘要〈TOP〉

A problem in performance assessments is the degree to which rater severity and leniency can affect the examinee's scores. In particular, fairness concerns related to performance systems include the exchangeability of raters. A possible resolution for addressing rater severity is for each rater to score each examinee's performance; thus, the difference in rater severity would affect each student at the same level. However, this is not always feasible in practice for fully crossed rating designs. In the context of performance assessments, equating procedures create links between raters when performing transformation with a fully crossed rating design is not feasible and could control for differences in rater severity. An effective equating procedure involves a strong statistical model and a systematic data collection approach. The Many- Facet Rasch model (MFRM) is a commonly used approach for adjusting rater differences. Although the use of the MFRM model has gained popularity as an equating approach for rater severity, several key considerations related to data collection designs and model data fit are also crucial. In particular, it is vital to ensure a sufficient level of connectivity in the rating design; that is, the raters can be linked to other assessment components, such as other raters, examinees, or tasks. Three types of data collection design are commonly used for equating. The first type is a complete network design, in which the data consist of complete designs with subjects of all assessment components. This is an ideal design for a rating system. The second type is an incomplete network design. Under an incomplete network design, examinees do not have scores on all assessment components, but a partial and systematic degree of connectivity exists for raters and tasks to produce a connected network of assessment components. The third type is a nonlinked network design, where no systematic linkage exists in the components of facets. Even if the unlinked scoring network has some potential problems, many important exams in Taiwan still use this rater design. The purpose of this study was to examine the effect of differences in data collection designs that could affect parameter estimation in the performance assessment. Using empirical data, this study explicitly related the central role of consideration to data collection designs for the interpretation of results when the MFRM is applied. The study had two main research objectives: (1) To examine the impact of different data collection designs on parameter estimates for examinees' ability, raters' severity, and the difficulty of scoring criteria. The indices included infit, outfit, separation index, reliability, and the chi-square test. (2) To evaluate the correlations of ability estimates between different designs and the magnitude of their impact on the ranking of the examinees' performance level. Examinees for the top 10, middle 10, and last 10 examinees in the complete network design were selected to evaluate their ranking differences for other designs. This study used the MFRM and oral performance score data of preservice principals to explore the effects of the three data collection designs. A total of four raters and 85 preservice principals participated. The raters scored seven criteria for each preservice principal's oral performance: content, structure, word usage, attitude, pronunciation, intonation, and time control. Each criterion was assigned a grade of 1-3, of which 1 represents the basic level, 2 represents a proficient level, and 3 represents an advanced level. The raters were trained before the actual rating was conducted. The specification of each grading level and the standards were explained; raters were also required to complete rating exercises before conducting the official rating. The anchored videos at various levels for the raters were discussed. Through these anchored videos, the raters could better understand the standards. Four equating designs were considered in this study. Design 1 was the complete network design; four raters rated all preservice principals in this design. Designs 2 and 3 were incomplete network designs. In these two designs, some rating scores overlapped to construct the connectivity of scoring components. In design 2, each student received scores from three raters, whereas in design 3, each student received scores from two raters. Design 4 was a nonlinked network design, in which each rater only reviewed his or her assigned class; there was no connection between raters' scores. The MFRM, a statistics model of the Rasch family, was employed to perform the four equating designs. When estimating the examinee's ability level, raters' severity and scoring criteria were simultaneously considered in the model. This study had two main findings: (1) For the incomplete and nonlinked network designs, some minor problems were related to the model fit, but overall, the infit and outfit indices were close to 1, which indicated that the use of the MFRM was feasible for analyzing the data used in this study. However, the reliability and separation indices for the nonlinked network design were low, and some chi-square tests did not reach significance-results that were quite different from the complete network design. (2) The lower the linkage between assessment components, the more biased the estimated stability of parameters. The fully connected network design provided the strongest connectivity at all levels (subjects, raters, and criteria), and this design was also the most ideal scenario for the data collection design. However, this design costs much in terms of rating time and money; thus, it is difficult to implement such a design in a large-scale test. By contrast, incomplete network designs are more feasible in large-scale tests, namely for establishing overlaps of the evaluation of some subjects of raters. The correlation between the complete network design and nonlinked network design was only 0.69, but the correlation between the complete network design and incomplete network design rose to 0.79-0.94. Moreover, a clear gap existed in participants' rankings between the ideal fully connected network design and nonlinked network design. For example, student #59 ranked 79th in the complete network design but 21st in the nonlinked network design, equaling a ranking difference gap of 58. The results revealed that even if the MFRM is used for correction, large errors will still exist in the estimation of ability values and the ranking results of examinees for a nonlinked network design. This study provided two suggestions: (1) Examination institutions should avoid using the nonlinked network rater design. Carefully constructed network assessment designs based on effective data collection designs have the chance of obtaining objective and fair measurements within systems with multiple facets. Regarding current large-scale tests, many do not use any statistical models for rater severity correction; furthermore, they use the nonlinked rater design. It is possible that examinees can experience bad luck and encounter a severe grader, resulting in them receiving a low score. Therefore, this study recommended that important examinations in the future should adopt a more complete rating plan. (2) This study used empirical data. A simulation study can be considered to further examine the impact of different designs of component connectivity on parameter estimates. In addition, different experimental designs are worth discussing; for example, if examinees are nested within tasks, would this nested relationship affect the parameter estimates of ability and rater severity levels? The impact of more complex data collection designs are worthy of future research.

参考文献 ( 55 ) 〈TOP〉

王暄博, H.-P.,郭伯臣, B.-C.,吕玉如, Y.-J.(2013)。大型测验等化群体不变性之探究—以 2007 年台湾学生学习成就评量资料库国中二年级数学科为例。测验学刊，60(3)，489-518。
连结：
吴慧珉, H.-M.,郭伯臣, B.-C.,许天维, T.-W.,陈婉宁, W.-N.(2015)。以可能值方法为基础之多向度能力值垂直等化探究。测验学刊，62(2)，95-126。
连结：
林小慧, H.-H.,林世华, S.-H.,吴心楷, H.-K.(2018)。科学能力的建构反应评量之发展与信效度分析：以自然科光学为例。教育科学研究期刊，63(1)，173-205。
连结：
林小慧, H.-H.,曾玉村, Y.-T.(2017)。科学多重文本阅读理解评量及规准之建构与信效度分析—以气候变迁与三峡大坝之间的关系题本为例。教育心理学报，49(2)，215-241。
连结：
姚汉祷, H.-D.,姚伟哲, W.-C.(2007)。应用多层面 Rasch 模式分析双不定向飞靶优秀选手的射击技术。测验学刊，55(1)，89-104。
连结：

被引用文献 ( 3 ) 〈TOP〉

国际计量

〈TOP〉

E-mail ：

文章公开取用时，将寄通知信至您填写的信箱地址