不同方法設定英文科決斷分數之實務性研究

The Comparison of Two Standard Setting Methods on an English Test

謝名娟(Ming-Chuan Hsieh)；謝進昌(Jin-Chang Hsieh)；林世華(Sieh-Hwa Lin)

書籤標定法 ； 標準設定 ； Yes/No Angoff ； bookmark method ； standard setting

測驗學刊

60卷3期（2013 / 09 / 01）

519 - 544

繁體中文

本研究之目的在比較書籤標定法與Yes/No Angoff標準設定方法在設定決斷分數上的差異，並使用TASA 2009年英文科來進行比較研究，研究結果顯示，兩種標準設定方法所做出的決斷分數略有不同，標準設定成員們認為，Yes/No Angoff法執行起來較為簡單，最終決定的決斷分數也較符合預期。然而，執行此方法所需要的會議時間較長；書籤標定法執行所需之時間雖然較短，然需要克服的技術層面較多，尤其在面臨題本難度順序與成員心中期望的難度順序不一致時，往往會影響成員放置書籤的決策。最後，本研究建議，雖Yes/No Angoff法執行上比較簡單，但是在時間與人力負荷考量下，若未來想要進行Yes/No Angoff法進行標準設定時，最好能預估兩天的時間，如此才能將題目充分進行討論，然而，對於題目較多的測驗或面臨人力不足的條件下，則較適合使用書籤標定法，並搭配加強標準設定成員訓練作為配套。

This paper presents a comparison on two prevalent method, bookmark and yes/no Angoff method, for setting cut scores on educational assessments. The comparison is presented through an application with a Grade 6 English Assessment in Taiwan. The implementations for each method are described in detail along with comparative results for the application. It is found that the resulting cutoff points from these two methods are somewhat different. Judges regarded that yes/no Angoff method is easier to implement and the resulting cutoff points are more close to their expectation. Comparatively, there are more difficulties need to be solved for the bookmark procedure, especially when the item difficulty order does not follow judges' expectation. It is found that although the implementation of Yes/No Angoff method is simple, the Bookmark method still has some promising features.

1. 吳宜芳、鄒慧英、林娟如(2010)。標準設定效度驗證之探究：以大型數學學習成就評量為例。測驗學刊，57(1)，1-27。
   連結：
2. 吳毓瑩、陳彥名、張郁雯、陳淑惠、何東憲、林俊吉(2009)。以常態混組模型討論書籤標準設定法對英語聽讀基本能力標準設定有效性之幅合證據。教育心理學報，41(1)，69-90。
   連結：
3. 謝進昌(2005)。碩士論文(碩士論文)。臺北市，國立政治大學教育研究所。
   連結：
4. Council of Chief State School Officers (2001). State student assessment programs annual survey (Data Vol. 2). Washington, DC: Author
5. 臺灣學生學習成就評量資料庫網站（2011）。臺灣學生學習成就評量資料庫建置計畫。2011年11月26日，取自http://tasa.naer.edu.tw/1about-1.asp?id=1
6. Andrew, B. J.,Hecht, J. T.(1976).A preliminary investigation of two procedures for setting examination standards.Educational and Psychological Measurement,36(1),45-50.
7. Brandon, P. R.(2002).Two versions of the contrasting-groups standard setting method: A review.Measurement and Evaluation in Counseling and Development,35(3),167-181.
8. Brandon, P. R.(2004).Conclusions about frequently studied modified Angoff standard setting topics.Applied Measurement in Education,17(1),59-88.
9. Brennan, R. L.(2001).Generalizability theory.New York, NY:Springer-Verlag.
10. Buckendahl, C. W.,Smith, R. W.,Impara, J. C.,Plake, B. S.(2002).A comparison of Angoff and bookmark standard setting methods.Journal of Educational Measurement,39(3),253-263.
11. Cizek, G. J.,Bunch, M. B.(2007).Standard setting: A guide to establishing and evaluating performance standards on tests.Thousand Oaks, CA:Sage.
12. Cizek, G. K.(Ed.)(2001).Setting performance standards: Concepts, methods, and perspectives.Mahwah, NJ:Lawrence Erlbaum Associates.
13. Department of Public Instruction, State of Wisconsin(1997).,Madison, WI:Office of Educational Accountability.
14. Giraud, G.,Impara, J. C.,Buckendahl, C.(2010).Making the cut in school districts: Alternative methods for setting cutscores.Educational Assessment,6(4),291-304.
15. Green, D. R.,Trimble, C. S.,Lewis, D. M.(2003).Interpreting the results of three different standard setting procedures.Educational Measurement: Issues and Practice,2(1),22-32.
16. Hurtz, G. M.,Auerbach, M. A.(2003).A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus.Educational and Psychological Measurement,63(4),584-601.
17. Huynh, H.(2006).A clarification on the response probability criterion RP67 for standard settings based on bookmark and item mapping.Educational Measurement: Issues and Practice,25(2),19-20.
18. Huynh, H.(1998).On score locations of binary and partial credit items and their application to item mapping and criterion referenced interpretation.Journal of Educational and Behavior Statistics,23(1),35-56.
19. Impara, J. C.,Plake, B. S.(1997).Standard setting: An alternative approach.Journal of Educational Measurement,34(4),353-366.
20. Kane, M.(1994).Validating the performance standards associated with passing scores.Review of Educational Research,64(3),425-461.
21. Lewis, D. M.,Mitzel, H. C.,Green, D. R.(1996).Standard setting: A bookmark approach.Council of Chief State School Officers National Conference on Large Scale Assessment,Boulder, CO:
22. Nedelsky, L.(1954).Absolute grading standards for objective tests.Educational and Psychological Measurement,14,3-19.
23. Pitoniak, M. J.(2003).Amherst, MA,University of Massachusetts.
24. Reckase, M. D.,Bay, L.(1999).Comparing two methods for collecting test-based judgments.Meeting of the National Council on Measurement in Education,Montreal, Quebec, Canada:
25. Shepard, L, A.(1995).Implications for standard setting of the National Academy of Education evaluation of the National Assessment of Educational Progress achievement levels.Proceedings of the joint conference on standard setting for large scale assessments of the National Assessment Governing Board (NAGB) and the national Center for Educational Statistics (NCES),Washington, DC:
26. Shepard, L. A.,Glaser, R.,Linn, R.,Bohrnstedt, G.(1993).Setting performance standards for student achievement.Stanford, CA:National Academy of Education.
27. Thorndike, R. L.(Ed.)(1971).Educational measurement.Washington, DC:American Council on Education.
28. Yin, P.,Schulz, E. M.(2005).A comparison of cut scores and cut score variability from Angoff-based and bookmark-based procedures in standard setting.Annual meeting of the National Council on Measurement in Education,Montreal, Canada:
29. 吳裕益(1986)。博士論文(博士論文)。臺北市，國立政治大學教育研究所。
30. 林惠芬(1993)。通過分數設定方法在護理人員檢覈筆試測驗之研究。測驗年刊，40，253-262。
31. 陳彥名(2006)。碩士論文(碩士論文)。臺北市，國立臺北教育大學教育心理與諮商學系。
32. 曾建銘、陳清溪(2009)。2007年臺灣學生學習成就評量結果之分析。教育研究與發展期刊，5(4)，1-38。
33. 鄭明長、余民寧(1994)。各種通過分數設定方法之比較。測驗年刊，41，19-40。

1. 林小慧,吳心楷(2019)。科學探究能力評量之標準設定與其效度檢核。教育心理學報，50(3)，473-502。