建構反應題的認知診斷模型

Cognitive Diagnostic Models for Constructed Response Items

郭伯臣(Bor-Chen Kuo)；洪裕堂(Yuk-Tarng Hong)；楊智為(Chih-Wei Yang)；馮文翠(Wen-Tsui Feng)

DINA ； 自動計分 ； 認知診斷 ； 建構反應題 ； automated scoring ； cognitive diagnosis ； constructed-response item ； DINA

測驗學刊

68卷2期（2021 / 06 / 30）

141 - 173

繁體中文

在線上學習中，如何能快速精準的了解學習者是否已經習得課程中的知識或概念，一直是個重要的問題。建構反應題可提供較為詳細的學習者訊息，但在線上測驗中建置建構反應題的自動計分機制需要耗用較多之人力與時間。因此，如何能合併選擇題及建構反應題進行測驗，以降低建置自動計分機制所需的人力與時間，同時能提升自動計分的精準度，也是待探討的議題。在本研究中，將提出一種應用在自動計分機制情境下的統計模型，用於提高數學建構反應題的自動評分診斷概念之準確性，以及比較不同自動計分方式的成效評估，並以模擬研究探討選擇題與建構反應題有效的組合方式，以實徵研究探討新方法的可行性。研究結果顯示，基於建構反應題的認知診斷模型和自動評分機制能提高診斷學習者概念之精準度，而使用適當題數的選擇題合併建構反應題進行測驗時，則可在不損及估計精準度的狀況下降低測驗的長度及施測所需的時間。

Multiple choice items are usually used for simple scoring. Constructed response items provide more diagnostic information about learners, however, it is costly and time-consuming to develop constructed response items for automated scoring process. To apply both multiple choices and constructed response items in online assessment would be more efficient. To reach this purpose, an automated scoring method for constructed response items and a statistical model for the assessment with both multiple choice and constructed response items are used in the present study. The study collected both simulated data and empirical data to evaluate the performance of the proposed model. The results show that the proposed cognitive diagnostic model for both multiple choice and constructed response items outperforms those only considering multiple choice items. As well, the efficacy of test development and data analysis was improved.

1. Bejar, I. I.(2011).A validity-based approach to quality control and assurance of automated scoring.Assessment in Education: Principles, Policy & Practice,18,319-341.
2. Burstein, J.(2003).The e-rater® scoring engine: Automated essay scoring with natural language processing.Automated essay scoring: A cross-disciplinary perspective,Hillsdale, NJ:
3. Chakraborty, U. K.,Konar, D.,Roy, S.,Choudhury, S.(2019).Automatic short answer grading using rough concept clusters.International Journal of Advanced Intelligence Paradigms,14(3/4),260-280.
4. de la Torre, J.(2009).DINA model and parameter estimation: A didactic.Journal of Educational and Behavioral Statistics,34,115-130.
5. de la Torre, J.,Douglas, J.(2004).Higher-order latent trait models for cognitive diagnosis.Psychometrika,69,333-353.
6. Doignon, J. P.,Falmagne, J. C.(1999).Knowledge spaces.New York, NY:Springer-Verlag.
7. Dziuban, C.,Moskal, P.,Johnson, C.,Evans, D.(2017).Adaptive learning: A tale of two contexts.Current Issues in Emerging eLearning,4(1),3.
8. Elliot, S.(2003).IntelliMetric: From here to validity.Automated essay scoring: A cross-disciplinary perspective,Hillsdale, NJ:
9. Heilman, M.,Madnani, N.(2015).The impact of training data on automated short answer scoring performance.Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications,Stroudsburg, PA:
10. Huebner, A.(2010).An overview of recent developments in cognitive diagnostic computer adaptive assessments.Practical Assessment, Research & Evaluation,15(3)
11. Junker, B. W.,Sijtsma, K.(2001).Cognitive assessment models with few assumptions, and connections with nonparametric item response theory.Applied Psychological Measurement,25,258-272.
12. Kuo, B. C.,Chen, C. H.,Yang, C. W.,Mok, M. M. C.(2016).Cognitive diagnostic models for tests with multiple choice and constructed response items.Educational Psychology,36,1115-1133.
13. Leacock, C.,Chodorow, M.(2003).c-rater: Automated scoring of short-answer questions.Computers and the Humanities,37,389-405.
14. Li, H.,Hunter, C. V.,Lei, P. W.(2016).The selection of cognitive diagnostic models fora reading comprehension test.Language Testing,33,391-409.
15. Liu, O. L.,Brew, C.,Blackmore, J.,Gerard, L.,Madhok, J.,Linn, M. C.(2014).Automated scoring of constructed-response science items: Prospects and obstacles.Educational Measurement: Issues and Practice,33(2),19-28.
16. Peredo, R.,Canales, A.,Menchaca, A.,Peredo, I.(2011).Intelligent web-based education system for adaptive learning.Expert Systems with Applications,38(12),14690-14702.
17. Ravand, H.,Barati, H.,Widhiarso, W.(2013).Exploring diagnostic capacity of a high stakes reading comprehension test: A pedagogical demonstration.Iranian Journal of Language Testing,3(1),12-37.
18. Risse, T.(2007).Testing and assessing mathematical skills by a script based system.10th International Conference on Interactive Computer Aided Learning,Villach, Austria:
19. Roberts, M. R.,Gierl, M.(2010).Developing score reports for cognitive diagnostic assessments.Educational Measurement: Issues and Practice,29(3),25-38.
20. Rupp, A.,Templin, J.(2008).Effects of Q-matrix misspecification on parameter estimates and misclassification rates in the DINA model.Educational and Psychological Measurement,68,78-98.
21. Schuwirth, L. W.,van der Vleuten, C. P.(2011).Programmatic assessment: From assessment of learning to assessment for learning.Med Teach,33,478-485.
22. Seifried, J.,Brandt, S.,Kögler, K.,Rausch, A.(2020).The computer-based assessment of domain specific problem-solving competence-A three step scoring procedure.Cogent Education,7(1),1719571.
23. Shute, V. J.(2008).Focus on formative feedback.Review of Educational Research,78,153-189.
24. Topol, B.,Olson, J.,Roeber, E.(2010).The cost of new higher quality assessments: A comprehensive analysis of the potential costs for future state assessments.Stanford, CA:Stanford University, Stanford Center for Opportunity Policy in Education.
25. Wang, S.,Yang, Y.,Culpepper, S. A.,Douglas, J. A.(2017).Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden markov model with covariates.Journal of Educational and Behavioral Statistics,43(1),57-87.
26. Williamson, D. M.(ed.),Mislevy, R. J.(ed.),Bejar, I. I.(ed.)(2006).Automated scoring of complex tasks in computer-based testing.Mahwah, NJ:Lawrence Erlbaum Associates.
27. Williamson, D. M.,Almond, R. G.,Mislevy, R. J.,Levy, R.(2006).An application of Bayesian networks in automated scoring of computerized simulation tasks.Automated scoring of complex tasks in computer-based testing,Mahwah, NJ:
28. Yang, C. W.,Kuo, B. C.,Liao, C. H.(2011).A HO-IRT based diagnostic assessment system with constructed response items.Turkish Online Journal of Educational Technology,10,46-51.
29. 馮文翠(2014)。臺中市，亞洲大學。