透過log數據探析澳門學生的問題解決行為：以PISA 2012的公開題為例

Analysis of Log File Data to Understand Macau's Student Problem-Solving Behavior: An Example of a Released Item from PISA 2012 Study

楊文佳(Man-Kai Ieong)；薛寶嫦(Pou-Seong Sit)；麥瑞琪(Soi-Kei Mak)；張國祥(Kwok-Cheung Cheung)

log數據 ； PISA電子化測驗 ； 問題解決能力 ； 教育數據探勘 ； 認知和情意特徵 ； cognitive and motivational characteristics ； educational data mining ； log file data ； PISA computer-based assessment ； problem solving competency

測驗學刊

63卷3期（2016 / 09 / 01）

153 - 178

繁體中文

問題解決能力為學生應對未來生活及工作挑戰的重要能力，然而，傳統紙筆測驗並未能探討問題解決過程的學生行為及其認知和情意特點。電子化問題解決測驗及其log數據則使解決上述困難成為可能。教育數據探勘是研究log數據等教育大數據的新興學科，該學科建基於商業數據探勘，其研究模式及方法皆有別於一般教育研究。有鑑於此，本研究首先闡述教育數據探勘的原理及技術，再使用PISA 2012之電子化問題解決能力測驗公開題的log數據進行探究。在研究方法上，本研究以PISA 2012問題解決表現最佳的十個經濟體為樣本，涉及三種典型的log數據，即解題時間、點擊次數，以及操作－回應路徑；進行預處理後，本研究使用分類和異常值檢測，及模式探勘等手法分析上述數據。結果顯示，澳門學生的解題時間及點擊次數均多於其他九個經濟體；研究樣本的操作－回應路徑則可分為六個問題解決群組，分別為：「最佳作答」、「自檢查後正確作答」、「不作答」、「誤解題意或不小心作答」、「嬉戲」，以及「其他」。其中，澳門在首兩個群組之總比例少於其他九個經濟體，缺乏解題動機或經過自檢查後仍然出錯者也多於其他經濟體。最後，研究者總結學生的問題解決特點，為教育工作者改善學生的問題解決能力提供實證依據，並為未來的教育數據探勘研究提出建議。

Problem-solving competency is essential for students to tackle challenges in future life. However, the traditional paper-pencil problem-solving test cannot reveal student behavior, nor the cognitive and motivational processes in solving problems. Fortunately, computer-based assessment and log file data analysis may be the silver bullet. Educational data mining (EDM) is an emerging discipline that investigates large-scale educational data, such as the log file data. It is developed on the basis of data mining techniques widely employed in business domains. The analytic tools and research methods of EDM are different from those in general educational studies. Under this backdrop, this research aimed to elucidate the principles and research methods of EDM, explored and analyzed the log file data of PISA 2012 computer-based problem-solving assessment. Regarding research methods, this study looked into the top ten high-performing economies in the PISA 2012 problem-solving assessment, analyzing three typical kinds of log file data, i.e. item response time, the number of mouse clicks, and operation-response paths. After pre-processing, techniques of classification and outliner detection, and pattern mining were applied to analyze the log file data. Research results are summarized as below: Macau students' item response time was obviously longer, and their number of mouse clicks was more, than those of the other high-performing economies. The operation- response paths further illustrated six typical log file response patterns, namely: ＂Perfect with High Effectiveness＂, ＂Correct Answer with Self-Checking＂, ＂No Response＂, ＂Misunderstanding and Carelessness＂, ＂Playing＂ and ＂Others＂. Macau's total proportions of the first two student groups were obviously smaller than those of the other high-performing economies, and Macau had more students who lacked motivation to solve problems, or answered incorrectly even after self-checking than the other economies. Finally, this research provided empirical evidences to enhance students' problem-solving competency and gave recommendations for further studies.

1. Baker, R.,Yacef, K.(2009).The state of educational data mining in 2009: A review and future visions.Journal of Educational Data Mining,1,3-17.
2. Baradwaj, B. K.,Pal, S.(2011).Mining educational data to analyze students' performance.International Journal of Advanced Computer Science and Applications,2,63-69.
3. Beck, J. E.,Mostow, J.(2008).How who should practice: Using learning decomposition to evaluate the efficacy of different types of practice for different types of students.Proceedings of 9th International Conference on Intelligent Tutoring Systems,Montreal, Canada:
4. Ben-Zadok, G.,Leiba, M.,Nachmias, R.(2011).Drills, games or tests? Evaluating students' motivation in different online learning activities, using log file analysis.Interdisciplinary Journal of E-Learning and Learning Objects,7,235-248.
5. Cocea, M.,Weibelzahl, S.(2009).Log file analysis for disengagement detection in elearning environments.User Modeling and User-Adapted Interaction,19,341-385.
6. Dringus, L. P.,Ellis, T.(2005).Using data mining as a strategy for assessing asynchronous discussion forums.Computer and Education,45,141-160.
7. Eysenck, M. W.(Ed.)(1990).The Blackwell dictionary of cognitive psychology.Oxford, UK:Basil Blackwell.
8. Frensch, P. A.(Ed.),Funke, J.(Ed.)(1995).Complex problem solving: The European perspective.Hillsdale, NJ:Lawrence Erlbaum Associates.
9. Hershkovitz, A.,Nachmias, R.(2008).Developing a log-based motivation measuring tool.Proceedings of the First International Conference on Educational Data Mining,Montreal, Canada:
10. Kankaras, M.,Moores, G.(2014).Analysis of cross-cultural comparability of PISA 2009 scores.Journal of Cross-Cultural Psychology,45,381-399.
11. Klosgen, W.,Zytkow, J.(2002).Handbook of data mining and knowledge discovery.New York, NY:Oxford University Press.
12. Lee, Y. J.(2015).Analyzing log files to predict students' problem solving performance in a computer-based physics tutor.Educational Technology & Society,18,225-236.
13. Mazza, R.,Milani, C.(2004).GISMO: A graphical interactive student monitoring tool for course management systems.International Conference on Technology Enhanced Learning,Milan, Italy:
14. McGaw, B.(Ed.),Peterson, P.(Ed.),Baker, E.(Ed.)(2010).International encyclopedia of education.Oxford, UK:Elsevier.
15. Merceron, A.,Yacef, K.(2003).A web-based tutoring tool with mining facilities to improve learning and teaching.Proceedings of the 11th International Conference on Artificial Intelligence in Education,Sidney, Australia:
16. Organisation for Economic Co-operation and Development=OECD(2004).Problem solving for tomorrow's world: First measures of cross-curricular competencies from PISA 2003.Paris, France:Author.
17. Organisation for Economic Co-operation and Development=OECD(2014).PISA 2012 results: Creative problem solving: Students' skills in tacking real-life problems.Paris, France:Author.
18. Organisation for Economic Co-operation and Development=OECD(2009).PISA data analysis manual.Paris, France:Author.
19. Organisation for Economic Co-operation and Development=OECD(2013).PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy.Paris, France:Author.
20. Romero, C.,Ventura, S.(2007).Educational data mining: A survey from 1995 to 2005.Expert Systems with Application,33,135-146.
21. Romero, C.,Ventura, S.(2010).Educational data mining: A review of the state of the art.Systems, Man, and Cybernetics-Part C: Applications and Reviews,40,601-618.
22. Romero, C.,Ventura, S.(2013).Data mining in education.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery,3,12-27.
23. Scherer, R.,Tiemann, R.(2012).Factors of problem-solving competency in a virtual chemistry environment: The role of metacognitive knowledge about strategies.Computers and Education,59,1199-1214.
24. Shen, R.,Han, P.,Yang, F.,Yang, Q.,Huang, J.(2003).Data mining and case-based reasoning for distance learning.Journal of Distance Education Technologies,1,46-58.
25. Shih, P. C.,Muñoz, D.,Sánchez, F.(2006).The effect of previous experience with information and communication technologies on performance in a web-based learning program.Computers in Human Behavior,22,962-970.
26. Srivastava, J.,Cooley, R.,Deshpande, M.,Tan, P.(2000).Web usage mining: Discovery and applications of usage patterns from web data.SIGKDD Explorations,1,12-23.
27. Tang, T.,McCalla, G.(2005).Smart recommendation for an evolving e-learning system.International Journal on E-Learning,4,105-129.
28. Zaïane, O.(2002).Building a recommender agent for e-learning systems.Proceedings of the International Conference on Computers in Education,Auckland, New Zealand:

1. 薛寶嫦,楊文佳,麥瑞琪,張國祥(2019)。從PISA科學素養評核框架及評核模式之轉變審視澳門學生科學素養的性別差異趨勢。測驗學刊，66(3)，249-284。