校務檔案怎麼評？校務經營檔案之信效度分析

Portfolio Assessment: Reliability and Validity in School Management

10.6251/BEP.202212_54(2).0004

謝名娟(Ming-Chuan Hsieh)

校務經營 ； 檔案評量 ； 信度 ； 效度 ； school management ； portfolio assessment ； reliability ； validity

教育心理學報

54卷2期（2022 / 12 / 01）

333 - 355

英文

校務經營能力為候用校長的必備職能之一。過去儲訓校長班繳交許多作業與文檔，內容聚焦為校長校務經營的各種面向的培養，但卻缺乏系統性的蒐集。本研究設計候用校長設計校務經營的檔案評量，並探討其信效度。評分者的評分嚴苛度、評分規準、評分者是否為師傅校長或是外聘專家對於受試者的影響亦加以探討。其結果顯示候用校長在校務檔案的製作中，對於學校背景資料掌握較好，對於創新經營的面向較差，外聘專家比師傅校長評分較為嚴格，但外聘與師傅校長評分均受到誤差的影響。總體而言，校務經營檔案評量具有大致良好的幅合效度證據，但區辨效度較不理想。

Portfolio assessment provides an opportunity to understand characteristics about a student that cannot be gleaned from responses to traditional test questions; the characteristics obtained through portfolio assessment are more integrated and reflect a more comprehensive picture of a student's learning experiences. Portfolio assessment has become more popular in recent years as a tool for measuring a student's suitability for admission into university. Portfolio assessment allows a learner to self-monitor their progress and achievement of learning goals through a systematic collection of assignments. The role of an instructor is to guide a student in constructing a portfolio, to set learning goals, and to provide opportunities for self-assessment and peer feedback. Many scholars and experts in Taiwan have built a rich theoretical foundation and performed practical research on portfolio assessment at the university, elementary, and secondary levels; few studies have investigated the use of portfolio assessment in other contexts. This study investigates the reliability and validity of portfolio assessment in the context of school management and, in particular, their use by preservice principals. The influence of scorer severity, scoring rubrics, and the influence of internal or external experts was investigated. In current preservice principal training courses, preservice principals are given many assignments. Although the assignment content focuses on the various aspects of school management, a systematic collection of assignments is lacking. In particular, challenges with practical implementation include the availability of too many assignments and the assignments having overlapping purposes. These disorganized and unsystematic assignments are a burden to both preservice principals and scorers. Based on the six professional competency indicators of preservice principals (visioning, strategic thinking, teamwork, communication and coordination, innovative management, and self-reflection), this study systematically constructs a course that uses portfolio assessment. However, evidence of the validity of the use of portfolio assessment in the context of school management must be further investigated. Preservice principal training courses cover a variety of topics, including school development, administration, professional responsibility, public relations, curriculum and instructional leadership, educational visitation, teacher learning, liberal arts, and integrated activities; such courses also often incorporate aspects of mentorship. Mentorship learning is designed to help preservice principals finish their assessments. Typically, six mentoring sessions from the senior school principals are provided on the content and practices required in the school development portfolio. In addition to written assignments, individual sharing is required, and mentors provide feedback. In total, 61 preservice principals who enrolled in the 8-week training course participated in this study. Each student was required to produce a portfolio of assignments related to school management that covered five major topics: (1) school background information, (2) goals and action plans, (3) school curriculum design, (4) school innovation plan, and (5) selfreflection. These five components are a primary focus of school development. Each class had two coaches and two external experts who scored the portfolios. A total of 12 experts were involved in the scoring process. At the start of the course, the coaches of each class introduced the function of the portfolios and explained the key content that must be included in each section. Students were asked to upload the content of their portfolios two to three times to the system as the course progressed. For example, after the "Data-Driven School Research" seminar, students were asked to complete an inventory of school records, followed by questions and forms to guide them through the process . At each step, the coaches shared, discussed, and advised students before moving on to the next section. An overall grade was given after completion of the entire portfolio. Reliability was assessed using the multifacet Rasch model (MFRM). Item difficulty and scorer severity were both considered when assessing ability. The validity dimension examined convergent and discriminant validity. Validity was examined by correlating portfolio scores with test scores and other assignment scores, and scores were based on daily observations by the coaches . The mean square of infit and outfit for each level are close to 1, which represents goodness of fit. In the Rasch model, reliability represents the reproducibility of the measure; ability as measured in a test with high reliability is more likely to be consistent with ability in the real world. Most reliability indices are relatively high; the lowest values occur at the estimation of the ability of preservice principals, which may be due to the high homogeneity and similarity of the preservice principals. Although the sample size was not large, the fit of the model was acceptable, which means that the data in this study are suitable for analysis using the MFRM. Students performed poorly in creating innovative plans but performed well in school background information, which may be because most of these students tend to come from backgrounds without much opportunity to make decisions on the direction of school affairs. For example, having a background as a director predisposes an individual to being uncertain about the direction of school innovation, whereas it is trivial for a student of any background to present information related to a school's background. The external experts were more severe than the coaches, probably because as mentors, the coaches developed close relationships with the students, and this made it difficult for the coaches to give the students low scores. The final study found a strong correlation between portfolio scores and oral scores. Oral tests were intended to provide an opportunity for students to present their school management philosophies. A significant correlation between portfolio scores and weekly reflection journal scores was also observed and is an indication of validity. A significant correlation between portfolio and behavioral performance scores was also observed, possibly because scoring of both of these items was performed by coaches and may therefore have been affected by the halo effect, resulting in lower discriminant validity. Although the scorers are experts in the field, they still need to be trained in scoring. In the future, we should design a rigorous scorer training program. For example, we can place some calibration cases in the scoring process to examine rater bias and make adjustments immediately. Exploring how to design a more convenient and easier-to-use model of scorer training to increase consensus among scorers and reduce subjective bias is noteworthy. In this study, both internal and external experts had different sources of errors in scoring; external experts were more biased when evaluating contextual data. Therefore, in future courses, external experts should be tasked with scoring factual information and internal experts (e.g., mentor principals) should be tasked with scoring contextual information, such as the rationale for or consistency of a portfolio. Exploring how the halo effect might be avoided is another potential area of improvement. Incorporating anonymous cross-class grading or peer-graded comments and suggestions might help avoid the halo effect. Cross-disciplinary collaboration should be considered as a way of enhancing innovative thinking. Team activities, such as brainstorming as a team, may be useful in helping students generate innovative ideas. In conclusion, this study evidenced the reliability and validity of portfolio assessment in the context of school management. However, careful consideration is still required to determine if it is suitable as an examination tool.

1. 余民寧, M.-N.(2011).教育測驗與評量—成就測驗與教學評量.心理出版社=Psychological Publishing.
   連結：
2. 林小慧, H.-H.,林世華, S.-H.,吳心楷, H.-K.(2018)。科學能力的建構反應評量之發展與信效度分析：以自然科光學為例。教育科學研究期刊，63(1)，173-205。
   連結：
3. 林素卿, S.-C.,葉順宜, S.-I.(2014)。檔案評量於國中英語教學應用之個案研究。教育科學研究期刊，59(2)，111-139。
   連結：
4. 張郁雯, Y.-W.(2010)。國小學童資訊素養檔案評量之發展研究。教育心理學報，41，521-550。
   連結：
5. 張郁雯, Y.-W.(2008)。國小學童資訊素養檔案評量之信度研究。教育心理學報，39，43-60。
   連結：
6. 張基成, C.-C.,吳炳宏, B.-H.(2012)。網路化檔案評量環境下教學者評量之信度與效度。科學教育學刊，20，393-412。
   連結：
7. 張基成, C.-C.,廖悅媚, Y.-M.(2013)。數位化學習歷程檔案之學習目標設定對自我調整學習之影響。科學教育學刊，21，431-454。
   連結：
8. 張新立, H.-L.,吳舜丞, S.-C.(2008)。多層面 Rasch 模式於學術研討會論文評分之應用。測驗學刊，55，105-128。
   連結：
9. 陳宏彰, H.-C.(2017)。跨域社會化經驗：候用校長教育局處行政實習實踐之分析與反省。當代教育研究季刊，25(3)，1-40。
   連結：
10. 劉祥熹, H.-H.,陳玉娟, Y.-C.,鄭筱慧, H.-H.(2016)。學校創新經營對家長選校意願影響之研究－以服務品質與學校形象為中介變項。教育科學研究期刊，61(4)，59-88。
    連結：
11. 謝名娟, M.-C.(2020)。從多層面Rasch模式來檢視不同的評分者等化連結設計對參數估計的影響。教育心理學報，52，415-436。
    連結：
12. 謝名娟, M.-C.(2017)。誰是好的演講者？以多層面 Rasch 來分析校長三分鐘即席演講的能力。教育心理學報，48，551-566。
    連結：
13. Arter, J. A.,Spandle, V.(1992).NCME instructional module: Using portfolios of student work in instruction and assessment.Educational Measurement, Issues and Practice,11(1),36-43.
14. Chang, C.-C.,Liang, C.,Chou, P.-N.,Liao, Y.-M.(2018).Using e-portfolio for learning goal setting to facilitate self-regulated learning of high school students.Behaviour & Information Technology,37(12),1237-1251.
15. Chang, C.-C.,Liang, C.,Tseng, K.-H.,Tseng, J.-S.,Chen, T.-Y.(2013).Were knowledge management abilities of university students enhanced after creating personal blog-based portfolios?.Australasian Journal of Educational Technology,29(6),916-931.
16. Chau, J.,Cheng. G.(2010).Towards the understanding the potential of e-portfolios for independent learning: A qualitative research.Australasian Journal of Educational Technology,26(7),932-950.
17. Coombe, C.,Barlow, L.(2004).The reflective portfolio: Two case studies from the United Arab Emirates.English Teaching Forum,42(1),18-23.
18. Diperna, J.,Derham, C.(2007).Digital professional portfolios of preservice teaching: An initial study of score reliability and validity.Journal of Technology and Teacher Education,15(3),363-381.
19. Gadbury-Amyot, C. C.,Kim, J.,Palm, R. L.,Mills, G. E.,Noble, E.,Overman, P. R.(2003).Validity and reliability of portfolio assessment of competency in a baccalaureate dental hygiene program.Journal of Dental Education,67,991-1002.
20. Higher Education Funding Council for England. (2008). Effective practice with e-portfolios: Supporting 21st century learning. JISC. https://research.qut.edu.au/eportfolio/wp-content/uploads/sites/186/2018/04/JISC_effective_practice_e-portfolios.pdf
21. Hughes, J.(2008).E-portfolio-based learning: A practitioner perspective.Enhancing Learning in the Social Sciences,1(2),1-12.
22. Joyes, G.,Gray, L.,Hartnell-Young, E.(2010).Effective practice with e-portfolios: How can the UK experience inform implementation?.Australasian Journal of Educational Technology,26(1),15-27.
23. LeMahieu, P. G.,Gitomer, D. H.,Eresh, J. T.(1995).Portfolios in large-scale assessment Difficult but not impossible.Educational Measurement: Issues and Practice,14(3),11-28.
24. Linacre, J. M.(2014).Linacre, J. M. (2014). Facets Rasch measurement [computer software]. Winsteps.com..Winsteps.com
25. Linacre, J. M.(2010).Linacre, J. M. (2010). A user’s guide to WINSTEPS: Rasch-model computer programs [Computer software manual]. Winsteps.com..https://winsteps.com
26. Linacre, J. M.(1998).Structure in Rasch residuals: Why principal components analysis?.Rasch Measurement Transactions,12(2),Article 636.
27. Linn, R. L.(Ed.)(1989).Educational measurement.Macmillan.
28. McNamara, T. F.(1996).Measuring second language performance.Longman.
29. Sharma, S.(2007).From chaos to clarity: Using the research portfolio to teach and assess information literacy skills.The Journal of Academic Librarianship,33(1),127-135.
30. Singh, O.,Ritzhaupt, A. D.(2006).Student perceptive of organizational uses of eportfolios in higher education.Proceedings of edmedia
31. Skawinski, S. F.,Thibodeau, S. J.(2003).A journey into portfolio assessment.The Education Forum,67(1),81-88.
32. Smith, E.V., Jr.(2002).Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals.Journal of Applied Measurement,3(2),205-231.
33. Spencer, L.,Spencer, S.(1993).Competency at work: Models for superior performance.John Wiley & Sons.
34. Tennant, A.,Pallant, J.(2006).Unidimensionality matters (a tale of two Smiths?).Rasch Measurement Transactions,20(1),1048-1051.
35. Wright, B. D.,Masters, G. N.(1982).Rating scale analysis.MESA Press.
36. 石文傑, W.-J.,馮啟峰, C.-F.,劉偉欽, W.-C.,羅聰欽, T.-C.(2014)。高職校長科技領導能力指標之探討。科技與工程教育學刊，47(2)，1-14。
37. 林信志（計畫主持人）, H.-C.(Principal Investigator),謝名娟（計畫主持人）, M.-C.(Principal Investigator)(2018)。科技部補助專題研究計畫成果報告科技部補助專題研究計畫成果報告,科技部=Ministry of Science and Technology。
38. 張美玉, M.-Y.(2000)。歷程檔案評量的理念與實施。科學教育，231，58-63。
39. 張基成, C.-C.,林俊宇, J.-Y.(2015)。網路化檔案評量系統內反思機制之設計及其對自我調整學習影響之評估。教育與心理研究，38(1)，31-64。
40. 張基成, C.-C.,彭星瑞, S.-R.(2008)。網路化檔案評量於國中電腦課程之使用及成效。師大學報：科學教育類，53(2)，31-57。
41. 張麗麗, L.-L.(2002)。檔案評量信度與效度的分析—以國小寫作檔案為例。教育與心理研究，25(1)，1-34。
42. 謝名娟（計畫主持人）, M.-C.(Principal Investigator),林信志（計畫主持人）, H.-C.(Principal Investigator)(2014)。國家教育研究院計畫成果報告國家教育研究院計畫成果報告,國家教育研究院=National Academy for Educational Research。