Validating the Cloze Items in the English Subtest of the General Scholastic Ability Test in Taiwan

台灣大學學科能力測驗英語科測驗克漏字題目效度的驗證

10.3966/181147172018120020001

劉玉玲(Yu-Ling Liu)；林文鶯(Wen-Ying Lin)；游錦雲(Ching-Yun Yu)

cloze test ； construct validity ； confirmatory factor analysis ； overall English proficiency ； 克漏字 ； 構念效度 ； 驗證性因素分析 ； 整體英語閱讀能力

語文與國際研究

20期（2018 / 12 / 01）

1 - 23

英文

In Taiwan, English is regarded a key subject in high school curriculum. Hence, before entering university, high school seniors are required to take an English subtest (ES) of the General Scholastic Ability Test (GSAT). Their scores on the GSAT-ES, besides other subtests, influence considerably their future academic pursuit and career path. To them, an important question is: What does the GSAT-ES really measure - several aspects of their English proficiency or just their overall English proficiency? This study was aimed to address, among others, such a question and focus particularly on the two types of cloze items in the GSAT-ES. For this study, a simpler version of Purpura's (2004) model of grammatical knowledge was adopted to classify the cloze items. Classification was done by a group of five experienced university-level English teachers. Then confirmatory factor analysis (CFA) was applied to investigate the degree of fit between the raters' classifications and the test takers' responses. The CFA findings indicated that the raters' classifications failed to fit the test takers' responses and that the one-component model best portrayed the response data. The finding of a one-component model agreed with the unitary competence hypothesis of Oller (1976, 1979), who advocated using cloze items for measuring overall language proficiency. This study concluded with some relevant implications for English teaching and testing.

本研究目的乃是藉由驗證性因素分析方法，探討台灣大學學科能力測驗英語科測驗中克漏字題目的構念效度。本研究邀請五位專家們根據Purpura（2004）理論架構將2015及2016年克漏字題目進行題目分類，主要研究問題是在檢視此分類架構與考生的答題資料是否有適配性？為回答此研究問題，本研究向大學考試入學中心申請2015及2016年隨機抽樣各5,500位學生每題克漏字答題的得分資料。本研究藉由驗證性因素分析探討專家將題目分類後產生的測量模型是否與考生的實徵資料相互配適，以期能找出實徵資料的最佳適配模型。研究結果顯示，專家的題目分配模型與實徵資料的適配度不是最佳的，最佳的適配度模型反而是單一整體特質模型。也就是說，本研究結果顯示，大學學科能力測驗英語科測驗克漏字題目乃是在測量整體英語閱讀能力。最後，本研究也根據研究結果，針對高中英語教師與大學考試入學中心編寫克漏字試題相關人員，提出具體建設。

1. Alderson, J. C.(1980).Native and non-native speaker performance on cloze tests.Language Testing,30(1),59-76.
2. Alderson, J. C.(1979).The cloze procedure and proficiency in English as a foreign language.TESOL Quarterly,13(2),219-227.
3. Anderson, D. C.(1980).Illinois,University of Illinois, Urbana-Champaign.
4. Bachman, L. F.(1982).The traits structure of cloze test scores.TESOL Quarterly,16(1),61-70.
5. Bagozzi, R. P.,Yi, Y.(1988).On the evaluation of structural equation models.Academic of Marketing Science,16(1),76-94.
6. Barnett, M. A.(1986).Syntactic and lexical/semantic skills in foreign language reading: Importance and interaction.Modern Language Journal,70,343-349.
7. Barry, S.,Lazarte, A. A.(1995).Embedded clause effects on recall: Does high prior knowledge of content domain overcome syntactic complexity in students of Spanish?.Modern Language Journal,79,491-504.
8. Beinbom, L.,Zesch, T.,Gurevych, I.(2015).Candidate Evaluation Strategies for Improved Difficulty Prediction of Language Tests.Proceedings of the Building Educational Applications Workshop at NAACL
9. Bensoussan, M.,Ramaz, R.(1984).Testing EFL reading comprehension using a multiple-choice rational cloze.Modern Language Journal,68(3),230-239.
10. Bollen, K. A.(1989).Structural Equations with Latent Variables.New York:John Wiley & Sons.
11. Buck, G.(2001).Assessing Listening.Cambridge:Cambridge University Press.
12. Burns, G. L.,Patterson, D. R.(2000).Factor structure of the Eyberg child behavior inventory: A parent rating scale of oppositional defiant behavior toward adults, inattentive behavior, and conduct problem behavior.Journal of Clinical Child Psychology,29(4),569-577.
13. Can, R.(2016).The relationship between readers' comprehension skills and awareness of cohesive relationships.Anthropologist,23(1-2),85-92.
14. Carroll, J. B.(1975).The Teaching of French as a Foreign Language in Eight Countries.New York:Wiley Center for Applied Linguistics.
15. CEEC. (2016). Test Manual of the English Subtest of the General Scholastic Ability Test. Retrieved from http://www.ceec.edu.tw/107學測英文考試說明定稿.pdf
16. Chou, S. Y.(2009).Chiayi, Taiwan,National Chung Cheng University.
17. De Beaugrande, R.,Dressier, W.(1981).Introduction to Text Linguistics.London:Longman.
18. Diamantopoulos, A.,Siguaw, J. A.(2000).Introducing LISREL: A Guide for the Uninitiated.London:Sage.
19. Fornell, C.,Larcker, D. F.(1981).Evaluating structural equation models with unobservable variables and measurement error.Journal of Marketing Research,18(1),39-50.
20. Fries, C. (1945). Teaching and Learning English as a Foreign Language. Ann Arbor: University of Michigan Press
21. Gable, R. K.,Wolf, M. B.(1993).Instrument Development in the Affective Domain: Measuring Attitudes and Values in Corporate and School Settings.Boston:Kluwer Academic Publishers.
22. Gardner, R. C.,Lambert, W. E.(1965).Language aptitude, intelligence, and second language achievement.Journal of Educational Psychology,56(4),191-199.
23. Gignac, G. E.(2005).Evaluating the MSCEIT V2.0 via CFA: Corrections to Mayer et al. (2003).Emotion,5,233-235.
24. Hale, G. A.,Stansfield, C. W.,Rock, D. A.,Hicks, M. M.,Butler, F. A.,Oller, J. W.(1988).Multiple-Choice Cloze Items and the Test of English as a Foreign Language (Research Reports No. 26).Princeton, NJ:Educational Testing Service.
25. Harsch, C.,Hartig, J.(2016).Comparing C-tests and Yes/No vocabulary size tests as predictors of receptive language skills.Language Testings,33(4),555-575.
26. Henning, S. D.(1992).Assessing literary interpretation skills.Foreign Language Annals,25,339-344.
27. Hosley, D.,Meredith, K.(1979).Inter- and intra-test correlates of the TOEFL.TESOL Quarterly,13(2),209-217.
28. Hoyle, R. H.(ed.)(1999).Structural Equation Modeling: Concepts, Issues, and Applications.London:Sage.
29. Huang, T. S.(1994).A Qualitative Analysis of the JCEE English Tests.Taipei:Crane Publishing Company.
30. Hughes, A.(2003).Testing far Language Teachers second edition.Cambridge:Cambridge University Press.
31. Joreskog, K.,Sorbom, D.(1993).LISREL 8: Structural Equation Modeling with the SIMPLIS Command Language.Hillsdale:Lawrence Erlbaum Associates, Inc..
32. Khodadad, E.(2012).Validity and tests developed on reduced redundancy, language components and schema theory.Theory and Practice in Language Studies,2(3),585-595.
33. Klein-Braley, C.(1997).C-Tests in the context of reduced redundancy testing: an appraisal.Language Testing,14(1),47-84.
34. Kline, P.(1994).An Easy Guide to Factor Analysis.New York:Routledge.
35. Lado, R.(1964).Language Teaching: A Scientific Approach.New York:McGraw-Hill.
36. Lado, R.(1961).Language Testing: The Construction and Use of Foreign Language Tests.London:Longman.
37. Landis, J. R.,Koch, G. G.(1977).The measurement of observer agreement for categorical data.Biometrics,33,159-174.
38. Lofgren, H.(1969).Measuring Proficiency in the German Language: A Study of Pupils in grade 7 (Didakometry No. 25).Malmo Sweden:School of Education.
39. Lonigan, C. J.,Hooe, E. S.,David, C. F.,Kistner, J. A.(1999).Positive and negative affectivity in children: Confirmatory factor analysis of a two-factor model and its relation to symptoms of anxiety and depression.Journal of Consulting and Clinical Psychology,67,374-386.
40. Lynn, M.(1986).Determination and quantification of content validity.Nursing Research,35,382-385.
41. MacKenzie, S. B.,Podsakoff, P. M.,Jarvis, C. B.(2005).The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions.Journal of Applied Psychology,90(4),710-730.
42. Ohnmacht, F. W.,Weaver, W. W.,Kohler, E. T.(1970).Cloze and closure: A factor study.Journal of Psychology,74,205-217.
43. Oller, J. W.(1976).Evidence for a general language proficiency factor: An expectancy grammar.Die Neueren Sprachen,75,165-174.
44. Oller, J. W.(1979).Language Tests at School.London:Longman.
45. Oller, J. W.(ed.)(1983).Issues in Language Testing Research.Rowley:Newbury House.
46. Oller, J. W.(ed.),Perkins, K.(ed.)(1978).Language in Education: Testing the Tests.Rowley:Newbury House.
47. Oller, J. W.(ed.),Perkins, K.(ed.)(1980).Research in Language Testing.Rowley:Newbury House.
48. Pimsleur, P.,Stockwell, R.,Comrey, A.(1962).Foreign language learning ability.Journal of Educational Psychology,53,15-26.
49. Purpura, J. E.(2004).Assessing Grammar.Cambridge:Cambridge University Press.
50. Rubio, D. M.,Berg-Weger, M.,Tebb, S. S.,Lee, E. S.,Rauch, S.(2003).Objectifying content validity: Conducting a content validity study in social work research.Social Work Research,27,94-104.
51. Saito, Y.(2003).Investigating the construct validity of the cloze section in the Examination for the Certificate of Proficiency in English.Spaan Fellow Working Papers in Second or Foreign Language Assessment
52. Scholz, G.,Hendricks, D.,Spurling, R.,Johnson, M.,Vandenburg, L.(1980).Is language ability divisible or unitary? A factor analysis of twenty-two English proficiency tests.Research in Language Testing,Rowley:
53. Shiotsu, T.,Weir, C. J.(2007).The relative significance of syntactic knowledge and vocabulary breadth in the prediction of reading comprehension test performance.Language Testing,24,99-128.
54. Spolsky, B.(1973).What does it mean to know a language; or how do you get somebody to perform his competence?.Focus on the Learner,Rowley:
55. Taylor, W. L.(1953).Cloze procedure: A new tool for measuring readability.Journalism Quarterly,30(4),414-438.
56. Waltz, C. F.,Strickland, O.,Lenz, E.(1991).Measurement in Nursing Research.Philadelphia:FA Davis.
57. Weaver, W W.,Kingston, A. J.(1963).A factor analysis of the cloze procedure and other measures of reading and language ability.Journal of Communication,13(4),252-261.
58. Yu, C. Y(2002).Los Angeles, CA.,University of California.