創意產品共識評量之信、效度析論

Consensual Assessment for Creative Products: A Review of Reliability and Validity

10.7108/PT.201003.0059

劉昆夏(Kun-Shia Liu)；鄭英耀(Ying-Yao Cheng)；王文中(Wen-Chung Wang)

Rasch多相模式 ； 共識評量技術 ； 創造力 ； 評分者效果 ； consensual assessment technique ； creativity ； many-facet Rasch model ； rater effect

測驗學刊

57卷1期（2010 / 03 / 01）

59 - 84

繁體中文

本研究經由創意產品共識評量之相關文獻的回顧與評析，歸納出當前創意產品共識評量所存在的信、效度問題，包括：(1)使用Cronbach's係數進行評分者一致性分析，導致評分者信度會因創作者能力分布以及評分者人數多寡，而呈現不穩定的現象；(2)僅以領域專家在「整體創造力」評量指標上的共識程度，做為效度之依據，缺乏創造力概念的聚斂效度與區辨效度之檢核；(3)忽略主觀評量過程中評分者效應所造成的評量誤差，導致相同專業背景的評分者，在不同的研究中所呈現的評分者一致性存有相當的差異。最後，本研究以試題反應理論的Rasch多相模式觀點，針對上述的問題，提出未來研究的可行方向，期能改善創意產品共識評量技術與提升評量的品質。

The study reviews the literature on consensual assessment of creative products and generalized problems of reliability and validity in current consensual assessment of creative products. Major issues are (a) problems of unstable rater reliability caused by creators' ability distribution and rater's number when Cronbach's was used to analyze rater reliability, (b) problems of lacking examination of convergent validity and discriminant validity in creativity concepts when most researches only used domain experts' consensual level in ”overall creativity” as the basis of validity, and (c) problems of ignoring measurement errors caused by raters' effects during subjective assessing process, resulting in significant differences of rater consistency for raters of the same background in different research findings. Finally, to resolve these problems, the study proposes a feasible direction of applying the many-facet Rasch model for future studies in order to improve consensual assessment technique of creative products and elevating the quality of assessment.

1. Amabile, T. M.(1983).Social psychology of creativity.New York:Spring-Verlag.
2. Amabile, T. M.(1979).Effects of external evaluation on artistic creativity.Journal of Personality and Social Psychology,37,221-233.
3. Amabile, T. M.(1982).Social psychology of creativity: A consensual assessment technique.Journal of Personality and Social Psychology,43,997-1013.
4. Amabile, T. M.(1996).Creativity in context.Boulder, CC:Westview Press.
5. Amabile, T. M.,Goldfarb, P.,Brackfield, S. C.(1990).Social influences on creativity: Evaluation, coaction, and surveillance.Creativity Research Journal,3,6-21.
6. Amabile, T. M.,Hennessey, B. A.,Grossman, B. S.(1986).Social influences on creativity: The effects of contracted-for reward.Journal of Personality and Social Psychology,50,14-23.
7. Andrich, D.(1978).A rating formulation for ordered response categories.Psychometrika,43,561-573.
8. Baer, J.,Kaufman, J. C.,Gentile, C. A.(2004).Extension of the consensual assessment technique to nonparallel creative products.Creativity Research Journal,16,113-117.
9. Caroff, X.,Besancon, M.(2008).Variability of creativity judgments.Learning and Individual Differences,18,367-371.
10. Chen, C.,Kasof, J.,Himsel, A. J.,Greenberger, E.,Dong, Q.,Xue, G.(2002).Creativity in drawings of geometric shapes: A cross-cultural examination with the consensual assessment technique.Journal of Cross-Cultural Psychology,33,171-187.
11. Congdon, P. J.,McQueen, J.(2000).The stability of rater severity in large-scale assessment programs.Journal of Educational Measurement,37,163-178.
12. Conti, R.,Coon, H.,Amabile, T. M.(1996).Evidence to support the componential model of creativity: Secondary analyses of three studies.Creativity Research Journal,9,385-389.
13. Dollinger, S. J.(2007).Creativity and conservatism.Personality and Individual Differences,43,1025-1035.
14. Dollinger, S. J.,Palaskonis, D. G.,Pearson, J. L.(2004).Creativity and intuition revisited.Journal of Creative Behavior,38,244-259.
15. Dollinger, S. J.,Shafran, M.(2005).Note on consensual assessment technique in creativity research.Perceptual and Motor Skills,100,592-598.
16. Dollinger, S. J.,Urban, K. K.,James, T. A.(2004).Creativity and openness: Further validation of two creative product measure.Creativity Research Journal,16,35-47.
17. Engelhard, G.(1992).The measurement of writing ability with a many-faceted Rasch model.Applied Measurement in Education,5,171-191.
18. Engelhard, G.(1996).Evaluating rater accuracy in performance assessments.Journal of Educational Measurement,33,56-70.
19. Engelhard, G.(1994).Examining rater error in the assessment of written composition with a many-faceted Rasch model.Journal of Educational Measurement,31,93-112.
20. Garaigordobil, M.(2006).Intervention in creativity with children aged 10 and 11 years: Impact of a play program on verbal and graphic-figural creativity.Creativity Research Journal,18,329-345.
21. Getzels, J.,Csikszentmihalyi, M.(1976).The creative vision: A longitudinal study of problem-finding in art.New York:Wiley-Interscience.
22. Hickey, M.(2001).An application of Amabile's consensual assessment technique for rating the creativity of children's musical compositions.Journal of Research in Music Education,49,234-244.
23. Kasof J.,Chen, C.,Himsel, A.,Greenberger, E.(2007).Values and creativity.Creativity Research Journal,19,105-122.
24. Kaufman, J. C.,Baer, J.,Cole, J. C.,Sexton, J. D.(2008).A comparison of expert and nonexpert raters using the consensual assessment technique.Creativity Research Journal,20,171-178.
25. Kaufman, J. C.,Baer, J.,Gentile, C. A.(2004).Differences in gender and ethnicity as measure by ratings of three writing tasks.Journal of Creative Behavior,38,56-69.
26. Kaufman, J. C.,Gentile, C. A.,Baer, J.(2005).Do gifted student writers and creative writing experts rate creativity the same way.The Gifted Child Quarterly,49,260-265.
27. Kaufman, J. C.,Lee, J.,Baer, J.,Lee, S.(2007).Captions, consistency, creativity, and the consensual assessment technique: New evidence of reliability.Thinking Skills and Creativity,2,96-106.
28. Linacre, J. M.(1989).Many-facet Rasch measurement.Chicago, IL:MESA.
29. Liu, K. S.,Cheng, Y. Y.,Wang, W. C.(2008).Rater effects in the cons ensued assessment of creative products.Paper presented at 2008 AERA Annual Meeting,New York:
30. Lumley, T.,McNamara, T. F.(1995).Rater characteristics and rater bias: Implications for training.Language Testing,12,54-71.
31. Lunz, M. E.,Stahl, J. A.,Wright, B. D.(1994).Interjudge reliability and decision reproducibility.Educational and Psychological Measurement,54,913-925.
32. Masters, C. N.(1982).A Rasch model for partial credit scoring.Psychometrika,47,149-174.
33. Mayer, R.E.,R. J. Stemberg (Ed.)(1999).Handbook of creativity.NY:Cambridge University Press.
34. Myford, C. M.,Wolfe, E. W.,E. V. Smith (Eds.),It M. Smith (Eds.)(2004).Introduction to Rasch measurement: Theory, models and applications.Maple Grove, MN:JAM Press.
35. Myford, C. M.,Wolfe, E. W.,E. V. Smith(Eds.),R. M. Smith (Eds.)(2004).Introduction to Rasch measurement: Theory, models and applications.Maple Grove, MN:JAM Press.
36. Priest, T.(2001).Using creativity assessment experience to nurture and predict compositional creativity.Journal of Research in Music Education,49,245-257.
37. Rasch, G.(1980).Probabilistic models for some intelligence and attainment tests.Chicago, IL:The University of Chicago Press.
38. Ruscio, J.,Whitney, D. M.,Amabile, T. M.(1998).Looking inside the fishbowl of creativity: Verbal and behavioral predictors of creative performance.Creativity Research Journal,11,243-263.
39. Sobel, R. S.,Rothenberg, A.(1980).Artistic creation as stimulated by superimposed versus separated visual images.Journal of Personality and Social Psychology,39,953-961.
40. Wang, W. C.,Wilson, M.(2005).Exploring local item dependence using a random-effects facet model.Applied Psychology Measurement,29,296-318.
41. Webster, P.,Hickey, M.(1995).Rating scales and their use in assessing children's music compositions.The Quarterly Journal of Music Teaching and Learning,6,28-44.
42. Wright, B. D.,Mok, M. M. C. ,E. V. Smith(Eds.),R. M. Smith (Eds.)(2004).Introduction to Rasch measurement: Theory, models and applications.Maple Grove, MN:JAM Press.
43. 王文中(2004)。Rasch測量理論與其在教育和心理之應用。教育與心理研究，27，637-694。
44. 王文中(1997)。評分客觀性與能力估計客觀性－傳統作法與試題反應理論作法之比較。測驗年刊，44(1)，29-52。
45. 教育部(2002)。創造力教育白皮書。台北市:教育部。

1. 陳明鴻、陳一平(2014)。產品設計領域中創造力組成成分之多元迴歸分析。設計研究學報，7，64-80。
2. 蕭佳純(2012)。國小學生內在動機、學科知識與創造力表現關聯之研究：教師創造力教學的調節效果。特殊教育研究學刊，37(3)，89-113。
3. 蕭佳純(2017)。國小學童正向情緒與創造力關聯之研究：以創造力傾向及創意自我效能為中介變項。教育研究與發展期刊，13(4)，57-84。
4. 蕭佳純、涂志賢(2012)。教師創造力教學行為量表之發展。課程與教學，15(2)，87-117。
5. 鄭英耀、何曉琪、王佳琪(2016)。科學想像力圖形測驗之發展。教育科學研究期刊，61(4)，177-204。