DOI: 10.3724/SP.J.1042.2019.00171

Advances in Psychological Science (心理科学进展) 2019/27:1 PP.171-180

Rater effects in creativity assessment

Rater effects refer to the impact of different raters' idiosyncrasies in their behaviors on the evaluation results in creativity assessment. Rater effects are due to the difference in raters' cognitive process of the evaluation, which are externally reflected in the difference of their scorings. This article first summarizes the studies of rater cognition and other influencing factors on creativity assessment, including characteristics of raters, information of creators and socio-cultural factors. It further examines inter-rater reliability indexes and their limitations, as well as the applications of Generalization Theory and Many-Facet Rasch Model in quantifying and controlling of rater effects. Finally, this paper specifies directions of future research based on the existing limitations, including deepening the investigation on rater cognition in creativity assessment, integrating the studies of rater effects on different levels, and developing new methods and techniques of creativity assessment.

Key words:creativity,subjective scoring,rater effects,rater cognition,inter-rater agreement

ReleaseDate:2019-01-28 10:22:55

贡喆, 刘昌, 沈汪兵. (2016). 有关创造力测量的一些思考. 心理科学进展, 24(1), 31-45.

晏子. (2010). 心理科学领域内的客观测量——Rasch模型之特点及发展趋势.心理科学进展, 18(8), 1298-1305.

Amabile, T. M. (1982). Social psychology of creativity:A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997-1013.

Amabile, T. M. (1983). The social psychology of creativity. New York, NY:Springer-Verlag.

American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing (2014 Edition). Washington, DC:AERA.

Baer, J., Kaufman, J. C., & Riggs, M. (2009). Brief report:Rater-domain interactions in the consensual assessment technique. The International Journal of Creativity & Problem Solving, 19(2), 87-92.

Beghetto, R. A., & Kaufman, J. C. (2007). Toward a broader conception of creativity:A case for "mini-c" creativity. Psychology of Aesthetics, Creativity, and the Arts, 1(2), 73-79.

Beketayev, K., & Runco, M. A. (2016). Scoring divergent thinking tests by computer with a semantics-based algorithm. Europe's Journal of Psychology, 12(2), 210-220.

Benedek, M., Mühlmann, C., Jauk, E., & Neubauer, A. C. (2013). Assessment of divergent thinking by means of the subjective top-scoring method:Effects of the number of top-ideas and time-on-task on reliability and validity. Psychology of Aesthetics, Creativity, and the Arts, 7(4), 341-349.

Benedek, M., Nordtvedt, N., Jauk, E., Koschmieder, C., Pretsch, J., Krammer, G., & Neubauer, A. C. (2016). Assessment of creativity evaluation skills:A psychometric investigation in prospective teachers. Thinking Skills and Creativity, 21, 75-84.

Birney, D. P., Beckmann, J. F., & Seah, Y. Z. (2016). More than the eye of the beholder:The interplay of person, task, and situation factors in evaluative judgements of creativity. Learning and Individual Differences, 51, 400-408.

Blair, C. S., & Mumford, M. D. (2007). Errors in idea evaluation:Preference for the unoriginal? The Journal of Creative Behavior, 41(3), 197-222.

Campbell, D. T. (1960). Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological Review, 67(6), 380-400.

Cheng, K. H. C. (2016). Perceived interpersonal dimensions and its effect on rating bias:How neuroticism as a trait matters in rating creative works. The Journal of Creative Behavior. February 16, 2017, Retrieved from

Cropley, A. (2006). In praise of convergent thinking, Creativity Research Journal, 18(3), 391-404.

Diedrich, J., Benedek, M., Jauk, E., & Neubauer, A. C. (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35-40.

Ellamil, M., Dobson, C., Beeman, M., & Christoff, K. (2012). Evaluative and generative modes of thought during the creative process. NeuroImage, 59(2), 1783-1794.

Fink, A., Benedek, M., Koschutnig, K., Pirker, E., Berger, E., Meister, S., … & Elisabeth M. W. (2015). Training of verbal creativity modulates brain activity in regions associated with language-and memory-related demands. Human Brain Mapping, 36(10), 4104-4115.

Finke, R. A., Ward, T. B., & Smith, S. M. (1992). Creative cognition:Theory, research, and applications.. Cambridge, MA:MIT Press.

Forthmann, B., Holling, H., Zandi, N., Gerwig, A., Çelik, P., Storme, M., & Lubart, T. (2017). Missing creativity:The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores. Thinking Skills and Creativity, 23, 129-139.

Galati, F. (2015). Complexity of judgment:What makes possible the convergence of expert and nonexpert ratings in assessing creativity. Creativity Research Journal, 27(1), 24-30.

Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking:Strategies and executive involvement in generating novel uses for familiar objects. British Journal of Psychology, 98(4), 611-625.

Goncalo, J. A., & Staw, B. M. (2006). Individualism-collectivism and group creativity. Organizational Behavior and Human Decision Processes, 100(1), 96-109.

Haller, C. S., Courvoisier, D. S., & Cropley, D. H. (2011). Perhaps there is accounting for taste:Evaluating the creativity of products. Creativity Research Journal, 23(2), 99-109.

Han, J. T., Long, H. Y., & Pang, W. G. (2017). Putting raters in ratees' shoes:Perspective taking and assessment of creative products. Creativity Research Journal, 29(3), 270-281.

Hao, N., Ku, Y. X., Liu, M. G., Hu, Y., Bodner, M., Grabner, R. H., & Fink, A. (2016). Reflection enhances creativity:Beneficial effects of idea evaluation on idea generation. Brain and Cognition, 103, 30-37.

Harbison, J. I., & Haarmann, H. (2014). Automated scoring of originality using semantic representations, Proceedings of the COGSCI, 36, 2327-2332.

Hennessey, B. A. (1994). The consensual assessment technique:An examination of the relationship between ratings of product and process creativity. Creativity Research Journal, 7(2), 193-208.

Hennessey, B. A., & Amabile, T. M. (2010). Creativity. Annual Review of Psychology, 61, 569-598.

Hong, S. W., & Lee, J. S. (2015). Nonexpert evaluations on architectural design creativity across cultures. Creativity Research Journal, 27(4), 314-321.

Hung, S. P., Chen, P. H., & Chen, H. C. (2012). Improving creativity performance assessment:A rater effect examination with many facet Rasch model. Creativity Research Journal, 24(4), 345-357.

Kaufman, J. C., Baer, J., Agars, M. D., & Loomis, D. (2010). Creativity stereotypes and the consensual assessment technique. Creativity Research Journal, 22(2), 200-205.

Kaufman, J. C., Baer, J., Cole, J. C., & Sexton, J. D. (2008). A comparison of expert and nonexpert raters using the consensual assessment technique. Creativity Research Journal, 20(2), 171-178.

Kaufman, J. C., Baer, J., Cropley, D. H., Reiter-Palmon, R., & Sinnett, S. (2013). Furious activity vs. understanding:How much expertise is needed to evaluate creative work? Psychology of Aesthetics, Creativity, and the Arts, 7(4), 332-340.

Kaufman, J. C., Beghetto, R. A., & Dilley, A. (2016). Understanding creativity in the schools. In Lipnevich, A. A., Preckel, F., & Roberts, R. D. (Eds.), Psychosocial skills and school systems in the 21st century (pp. 133-153). Springer.

Kozbelt, A., & Serafin, J. (2009). Dynamic evaluation of high-and low-creativity drawings by artist and nonartist raters. Creativity Research Journal, 21(4), 349-360.

Lan, L., & Kaufman, J. C. (2012). American and Chinese similarities and differences in defining and valuing creative products. The Journal of Creative Behavior, 46(4), 285-306.

Lebuda, I., & Karwowski, M. (2013). Tell me your name and I'll tell you how creative your work is:Author's name and gender as factors influencing assessment of products' creativity in four different domains. Creativity Research Journal, 25(1), 137-142.

Licuanan, B. F., Dailey, L. R., & Mumford, M. D. (2007). Idea evaluation:Error in evaluating highly original ideas. The Journal of Creative Behavior, 41(1), 1-27.

Linacre, J. M. (1994). Many-facet Rasch measurement (2nd Edition). Chicago, IL:MESA.

Long, H. Y. (2014a). An empirical review of research methodologies and methods in creativity studies (2003-2012). Creativity Research Journal, 26(4), 427-438.

Long, H. Y. (2014b). More than appropriateness and novelty:Judges' criteria of assessing creative products in science tasks. Thinking Skills and Creativity, 13, 183-194.

Long, H. Y., & Pang, W. G. (2015). Rater effects in creativity assessment:A mixed methods investigation. Thinking Skills and Creativity, 15, 13-25.

Lu, C. C., & Luh, D. B. (2012). A comparison of assessment methods and raters in product creativity. Creativity Research Journal, 24(4), 331-337.

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46.

Mueller, J. S., Melwani, S., & Goncalo, J. A. (2012). The bias against creativity:Why people desire but reject creative ideas. Psychological Science, 23 (1), 13-17.

Mueller, J. S., Wakslak, C. J., & Krishnan, V. (2014). Construing creativity:The how and why of recognizing creative ideas. Journal of Experimental Social Psychology, 51, 81-87.

Mumford, M. D., Lonergan, D. C., & Scott, G. (2002). Evaluating creative ideas:Processes, standards, and context. Inquiry:Critical Thinking Across the Disciplines, 22(1), 21-30.

Plucker, J., Beghetto, R. A., & Dow, G. (2004). Why isn't creativity more important to educational psychologists? Potential, pitfalls, and future directions in creativity research. Educational Psychologist, 39(2), 83-96.

Plucker, J. A., & Makel, M. C. (2010). Assessment of creativity. In Kaufman, J. C. & Sternberg, R. J. (Eds.), The Cambridge handbook of creativity (pp. 48-73). New York, NY:Cambridge University Press.

Primi, R. (2014). Divergent productions of metaphors:Combining many-facet Rasch measurement and cognitive psychology in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 8(4), 461-474.

Runco, M. A., & Jaeger, G. J. (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92-96.

Runco, M. A., & Smith, W. R. (1992). Interpersonal and intrapersonal evaluations of creative ideas. Personality and Individual Differences, 13(3), 295-302.

Silvia, P. J. (2008). Discernment and creativity:How well can people identify their most creative ideas? Psychology of Aesthetics, Creativity, and the Arts, 2(3), 139-146.

Silvia, P. J. (2011). Subjective scoring of divergent thinking:Examining the reliability of unusual uses, instances, and consequences tasks. Thinking Skills and Creativity, 6(1), 24-30.

Silvia, P. J., Martin, C., & Nusbaum, E. C. (2009). A snapshot of creativity:Evaluating a quick and simple method for assessing divergent thinking. Thinking Skills and Creativity, 4(2), 79-85.

Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram, J. T., Hess, K. I.,... Richard, C. A. (2008). Assessing creativity with divergent thinking tasks:Exploring the reliability and validity of new subjective scoring methods. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 68-85.

Sowden, P. T., Pringle, A., & Gabora, L. (2015). The shifting sands of creative thinking:Connections to dual-process theory. Thinking & Reasoning, 21(1), 40-60.

Storme, M., Myszkowski, N., Çelik, P., & Lubart, T. (2014). Learning to judge creativity:The underlying mechanisms in creativity training for non-expert judges. Learning and Individual Differences, 32(4), 19-25.

Tan, M., Mourgues, C., Hein, S., MacCormick, J., Barbot, B., & Grigorenko, E. (2015). Differences in judgments of creativity:How do academic domain, personality, and self-reported creativity influence novice judges' evaluations of creative productions? Journal of Intelligence, 3(3), 73-90.

Wilson, R. C., Guilford, J. P., & Christensen, P. R. (1953). The measurement of individual differences in originality. Psychological Bulletin, 50(5), 362-370.

Wolfe, E. W. (2004). Identifying rater effects using latent trait models. Psychology Science, 46(1), 35-51.

Wolfe, E. W., & McVay, A. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement:Issues and Practice, 31(3), 31-37.

Yang, Y. Y., Oosterhof, A., & Xia, Y. (2015). Reliability of scores on the summative performance assessments. The Journal of Educational Research, 108(6), 465-479.

Zhou, J., Wang, X. M., Song, L. J., & Wu, J. (2017). Is it new? Personal and contextual influences on perceptions of novelty and creativity. Journal of Applied Psychology, 102(2), 180-202.

Zhu, Y. X, Ritter, S. M., Müller, B. C. N., & Dijksterhuis, A. (2017). Creativity:Intuitive processing outperforms deliberative processing in creative idea selection. Journal of Experimental Social Psychology, 73, 180-188.