DOI: 10.3724/SP.J.1042.2016.00335

Advances in Psychological Science (心理科学进展) 2016/24:3 PP.335-350

Visual text or narration? Meta-analysis of the modality effect in multimedia learning

The modality effect in multimedia learning showed that learning outcomes were enhanced if words were presented in an audial format with pictures, rather than a visual format (see Mayer, 2009). However, research on the reverse modality effect indicated a totally opposite result (e.g. Crooks, Cheon, Inan, Ari, & Flores, 2012; Tabbers, Martens, & van Merriënboer, 2004). A meta-analysis based on 91 empirical studies was conducted to investigate the effect of text modality on both retention and transfer tests. After pooling data preliminarily, ninety-four independent effect sizes (8088 participants) were finally included in retention-related meta-analysis while 83 independent effect sizes (6664 participants) in transfer-related meta-analysis. The results suggested that participants who learned from narration outperformed those who learned from visual text both on retention test (dretention = 0.24) and transfer test (dtransfer = 0.25) with different effect sizes from Ginns's (2005). Further moderator analyses indicated that modality effect on learning outcomes was significantly moderated by the pace of presentation, dynamism of pictures and duration of learning materials. Specifically, the modality effect occurred mainly in conditions of system-paced presentation (dretention = 0.43, dtransfer = 0.44), dynamic pictures (dretention = 0.50, dtransfer = 0.59) and short learning materials (dretention = 0.38, dtransfer = 0.33). All of the results didn't reveal a reverse modality effect. The replicated strong modality effect suggested that performance on recall and comprehension tasks was better when words and pictures were presented in a dual modality, rather than a single one, which supported Mayer's Cognitive Theory of Multimedia Learning (CTML). Moreover, the pace of presentation, dynamism of pictures and duration of learning materials should be considered as vital boundary conditions of modality effect.

Key words:modaity effect,reverse modality effect,boundary condition,multimedia learning,meta-analysis

ReleaseDate:2016-12-19 16:07:58

刘颖. (2014). 感知学习风格对通道效应的影响研究(硕士学位论文). 河北大学, 保定.

罗淑娴. (2012). 通道效应在老年人多媒体学习中的有效性研究(硕士学位论文). 湖南师范大学, 长沙.

肖玉琴. (2010). 通道效应在交互性多媒体学习环境中的有效性研究(硕士学位论文). 湖南师范大学, 长沙.

邹琴. (2013). 内外部线索对多媒体学习效果的影响: 来自眼动的证据(硕士学位论文). 华中师范大学, 武汉.

Abel, R. R., & Kulhavy, R. W. (1986). Maps, mode of text presentation, and children's prose learning. American Educational Research Journal, 23(2), 263-274.

AbuSaada, A. H., Lee, L. P. L., & Fong, S. F. (2013). Effects of modality principle in tutorial video streaming. International Journal of Academic Research in Business and Social Sciences, 3(5), 456-466.

Adegoke, B. A. (2010). Integrating animations, narratives and textual information for improving physics learning. Electronic Journal of Research in Educational Psychology, 8(2), 725-748.

Adegoke, B. A. (2011). Effect of multimedia instruction on senior secondary school students' achievement in Physics. European Journal of Educational Studies, 3(3), 537-550.

Al-Abbasi, D. (2012). The effects of modality and multimedia comprehension on the performance of students with varied multimedia comprehension abilities when exposed to high complexity, self-paced multimedia instructional materials. Journal of Educational Multimedia and Hypermedia, 21(3), 215-239.

Aldalalah, O. M., & Fong, S. F. (2010). Effects of modality and redundancy principles on the learning and attitude of a computer-based music theory lesson among Jordanian primary pupils. International Education Studies, 3(3), 52-64.

Alty, J. L., Al-Sharrah, A., & Beacham, N. (2006). When humans form media and media form humans: An experimental study examining the effects different digital media have on the learning outcomes of students who have different learning styles. Interacting with Computers, 18(5), 891-909.

Atkinson, R. K. (2002). Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology, 94(2), 416-427.

Austin, K. A. (2009). Multimedia learning: Cognitive individual differences and display design techniques predict transfer learning with multimedia learning modules. Computers & Education, 53(4), 1339-1354.

Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In G. H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (pp. 47-89). New York: Academic Press.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley.

Brünken, R., & Leutner, D. (2001). Aufmerksamkeitsverteilung oder Aufmerksamkeitsfokussierung? Empirische Ergebnisse zur "Split-Attention-Hypothese" beim Lernen mit Multimedia. (Attention Splitting or Attention Focussing? Empirical Results Concerning the "Split-Attention Hypothesis" in Learning with Multimedia). Unterrichtswissenschaft, 29(4), 357-366.

Brünken, R., Plass, J. L., & Leutner, D. (2004). Assessment of cognitive load in multimedia learning with dual-task methodology: Auditory load and modality effects. Instructional Science, 32(1-2), 115-132.

Cheon, J., Crooks, S. M., & Chung, S. (2014). Does segmenting principle counteract modality principle in instructional animation? British Journal of Educational Technology, 45(1), 56-64.

Cheon, J., Crooks, S. M., Inan, F. A., Flores, R., & Ari, F. (2011). Exploring the instructional conditions for a reverse modality effect in multimedia instruction. Journal of Educational Multimedia and Hypermedia, 20(2), 117-133.

Chuang, H.-Y., & Ku, H.-Y. (2011). The effect of computer- based multimedia instruction with Chinese character recognition. Educational Media International, 48(1), 27-41.

Chung, S., Cheon, J., & Lee, K.-W. (2015). Emotion and multimedia learning: An investigation of the effects of valence and arousal on different modalities in an instructional animation. Instructional Science, 43(5), 545-559.

Clark, R. C., & Mayer, R. E. (2011). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning (3rd ed.). Hoboken, NJ: John Wiley & Sons.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey: Lawrence Erlbaum Associates.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159.

Cooper, H. M. (1989). Integrating research: A guide for literature reviews (2nd ed.). Newbury Park, CA: SAGE.

Craig, S. D., Gholson, B., & Driscoll, D. M. (2002). Animated pedagogical agents in multimedia educational environments: Effects of agent properties, picture features and redundancy. Journal of Educational Psychology, 94(2), 428-434.

Crooks, S. M., Cheon, J., Inan, F., Ari, F., & Flores, R. (2012). Modality and cueing in multimedia learning: Examining cognitive and perceptual explanations for the modality effect. Computers in Human Behavior, 28(3), 1063-1071.

Crooks, S. M., Inan, F., Cheon, J., Ari, F., & Flores, R. (2009). When text should be seen and not heard: An instance of the reverse modality effect in multimedia learning. In Proceeding of the annual convention of the association for educational communications and technology (Vol. 1, pp. 91-93). Louisville, KY.

de Oliveira Neto, J. D., Huang, W. D., & de Azevedo Melli, N. C. (2015). Online learning: Audio or text? Educational Technology Research and Development, 63(4), 555-573.

Driver, J., & Spence, C. (2000). Multisensory perception: Beyond modularity and convergence. Current Biology, 10(20), R731-R735.

Dunsworth, Q., & Atkinson, R. K. (2007). Fostering multimedia learning of science: Exploring the role of an animated agent's image. Computers & Education, 49(3), 677-690.

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7109), 629-634.

Erlandson, B. E., Nelson, B. C., & Savenye, W. C. (2010). Collaboration modality, cognitive load, and science inquiry learning in virtual inquiry environments. Educational Technology Research and Development, 58(6), 693-710.

Fiorella, L., Vogel-Walcutt, J. J., & Schatz, S. (2012). Applying the modality principle to real-time feedback and the acquisition of higher-order cognitive skills. Educational Technology Research and Development, 60(2), 223-238.

Flores, R., Coward, F., & Crooks, S. M. (2010). Examining the influence of gender on the modality effect. Journal of Educational Technology Systems, 39(1), 87-103.

Furnham, A., de Siena, S., & Gunter, B. (2002). Children's and adults' recall of children's news stories in both print and audio-visual presentation modalities. Applied Cognitive Psychology, 16(2), 191-210.

Gerjets, P., Scheiter, K., Opfermann, M., Hesse, F. W., & Eysink, T. H. S. (2009). Learning with hypermedia: The influence of representational formats and different levels of learner control on performance and learning behavior. Computers in Human Behavior, 25(2), 360-370.

Ginns, P. (2005). Meta-analysis of the modality effect. Learning and Instruction, 15(4), 313-331.

Grimley, M. (2007). Learning from multimedia materials: The relative impact of individual differences. Educational Psychology, 27(4), 465-485.

Gyselinck, V., Jamet, E., & Dubois, V. (2008). The role of working memory components in multimedia comprehension. Applied Cognitive Psychology, 22(3), 353-374.

Harskamp, E. G., Mayer, R. E., & Suhre, C. (2007). Does the modality principle for multimedia learning apply to science classrooms? Learning and Instruction, 17(5), 465-477.

Hassanabadi, H., Robatjazi, E. S., & Savoji, A. P. (2011). Cognitive consequences of segmentation and modality methods in learning from instructional animations. Procedia-Social and Behavioral Sciences, 30, 1481-1487.

Higgins, J. P. T., & Green, S. (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]: The Cochrane Collaboration. Retrieved August 22, 2015, from

Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327(7414), 557-560.

Höffler, T. N., & Leutner, D. (2007). Instructional animation versus static pictures: A meta-analysis. Learning and Instruction, 17(6), 722-738.

Hoogerheide, V., Loyens, S. M. M., & van Gog, T. (2014). Comparing the effects of worked examples and modeling examples on learning. Computers in Human Behavior, 41, 80-91.

Inan, F. A., Crooks, S. M., Cheon, J., Ari, F., Flores, R., Kurucay, M., & Paniukov, D. (2015). The reverse modality effect: Examining student learning from interactive computer- based instruction. British Journal of Educational Technology, 46(1), 123-130.

Jeung, H. J., Chandler, P., & Sweller, J. (1997). The role of visual indicators in dual sensory mode instruction. Educational Psychology, 17(3), 329-345.

Kalyuga, S. (2012). Instructional benefits of spoken words: A review of cognitive load factors. Educational Research Review, 7(2), 145-159.

Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13(4), 351-371.

Kalyuga, S., Chandler, P., & Sweller, J. (2000). Incorporating learner experience into the design of multimedia instruction. Journal of Educational Psychology, 92(1), 126-136.

Koroghlanian, C., & Klein, J. D. (2004). The effect of audio and animation in multimedia instruction. Journal of Educational Multimedia and Hypermedia, 13(1), 23-46.

Kühl, T., Eitel, A., Damnik, G., & Körndle, H. (2014). The impact of disfluency, pacing, and students' need for cognition on learning with multimedia. Computers in Human Behavior, 35, 189-198.

Kühl, T., Scheiter, K., Gerjets, P., & Edelmann, J. (2011). The influence of text modality on learning with static and dynamic visualizations. Computers in Human Behavior, 27(1), 29-35.

Leahy, W., Chandler, P., & Sweller, J. (2003). When auditory presentations should and should not be a component of multimedia instruction. Applied Cognitive Psychology, 17(4), 401-418.

Leahy, W., & Sweller, J. (2011). Cognitive load theory, modality of presentation and the transient information effect. Applied Cognitive Psychology, 25(6), 943-951.

Leopold, C., & Mayer, R. E. (2015). An imagination effect in learning from scientific text. Journal of Educational Psychology, 107(1), 47-63.

Levin, J. R., & Divine-Hawkins, P. (1974). Visual imagery as a prose-learning process. Journal of Literacy Research, 6(1), 23-30.

Lindow, S., Fuchs, H. M., Fürstenberg, A., Kleber, J., Schweppe, J., & Rummer, R. (2011). On the robustness of the modality effect: Attempting to replicate a basic finding. Zeitschrift für Pädagogische Psychologie, 25(4), 231-243.

Main, R. E., & Griffiths, B. (1977). Evaluation of audio and pictorial instructional supplements. AV Communication Review, 25(2), 167-179.

Mann, B. L. (1997). Evaluation of presentation modalities in a hypermedia system. Computers & Education, 28(2), 133-143.

Mann, B. L., Newhouse, P., Pagram, J., Campbell, A., & Schulz, H. (2002). A comparison of temporal speech and text cueing in educational multimedia. Journal of Computer Assisted Learning, 18(3), 296-308.

Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.

Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167-171.

Mayer, R. E., Dow, G. T., & Mayer, S. (2003). Multimedia learning in an interactive self-explaining environment: What works in the design of agent-based microworlds? Journal of Educational Psychology, 95(4), 806-813.

Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology, 90(2), 312-320.

Mayer, R. E., & Moreno, R. (2002). Animation as an aid to multimedia learning. Educational Psychology Review, 14(1), 87-99.

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43-52.

Mayrath, M. C., Nihalani, P. K., & Robinson, D. H. (2011). Varying tutorial modality and interface restriction to maximize transfer in a complex simulation environment. Journal of Educational Psychology, 103(2), 257-268.

McNeill, A. L. (2004). The effects of training, modality, and redundancy on the development of a historical inquiry strategy in a multimedia learning environment (Unpublished master's thesis). Virginia Polytechnic Institute and State University.

Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91(2), 358-368.

Moreno, R., & Mayer, R. E. (2002). Learning science in virtual reality multimedia environments: Role of methods and media. Journal of Educational Psychology, 94(3), 598-610.

Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177-213.

Mousavi, S. Y., Low, R., & Sweller, J. (1995). Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology, 87(2), 319-334.

Nugent, G. C. (1982). Pictures, audio, and print: Symbolic representation and effect on learning. Educational Communication and Technology Journal, 30(3), 163-174.

Park, B., Flowerday, T., & Brünken, R. (2015). Cognitive and affective effects of seductive details in multimedia learning. Computers in Human Behavior, 44, 267-278.

Park, B., Moreno, R., Seufert, T., & Brünken, R. (2011). Does cognitive load moderate the seductive details effect? A multimedia study. Computers in Human Behavior, 27(1), 5-10.

Rias, R. M., Zaman, H. B., & Norhana. (2011). Modality effects in multimedia learning: A case study. Paper presented at the 2011 3rd International Congress on Engineering Education (ICEED). Kuala Lumpur, Malaysia.

Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Hoboken, NJ: John Wiley & Sons.

Rummer, R., Schweppe, J., Fürstenberg, A., Scheiter, K., & Zindler, A. (2011). The perceptual basis of the modality effect in multimedia learning. Journal of Experimental Psychology: Applied, 17(2), 159-173.

Rummer, R., Schweppe, J., Fürstenberg, A., Seufert, T., & Brünken, R. (2010). Working memory interference during processing texts and pictures: Implications for the explanation of the modality effect. Applied Cognitive Psychology, 24(2), 164-176.

Sabet, M. K., & Shalmani, H. B. (2010). Visual and spoken texts in MCALL courseware: The effects of text modalities on the vocabulary retention of EFL learners. English Language Teaching, 3(2), 30-36.

Sanchez, E., & Garcia-Rodicio, H. (2008). The use of modality in the design of verbal aids in computer-based learning environments. Interacting with Computers, 20(6), 545-561.

Savoji, A. P., Hassanabadi, H., & Fasihipour, Z. (2011). The modality effect in learner-paced multimedia learning. Procedia-Social and Behavioral Sciences, 30, 1488-1493.

Schüler, A., Scheiter, K., & Gerjets, P. (2010). Does spatial verbal information interfere with picture processing in working memory? The role of the visuo-spatial sketchpad in multimedia learning. Paper presented at the Proceedings of the 32nd Annual Meeting of the Cognitive Science Society.

Schüler, A., Scheiter, K., & Gerjets, P. (2013). Is spoken text always better? Investigating the modality and redundancy effect with longer text presentation. Computers in Human Behavior, 29(4), 1590-1601.

Schüler, A., Scheiter, K., Gerjets, P., & Rummer, R. (2008). The modality effect in multimedia learning: Theoretical and empirical limitations. Paper presented at the Eighth International Conference for the Learning Sciences-ICLS 2008, Utrecht, The Netherlands.

Schüler, A., Scheiter, K., Rummer, R., & Gerjets, P. (2012). Explaining the modality effect in multimedia learning: Is it due to a lack of temporal contiguity with written text and pictures? Learning and Instruction, 22(2), 92-102.

Scheiter, K., Schüler, A., Gerjets, P., Huk, T., & Hesse, F. W. (2014). Extending multimedia research: How do prerequisite knowledge and reading comprehension affect learning from text and pictures. Computers in Human Behavior, 31, 73-84.

Schmidt-Weigand, F. (2005). Dynamic visualizations in multimedia learning: The influence of verbal explanations on visual attention, cognitive load and learning outcome (Unpublished doctorial dissertation). Universitätsbibliothek Gieβen.

Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010a). A closer look at split visual attention in system- and self-paced instruction in multimedia learning. Learning and Instruction, 20(2), 100-110.

Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010b). Explaining the modality and contiguity effects: New insights from investigating students' viewing behaviour. Applied Cognitive Psychology, 24(2), 226-237.

Schnotz, W., Mengelkamp, C., Baadte, C., & Hauck, G. (2014). Focus of attention and choice of text modality in multimedia learning. European Journal of Psychology of Education, 29(3), 483-501.

Schoor, C., Bannert, M., & Jahn, V. (2011). Methodological constraints for detecting the modality effect. Electronic Journal of Research in Educational Psychology, 9(3), 1183-1196.

Schweppe, J., & Rummer, R. (2014). Attention, working memory, and long-term memory in multimedia learning: An integrated perspective based on process models of working memory. Educational Psychology Review, 26(2), 285-306.

Segers, E., Verhoeven, L., & Hulstijn-Hendrikse, N. (2008). Cognitive processes in children's multimedia text learning. Applied Cognitive Psychology, 22(3), 375-387.

Seufert, T., Schütze, M., & Brünken, R. (2009). Memory characteristics and modality in multimedia learning: An aptitude-treatment-interaction study. Learning and Instruction, 19(1), 28-42.

She, H.-C., & Chen, Y.-Z. (2009). The impact of multimedia effect on science learning: Evidence from eye movements. Computers & Education, 53(4), 1297-1307.

Smith, A., & Ayres, P. (2014). Investigating the modality and redundancy effects for learners with persistent pain. Educational Psychology Review, 1-24.

Stein, B. E., London, N., Wilkinson, L. K., & Price, D. D. (1996). Enhancement of perceived visual intensity by auditory stimuli: A psychophysical analysis. Journal of Cognitive Neuroscience, 8(6), 497-506.

Stiller, K. D., Freitag, A., Zinnbauer, P., & Freitag, C. (2009). How pacing of multimedia instructions can influence modality effects: A case of superiority of visual texts. Australasian Journal of Educational Technology, 25(2), 184-203.

Sullins, J., Craig, S. D., & Graesser, A. C. (2010). The influence of modality on deep-reasoning questions. International Journal of Learning Technology, 5(4), 378-387.

Sweller, J., van Merriënboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251-296.

Sydorenko, T. (2010). Modality of input and vocabulary acquisition. Language Learning & Technology, 14(2), 50-73.

Tabbers, H. K. (2002). The modality of text in multimedia instructions: Refining the design guidelines (Unpublished doctorial dissertation). Open University of the Netherlands, Heerlen, NL.

Tabbers, H. K. (2006). Where did the modality effect go? A failure to replicate one of Mayer's multimedia learning effects and why this still might make some sense. Paper presented at the Proceedings of the EARLI SIG 2 meeting “Learning by interpreting and constructing educational representations”, Nottingham, UK.

Tabbers, H. K., & De Koeijer, B. (2010). Learner control in animated multimedia instructions. Instructional Science, 38(5), 441-453.

Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2000). Multimedia instructions and cognitive load theory: Split-attention and modality effects. Paper presented at the National Convention of the Association for Educational Communications and Technology, Long Beach, CA.

Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2001). The modality effect in multimedia instructions. Paper presented at the Proceedings of the 23rd annual conference of the Cognitive Science Society, Mahwah, NJ: Lawrence Erlbaum Associates.

Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2004). Multimedia instructions and cognitive load theory: Effects of modality and cueing. British Journal of Educational Psychology, 74(1), 71-81.

Tabbers, H. K., & van der Spoel, W. (2011). Where did the modality principle in multimedia learning go? A double replication failure that questions both theory and practical use. Zeitschrift für Pädagogische Psychologie, 25(4), 221-230.

Tata, M. S., Prime, D. J., McDonald, J. J., & Ward, L. M. (2001). Transient spatial attention modulates distinct components of the auditory ERP. Neuroreport, 12(17), 3679-3682.

Tindall-Ford, S., Chandler, P., & Sweller, J. (1997). When two sensory modes are better than one. Journal of Experimental Psychology: Applied, 3(4), 257-287.

Tindall-Ford, S., & Sweller, J. (2006). Altering the modality of instructions to facilitate imagination: Interactions between the modality and imagination effects. Instructional Science, 34(4), 343-365.

Tversky, B., Morrison, J. B., & Betrancourt, M. (2002). Animation: Can it facilitate? International Journal of Human-Computer Studies, 57(4), 247-262.

van den Broek, G., Segers, E., & Verhoeven, L. (2014). Effects of text modality in multimedia presentations on written and oral performance. Journal of Computer Assisted Learning, 30(5), 438-449.

van Gerven, P. W. M., Paas, F., van Merriënboer, J. J. G., & Schmidt, H. G. (2006). Modality and variability as factors in training the elderly. Applied Cognitive Psychology, 20(3), 311-320.

Witteman, M. J., & Segers, E. (2010). The modality effect tested in children in a user-paced multimedia environment. Journal of Computer Assisted Learning, 26(2), 132-142.

Wong, A., Leahy, W., Marcus, N., & Sweller, J. (2012). Cognitive load theory, the transient information effect and e-learning. Learning and Instruction, 22(6), 449-457.

Wouters, P., Paas, F., & van Merriënboer, J. J. G. (2009). Observational learning from animated models: Effects of modality and reflection on transfer. Contemporary Educational Psychology, 34(1), 1-8.