Multimodality

Multimodality studies how meaning is made through the combination of semiotic modes—language, image, sound, gesture, layout—rather than language alone.

Definition

Multimodality is the study of communication and meaning-making that attends to the full range of semiotic modes—including writing, image, sound, gesture, and spatial design—and how they combine.

Scope

This topic covers the multimodal approach to communication and rhetoric, developed largely from social semiotics. It treats the concept of mode and its affordances, the orchestration of multiple modes in texts and performances, and the implications for composition and literacy. The relation of multimodality to visual rhetoric and to digital media is included.

Core questions

• What is a semiotic mode, and what can each mode do?
• How do modes combine and divide labor in a text?
• How does the choice of mode shape meaning and rhetorical effect?
• What does multimodality imply for writing and literacy?

Key concepts

• mode and medium
• affordance
• semiotic resource
• orchestration of modes
• design

Key theories

Social semiotics of mode

Kress and van Leeuwen treat each mode as a culturally shaped resource with distinct affordances, analyzing how meaning is distributed across and orchestrated among modes in any communicative act.

History

Multimodality grew from social semiotics and systemic functional linguistics, especially the work of Kress and van Leeuwen from the 1990s onward, as digital media made the combination of modes ubiquitous. Their 2001 Multimodal Discourse and Kress's 2010 Multimodality set out the theory, and handbook treatments consolidated it as an interdisciplinary field spanning rhetoric, education, and communication.

Debates

Are all modes equally analyzable?

Researchers debate whether the analytic apparatus developed for language and image transfers to modes such as sound, gesture, and movement, and whether modes are universal or culturally specific.

Key figures

• Gunther Kress
• Theo van Leeuwen
• Carey Jewitt

Seminal works

• kress2001
• kress2010

Frequently asked questions

What is an 'affordance' of a mode?

An affordance is what a mode makes it easy or hard to express. Writing unfolds in time and is good for sequence and logic, while images present things simultaneously and are good for spatial relations; each mode therefore lends itself to different meanings.