Week 3_RNA (The What of Multiliteracies)

TSLB3193
MULTILITERACIES

Cultural Literacy Skills
in Context and Technology

Week 3
Multimodality and Literacy

Rozanna Noraini Albakri

The ‘What’ of Multiliteracies:

Multimodality - written, oral, visual,
audio, tactile, gestural, spatial

Multimodality involves the
complex interweaving of word,
image, gesture and movement,
and sound, including speech.

These can be combined in
different ways and presented
through a range of media.

Bearne, E. And Wolstencroft, H. 2007

Multimodal Literacy

• Understanding the different ways of knowledge
representations and meaning-making.

• Focuses on the design of discourse by investigating
the contributions of specific semiotic resources (e.g.
language, gesture, images) co-deployed across
various modalities (e.g. visual, aural), as well as their
interaction and integration in constructing a
coherent multimodal text( such as advertisements,
posters, news report, websites, films).

Multimodal Text

• According to Bull and Anstey (2010) there
are five semiotic (sign, symbol and/or code)
systems that the multiliterate needs to have
knowledge of when exploring and
examining multimodal texts.

• Multimodal texts combine two or more of
these systems.

The understanding of the different ways of knowledge
representations and meaning-making can be achieved

through:

Visual Mode

Use of images and other characteristics
we can see:

– Colour
– Layout
– Style
– Size/perspective
– Foreground/background

Linguistic Mode

❖Word Choice (Vocabulary options)
❖Delivery (spoken, written)
❖Organization (phrases, sentences,

paragraphs, etc)
❖Coherence (how individual words

and ideas are developed and
related)

Audio (Aural) Mode

Focuses on sound and include
elements like :

- music
- sound effects/ambient
- noise/sounds
- silence
- tone of voice
- volume
- emphasis and accent

Gestural Mode

The way movement, such as body language can
make meaning, which includes:

facial expression hand gestures
body language
interactions
between people

Spatial Mode

Physical arrangement which includes:-

arrangement organization

proximity between people or objects



Dimensions of Multimodal Literacy

1st Dimension of Multimodal Literacy:
Media Literacy

2nd Dimension of Multimodal Literacy:
Multisemiotic Experience

1st Dimension Media Literacy
of Multimodal

Literacy

Media Literacy

• The first dimension is with respect to the
prevalence of multimodal texts, specifically
through multimedia texts afforded by the digital
media, hence stressing the need for a literacy to
produce and access information.

.

Media Literacy (cont.)

• Multimodal literacy acknowledges the
significance of all the semiotic resources and
modalities in meaning making..

• Semiotic resources are conferred the same
status as language and are just as effective in
semiosis.

Media Literacy (cont.)

• A multimodal text conveys meaning through a
combination of two or more modes

For example, a poster conveys meaning through
a combination of written language, still image,
and spatial design.

Media Literacy (cont.)

• Each mode has its own specific task and function
(Kress, 2010, p. 28) in the meaning making process,
and usually carries only a part of the message in a
multimodal text (Kress, 2010, p. 28).

For example in a picture book, the print and the image
both contribute to the overall telling of the story but do
so in different ways.

Media Literacy (cont.)

• Effective multimodal authors creatively integrate
modes in various configurations to coherently
convey the meaning required, ‘moving the emphasis
backwards and forwards between the various
modes') throughout the text (Cope and Kalantzis,
2009. p. 423.

Media Literacy (cont.)

“A ‘multimodal literate’ student must be
sensitized to the meaning potential and choices
afforded in the production of the text, rendering

an enhanced ability to make deliberate and
effective choices in the construction and
presentation of knowledge”.

O’Halloran, K.L. & Lim, F. V. (2011), ‘Dimensioner af Multimodal Literacy’, Viden om
Læsning 10, September 2011: 14–21, Copenhagen, Denmark: Nationalt Videncenter
for Laesning

Media Literacy (cont.)

• ‘Multimodal literate’ students will not only
become discerning consumers of multisemiotic
texts but they also will become competent
producers of multimodal texts themselves.

Media Literacy (cont.)

“As educators, we need to develop the knowledge
and pedagogy to teach multimodal literacy. We
cannot assume that just because our young are

growing up in a media-rich world, they will be able to
view multimodal representations critically and not be

naive consumers of media texts.”

Lim, F.V. & Tan, K.Y.S. (2018). Developing Multimodal Literacy Through Teaching the
Critical Viewing of Films in Singapore. Journal of Adolescent & Adult Literacy.

2nd Dimension Multisemiotic
of Multimodal Experience

Literacy

Multisemiotic Experience

• The recognition that the experience of teaching and
learning is intrinsically multisemiotic and multimodal.

• O’Toole (1994: 15) observes, “[w]e ‘read’ people in
everyday life: facial features and expression,
stance, gesture, typical actions and clothing”.

Multisemiotic Experience (cont.)

• All interactions are multimodal. Our communication is
more than what is said and heard but by what we
perceive through expressions, gazes, gestures and
movements.

• Norris (2004: 2) observes that “[a]ll movements, all
noises, and all material objects carry interactional
meanings as soon as they are perceived by a person”.

Multisemiotic Experience (Cont.)

Multimodal literacy “focuses on the design of
discourse by investigating the contributions of

specific semiotic resources (e.g. language,
gesture, images) co-deployed across various
modalities (e.g. visual, aural, somatic), as well as
their interaction and integration in constructing a

coherent text.”

Lim, F. V. , K.L. O’Halloran, S. Tan and M.K.L. E (2015: 917), ‘Teaching Visual Texts with Multimodal
Analysis Software’, Educational Technology Research and Development 63(6), 915–935.

Multisemiotic Experience (cont.)

• There is a need to understand how the lesson
experience is constructed through the teacher’s use of a
repertoire of semiotic resources as embodied in his/her
pedagogy.

• The combination of semiotic resources and modalities in
the implementation of the lesson can provide
understandings which may lead to more effective
teaching and learning in the classroom.

Multisemiotic Experience (Cont.)

“The multimodal approach takes into account
how linguistic and visual (and other) choices
fulfill the purposes of the text, the audience
and context, and how those choices work
together in the organisation and
development of information and ideas.”

Lim, F. V. & Tan, K.Y.S. (2017). Multimodal Translational Research: Teaching Visual Texts. IN
Seizov, O. & Wildfeuer, J. (EDS.), New Studies in Multimodality: Conceptual and
Methodological Elaborations (PP.175-200). London/New York: Bloomsbury.

Multimodal texts

• Multimodal is not synonymous with digital.
• It can be:-

Paper - books, comics, posters
Digital - slide presentations, e-books, blogs, etc
Live- a performance or an event
Transmedia-multiple delivery channels, through

combination of media platforms

Examples of multimodal texts

❑ a picture book, in which the textual and visual elements
are arranged on individual pages that contribute to an
overall set of bound pages

Examples of multimodal texts

❑ a webpage, in which elements such as sound effects, oral
language, written language, music and still or moving
images are combined

Examples of multimodal texts

❑ a live ballet performance, in which gesture, music, and
space are the main elements.

Examples of multimodal texts

❑ Picture books ❑ Music videos
❑ Non-fiction books ❑ Video Games
❑ Pop-up and lift the flap ❑ Signages
books (non linear) ❑ Comic books
❑ Web pages / Websites ❑Advertisements
❑ Blogs / Vlogs ❑Newspapers
❑ Advertisements ❑Magazines
❑ Movies / Films ❑Poetry

Why adopt New New
multimodal ways of ways of
approach reading writing

‘literacy teaching and learning needs to change
because the world is changing’

Cope and Kalantzis. 2000: 41

Why adopt ❑ The concept of
multimodal
approach multimodality is a useful
yardstick to measure and
evaluate the diversity ways
of meaning making

❑ Multimodal texts, which convey information by
means of various modes such as visual images,
written language, design elements and other semiotic
resources, are more complex than written texts.

❑ The future of Why adopt ❑ Supporting
multimodal
reading and writing are approach children with reading,
closely interwoven with analysing and using
the future of digital modes will enable them
technologies. to develop literacy skills
for today and the future.

❑ Children already know much about multimodal texts from their home

experiences. As teachers it is our responsibility to build on these
experiences and the children’s knowledge of multimodality in the
classroom. This means we need to recognise the relationships between
different modes: text and image, sound and gesture and use this in our
teaching.

Helping teachers support students’ facility
with multimodal texts

Give knowledge and Focus particular understandings
understanding about reading about the design of multimodal
and writing multimodal texts that texts which support the effective
are delivered in different ways classroom use.
(paper, live and digital electronic)

These include an understanding An understanding of a text’s purpose,
that texts perform a particular audience and method of communication
function over time or within a is key; not only on what is included in a
specific context, and they are text, but how different elements relate to
designed to achieve particular each other, and the effect they are
communicative purposes designed to achieve.

Knowledge of the five semiotic systems of
which a multimodal text can be composed

❑ Teachers and students
need to understand the
codes and conventions
of each of the five
semiotic systems in
order to make or convey
meaning through them.

Cont…

❑ Teachers and students should be taught a linguistic

grammar that enables them to select and use semiotic
systems effectively in a multimodal text.

For example, when composing a multi-modal text they
will need to make decisions about whether to show a
character’s emotions through sound, gesture, facial
expression or descriptive words, or some combination of
these.

The New London Group believe that
multimodality can bring benefit to

teaching and learning process. This
new pedagogy emphasizes on
“modes of representation much
broader than language alone”

(The New London Group, 1996, p.64)

The idea is that classroom setting can be
a good place “to develop

communication and awareness of
discourse practices that provides real
opportunities for students to express
their individual cultural experiences built

on their linguistic resources”

(The New London Group, 1996, p.69)

Multimodal ➢ In an explicit pedagogical approach,
Approach classroom interaction has particular
characteristics and it is recognised that
in the every literate practice in the classroom
Classroom builds students’ understandings about what

counts as literacy or literate practices.

➢ Communication and discourse practices (literacy practices),
shaped through text, image or a combination of resources could
engage learners actively, help them to effectively understand
and produce texts in the target language, enhance their
awareness, facilitate comprehension, and more importantly
motivate them.

Tutorial Task

In your respective groups:-
Collect samples of multimodal texts:

-Print
-Digital
-Live
-Transmedia

Discuss how the different modes (linguistic, visual, audio,
gesture, and spatial) are used in the multimodal texts to
convey meaning.

Thank You

Rozanna Noraini Albakri