1 Multimodality Dr. Charlotte van Hooijdonk LCC Meeting, 10 september 2008.

1

Multimodality

Dr. Charlotte van Hooijdonk

LCC Meeting, 10 september 2008

2

Program

Multimodality

Research done in my thesis

Multimodality research within LCC context

3

What is multimodality?

A multimodal information presentation can be classified on the basis of three criteria or perspectives:

Delivery medium: devices used to deliver the information

Presentation mode: format of the message or the sign system used

Sensory mode: the human senses employed to process information

4

Why is multimodality important?

Presenting information using multiple modalities is more effective than presenting information using a single modality (e.g., Mayer, 2001; Oviatt, 1999).

Moreover, recent developments in computer technology have led to new possibilities of presenting information and to a renewed interest in the effects of different presentation modes.

5

Thesis: research questions

Research questions

When and how do people present information in a multimodal way?

Which presentation modes are most effective for learning and executing procedural instructions?

6

Thesis: research questions

Research questions

When and how do people present information in a multimodal way?

Which presentation modes are most effective for learning and executing procedural instructions?

7

Answer modalities

Monitor and keyboard should be in line with the user. Adjust the height of your chair in such a way that when you sit, the lower arms are level with the keyboard, without you pulling up your shoulders. The latter can be achieved easily by ensuring that the chair's armrests just reach the elbows when you sit with your shoulders in a relaxed position. Adjust the height of the chair in such a way that the armrests are level with or slightly above the top of the desk. Use the mouse and the keyboard without stretching your arms and/or bending your wrist.(www.unimaas.nl)

How should my workspace be organized to prevent RSI?

8 http://www.geckil.com/~harvest/rsi/

How should my workspace be organized to prevent RSI?

Answer modalities

9

Answer lengthWhat does RSI stand for?

RSI stands for Repetitive Strain Injury.

RSI stands for Repetitive Strain Injury. This

disorder involves damage to muscles, tendons and

nerves caused by overuse or misuse, and affects

the hands,wrists, elbows, arms, shoulders, back,

or neck.

10

Research design

Participants 111 Students from Tilburg University participated for course credits (46

male and 65 female).

Task For 8 medical questions, provide two answers (one brief, one extended)

-as you would like to have them- using a PowerPoint template with whatever (combination of) media you feel is best for a given question.

Coding system In total 1775 answers were collected Each answer was coded on the presence of visual media (and on the

function of visual media.

11

Results: functions of visual media

Decorative visuals did not add any information to the answer.

12

Representational visuals represented parts of the textual answer. Their presence clarified the text.

Results: functions of visual media

13

Informative visuals added information to the answer.

Results: functions of visual media

14

Results: answer length

15

Results: question type

16

Thesis: research questions

Research questions

When and how do people present information in a multimodal way?

Which presentation modes are most effective for learning and executing procedural instructions?

17

Research design

We conducted an experiment studying a specific kind of procedural instructions, namely exercises for the prevention of RSI.

We varied the information modality (text versus static visual versus dynamic visual) and the difficulty degree of the exercises (easy versus difficult) into account.

Participants had to learn RSI exercises and were asked to execute them.

18

Instruction in a text

Place your right hand in front of your body

with the elbow straight. Place your left

hand on the palm of your right hand. Gently

push the your right hand towards you until

you feel a mild stretch.

19

Instruction in a static visual

20

Instruction in a dynamic visual

filmclip.avi

21

Results

None of the presentation formats was superior on all dimensions of interest, each had some disadvantages.

Certain RSI exercises seemed to be inherently easier to represent than others, which varied across different presentation formats.

22

Multimodality research within LCC context

What?Effectiveness of (combinations of) different

presentation formats in different genresProcessing of (combinations of) different

presentation formats Adapting presentation formats to different

user groups / needs How?

Quantitative Qualitative