A Study on the Behavior under Multitasking Conditions in a ...After introducing into the topic labeled as "human multitasking", embedded in situation of routine life, ... task (a test

A S T U D Y O N T H E B E H AV I O R U N D E RM U LT I TA S K I N G C O N D I T I O N S I N AD Y N A M I C TA S K S C E N A R I O I N T H E

C O N T E X T O FH U M A N - M A C H I N E - I N T E R A C T I O N

vorgelegt vonDipl.-Psych. Jürgen Kiefer

von der Fakultät V - Verkehrs- und Maschinensystemeder Technischen Universität Berlin

GRK PROMETEIzur Erlangung des akademischen Grades

Dr. phil.genehmigte Dissertation

Promotionsausschuss:Berichterstatter: Prof. Dr. Ing. Leon UrbasBerichterstatter: Prof. Dr. Phil. Manfred ThüringTag der wissenschaftlichen Aussprache: 05.10.2009

Berlin 2010

D 83

This document was created using the tools:

• MikTex 2.7 (Freeware)

• WinEdt v5.60 (Shareware)

• Ghostscript, Ghostview, GSview

Special thank to Tony Jameson for providing the style-files:

• sfb378.sty

• sfb378-apacite.sty

• sfb378-apacite.bst

Dipl.-Psych. Jürgen Kiefer: A Study on the Behavior under MultitaskingConditions in a Dynamic Task Scenario in the Context of Human-Machine-Interaction , vorgelegt von , © Tag der wissenschaftlichen Aussprache:05.10.2009

pour JULES

A B S T R A C T

The work presented here is focusing on the behavior of participantsin situations of daily life, in which several demands apparently at thesame time need to be dealt with. After introducing into the topiclabeled as "human multitasking", embedded in situation of routine life,reasons for choosing the topic and approaching it are provided (chapter1). In chapter 2, an overview of the history in human multitasking andtask switching is given. First approaches starting at the beginning oflast century up to recent approaches and ideas are presented and theirimpact for psychological science is displayed. Entering chapter 3, theempirical work is presented: study 1 portrays a driving simulation in adriving simulator, in which a primary task (driving) plus a concurrenttask (a test of attention which was adapted for the in-car scenario) areapplied. The secondary task featured three different levels. The firststudy gives an impression about how people manage such scenarios.Please note that the main task (driving in the simulator) was considereda dynamic task. The second study mimics study one and is a replicationwith the additional aspect of training and its impact on performance.With the help of study two, strategies how to handle the scenarioare derived and a heuristic is described which is applied by a bunchof people. Setting of the study was taken from study one. As taskconfiguration was expected to strongly moderate the behavior of theparticipants during the task scenario, main task (driving) and secondarytask (test of attention) were varied: applying the lane change task (LCT),a computer-simulation for driving behavior, it was possible to betteranalyze the lane derivation during driving (which was taken as ameasure of performance for the main task). As for the secondary task,the variations of the test of attention used in the previous two studieswere systematically extended. Results of study 3 were used to illustratethe impact of task configuration for the scenario. In the last study,time pressure as additionally component was applied and its impactwas measured on task performance both for main task (driving) andsecondary task (test of attention). The last chapter (chapter 4) resumesthe results and provides design recommendations. The work closeswith a conclusion and mentions aspects that were not considered dueto time constraints.

v

ich hatte nur diese zeit (rainer werner fassbinder)

A C K N O W L E D G M E N T S

thank you:

• Dr. Dirk Schulze-Kissing ("Ulysses")

• Joachim ("Jolle") Wutke

• Prof. Leon Urbas

• Prof. Manfred Thüring

• Prof. Hartmut Wandke

• Prof. Anthony Jameson

• Prof. Werner H. Tack

• Dipl.-Ing. Marcus Heinath

• Dipl.-Psych. Robert Lischke

• Daniel Doering

• Dipl.-Psych. Tobias Katus

• Dipl.-Psych., Dipl.-Ing. Holger Schultheis

• Dipl.-Psych. Cordula Krinner ("Miss LateX")

• Dipl.-Psych. Nicola Fricke

• Dipl.-Psych. Diana Woelki (apt pupil)

• Dipl.-Psych. Necla Soyak

• (cand.) Dipl.-Psych. Bob Kaldasch

• (cand.?) Dipl.-Psych. Michael Schulz

• Antti Oulasvirta (PhD)

• Adam Chuderski (PhD)

• Jing Qian (PhD)

• Joscha Bach (PhD)

• Inessa Seifert (PhD)

• Dipl.-Psych. Nadya Dich

• Michal B. Paradowski (PhD)

• Stefan Mattes (Daimler)

• Dirk Weishaar (pour Messiaen)

• Karin Scherinsky-Pingel

• Birgit Trogisch

• Elke Fadel

• Mario Lasch

• ... and a few others

vii

C O N T E N T S

i introduction 1

1 introductory note 5

1.1 Preface 5

1.2 Multitasking and human-machine interaction 7

1.3 Technology can do multitasking 8

1.4 An excerpt of recent studies 9

1.5 An ability to multitask? 10

ii theory 11

2 theoretical background 15

2.1 A short historical survey 16

2.1.1 The 1920‘s 16

2.1.2 The 1930‘s 17

2.1.3 The 1940‘s 17

2.1.4 The 1950‘s 17

2.1.5 The 1960‘s 18

2.1.6 The 1970‘s 18

2.1.7 The 1980‘s 19

2.1.8 The 1990‘s 20

2.1.9 2001: The cognitive bottleneck 20

2.1.10 2005: A general multitasking component 21

2.2 Multitasking or task interruption? 23

2.2.1 Characteristics of task interruption 23

2.2.2 The task switching question 25

2.2.3 A grammar for task scheduling 26

2.3 Single or multiple resources? 26

2.4 Summary and criticism 28

2.4.1 Need for continuous tasks in multitasking stud-ies 29

2.4.2 Training and task repetition 29

2.4.3 Cognitive heuristics - human multitasking 29

iii studies 31

3 empirical studies 35

3.1 Study I: Identification of multitasking heuristics 35

3.1.1 Method in study I 36

3.1.2 Hypothesis: D2-Drive under multitasking 39

3.1.3 Results of study I 40

3.1.4 Discussion of study I 41

3.2 Study II: Practice - multitasking heuristics 42

3.2.1 Method in study II 42

3.2.2 Hypotheses for study II 44

3.2.3 Results of study II 44

3.2.4 Discussion of study II 46

3.3 Study III: The role of task configuration 47

3.3.1 Method in study III 47

3.3.2 Hypotheses for study III 50

3.3.3 Results of study III 51

3.3.4 Discussion of study III 52

3.4 Study IV: Amplification via time pressure 52

ix

x contents

3.4.1 Method in study IV 53

3.4.2 Hypotheses for IV 55

3.4.3 Results of study IV 56

3.4.4 Discussion of study IV 57

iv discussion 59

4 critical discussion 63

4.1 Scope and findings 63

4.2 Cognitive modeling 64

4.3 Design recommendations 65

4.4 Criticism and outlook 67

4.4.1 The role of memory in human multitasking 68

4.4.2 Domain independence 69

4.4.3 Need for a computational model of human multi-tasking 69

4.5 fMRI studies on multitasking 70

4.6 Popular stereotypes about multitasking 71

4.6.1 Multitasking and happiness 71

bibliography 73

a appendix 81

a.1 Appendix: Structured interview 81

L I S T O F F I G U R E S

Figure 1 A sketch on MT 6

Figure 2 Human-machine-interaction in daily life 7

Figure 3 Advertisement, Berlin (2007) 8

Figure 4 Robert Rauschenberg: First Landing Jump 10

Figure 5 A cartoon on multitasking 15

Figure 6 Jersild (1927): task switching paradigm 16

Figure 7 Model of Rasmussen (1983) 19

Figure 8 Cognitive bottleneck (Pashler, 1993) 21

Figure 9 Multitasking models - Salvucci (2005) 22

Figure 10 Interruption scenario 25

Figure 11 A grammar for task scheduling 27

Figure 12 Wickens‘ Model of multiple resources 28

Figure 13 Study I: Scenario 36

Figure 14 Study I: D2 test of attention 37

Figure 15 Study I-IV: D2-Drive 38

Figure 16 Study I: performance D2-Drive 40

Figure 17 Study I-IV: the merge heuristic 41

Figure 18 Study II: performance D2-Drive 45

Figure 19 Study II: the impact of training 46

Figure 20 Study III: the lane change task 48

Figure 21 Study III: analyzing LCT 49

Figure 22 Study III: performance D2-Drive 51

Figure 23 Study IV: scenario 53

Figure 24 Study IV: performance D2-Drive 54

Figure 25 Study IV: the impact of time pressure 56

Figure 26 Study IV: performance D2-Drive 57

Figure 27 Study IV: time pressure and D2-Drive 57

Figure 28 Study IV: eye movements 58

Figure 29 Overview of cognitive modeling 64

Figure 30 Pattern processing in D2-Drive 65

Figure 31 Modeling of D2-Drive 66

Figure 32 Pandoras box (Source: www) 67

Figure 33 Modification of LCT (Soyak, 2008) 68

Figure 34 General executive for multitasking 70

L I S T O F TA B L E S

Table 1 Controlled vs. automatic processing 18

xi

xii List of Tables

Table 2 Study II: Amount of attention 47

A C R O N Y M S

UCD User-centered design

LCT Lane change task

IRG Information-requirement grammar

AOI Areas of interest

Part I

I N T R O D U C T I O N

3

1I N T R O D U C T O RY N O T E

1.1 preface

efficiency matters! In modern western society (and not onlythere), time is money, and the less time required to do a job or task themore efficient your work is considered to be. Even before the wordmultitasking itself was applied, psychological approaches towards thephenomenon of how to handle the demand of multiple tasks werereported. The intellectual debate goes back even to the ancient Greektimes. "To do two things at

once - is to do neither."

to do two things at once is to do neither. With these words,roman philosopher Publilius Syrus 1 describes a phenomenon whichthousand of years later released a core discussion in psychology lastingalmost a hundred years, and a plethora of studies investigate whetherpeople in fact turn out to be able to perform several tasks concurrentlyor not. Multitasking madness, some scientists say, leads to a waste oftime. The myth of human multitasking not only remains but even moregains popularity in the era of mobile computing and human computerinteraction. Is there in fact an illusion of concurrency? This questiondoes not define the main issue of this work albeit it plays a (minor) role.Moreover, the author aims to show how people handle the demandsof multiple tasks at the same time, put in a real-life situation whichmight occur day by day. This work starts with the word "efficiency",meaning the degree of target achievement in relation to necessary costs,be them mental, physical, or financial. It is efficient to save time, butdo we really save time via multitasking?

procrastination is a type of behavior which is characterized by defer-ment of actions or tasks to a later time. Psychologists often cite procrastina-tion as a mechanism for coping with the anxiety associated with starting orcompleting any task or decision. (taken from: http://en.wikipedia.org/wiki/Procrastination.

efficiency matters! According to Steel [2007], procrastination isclosely related to perfectionism and workaholism. Ego syntonic perfec-tionists tend to be less likely to procrastinate than non-perfectionists.This is remarkable: employers more and more concentrate on efficiencyand most of them assert the more we can do at the same time thebetter our job. On the opposite, researchers in the field of psychologyfind in their studies that multitasking seems to be counterproductiveor inefficient. Fast switching between tasks leads not only to lowerperformance in each task individually but also takes more time, as, forinstance, Rogers and Monsell [1995] report.A variety of empirical investigations on multitasking do already existand one might ask: "Why more studies?" or "What is the benefit of

1 Publilius Syrus was a native of Syria and a Latin writer of maxims in the 1st century BC.The legacy of his work is a collection of sentences (Sententiae), a series of moral maximsin iambic and trochaic verse.

5

http://en.wikipedia.org/wiki/Procrastination

http://en.wikipedia.org/wiki/Procrastination

6 introductory note

Figure 1.: To do two things at once is to do neither?(Source: unknown)

this investigation?". To reply to these objections, let me analyze com-mon problems of former studies on task switching, dual-task - andmultitasking scenarios:

1. Most studies in the context of psychology are not applied stud-ies. Systematic control as a precondition for proper researchprevents from a direct connection to real life. Only recently (andwith the help of the vast development in computer technology),more and more researchers dare to investigate applied studies inthe field.

2. Most studies lack task repetition and systematic analysis. Inorder to derive how humans perform several tasks which arepresented concurrently, it is necessary to provide a large numberof task execution. Otherwise, the conclusions drawn from theexperiments are based solely on a small number of observations.

In his studies, Saluvicci [2005] uses cell phone dialing as a secondarytask in a driving situation. Although the studies within his paradigmprovide a deep inside into human multitasking in real life, the sec-ondary task in his scenarios (dialing) misses issues which are includedin the studies reported within my work. To properly investigate andanalyze a multitasking scenario including two tasks, I claim that asecondary task should be

1. fully controllable

1.2 multitasking and human-machine interaction 7

2. context-independent

3. interruptable

4. observable

Finding a balance between proper, science-based research on the oneside without losing touch to reality on the other side is the challengeof this work. Nowadays, multitasking takes place everywhere: mostpeople walking on the street use their phone and speak to friends.Rapid switches from one task to another, even without being aware ofit, occur. Most of us do not necessarily become aware of simultaneous"actions" (e.g., walking and speaking at the same time). To my mind,many situations closely relate to the use and application of moderntechnological systems. This is the starting point of the following work.

1.2 multitasking and human-machine interaction

Figure 2.: Human machine interaction in daily life

An increasing development of technological systems in the beginningof the 21st century puts us into situations in which we have the lureof choice when interacting with a mobile system, a cash machineor a portable data assistant (PDA). This is especially true for newtechnology features in cars Strayer and Johnston [2001], and McCarleyet al. [2004] describe this phenomenon as "burgeoning popularity of in-vehicle technology". In the last decades, so-called in-vehicle informationsystems (IVIS) aim to support the driver by offering a multitude ofpossibilities while driving. Green [1999], for instance, postulates a"15-second rule for driver information systems", meaning 15 seconds tobe the maximum time for a task duration while driving. According toGreen, the time a task requires is strongly correlated to crash risk. Also,Green promotes to measure task time instead of applying eye tracking(e.g., eyes-off-the-road-time) due to its instability. His rule became aninternational standard and highly focuses on in-car security to preventhuman errors. According to Green [1999], the "15-Second rule"

8 introductory note

• is consistent with existing national and trade association guide-lines

• is consistent with accepted vehicle design practice

• is a feature to minimizes harm to drivers

Technological development is inevitable. Using the availability of(more or less) intelligent systems can support us in daily routines, butit can also turn out to be a burden. Almost twenty years ago, this hasbeen referred to as "techno stress".

Figure 3.: Advertisement, Berlin (2007)

1.3 technology can do multitasking

Technology can do multitasking forever but humans can not! Caig Brod,author of the ground-breaking bestseller Techno Stress: The Human Costof the Computer Revolution Brod [1984], directed to the implicit dangerof the vast development of modern technology. Rosen and Weil 1 pro-vide an excellent explanation why nowadays, modern communicationlike email is so distracting. They call this phenomenon "multitaskingmadness": Human beings have brains that allow them to appear as thoughthey can comfortably perform more than one task at a time. In reality, ourbrains have an excellent filtering mechanism that helps switch our attentionrapidly from one thought to the next.

To overcome this problem and stop the multitasking madness, Rosenand Weil recommend the following:

1. Precise time estimation: typically, we underestimate time neededto perform or fulfill a task. This bias creates expectations wecannot meet. Realistic time estimation is a first step towardshandling the demand of several tasks (almost) concurrently.

2. External memory: letting go off memory traces reduces cognitiveload and helps us to focus more intensively to the current task.

1 published on www.contextmag.com

1.4 an excerpt of recent studies 9

3. Task perseverance: a full focus on one task at a time withoutmaintaining thoughts for other tasks decreases time needed toperform a task and increases task accuracy.

4. Down time: work will be more efficient after a refresh, be itplaying with children, watching TV, or reading a book.

A study by the Institute for the Future reported that employees of Fortune1.000 companies send and receive 178 messages a day and are interrupted anaverage of at least three times an hour!

1.4 an excerpt of recent studies

In chapter two, an overview on the history of task switching andmultitasking is given. But before that, let me mention a few exemplarystudies to illustrate the importance nowadays.

• Rogers and Monsell [1995] point out that people are faster inrepeating a task compared to task switching. This is also truefor familiar tasks which can easily be anticipated. Given moretime between the trials did not help to completely eliminate theswitching costs. According to Rogers and Monsell, switch costsare explained by (a) the need for mental control for the newsetting, and (b) carry-over effects from the previous trial. Properpreparation did not have any significant improvements.

• Meuter and Allport [1999] did a study in which subjects wereasked to name digits in their first or second language (dependingon the background color). Not surprisingly, response time fordigits in the first language was faster compared to the secondlanguage (in a repetitive task setting). But also, subjects wereslower when the language changed (task switching).

• Rubinstein et al. [2001] showed in a serial of four famous studiesusing a variety of tasks (e.g., maths, geometry) that task switchingcauses tremendous time loss. Additionally they were able to showthat performance was strongly influenced by task complexity.

• Yeung and Monsell [2003] present a modeling of experimentalinteractions between task dominance and task switching, illus-trating the importance of so-called prospective memory (we willcome back to that concept later within this work). It seems thatremembering where to continue a task plays a key role in thecontext of task switching.

These four excerpts demonstrate that multitasking and task switch-ing play a key role, both in scientific research as well as in real life.In context of the design of new technological products, insights howpeople handle several, apparently concurrent tasks, is of special im-portance: knowing the cognitive mechanisms behind allows to adapthuman-machine-systems to the user‘s need already in early stages ofthe design process. This we call prospective design.

Prospective design means to develop and integrate tools and meth-ods in order to investigate the human-machine interaction already inthe early development stages of technical systems. Design includesaesthetic as well as functional aspects. User-centered design (UCD) is

10 introductory note

an approach to integrate knowledge about and aspects of the user, e.g.cognitive limitations. UCD includes multiple stages, such as analyzes,development, testing, or re-design. Norman [1999] simply describesUCD as design based on the needs of the user. The earlier a designeris familiar with these insights (i.e., the users‘ needs) before a productis finalized, the better (see also Chin et al. [1988]). This work aims toprovide exactly this knowledge about the user. In four empirical stud-ies, multitasking scenarios are applied to analyze how users interactwith a system (i.e. doing a secondary task) while concentrating on adynamic activity (i.e., the primary task). Findings will provide helpfulrecommendations for a prospective design of human-machine-systems.

1.5 an ability to multitask?

More than half a century ago, Cherry [1953] mentioned that we have anatural ability and predisposition to multitask. At that time, for sure, thetechnological demands remained rather limited. Almost at the sametime, Robert Rauschenberg1 argued that "technology is contemporarynature". The author would rather go with the last quote than with thefirst one. And to show how we adapt to this "nature" is the focus ofthis work."There’s been an

exponential explosionof availableinformation. Part ofthe responsibility ofpeople developing thistechnology -computermanufacturers, ’bigBill’ Gates out inSeattle - should betaking into accountthe multitaskinglimitations of peopleusing it." DavidMeyer (2001,Interview)

Figure 4.: Combine painting: cloth, metal, leather, electric fixture, cable, and oilpaint on composition board, with automobile tire and wood plan (R.Rauschenberg, 1961)

1 In the 1950s, artist Robert Rauschenberg (1925-2008) created the concept known as"Combines": he put non-traditional materials and objects in innovative combinations,thus combining both painting and sculpture

Part II

T H E O RY

13

2T H E O R E T I C A L B A C K G R O U N D

For all but the most routine tasks (and few mental undertakings are trulyroutine) it will take more time for the brain to switch among tasks than itwould have to complete one and then turn to the other. (David Meyer)1

In his LA Times article, David Meyer - an expert in the field ofresearch on human multitasking - communicates a strong and directmessage, which is already summarized in the headline of his article: weare all multitasking, but what is the costs? Meyer points out that "costs" donot only refer to time but also to mental fatigue and a loss of attention,i.e. accuracy. His quote opens this chapter which is meant to providean overview of the history of (human) multitasking, with other words,the theoretical background of this work. Before the journey starts, letus be aware that we are interested in human multitasking. However, theterm multitasking has been used in several areas, such as:

computer multitasking - the apparent simultaneous performanceof two or more tasks by a computer’s central processing unit.

media multitasking could involve using a computer, mp3, or anyother media in conjunction with another.

human multitasking: the ability of a person to perform more thanone task at the same time

Figure 5.: A cartoon on multitasking(Source: EDUCATION 2.0, http://atedu20.blogspot.com)

1 taken from LA Times, Monday, July 19, 2004

15

16 theoretical background

2.1 a short historical survey

Defining the concept of "multitasking" has been challenging scientistsfrom various disciplines for decades (or even centuries). Long timebefore the word itself was used, psychological approaches towardsthe phenomenon of how to handle the demand of multiple tasks werereported. Chapter 2.1 gives a short introduction into the history oftask switching and multitasking, from the early beginning (Jersild andearly task switching paradigms) to nowadays studies (Saluvicci [2005],Taatgen [2005], etc.). This summary is not meant to provide a fulldescription of all studies in this area, moreover I mention and describeimportant steps towards the current state of the research by giving anoverview of the history of human multitasking.

2.1.1 The 1920‘s

1927. Jersild [1927] confronted participants with a list of stimuli toinvestigate the ability to alternate between different tasks. He wasinterested in how people switch from one task to another. In hisstudies, two conditions had to be executed, one in which the same taskwas performed on each item (so-called pure task blocks) and a secondone in which different tasks were performed (alternating task blocks).

STIM - 3 STIM - 3

task A task A

r(A-A)

STIM - 3

task A

STIM - 3

task A

TASK

REPETITIO

NTA

SK R

EPETITION

TASK

SWITC

HIN

GTA

SK SW

ITCH

ING

r(A-A) r(A-A)

STIM - 3 STIM + 6

task A task B

r(A-B)

STIM - 3

task A

STIM + 6

task B

r(B-A) r(A-B)

PURE TASK BLOCK CONDITION

ALTERNATING TASK BLOCK CONDITION

Figure 6.: Early task switching study by Jersild (1927)

In the pure task block condition, task A was "subtracting three fromeach number on the list". In the alternating task block condition, Task Bwas "adding six to the number on the list". As we can see, already in thisearly period of psychological studies, Jersild assumed that, though bothtasks being mathematical operations, he assumed them to be differentin terms of cognitive processing, long time before the term "cognitive"was used. A further, second distinction was between univalent (i.e.,each stimulus is a potential input only for the appropriate task) andbivalent (i.e., every stimulus is a potential input for either task) itemlists (see also Fagot, 1994). The tasks mentioned above imply a bivariatecondition. Univariate lists, in contrast, contained words (input for task

2.1 a short historical survey 17

A) and numbers (input for task B). Switch costs (in terms of reactiontimes) were measured as the difference between a switch (r(A-B) orr(B-A)) and a no-switch (r(A-A)). To my best knowledge, Jersild was thefirst one to introduce the term switch costs. Even nowadays, the notion"switch cost" still holds (Pashler [2000], Rubinstein et al. [2001]), as wellas "mental set" (see Spector and Biederman [1976], Meiran [1996]).Main findings in the studies by Jersild [1927] are:

1. For bivalent item lists, performance time is slower in the alternat-ing condition.

2. For univalent item lists, performance is slower in the pure condi-tion

These early, surprising findings remark a first step towards the dis-tinction between modalities required to properly execute a specific task.The definition of switching costs still nowadays is used in many studieson task switching. A few years after Jersild and his task switchingparadigm, the aspect of concurrency became of deeper interest.

2.1.2 The 1930‘s

1931. Telford [1931] asked the question what happens if two tasksoverlap, i.e. the second task appears with a temporal delay to thefirst one. In contrast to reported studies in which tasks are presentedsequentially (task switching), i.e. one after the other, this scenario iscalled the psychological refractory period (PRP). The PRP - paradigm is asfollows: a stimulus S2 of a task T2 is presented shortly after the onsetof a stimulus S1 of a task T1. The difference of these two onsets (S2-S1) is called "simulus onset asynchrony" (SOA). This extension of taskcoordination is a first step into the investigation of task concurrency, i.e.the fact that for a short moment two tasks appear concurrently. Mainresults in the context of PRP is that the smaller the (temporal) distancebetween S1 and S2 (the shorter the stimulus onset asynchrony, SOA), thelonger the reaction time (RT) for S2. RT is measured for both T1 andT2. Following the argumentation of Jersild [1927], RT for S2 should beshorter after T1 than T2. Unfortunately, many studies in the context ofPRP do not include a pure task condition (task repetition) as applied inJersild [1927].

2.1.3 The 1940‘s

Vince, M. (1949). The connection between the psychological refractoryperiod and rapid response sequences (Vince [1949]) is investigated byMargaret Vince. She could confirm PRP - results by Telford [1931].

2.1.4 The 1950‘s

1952. A processing bottleneck is proposed by Welford [1952]. Twodecisions about two responses to two different stimuli at the same time,Welfold claims, is impossible. Imagine two visual stimuli and twonecessary responses (e.g., button presses). Participants had to respondboth stimuli by pressing a button, respectively. Welford found thatthe reaction time for the second stimulus is slower than reaction timefor the first stimulus. He called this delay psychological refractory period


controlled processes automatic processes

slow fast

flexible use no easy modification

reduce capacity do not reduce capacity

conscious (attention) unconscious (no attention)

Table 1.: Controlled vs. automatic processing

and argues that it is always present, even for quite different stimuli.Welford [1952], to my best knowledge, was the first one to introducethe concept of a bottleneck. Many subsequent studies support hisassumptions for a general bottleneck in dual-task processing, implyingserial processing of cognitive steps. Welford did not disclaim thatthis bottleneck is sometimes small or can be reduced, e.g. by training.The impact of training (which is also a denotative feature within thepresented studies) will get more attention later within this chapter, incontext of assumptions about single vs. multiple resources in dual taskperformance.

2.1.5 The 1960‘s

1963. Borger [1963] investigated the refractory period and serial choicereactions. He found PRP - effects with visual and auditory stimuli. Inthe studies, some participants applied a queuing strategy, i.e. reaction totask one (R1) is buffered and given shortly before reaction to the secondtask (R2). Pashler [2000] calls this behavior grouping strategy: it can beavoided by giving appropriate instructions. Meyer and Kieras [1997a]instructed to produce R1 as fast as possible. This, one might object,potentially evokes time pressure but prevents from answer queuing (seealso Meyer and Kieras [1997b]. Learning from instructions became evenmore important in recent years (Taatgen et al., Taatgen et al. [2006]). Inthe presented studies in chapter three, to come to the point, the appliedmain task will be instructed as priority task. By doing so, "grouping"effects of task response are implicitly excluded. Before further PRP -or task switching studies are presented, it is necessary to address ourattention to the way a task is processed - a diminutive but not exiguousaspect in context of human multitasking.

2.1.6 The 1970‘s

1977.Schneider and Shiffrin [1977] emphasize the necessity of a distinc-tion between controlled and automatic processes. Later in this chapterwe will see why this distinction is of deeper interest for human multi-tasking and dual task performance.

The price for flexibility (controlled processes) is a reduce of speed.Of special interest related to the studies present in the next chapter andthe issue (human multitasking), automatic processes are not necessarilyconsciously accessible. Automatic processing is the result of trainingand practice. Automatic processing is a typical feature of skill acqui-


sition (e.g., Anderson [1982], Lee and Taatgen [2002]). One model tocontribute to this phenomenon was proposed by Rasmussen.

Figure 7.: Skills, rules, and behavior (Rasmussen [1983])

2.1.7 The 1980‘s

1983. Rasmussen [1983] developed a model about skill, rules andknowledge to explain the essential features of human skilled behaviour.In his eyes, a skill is a combination of open- and closed-loop behavior.In his model, he claims three levels of behavior:

1. Skill-based behavior (SBB): Automatic processing, without con-scious attention or control, relying on signals.

2. Rule-based behavior (RBB): Behavior is based on familiar rulesand consists of a sequence of subroutines (e.g., mathematicalproblem solvin, driving)

3. Knowledge-based behavior (KBB): Relying upon a "mental model"(of the system), no rules needed.

With other words, behavior turns from a cognitive stage to an asso-ciative stage and finally to an autonomous stage. Conscious processing,thus, becomes unconscious processing, including human error behavior(from error-prone to error-free) and speed (from slow to fast processing). Incontrast to the assumptions from Schneider and Shiffrin [1977] whosetheoretical "explanation" remains more descriptive than explanatory,the approaches on skill acquisition from the decade of the 1980s providea comprehensive and plausible framework for the question how pro-cesses become automatic and thus resource-saving. Later in this chaper,a model by Chris Wickens will be introduced. This model postulatesmultiple cognitive resources. But before that, let us have a deeper lookinto further relevant studies on task switching and dual tasking.


2.1.8 The 1990‘s

1995. Rogers and Monsell [1995] introduce task set reconfiguration to ex-plain the phenomenon of alternation costs even without item repetitions.Monsell [1967] claims that different processing modules are needed fordifferent aspects of a task. Alternation costs are defined as differencebetween so-called pure and alternating-task blocks. Monsell used an"alternating runs" procedure. Though the empirical approach is ratherabstract and not intuitively transferable into a real-life context, Monsellalso provides an illustrating example to explain what is meant by a taskset:

a professor sits at a computer, attempting to write a paper. The phonerings, he answers. It is an administrator, demanding a completed module re-view form. The professor sighs, thinks for a moment, scans the desk for theform, locates it, picks it up and walks down the hall to the administratorsoffice, exchanging greetings with a colleague on the way. Each cognitive taskin this quotidian sequence (sentence-composing, phone-answering, conversa-tion, episodic retrieval, visual search, navigation, social exchange) requiresan appropriate configuration of mental resources, a procedural "schema" or"task-set".1992-2000. Hal Pashler summarizes recent analyzes of dual-task studiesin Pashler [2000] by putting them into two categories, namely

1. studies of task switching or mental sets

2. studies on divided attention or dual task performance

Pashler claims that people show limitations when they have to per-form two tasks concurrently, and these limitations are strongest incentral stages of decision, memory retrieval, and response selection,with other words, in cognitive aspects where tasks are "intellectuallydemanding" (p. 287). It is widely known that training and practicesupports performance, especially in perception and motor response.Hazeltine et al. [2002], for instance, strongly promote a simultaneousdual-task performance with parallel response selection afer sufficienttraining (see also Ruthruff et al. [2003]). Instead of practice or training,some authors (e.g., Meiran and Daichman [2005], Sohn and Anderson[2005]) use the expression advanced task preparation to emphasize thepreparatory control. In their study, task switching produced a perfor-mance decrease ("task errors") which disappeared after "advanced taskpreparation" (i.e., extensive task practising).

Doing two things at the same time (Pashler [1993]) is inevitably con-nected to restrictions of a so-called bottleneck (Pashler [1984], Greenwaldand Shulman [1973], Greenwald [1972]). A few year ago, a discussionabout the central bottleneck took place starting after a paper publishedby Meyer and Kieras [1997a] (see also: Meyer and Kieras [1997b]).

2.1.9 2001: Uncorking the central cognitive bottleneck

2001. A ground-breaking article published in Psychological Science reani-mated an old discussion about the simultaneous performance of two ormore tasks involving perception, cognition and action. As mentionedby Pashler [2000], human multitasking is restricted and main reason forthis restriction is a central bottleneck. The response-selection bottleneck


PERCEPTION EXECUTIONRESPONSE SELECTION

AND PROGRAMMING

S1 R1

PERCEPTION EXECUTIONRESPONSE SELECTION

AND PROGRAMMING

S2 R2

RT 1

SOA

RT 2

Figure 8.: Bottleneck theory (adapted from Pashler(1993)

(RSB) hypothesis assumes the steps "perception - response selectionand programming - execution" and claims that response selection toa stimulus S2 from a Task T2 can only be executed after the responseselection to a stimulus S1 from a Task T1 has been finished (see Pashlerand his bottleneck theory, Pashler [1993]). According to that, parallelprocessing is possible during perception (early stage of informationprocessing) and execution (late stage of information processing). How-ever, processing of response selection is serial.Schumacher et al. [2001] argue in their article that even after "moderate"training, people reach a state in which they perform two tasks in paral-lel and the authors call this virtually perfect time sharing. But they alsomention that not all participants in their studies were able to reach thisstate (individual differences) and the question arises whether extensivepractice would enable virtually perfect time sharing for all participants.Dual-task interference is explained by conservative executive controlpostponing one task while another one is not yet executed. Main claimin their approach, in sum, is that intensive training and practice allowhuman multitasking without dual tasking costs for switching or re-sponse selection time according to a central bottleneck. Based on theseresults, study II of my work investigates allows participants a largeamount of practice in order to overcome limitations and to becomeskilled for the applied multitasking scenario.

2.1.10 2005: A general multitasking component

2005. Salvucci categorizes multitasking studies related to real-worldtasks as illustrated in Fig. 9 (taken from Saluvicci [2005]). In contrastto many psychological approaches within this subject, he highlightsthe fact that in "real life", many situations should be understood asmultitasking scenarios.

While Lee and Taatgen [2002] define multitasking as "the ability tohandle the demands of multiple tasks simultaneously", Saluvicci [2005] seeshuman multitasking as the "ability to integrate, interleave, and perform


Figure 9.: Examples of multitasking models developed in a cognitive architec-ture (from: Saluvicci [2005])

multiple tasks and/or component subtasks of a larger complex task". For aclassification, he divides discrete (duration < 10 s) and continuous(duration > 10 s) tasks and proposes four categories:

1. Models of discrete successive tasks

2. Models of discrete concurrent tasks

3. Models of elementary continuous tasks

4. Models of compound continuous tasks

Models of discrete successive tasks are task switching studies likethose already examined in the 1920‘s. Alternating simple choice-reaction tasks are applied to investigate switching costs. In thesescenarios, the aspect of concurrency is not given. For this reason, i donot consider them as multitasking studies per se.Models of discrete concurrent tasks include a temporal delay. PRP-studies in the context of dual task performance belong to this section.Stimulus onset asynchrony defines when the second task begins. Asalready mentioned before, Pashler [2000] and others assume a centralbottleneck which does not allow absolute concurrency.Elementary continuous tasks build the bridge to multitasking in dailylife: one continuous task (e.g., driving) is performed while at somepoints a discrete task (e.g., a simple choice reaction task) is presented.To the authors belief, integrating these aspects of concurrency is a first

2.2 multitasking or task interruption? 23

step into human multitasking in a realistic context.Even more important and relevant for this work are compound con-tinuous tasks. As the former category refers to tasks with a durationshorter than 10 seconds, this last section captures many scenarios, beit in the context of air traffic control, driving, or mobile computing.Salvucci mentions multiple recent examples (see Fig. 9) such as a modelof manual tracking by Meyer and Kieras [1997a] and Meyer and Kieras[1997b], a radar-operator model, identification if new aircraft on radar,or a scenario in which driving and phone dialing is modeled (Salvucci[2001]. According to Saluvicci [2005], "...all these efforts contribute to abroader understanding of multitasking through study of both overall measuresof task performance and particular measures of multitasking performance".

2.2 multitasking or task interruption?

The following section focuses on task interruption, on its characteristicfeatures and gives some example of task interruption studies. To theauthor‘s mind, task switching from an unfinished task (which later isresumed) to a secondary task is, strictly spoken, a task interruption ofthis primary task. For this reason, both cases directly refer to humanmultitasking.

2.2.1 Characteristics of task interruption

Already in 1927, Bluma Wulfovna Zeigarnik, a soviet psychologistand student of Kurt Lewin and Lev Vygotsky, showed that peopleremember interrupted tasks better than uninterrupted tasks (Zeigarnikeffect, see Zeigarnik [1927] and Zeigarnik [1967]). She observed thatwaiters seem to have a better memory for unpaid orders (van Bergen[1968]). Similar to Zeigarnik, Maria Ovsiankina showed that peopletend to resume unfinished, but not finished tasks (Ovsiankina [1928].Her assumptions closely relate to Zeigarnik and this effect is referredto as Ovsiankina effect. A few years later, H. [1941] was interested inthe impact of feedback (success and failure) on the resumption ofa task (motivational component). However, these effects could notalways be reproduced, but at least it shows that interrupting peopleaffects their task performance, be it in terms of stress (Cohen, 1980),decrease in task performance (Gillie and Broadbent [1989]), producingmistakes (McFarlane and Latorella [2002]) or recalling information.McFarlane [1998] defines an interruption as "the process of coordinatingabrupt changes in people’s activities" and in reference to this definition,McFarlane and Latorella [2002] classify interruptions using a taxonomywhich is presented here with slight modifications modified by theauthor:

source of interruption The interruption can be taken by the per-son who is doing a task (i.e., self, internal interruption), by anotherperson (i.e., external interruption), or by a machine (e.g., computer,external interruption). In many classical studies on task switch-ing, the source of the interruption can easily be controlled, forinstance by stopping task one and allowing to fully focus on tasktwo. However, in the context of human multitasking, the situationlooks rather different when one ongoing task is not stopped eventhough a second task starts.


individual differences Humans are bounded and rely on per-sonal limitations. Cognitive processing is limited, and so areprocesses of perception and motor response. Individual differ-ences play an important role in the field of human interruption,but are not of deeper interest within this work.

method of coordination Immediate interruptions occur withoutcoordination, in contrast to negotiated interruption. Further meth-ods are (human- or machine-) mediated interruption and sched-uled interruption based on an explicit agreement or by conventionfor repetitive interruptions.

meaning of interruption We all know the most common mean-ing of interruptions in our daily life. Alarms clocks during ameeting remind us to stop the current activity (task) and turnto another task/appointment/activity. Simply spoken, we arereminded that now, starting with the alert, our attention has to befocused on a specific action. Interruptions can also beckon us toultimately stop our current task.

method of expression Physical expression (verbal, paralinguistic,kinesic), expression for effect on face-wants (politeness),a signal-ing type (by purpose, availability, and effort), metal-level expres-sions to guide the process, adaptive expression of chains of basicoperators, intermixed expression, expression to afford control.

channel of conveyance Face-to-face, other direct communica-tion channel, mediated by a person, mediated by a machine,meditated by other animate object.

changed human activity Internal or external, conscious or sub-conscious, asynchronous parallelism, individual activities, jointactivities (between various kinds of human and non-human par-ticipants), facilitation activities (language use, meta-activities, useof mediators).

effect of interruption An interruption can cause multiple chang-es in human activity. It can influence motor behavior but alsomemory, awareness and the focus of attention. Especially in thecontext of multitasking situations, an interruption might create acomplete new situation to which the person who is interruptedmust adapt.

2002. Beginning of this decade, Altmann and Trafton [2002] describea sequence of actions within an interruption situation, as illustratedin Fig. 10. A primary task is performed and interrupted by an alert.The period between this alert and the start of a secondary task is theinterruption lag. In their studies, Trafton et al. [2003] focus on twocharacteristics of an interruption lag:

• the availability of the primary task during the interruption period

• the duration of the interruption lag

Quite obviously, this time lag is a function of both the time thesecondary task starts and a person‘s reaction time that defines whento start the secondary task. The secondary task itself is performed andends at a certain moment in time. Before the primary task is resumed,

2.2 multitasking or task interruption? 25

BEGINOF 1st TASK

ALERT FOR 2nd

TASK

BEGINOF 2nd TASK

ENDOF 2nd TASK

RESUMEOF 1st

TASK

INTERRUPTIONLAG

RESUMPTION LAG

Figure 10.: Interruption situation (taken from: Altman and Trafton(2002))

time passes. This interval is called the resumption lag. Fig. 10 clearlyillustrates the complete sequence of a scenario in which two tasks areexecuted, but they lack the integration of one or even more continuoustasks which might be partially executed in parallel. While driving, forinstance, people seem to be able to switch their visual attention to thephone for a split second and dial a number. Nevertheless, they continuedriving (without visual awareness for that moment).

2.2.2 The task switching question

One of the core questions in the context of task switching and multi-tasking is: "when do people switch between tasks?" Kushleyeva et al.[2005] mention three criterions (referred to as "major skill sets") whichhave to be met for "satisfactory" multitasking performance, namely

1. the ability to create and schedule future intentions

2. the facility to remember and prioritize these intentions

3. the ability to switch from carrying out one to another task whenthe appropriate moment in time is finally reached

The first and the second condition resemble a concept which gainsmore and more interest recently: memory for future intentions is of-ten named prospective memory (remembering to remember, Winograd[1988]). McDaniel and Einstein [2000] distinguish between event-basedprospective memory (cue is an event, e.g., pressing a button or an-swering a question) and time-based prospective memory (recalling tocontinue a task at a certain time, e.g. going to a meeting at 4pm).Following Smith and Bayen [2004] and Smith [2003], maintaining anintention always requires attention resources, whereas McDaniel andEinstein [2000] note that cue identification can be automatic or effort-ful, depending on a variety of parameters. The connection betweenprospective memory and the area of interruption becomes quite visi-


ble: Altmann and Trafton [2002] highlight that in order to resume aninterrupted task, two essential conditions have to be met:

1. prospective goal encoding ("what was I about to do?")

2. retrospective rehearsal ("what was I doing?")

In their eyes, prospective goal encoding constitutes "a key mechanismbehind prospective memory". Retrospective rehearsal is connected towhat Kushleyeva (Kushleyeva et al. [2005]) calls "facility to remember"and can be suppressed by tasks which prevent from rehearsal, such asthe n-Back task (McElree [2001], Owen et al. [2005], Juvina and Taatgen[2007]). The third and last criterion mentioned above focuses on themoment in time when precisely a switch has to be executed. Thisdecision, however, determines not only a switch of attention but alsothe activation of another task set, i.e. the task set which is necessary toperform the resumed task. In Chapter 3, I claim and provide evidencethat this decision is both conscious (and reported) but also unconscious(and thus only available in eye tracking data). In their approach, DarioSalvucci and his team use a computational grammar (an algorithm)called "information requirements grammar" to describe when peopleswitch from one task to another.

2.2.3 A grammar for task scheduling

Andrew Howes and his collegues propose a "theory of competencefor tasks", so-called information requirements grammar, IRG (Howeset al. [2005]). IRG implies the assumptions that (a) information andcontrol requirements constrain the execution of a task and (b) availableresources constraint the performance of a task (i.e., of their compo-nent processes and the necessary information). Two different kindsof constraints define task scheduling, namely information and controlconstraints on the one side and resource constraints on the other side.This is illustrated in the following example:

IRG, however, does not allow a delay in performance when theinformation necessary to execute a task or subtask is available. Inaddition, this grammar proposed perfect task switching.

2.3 single or multiple resources?

Single-resource theories assume one central, unique resource (General-Purpose-Limited-Capacity Central Processor). Nobel prize winnerDaniel Kahneman claims that we have only one global resource, and ifwe reach this available capacity, e.g. by demands of multiple concur-rent tasks, we feel cognitive load (Kahneman [1992]). Single-resourcetheories postulate a direct connection between number and difficultyof concurrent tasks on the one side and resulting cognitive load on theother side. Performance decreases with increasing number of tasksand directly influences our limited, cognitive resource. The more dif-ficult a task, the more reduced the available cognitive resource andconsequently the performance. Especially in dual task studies, thistheoretical assumptions by Kahneman [1992] became popular. Twotasks can be performed concurrently until the limit is exceeded. If thishappens, cognitive resources are no longer available and both taskscannot be performed in parallel. Mainly, a decrease in reaction time

2.3 single or multiple resources? 27

Figure 11.: Howes (2005): a grammar for task scheduling

results from that. Performance errors are expected to increase in thiscase.

In contrast to theories on one central cognitive resource, so-calledtheories of multiple resources postulate different and specific modulsfor information processing (see Fig. 12). Similar to assumptions byKahneman [1992], a limitation of the cognitive system is not denied.But the main difference to single-resource theories is that the centralcapacity is a product of different, independent individual capacities.Wickens (Wickens [2002], Wickens [1984], Wickens and Liu [1988], andWickens [2004]) proposes a model of multiple resources, as shown inFig. 12 with the following categorial, dichotomic dimension:

1. processing stages (perception, cognition, responding)

2. perception modalities (visual vs. auditory)

3. cisual processing (focal vs. ambient)

4. processing codes (spatial vs. verbal)

Following Wickens and his model of multiple resources, it is generallypossible to perform multiple tasks under the same conditions withoutdistraction or loss in performance. Division of attention, for instance,turned out to be more robust under "cross-modal time-sharing" com-pared to "intra-modal time-sharing". The model by Wickens (Wickens[2004], Wickens [2002]) allows different, divided resources for indi-vidual stages of information processing: resources for perception andresources for response do not interfere,thus both processes theoreticallyrun in parallel without performance loss. For the presented studies inthe next chapter, these implications play a key role.


Figure 12.: Model of multiple resources, taken from: Wickens and Liu [1988]and Wickens [2004]

2.4 summary and criticism

The presented theoretical approaches towards human multitaskingserve to give an short overview and can be summarized in the followingmain messages:

1. First empirical approaches to investigate human multitasking goback to the task switching studies (Jersild [1927]) in which simplechoice reaction tasks were applied.

2. These studies were enriched by the aspect of overlapping tasks(Telford [1931]) and the concept of psychological refractory periodwas introduced.

3. In the following decades until the late 1980s, many variations ofthe early findings systematically analyzed human task switching.

4. Pashler [2000] emphasizes the role of a central bottleneck andaims to show the inevitability to fully parallelize two tasks.

5. With their Psychological Science article on how to reach virtuallyperfect time-sharing, Schumacher et al. [2001]

6. In recent studies (Saluvicci [2005], Taatgen [2005]), real-life scenar-ios in the context of human multitasking gain increasing attentionand the importance of continuous tasks in such investigations ishighlighted.

7. Core questions in explaining human multitasking still remain:a) Is it sufficient to apply discrete tasks in multitasking studies?b) How much training do people need to optimize task schedul-ing?c) When and how do people decide to switch from one task toanother?d) Which strategies do people apply in human multitasking situa-tions?

2.4 summary and criticism 29

2.4.1 Need for continuous tasks in multitasking studies

In Pashler [2000], the main focus is on discrete tasks. This is a rathercommon handling in the context of studies on dual task performance.Altmann and Trafton [2002] use discrete tasks in their task switchingscenario. As Saluvicci [2005] remarks, to draw conclusions abouthuman multitasking behavior in real life, it is necessary to investigatecontinuous tasks in a dynamic environment. Driving as the mostprominent example for a continuous task underlines this demand. Butalso various other situations in daily life like walking in the streetand concurrently using a mobile phone support a call for appliedstudies including continuous tasks. For this reason, my studies rely onscenarios of "compound continuous tasks", as proposed by Saluvicci[2005].

2.4.2 The importance of training and task repetition

Schumacher et al. [2001] highlight the importance of task training in thecontext of multitasking studies. They claim that participants becomeskilled in that specific task set. Lee and Taatgen [2002] considers skillacquisition as a method to "learn" how to do multitasking. Within thecognitive architecture ACT-R (Anderson [2007]), Taatgen refers to amechanism called "production compilation" as main explanation howto perform successful multitasking. The following nice example (takenfrom: Taatgen [2005]) helps to illustrate how this works:Put water in kettle, put water on stove until it boils, put tea leaves in teapot,pour boiling water in teapot, and wait 3 to 5 min. These five instructionsfor making tea can be stored almost literally in declarative memory. The sim-plicity of the representation explains why this is the starting point for a newskill: Declarative items of knowledge can be added as single items to mem-ory. The disadvantage of declarative representations is that they cannot actby themselves; instead they need, according to Anderson’s theory, produc-tion rules to be retrieved from memory and interpreted. This explains whyinitially processing is slow, because the declarative representations must beretrieved before they can be carried out, and it is prone to errors because theright declarative fact might not be retrieved at the right time.

In study II within chapter III, the role of training and practise isillustrated. We will see to what extend this will contribute to handlethe demands of concurrent multiple tasks in a dynamically changingenvironment.

2.4.3 Cognitive heuristics under human multitasking

Even though the approaches by Taatgen [2005], Saluvicci [2005] andothers provide (computational) models which accurately predict hu-man performance under multitasking in their concrete task scenarios, Idoubt that human task scheduling behavior follows a grammar like IRG(see Howes et al. [2005]) or a formal description. Instead, I claim that ina human multitasking scenario, people adapt to the environment anddevelop strategies to optimally "survive" in such situations. Addition-ally, people might not necessarily be aware of their applied strategies,i.e. they arise either consciously (strategic) or unconsciously. They donot need to be precise, either. Therefore, I use the word "heuristic"which comes from the same Greek root as Eureka! And means "to find".


In my understanding, a heuristic is a "rule of thumb", with other wordsa rule which is simple, efficient and can easily be learned throughexperience and training. Kahneman and Tversky [1973] propose theavailability heuristic as a a heuristic for judging frequency and proba-bility where people base their prediction of the frequency of an eventor the proportion within a population based on how easily an examplecan be brought to mind. With other words, the ease of imagining anexample has more weight for the judgment than the actual statisticalprobability. Because an example is easily brought to mind or mentally"available", the single example is considered as representative of thewhole rather than as just a single example in a range of data. StuartSutherland illustrates this heuristic using a plausible example (Stewartet al. [1994]). Asked whether there are more words with "r" as thefirst letter than with "r" in the third position and also whether thereare more words beginning with "k" than with "k" as the third letter,most people tend to reply that in both cases there are more wordswith "r" on 1st position than on the 3rd position. The same counts forthe "k" - example. Nevertheless, people make a mistake, because inboth cases, there are more words with "r" (same for "k") on the 3rdposition. Words are arranged according their initial letter. Retrievingfrom memory thus is facilitated for words starting with a letter, e.g.with "r" (road, run) whereas it is more difficult to retrieve words withthis letter on the 3rd position (like street, care). The statistical frequencyis completely ignored and people judge based on the availability ofwords in their mind. This heuristic is used to explain findings in thearea of probability judgment and people in general are not aware thatthey make use of a cognitive heuristic. The unconscious application ofa heuristic implies that cognitive resources should suffer less comparedto a strict propositional processing. Heuristics furthermore are not, assome scientists state, a bias or a failure: According to Gigerenzer andSelten [2002], heuristics can be fast, frugal and accurate all at the same timeby exploiting the structure of information in natural environments (page 9).With other words, the development is an adaptation mechanism to theenvironment. This characteristic is of main importance for the fourempirical studies, now presented in the upcoming chapter.

Part III

S T U D I E S

33

3E M P I R I C A L S T U D I E S

Purpose of the following four studies and scope of the entire workis to convince and give ample evidence that, in a real life scenario,people do not follow principles of pure optimization when doingmultitasking. Multiple approaches (e.g., Brumby et al. [2007]; Saluvicci[2005]) propose a task switching behavior according to free resources(e.g., visual attention or manual action) and available informationrequired for performing a specific task. In study I-IV, the author triesto illustrate how people do multitasking by adapting to the structure ofthe environment. In contrast to many (if not even most) psychologicalstudies in the last century, a dynamic main task in man machineinteraction will be used. Study I introduces the multitasking-scenariowhich was applied in all of the four studies. Main focus in the firststudy lies on investigating how people manage a multiple task situationin a real-life context. Study II goes one step further by concentrating onthe impact of practice and thus analyzes multitasking strategies to adeeper extend. Study III shows the importance of task configuration:with a systematic variation of the involved secondary task, it becomesobvious how people manage the demand of several tasks according totheir design. The last and final study (IV) refers to an aspect of dailylife, i.e. time pressure: in dynamic man-machine-interaction, availabletime is often rather precious and especially high time pressure plays akey role in how we handle multitasking in everyday life.

Study I and II were conducted in a driving simulator. In study III andIV, a driving simulation on a PC was used. Please note that for the firsttwo studies, ecological validity turns out to be higher compared to thelast two studies, but these studies (I and II) lack a complete systematiccontrol of external influences. For this reason, study III and IV were runon a PC with a simulated driving environment and perfectly controlledconditions.

Findings of the four studies provide helpful insight and consequentlysupport the prospective design of human machine interaction alreadyin early stages of system development. Understanding how peoplebehave in a multitasking scenario, what kind of strategies they use andhow to support their performance allows a direct intervention duringthe design process and helps to reduce human errors in daily life.

3.1 study i: identification of multitasking heuristics

Most studies in the context of human multitasking mainly includestandard, PC-based psychological tasks. As illustrated in 2, psycholo-gists have been analyzing multiple-task coordination for quite a longtime, starting already in the 1920s. Task shifting studies (Altmann andTrafton [2002]) or PRP-studies (Pashler [2000], Pashler [1993]), however,lack a direct connection to human behavior in daily life. The catego-rization of multitasking studies by Saluvicci [2005] nicely reflects thatonly recently, researchers started to pay more and more interest indynamic task environments. Study I, for this reason, is interested inthe adaptation and the allocation of attention in a scenario in which

35

36 empirical studies

people have to perform a continuous task (driving) and concurrentlysomehow handle the demands of a secondary task which is connectedto the main task. It is well-known that a driver is multitasking whiledriving: everywhere, everyday. Be this eating, reading email, or talkingon the phone. Inattention, distraction, and mental fatigue still are themost dangerous contributing factors leading to an accident. The designof an in-car-system which requires less attention and is easy to use,thus, can prevent from vehicle crashes. To do this, it is necessary to un-derstand how people behave in a naturalistic multitasking environment.

3.1.1 Method in study I

Participants in study I

Twenty four male and female undergraduate students (age between20 and 30) of Technical University Berlin participated the first study.All participants had a driving license and were experienced in driving.Wearers of glasses were excluded from the study. Gender effects wereof no interest for the study.

Figure 13.: Car interior in study I

Involved tasks in study I

Primary task in study I was a driving task (with lane derivation beingconsidered a dependent variable): participants were asked to drive withconstant speed (130 km/h) in a driving simulation in a car. The taskitself was quite trivial (keeping the lane): participants were instructedto keep the lane (a simulation which does not require a deeper spec-ification here). Performing this task (after being instructed to do so)turned out to be feasible for all subjects. Also, the focus on drivingas primary task was understood by all subjects and they acted accord-ingly, considering driving with main importance. As secondary task,the MODYS research group, especially Marcus Heinath and JeronimoDzaack, build a derivation from the "D2 test of attention" (Brickenkamp

3.1 study i : identification of multitasking heuristics 37

[1992]), an in-car version adapted to be displayed on a 10inch screenin the car interior. In what follows, this test is hence referred to as"D2-Drive". In comparison to other "in-vehicle information systems"(IVIS), D2-Drive is a model of a secondary task on a screen-orienteddriver information system and captures the following characteristics:

• Attention: D2-Drive requires full visual attention.

• Access: D2-Drive can easily be learned.

• Interruptability: D2-Drive can be interrupted and resumed.

• Resources: D2-Drive is a measuring tool for residual resources.

• Cognition: D2-Drive requires perception (read), cognition (de-cide) and action (motor response).

In this sense, D2-Drive is cultural independent, scalable in termsof complexity and serves as an optimal tool to measure individualattention. In this case, attention is needed for the secondary task forperception ("reading") and action (manual response). As a dependentvariable, Dr-Drive performance was considered in study I and allupcoming studies.

Figure 14.: Original D2 test of attention (Brickenkamp [1992])

Similar to the original paper and pencil version of the D2-test devel-oped by Brickenkamp (Brickenkamp [1992], in D2-Drive people have tojudge whether a pattern contains the letter "d" and concurrently twostrokes. However, in contrast to the original version, D2-Drive requiresto press a button (Yes or No) instead of crossing the pattern out. Thisfeature underscores the similarity to many situations in which we inter-act with systems in daily life. Based on the fact that D2-Drive requiresfull visual attention and asks for a decision (a cognitive evaluation),performing D2-Drive in context of a multitasking scenario means avisual interruption of a concurrent task.

Three versions of D2-Drive were used in study I (see Fig. 15):

d2-drive-v1 .1: Presentation of a complete row of (five) patterns Fo-cus only on the pattern in the middle (third pattern) Executiononly of pattern in the middle (1 pattern)

d2-drive-v1 .2: Presentation of a complete row of (five) patterns Fo-cus on complete row Execution of complete row (5 patterns)

d2-drive-v1 .3: Presentation of a matrix of patterns (rows and col-umns) Focus on the row whose number was presented Executionof this complete row

As can be seen, D2-Drive-v1.2 is close to the original version of the D2-test. D2-Drive-v1.3 contains an additional memory element, i.e. while


performing a complete row on the n-th screen, the row which needs tobe performed on the next (i.e., n+1 - th) screen has to be read and keptin mind. The design of all three versions assumes increasing complexityand the demand of different resources. Whereas version D2-Drive-v1.1requires visual fixation, a cognitive decision process and a response,the other two versions further include a reading element. D2-Drive-v1.3additionally relies on vertical as well as horizontal aspects.

Figure 15.: D2-Drive (Urbas et al., 2005)

Design of study I

Complexity of D2-Drive was treated as between-subjects factor (threegroups with 8 participants per version) and condition (single- vs. multi-tasking) as within-subject factor. Measure of performance for main task(Driving) was lane derivation, for secondary task (D2-Drive) number ofcorrect patterns per minutes (i.e., trial). Please note that the error rate inall three versions of D2-Drive approximated zero: hence, the decreasein performance is reflected in the number of performed patters (i.e.,patterns per min).

Procedure in study I

In study I, participants were first introduced to the complete procedureand instructed to perform the primary task, i.e. driving in the simulatorenvironment with constant speed (130 km/h) and main priority. First,participants trained the primary task and afterwards, baseline measures(single task condition) for driving were recorded. Subsequently thesecondary task (D2-Drive) was explained, trained and performed undersingle task condition (pretest). Following, the multitasking conditionstarted: while driving, at four different positions a sound indicated aD2-Drive test (duration of 60sec). As soon as participants heard thissound, they were requested to perform the secondary task (D2-Drive)without neglecting the primary task (driving). Within a lap, D2-Drivewas presented four times. The study concluded with a post test forD2-Drive and a structured interview in which participants were askedquestions about their experience while performing the study. For dataanalysis, a multivariate analysis of variance was conducted.

1. Welcome, introduction and instruction

2. Training "Driving"

3. Baseline "Driving"

4. Training "D2-Drive"


5. Single-Task "D2-Drive" (Pretest)

6. Dual-task session (4 x "D2-Drive")

7. Single-Task "D2-Drive" (Posttest)

8. Structured interview

Hypotheses for study I

In collaboration with the MODYS research group as well as from aexplorative perspective, the following hypotheses, mentioned below,were tested.

Hypothesis: Driving under multitasking

The primary task itself (driving) is a continuous tracking task anddoes not require deep cognitive processing. Driving is instructed tobe considered as priority task. Therefore, no performance decrease isexpected under multitasking compared to single tasking (baseline fordriving).

3.1.2 Hypothesis: D2-Drive under multitasking

• For the three versions of the secondary task, complexity is as-sumed to increase from version 1 to 3 consecutively and henceperformance to decrease. This assumption is based on a deepercognitive processing (visual perception, cognitive processing interms of decision making, and action via motor response), espe-cially for D2-Drive-v1.2 and D2-Drive-v1.3. It can be expectedthat performance (patterns per minute) in D2-Drive-v1.2 is com-parable to results from the paper and pencil version developedby Brickenkamp [1992]. Increasing complexity (D2-Drive-v1.1< D2-Drive-v1.2 < D2-Drive-v1.2) should be reflected in perfor-mance data (correct patterns per minute) and result in significantdifferences.

• D2-Drive requires visual attention, cognitive processing as wellas motor action (response). Therefore performance in D2-Drive isexpected to decrease under multitasking performance. Driving(main task) was instructed to be performed with priority andconsequently a decrease in the primary task cannot be expected.In addition, the three different versions of D2-Drive require a dif-ferent amount of visual attention. For D-Drive-v1.2, performanceis expected to be best: in this version, visual-motor coordination(as described by Wickens [2002]) is possible. D2-Drive-v1.1 andD2-Drive-v1.2 should not differ in performance significantly inthe pretest but in the dual task condition and even more in thepost-test due to stronger learning effects and development of a"multitasking skill" (Taatgen [2005]). D2-Drive-v1.3 is supposed tobe the most complex and thus most demanding version, leadingto worst performance. This is a result of the additional cognitive(memory) element in it.


3.1.3 Results of study I

Results: Driving under multitasking

Not surprisingly, driving was not affected by an additional secondarytask. Performance under multitasking was similar compared to singletask performance. Recordings of baseline driving after training confirmthat under multitasking, participants do not improve performance, i.e.after single tasking, they reached their maximum and learning effectsdue to longer practice can be excluded. The results further supportthe assumption that participants follow the instructions and considerdriving as main task with main priority.

Results: D2-Drive under multitasking

Figure 16.: Performance in D2-Drive in study I

Pre- and post-tests (single task condition) were applied to checkwhether learning effects (see Taatgen [2005]) occur. For D2-Drive-v1.1, performance did not change significantly whereas in both D2-Drive-v1.2 and D2-Drive-v1.3, participants improved significantly aftermultitasking (for both, p < 0.05). Overall, there was a strong significantdifference between single and multitasking for all three versions (forD2-Drive-v1.2: p < 0.01, for D2-Drive-v1.2: p < 0.05, for D2-Drive-v1.3:p < 0.06). Surprisingly, in D2-Drive-v1.2 performance was best in allconditions and performance data are comparable to the paper andpencil version.

After the complete scenario, participants were asked several questionsconcerning the manner in which they performed both tasks individuallyas well as concurrently. Based on these interviews, the most prominent"strategies" how people performed the secondary task were:

1. (1) For D2-Drive-v1.1, no specific strategy was applied. Partic-ipants fixated the pattern in the middle and from time to timeturned their visual attention to the street. The behavior remainedconstant during the complete procedure of the study.


2. (2) For D2-Drive-v1.2, participants started to perform the sameway they did in D2-Drive-v1.1, but over time, several patternswere read sequentially and responses were given sequentially aswell. 14 out of 24 confirmed to use this "strategy", some did soalready in the beginning of the dual task condition while others"developed" this strategy during the multitasking scenario.

3. (3) For D2-Drive-v1.3, the interviews state that participants usedthe same strategy. One additional "adaptation" was an external-ization of the memory element: 10 out of 24 participants reportedthat they did not keep this number in their mind but used thecorresponding finger of their left hand (which was on the wheel).

Mostly, for the secondary task, participants apply a processing ora strategy which here in this context is referred to as "merge heuris-tic": at the beginning, participants perform the test pattern by pattern.After a while, participants seem to have "understood" that mergingpatterns together and replying them in a sort of block (e.g., reading 2

or 3 patterns, scanning the street, responding these 2 or 3 patterns bymanually pressing keys) is a resource-friendly and clever adaptation tothe environment, i.e. the multitasking situation.

Figure 17.: The "merge heuristic"

3.1.4 Discussion of study I

In sum, study I shows that performance under multitasking does notdecrease in both tasks. One reason for this phenomenon is the impact oftraining (practice effects) as mentioned by the participants in the struc-tured interviews. A second, maybe even more important reason, is thedevelopment and application of multitasking heuristics. These heuris-tics can potentially be derived not only from verbal reports (structuredinterviews). In study II, eye tracking data will be applied additionally.The observation that performance improves over time (effect of practice)implies a need for a second study in which (a) participants are givenmore time to practice both tasks concurrently and (b) eye tracking dataare used to support assumptions about the "merge heuristic". Addition-ally to these aspects, a third extension is use of another 3 versions ofD2-Drive with nine instead of five patterns. These version are referred


to as D2-Drive-v2.1(5), D2-Drive-v2.2(5), D2-Drive-v2.3(5) and conse-quently D2-Drive-v2.1(9), D2-Drive-v2.2(9), D2-Drive-v2.3(9), where thenumber in brackets indicates the corresponding pattern lengths.

3.2 study ii: practice and multitasking heuristics

In study I, participants used cognitive heuristics which developed with-out instruction and not necessarily consciously. The "merge heuristic"was detected and described in detail in the previous paragraph. Maincritical issue of study I is based on the observation that participantsincreased in performance over time, especially in D2-Drive-v1.2. Thus,the question is whether the detected heuristic is moderated by practiceand how practice supports multitasking heuristics. In many studieson skill acquisition, practice has a tremendous impact on performance.For this reason the same scenario was applied with some slight modi-fications. Does more intensive driving amplify the use of the "mergeheuristic"? Will performance increase in a second driving lap with thehelp of the "merge heuristic"? Or will participants feel mental fatigueand produce worse performance under multitasking? And also, inanalogy to the original paper and pencil test of D2, it is of furtherinterest if longer patterns per version (9 instead of 5) even amplify thegap between the three versions. Especially D2-Drive-v1.2 is expectedto benefit from such a modified configuration. Therefore, what is theimpact of pattern length in the secondary task? Is there any connectionbetween pattern length and the "merge heuristic"? These questions willbe answered in this paragraph.

3.2.1 Method in study II

In study II, the same method was applied as in study I.

Participants in study II

Thirty-six undergraduate students of Technical University Berlin tookpart in the second study. Gender was equally distributed. Age wasof no further interest. As in study I, all participants of study II hada driving license and were experienced in driving. Because of themeasuring of eye tracking data, all participants had to confirm to wearneither glasses nor contact lenses. Participants were male and female,gender effects were not of any interest in this context.

Involved tasks in study II

Primary task for study II was driving, using the same environment(driving simulator with identical lane) as in study I. Lane derivationand performance in D2-Drive are dependent variables in study II. Incontrast to the first study, this time participants were asked to drivetwo laps. In fact, study II is a slight variation and a design similar tothe one applied in study I, with the following extensions:

• Providing more training by extensive driving (2 laps)

• Investigating the impact of pattern length in the secondary task(5 vs. 9 patterns)

3.2 study ii: practice - multitasking heuristics 43

As in the previous study, secondary task was D2-Drive. This is anecessary condition in order to compare participants behavior and tosee whether any possible effect is in fact based on practise.

d2-drive-v2 .1 Same as in study I (for 5 as well as 9 patterns)



In study II, subjects were instructed to pay main attention to the pri-mary task (driving in the simulation environment) and at the same time- without neglecting the driving - to perform D2-Drive as appropriateas possible. The three versions described in the previous section werepresented with either five patterns (as in the first scenario) or with 9

patterns. This is denoted by an according number in brackets, e.g., D2-Drive-v2.2(9), indicating that the second version was presented usingnine patterns. Please note that not coincidentally, the numbers refer toassumptions about working memory capacity: Cowan (2001) claimsthat people are able to maintain 4 (plus/mins 1) chunks, whereas for-mer studies (Miller, 1957) promote a "magical number seven", implyingthat working memory capacity features a size of seven. Nevertheless,this work is not meant to focus on people‘s capacity limit when doingmultitasking. Choosing a nine-patterns version just plays to the vari-ance in working memory performance: participants with a size biggerthan five "have the chance" to make use of their full capacity and donot find themselves restricted by a five-patterns version.

Design in study II

Exactly like before, "complexity in D2-Drive" (3 versions) was treatedas between-subject factor and "pattern length" (five vs. nine) as within-treatment.

Procedure in study II

As in the previous study, participants were welcomed and introducedto the scenario. As in study I, it was important for them not to have par-ticipate in a similar driving study (e.g., study I) before (inexperienced).

procedure for extended driving and d2-drive

1. Welcome, introduction and instruction

2. Training "Driving"

3. Baseline "Driving"

4. Training "D2-Drive"

5. Single-Task "D2-Drive" (Pretest)

6. Dual-task session (4 x "D2-Drive", lap 1)

7. Dual-task session (4 x "D2-Drive", lap 2)

8. Single-Task "D2-Drive" (Posttest)



3.2.2 Hypotheses for study II

Hypothesis: Intensive driving under multitasking

In the previous study, driving performance was not influenced bymultitasking. Therefore, in study II no significant differences wereexpected under multitasking compared to driving in the single taskcondition. The instruction (i.e., considering driving as man task withmain priority) was expected to fully work, meaning that participantsshould perform D2-Drive without neglecting the driving task. Mentalfatigue should not occur.

D2-Drive under multitasking (pattern lengths and practice)

Schneider and Shiffrin [1977] claim that tasks become "automized" afterpractice, thus no voluntary control is needed and no interference withmental operations appears, i.e. simultaneous performance withoutinterference is possible. However, in their studies, discrete dual tasksituations were investigated. In this study, a dual task scenario withcontinuous tasks is used. With reference to results from Schumacheret al. [2001], Schneider and Shiffrin [1977] and Taatgen [2005], practicenot only improves performance but also turns controlled processing of(part of) a task into automatic processing (from error-prone to error-freebehavior with an increase in speed). Intensive training should thus pro-mote using the "merge heuristic" (consciously as well as unconsciously)and thus lead to a better performance for D2-Drive-v2.2, but not forD2-Drive-v2.1 (due to its configuration not supporting the use of thedescribed heuristic). This counts for the dual task condition (mergingof action parts adaptively to the multitasking situation) as well as forthe post-test (participants will have learned how to optimally use the"merge heuristic"). However, D2-Drive-v2.3 is not expected to show sig-nificantly better performance after training. The memory element in itprevents from automatically enter responses by demanding resourcesin working memory.The "merge heuristic" (described in the previous section) is a smart, cog-nitive tool to adapt to the multitasking environment. Similar to studieson perfect-time sharing by Schumacher et al. [2001], two tasks coalesceand almost become one (as much as possible). Visual attention, ob-viously, cannot be shared, and both tasks require visual perception:the driving task requires, even if it is only a minimum, control scansof the street, and by nature, D2-Drive is based on D2 (Brickenkamp[1992]) which is a test of attention. However, participants in the firststudy somehow managed to separate those elements of both tasks thatallow parallel processing (control scans on the street and concurrentlyresponse in terms of manual action). Even more surprisingly, this allhappens without instruction: people understand the structure of their(dynamically changing) task environment and automatically adapt toit.

3.2.3 Results of study II

Results: Intensive driving under multitasking

Not surprisingly, driving remained constant and was not influencedby a secondary task. Even more, under dual task condition, driving

3.2 study ii: practice - multitasking heuristics 45

Figure 18.: Performance in D2-Drive in study II

performance did not significantly change from lap 1 to lap 2. This con-stancy of performance goes in line with results from the previous studyand again shows that participants follow the instructions and considerdriving as main task with main priority. It also shows that mentalfatigue, in case participants felt it, did have no effect on performance.

Results: D2-Drive under multitasking: pattern lengths and learning

The factor "pattern length" (five vs. nine patterns) did not have asignificant influence on performance in the secondary task. In thenine-patterns version, participants were only slightly better. However,this result is not worth further mentioning. For this reason, all data isaggregated, subsumed and hence I refer to D2-Drive overall. As alreadyconfirmed by study I, D2-Drive-v2.2 outperforms D2-Drive-v2.1 as wellas D2-Drive-v2.3. As can be seen, participants perform rather poor inD2-Drive-v2.3 (the obviously most difficult version). In this version,performance under dual task performance does not improve (in contrastto the other versions), only small, but not significant learning effects(pretest vs. post-test) were found. Comparing D2-Drive-v2.1 and D2-Drive-v2.2, the power of the "merge heuristic" becomes transparent: thisversion of the secondary task allows to optimally perform both tasks(or part of them) concurrently. Combining single parts of each of them(visual perception, cognition, motor action) is possible via the abilityto separate individual parts of the secondary task and reconfigurethem. Also, please note that the level of performance can be carriedover to the posttest in D2-Drive-v2.2, i.e. learning effects are strongerin this version than in D2-Drive-v2.1. Remarkably, in D2-Drive-v2.3,performance in the post-test is even betterParticipants in study II passed eight trials, i.e. while driving they werepresented eight times D2-Drive (two laps). 19 displays the performanceover time and clearly shows the improvement from trial 1 until trial8.Interestingly, this effect is strongest for D2-Drive-v2.2. A closer look atthe trials gives insight about the zigzag - sequence in the performancedata: even though it was not applied as a factor, a post observationgives insight about this phenomenon. Odd trials (i.e., trial one, trial


three, trial five (or trial one in the second lap), trial, seven (or trialtwo in the second lap)) correspond to a lane curve whereas even trialinclude a straight lane. There was no assumption about street behaviorand thus the significance and meaning of this "factor" is rather weak.Driving behavior on a curved vs. straight lane remains stable.

Figure 19.: Improvement over time in study II

Results: Cognitive heuristics under multitasking

In 3.1, one main critical issue was the lack of eye-tracking data. StudyII accounts for this measure, a comparison of visual time spend onthe primary vs. secondary task was done. On average, participantsspend 47 percent of visual attention (time of gaze in percent for thedefined area of interest, AOI, "lane" and for AOI "D2-DRIVE" duringmultitasking) on D2-DRIVE and 51 percent on the lane. The residualamount (2 percent) can be considered as noise. Interestingly, D2-Drive-v1.2 requires only 36 percent of eye gazes, whereas D2-Drive-v1.1 aswell as D2-Drive-v1.3 both require 48 percent. In combination with thehigh performance in D2-Drive-v1.2, this supports the assumption thatparticipants perform the secondary task while their visual attention ison the primary task. This adaptive task coordination can be explainedwith the help of the "merge heuristics" mentioned in paragraph 3.1.The explanations of participants of how they perform the individualtasks and both tasks together (structured interview) again support thatmost people use the "merge heuristic" (reported by 22 out of 36), and itcan be assumed that even people who were not able to verbalize theirapplied strategy/heuristic, might have used them (as can be derivedfrom eye tracking data).

3.2.4 Discussion of study II

In study II, the goal was to investigate the impact of intensive practise.Main findings of the second study are:

3.3 study iii: the role of task configuration 47

version of d2 gazes on d2-drive gazes on lane

Total 47 percent 51 percent

D2-Drive-v1.1 48 percent 51 percent



Table 2.: Study II: Amount of attention (eye gazes, AOI)

1. Practice supports the development of cognitive heuristics.

2. Task complexity seems to have an decisive influence.

3. Different task configurations require different amounts of visualattention.

4. The driving task (tracking) remains unaffected by the secondarytask.

It seems that the more practice participants have, the stronger theimpact of the task configuration. Eye tracking analyzing supports theassumption that cognitive heuristics (in particular, the "merge heuristic")are applied. For a deeper analysis of human multitasking, study IIIuses a standardized driving simulation as primary task.

3.3 study iii: the role of task configuration

Study I and II provide ample evidence that:

1. People adapt to the environment with the help of cognitive heuris-tics.

2. This behavior is more pronounced under intensive training.

The applied secondary task in the previous studies varied in terms ofcomplexity (three difficult levels) but also in terms of its task configura-tion. In other words, the task itself determines how people perform it.To further investigate the impact of task configuration, four versions ofthe "D2-Drive"-test were build which systematically offer the potentialfor cognitive heuristics to be applied. Even though the first two stud-ies contain a rather high degree of ecological validity, the systematiccontrol in a driving simulator is strictly limited. To further investigatehuman multitasking in a real life context, in the subsequent studies the"lane change task" (Mattes [2003]), hence LCT, was applied.

3.3.1 Method in study III

Study III has not been conducted in a driving simulator. Instead, astandardized tool for measuring lane derivation (LCT) was used. Anadditional feature of LCT is a (cognitive demanding) task involvedin it (i.e., to change the lane on the appropriate moment in time).LCT is a standard tool in a dynamically changing task environmentand has often been used in the context of human machine interactionin the area of driving (see Mattes [2003]). In september 2005, LCT


was submitted as measuring tool to the international organization forstandards (ISO) and for the time being, LCT is in a testing routinewith the title "ISO/DIS 26022: Simulated Lane Change Test To AssesDriver Distraction" (International Organization for Standardization,2007). Based on the fact that LCT is "freeware", easy to install andmanage, and also features a tool to analyze driving data (called LCTA),it is a perfect main task for investigating human multitasking like instudy I and II.

Participants

Forty students of Technical University Berlin (same characteristics asin study I and in study II) fulfilling the same premises (no glasses, age20-30, experienced in driving) participated study III.

Involved tasks

Figure 20.: Lane Change Task (taken from Mattes, 2003)

Figure 20 illustrates LCT: participants drive on a simulated highwayand are asked to change the lane according to predefined road signs.These signs contain three columns, two with crosses (i.e., the letter "x")and one with an arrow. This arrow indicates the position on whichparticipants have to switch. To do LCT properly, four steps are included:

1. (1) Perception (road sign)

2. (2) Action (start maneuver)

3. (3) Action (perform lane change)

4. (4) Action (keep lane)

LCT measures the lane derivation: in Fig. 21, the green (even) lineis the optimal lane calculated from the system itself based on a largeamount of driving samples. This optimal lane can be modified bychanging individual parameters (for a deeper insight, please have alook at the manual of LCT). The red, wiggly line shows the driving


behavior of a participant. The density between the two lines constitutesa value for the lane derivation. Lane derivation is the dependentvariable for the main task (driving).

LANE DERIVATION UNDER SINGLE TASKING

LANE DERIVATION UNDER DUAL TASKING

Figure 21.: How to calculate lane derivation in the LCT

LCT is a standardized tool often applied in the context of drivingstudies. For a further description of the LCT, please see Mattes [2003].In study III and IV, primary task in the scenario is the LCT, secondarytask is the D2-Drive.Secondary task in study III was a systematic variation of D2-Drive-v1.2(i.e., D2-Drive-v2.2) with the following features:

d2-drive-v3 .1 Constant row, no visual support

d2-drive-v3 .2 Changing row, no visual support

d2-drive-v3 .3 Constant row and visual support

d2-drive-v3 .4 Changing row and visual support

Visual support means that the current pattern was highlighted. Thishelps to remember the current pattern and there is no need to keep thatposition in mind. Also, a constant row allows (technically) to applythe "merge heuristic", i.e. the answers for several patterns can be keptin mind and replied combined. Please note that for all four versions,different assumptions define the hypotheses for study III (as furtherdescribed in section 3.3.2).

Design

In contrast to the previous two studies, complexity of D2-Drive wastreated as within-subjects factor. To avoid learning effects, the orderof the D2-Drive versions was balanced. The condition "single- vs.multitasking" was also treated as within-subject factor. Measure ofperformance for main task (Driving) was lane derivation, for secondarytask (D2-Drive) number of correct patterns per minutes (i.e., trial). Asbefore, the error rate in all three versions of D2-Drive approximated


zero: hence, the decrease in performance is reflected in the numberof performed patters (i.e., patterns per min). Pre- as well as post-testswere applied to investigate possible learning effects.

Procedure

Each participant had to perform the following (sequence of) steps:

1. Welcome and general introduction

2. Introduction of LCT

3. Training "Driving" (LCT)

4. Single task "Driving" (LCT, Baseline)

5. Introduction and Training "D2-Drive-v3.1"

6. Single task "D2-Drive-v3.1" (Pretest)







13. Introduction to eye tracking measuring

14. Checking of eye tracking system

15. Dual task (session) (including "D2-Drive-v3.1" - "D2-Drive-v3.4"

16. Single task "D2-Drive-v3.1" (Posttest)





3.3.2 Hypotheses for study III

The following hypotheses in this subsection were derived from resultsof the previous studies in 3. As in study I and II, hypotheses refer bothto driving behavior and to performance in the secondary task, undersingle as well as dual tasking.

Hypothesis: Independence of primary task (stability)

As in the previous two studies, in study III driving is expected toremain stable under multitasking as it is instructed as primary task.No performance decrease is expected. Same counts for lane changebehavior: as an effect of proper instruction and training, participantsshould not produce errors (i.e., assuming the constantly change to thecorrect lane).


Hypothesis: Influence of task configuration

The investigated main task in study III contains higher cognitive de-mands than the main tasks in the previous two studies. For this reason,performance in D2-Drive under multitasking is expected to stronglydecrease under multitasking in comparison to single task performance.In D2-Drive-v3.1 and D2-Drive-v3.3, the pattern row does not changewhich offers the possibility to apply the "merge heuristic": responsescan be anticipated and entered in a unit of several patterns. During thisaction (motor response), visual resources are "free" and participants canscan the lane and (visually) resume the primary task, i.e. driving.

3.3.3 Results of study III

Results are divided into driving behavior and performance in D2-Drive.

Results: Performance in driving

Under multitasking condition, driving was not affected by the sec-ondary task. Performance in LCT while D2-Drive was presented didnot differ significantly from LCT - performance under single task con-dition.

Results: Performance in D2-Drive

Figure 22.: Performance in D2-Drive in study III

Figure 22 shows that performance with D2-Drive is better if thepattern row remains the same (anticipation and merging is possible)compared to a version in which the row changes after each response.With other words, D2-Drive-v3.1 and D2-Drive-v3.3 outperform D2-Drive-v3.2 and D2-Drive-v3.4. This effect is statistically significant forsingle tasking as well as multitasking.


Results: Eye tracking

D2.Drive-vrs. Gazes atLCT

Gazes at D2-Drive

Gazes at En-vironment

D2-Drive-v3.1 60 35 5

D2-Drive-v3.2 44 51 5

D2-Drive-v3.3 71 26 3

D2-Drive-v3.4 50 46 4

The presented eye tracking data in study III support the assumptionof the applied heuristics when doing multitasking. However, a deeperanalysis and interpretation of the precise meaning and implication ofthe eye tracking data is not provided due to a missing embeddingwithin a theoretical context. For upcoming studies, i highly recommendto synchronize the formulated hypotheses with the correspondingassumptions on expected eye tracking results.

Results: Structured Interview

The structured interviews from the previous two studies were adaptedand applied in study III.

3.3.4 Discussion of study III

In study III,

• performance in primary task (LCT) is not affected by multitasking

• performance in secondary task (D2-Drive) is highly influenced bythe presence of a higher demanding primary task (LCT)

• configuration of secondary task (D2-Drive) effects performance,even though this influence reduces under multitasking

• configuration of secondary task (D2-Drive) strongly influencesmultitasking heuristic

Especially in those versions of D2-Drive in which the "merge heuris-tic" could not be applied (changing row), participants reported a (sub-jective) feeling of time pressure which is an inherent component of thestructure of the task itself. Performance under multitasking thus seemsto depend (a) on the task environment as well as (b) on situationalcomponents such as time pressure. These two issues are considered instudy IV, investigating the impact of time pressure and system design.For this reason, the next study focused on the impact of time pres-sure under human multitasking, applying the identical experimentalscenario.

3.4 study iv: amplification via time pressure

As study III illustrates, the structure of the involved tasks seems to havea tremendous influence on participants performance under multitask-ing. The way D2-Drive is designed supports or hinders performance butnevertheless, driving performance remains untouched by the appliedsecondary tasks.

3.4 study iv: amplification via time pressure 53

Figure 23.: Scenario in study IV

3.4.1 Method in study IV

Participants

Forty subjects participated in study IV. Due to technical problems,four participants were excluded from the data analysis. Therefore,sample size for this study reduced to thirty six, with an equal genderproportion. Age distribution (within a range from 20 to 40 years) wasnot analyzed as not being of deeper interest for the research questions.

Involved tasks

As in study III, primary (main) task is LCT. Secondary task again isD2-Drive. With reference to the structure of D2-Drive in the previousstudies, two versions of D2-Drive were designed by cand. Dipl.-Psych.Robert Lischke, with the following requirements:

1. Clear distinction in terms of ease of use (subjective impression)

2. Clear distinction in terms of required time (reaction time)

3. Clear distinction in terms of necessary visual attention (eye trackingdata)

Based on these requirements, the design was as follows:

d2-drive-v4 .1 Constant row of patterns and constant reply buttons

d2-drive-v4 .2 Changing row of patterns and changing reply buttons

D2-Drive-v4.1 allows participants to use the "merge heuristic". Sev-eral patterns can be scanned and replied together. Also, after a while,entering the solution without visual attention is expected due to thefact that both reply buttons (yes, no) remain at the same position.D2-Drive-v4.2 does not allow the use of the "merge heuristic" due to itsstructure: after each reply (motor action), the patterns change, i.e., a


new row with patterns appears. For instance, if a person performs thefirst pattern (at the beginning of the row), after pressing a button (yesor no) this row changes and the person now has to perform the secondpattern (which has not been shown before). The same counts for thereply buttons: after each button press, the positions of the buttons (yes,no) change, either one of them or both. Four possible configurationsfor the reply buttons are possible, as is shown in Fig. 24.

To check the three claimed preconditions (ease of use, required time,necessary visual attention), in a pretest of these two versions, eightparticipants of the graduate program prometei tested both versionsunder single task condition.

YES

NO

YES

NO

YES

NO

YES

NO

D2-DRIVE-v4.1 D2-DRIVE-v4.2

YES

NO

Figure 24.: D2-Drive in study IV

Design

Two independent variables (time pressure, complexity level in D2-Drive) were investigated, both as within-subject factors. Please notethat complexity of D2-Drive is within the secondary task and shouldnot lead to any confusion plus not be considered the same as doingmultitasking. Complexity in D2-Drive it the label for the difficultylevels in D2-Drive. As in the previous studies, dependent measureswere performance in main task (LCT) and performance in secondarytask (D2-Drive). For each participant, four conditions were applied.

Procedure

1. Welcome

2. Technical preparation (for physiological data measurement)

3. Relaxation period for participants

4. Baseline (Physiology)

5. Calibration of eye tracking equipment

6. Training main task (LCT)


7. Training secondary task (D2-Drive-v4.1)

8. Single task (D2-Drive-v4.1, Pretest)

9. Training secondary task (D2-Drive-v4.2)

10. Single task (D2-Drive-v4.2, Pretest)

11. Dual task session (driving and D2-Drive-v4.1): low time pressurecondition

12. Dual task session (driving and D2-Drive-v4.2): low time pressurecondition

13. Dual task session (driving and D2-Drive-v4.1): high time pressurecondition

14. Dual task session (driving and D2-Drive-v4.2): high time pressurecondition

15. Single task (D2-Drive-v4.1, Posttest)

16. Single task (D2-Drive-v4.2, Posttest)

17. Demographical questionnaire (age, driving experience, etc.)

the application of time pressure : Time pressure was appliedvia instruction as follows: participants were asked to consider drivingas main task (priority A), and at the same time to perform (1) as manylance changes as possible and (2) as many patterns in D2-Drive aspossible.

low time pressure Prioritize (safe) driving and perform D2-Drivewithout neglecting LCT.

high time pressure Consider driving as main task, but at the sametime aim to perform as many lane changes (LCT) and as manypatterns (D2-Drive) as possible.

3.4.2 Hypotheses for IV

Hypothesis I: The impact of time pressure

Hypothesis I claims that time pressure has a negative impact, both ondriving as well as on D2-Drive. Reported statements from study IIIrecommend to consider time pressure as one of the main situationalinfluences.

Hypothesis II: The impact of task complexity

As suggested from the quantitative and qualitative data of the previousstudies, D2-Drive-v4.1 is expected to provide a better performancecompared to D2-Drive-v4.2. This is based on (1) the possibility to applythe "merge heuristic".


Figure 25.: The influence of time pressure on driving

3.4.3 Results of study IV

Results: LCT and time pressure

Performance in LCT was not influenced by the secondary task: nosignificant difference in lane derivation was found between singletasking and multitasking. This goes in line with previous resultsfrom study I-IV. However, time pressure highly effects the drivingbehavior (p < 0.01), as can be seen in figure 27: under time pressure,lane derivation is pronounced twice as much as without time pressure.This indicates that the instruction given to the participants was takenseriously. It is also remarkable that after the driving (single task),the performance in LCT did even imporve (though not statisticallysignificant). Lane derivation was lower under multitasking withouttime pressure compared to the performance under baseline driving.This can be explained in terms of learning effects.

Results: D2-Drive

An overall comparison shows that D2-Drive-v4.1 is permanently better(in terms of number of correct patterns performed) compared to D2-Drive-v4.2. Please also note that learning effects (comparing pre-test vs.post-tests) occur for D2-Drive-v4.1 (p < 0.05) but not for D2-Drive-v4.2.Also, for D2-Drive-v4.1, number of performed patterns is significantlylower under multitasking (p < 0.05), but for D2-Drive-v4.2 there is 8sta-tistically) not difference between single- vs. multitasking. Interestingly,under multitasking, D2-Drive-v4.1 seems to receive some losses. Forthis reason, a more detailed analysis of the interplay between timepressure (factor 1) and task difficulty (factor 2) is necessary.

Overall, time pressure has a significant influence on performance ofD2-Drive (p < 0.05). Additionally, time pressure shows a stronger im-pact on D2-Drive-v4.2 (p < 0.01). This is surprisingly, also consideringthe fact that multitasking does not negatively influence D2-Drive-v4.2


Figure 26.: Performance in D2-Drive in study IV

Figure 27.: Time pressure and performance in D2-Drive

to the same amount as for D2-Drive-v4.1. Taken together all results,we can conclude that under multitasking and concurrent time pres-sure, D2-Drive-v4.2 performs quite worse and participants seem to failcompletely. A look at figure 25 confirms this assumption.

Results: Eye tracking

Eye tracking data in study IV did not deliver any additional information.However, as illustrated in figure 28, gaze analysis helped identifyingthe (development and) application of the "merge heuristic", which wasalso mentioned in verbal reports by the participants.

3.4.4 Discussion of study IV

The two main findings in study IV are:

1. Time pressure highly influences both primary as well as secondarytask.


Figure 28.: Tracking eye movements in D2-Drive)

2. Time pressure accelerates the development of cognitive heuristics.

Participants in study IV reported a strong feeling of time pressureand felt highly motivated to "optimize" their performance in termsof (1) number of lane changes (LCT), and (2) number of performedpatterns (D2-Drive).

in chapter iv the results of the four empirical studies will besummarized and a critical analysis is given.

Part IV

D I S C U S S I O N

61

4C R I T I C A L D I S C U S S I O N

4.1 scope and findings

Observations of human behavior in real life gave birth to the idea ofthis work. Modern technology as the core domain in which multipledaily tasks occur at the same time was chosen as research area for theempirical studies presented in this work. An overall review of studiesin the field (see chapter II) of human multitasking showed that

• many (if not most) psychological studies remain artificial. Espe-cially the reported task switching or PRP approaches (an excellentoverview provides Pashler [2000]) do not offer the possibility todraw conclusions about human behavior in daily life.

• applied studies in the field of human machine interaction mainlylack task repetition and systematic control (as in the studiesreported by Saluvicci [2005]). Also, theory embedment is missingquite often.

• contemporary approaches (e.g., Brumby et al. [2007]) mainlyfocus on optimization and do not fully integrate qualitative as-sumptions about human (conscious and unconscious) informationprocessing.

• the aspect of dynamically changing task environments has hardlybeen considered.

These critical remarks serve to show the starting point of this work.After an introduction (chapter I) and a general overview of the historyof human multitasking (chapter II), four applied studies in the field ofhuman machine interaction were presented (chapter III), illustrating

1. ... how people in general do human multitasking using heuristicsinstead of trying optimization (study I).

2. ... to what extend training and extensive task repetition impactshuman multitasking and applied heuristics (study II).

3. ... the importance of task configuration (study III).

4. ... what happens with human behavior given people have asubjective feeling of time pressure (study IV).

This work, for sure, is not the final cut in studying human multitask-ing. A specific domain (driving plus a secondary task) was chosen asresearch domain. Based on the empirical studies of this work, I wouldlike to summarize the findings. I supported evidence for the followingresults:

cognitive heuristics highly support human multitasking in dy-namically changing environments (study I). This was shown fortwo different main tasks (a simple driving task in a simulator andusing the lane change task).

63

64 critical discussion

extensive training does not only support the use of cognitiveheuristics but also improves overall performance (study II).

the configuration of a task turns out to be determining howpeople manage multiple, concurrent tasks (study III).

time pressure tremendously influences multitasking performance(study IV). Both primary as well as secondary task highly sufferunder instructed time pressure.

As reported in the theoretical part (chapter II), recent approacheshave been started to simulate human multitasking behavior in a cog-nitive architecture. To fully develop a multitasking mechanism forcognitive modeling was not the scope of this work. However, partiallyan implementation has been done, as will be reported in the nextsection.

4.2 cognitive modeling

To additionally confirm theoretical assumptions about the "mergeheuristic", a computational model within the cognitive architectureACT-R (Anderson [2007]) was built, based on version D2-Drive-v1.2.ACT-R is a production system, information processing is simulated viaa collection of rules. ACT-R contains "condition-action-pairs", i.e. if (acondition is met), then (do the following action) - pairs, and is a hybridcognitive architecture: it features a symbolic level (production system,sequential processes) as well as a sub-symbolic level (parallel processes,utility-based selection). Main components of ACT-R are modules (e.g.,the perceptual-motor (PM) module is the interface with the - simulationof - the real world), buffers, and a pattern matcher. Memory in ACT-Ris either declarative (facts about the world) or procedural (knowledgeabout how we do things). Many successful models have been imple-mented in ACT-R, mainly in the areas of learning and memory, problemsolving and decision making, language and communication, perceptionand attention, cognitive development and individual differences. Con-ceptual reasons for choosing ACT-R in my work are its assumptionson human memory, the psychological plausibility and the interactionwith environment. Before I introduce results of the ACT-R model ofD2-Drive, let me give a short explanation of cognitive modeling, asillustrated in Fig. 29:

Figure 29.: Cognitive modeling (following Taatgen [1999])

According to Werner H. Tack (personal communication), cognitive mod-eling is meant to be a "simulation of human problem solving". Tack

4.3 design recommendations 65

[1995] refers to it as "defining symbol structures for specific cognitivetasks". Cognitive modeling is not a metaphor for the mind. Its goal isthe prediction of human behavior. Mental task processes are specifiedand cognitive tasks are performed by computing (Lewis [1999]).A cognitive architecture 1 contains theoretical assumption (theory)about cognitive processes. In combination with task knowledge, a taskmodel is build and derived. This task model produces performancedata (see Fig. 29). Traditional measures are time to perform the task,accuracy in the task, or neurological (fMRI) data. From another side,a (psychological) experiment produces data as well. This data is ana-lyzed and compared to the data produced by the model. The matchingbetween model data and empirical data defines the goodness of themodel.Both on a micro-level (performing of a single pattern) as well as ona macro-level, the model describes the processing how participantsperformed D2-Drive.

YES NO

YES

NO

pattern processingat the beginning

pattern processingafter training

implicitlearning

READ upper

READ lower

READ middle

READ upper

READ lower

READ middle

d-pattern p-pattern

READ upper

READ lower

READ middle

READ upper

READ lower

READ middle

REPLY (p)REPLY (d)

REPLY (p)REPLY (d)

Figure 30.: Processing of a single pattern in D2-Drive

Fig. 30 shows the basic assumptions for the corresponding ACT-Rmodel (see also Kiefer et al. [2006], for more details). We generallydistinguish "d-patterns" (patterns containing the letter "d") from "p-patterns" (patterns containing the letter "p"). In both cases, participantsstart on one position (i.e., the middle, which is at the same time theletter-component in the pattern). Next, they scan the upper part andthen the lower part. After a while, implicit learning takes place, partic-ipants "understand" that with "p-patterns", the last two steps are notnecessary. This leads to shorter reaction times for p-patterns (see leftpart of Fig. 31). The implemented Act-R model fits the data pretty well,especially performance of "p-patterns".

4.3 design recommendations

Visiting national and international conferences to present preliminaryresults of my work, it happened quite often that after my presenta-

1 an algorithm that simulates a non-linear theory of cognition, based on Taatgen [1999]


Figure 31.: Modeling of D2-Drive

tion/talk, people coming from different disciplines (e.g., designers,engineers) came to me asking me about advice. "What would yourecommend me if i tell you that i am planning to develop an in-vehileinformation system?" Or: "Which mistakes can i avoid when I designan information system which should be used in daily life?"Derived from the results described in the previous chapter and fromthe insight i won during the last years doing theoretical and practicalwork in that area, here is my advice. Some of the recommendationsmight be helpful ideas, with the hope to prevent from avoidable errorsin multitasking situations:

do not force people to multitask! Giving advice how to per-form two tasks concurrently not only prevents from developingindividual strategies or heuristics, but also leads to a direction inprocessing which might be suboptimal for participants‘ individ-ual style. Learning, from a psychological perspective, happensconsciously or unconsciously. Remember that some participantsin the previous four studies could not even report that they usedthe "merge heuristic", i.e., they learned implicitly i instead ofrule-based. Instructions do not allow such a possibility.

reduce complexity! The more complex a system, i.e. the higherthe functionality, the more difficult it is to comprehend. Wesaw in the first two studies that D2-Drive-v1.3 (and D2-Drive-v2.3 consequently) performed worst. Please keep in mind that amore demanding system requires a longer learning period untilindividual steps can be taken without mental load or cognitivedemand. What is the benefit of a extremely functional mobilephone if it takes you months (or years) to understand the structureand the navigation? Cognitive demanding tasks produce humanerrors, so always ask yourself: is the benefit worth the effort?

support familiarity! As study III and IV show, a dynamicallychanging secondary task requires visual (re-)orientation as well ascognitive (resumption) costs. Part of a task needs to be resumed,but the mental set has changed (i.e., the positions of the answer-buttons). Familiarity does not require many cognitive resources.Therefore, a system which is easy to understand, learn and be-come familiar with, can be accessed immediately and promoteshuman multitasking.

apply prospective design! Do not mainly focus on results fromstudies in literature. Even if prominent theories advice you to

4.4 criticism and outlook 67

do this or that (this work is not an exception, by the way), try toinvestigate the human-machine interaction already in the earlydevelopment phases of your planned technical systems.

These recommendations mainly result from an overall impression ofthe studies in the previous chapter, supported by quantitative (perfor-mance data) as well as qualitative (interview data) measures.

4.4 criticism and outlook

Aim of the presented work was to approach human behavior in multi-tasking scenarios from a human machine interaction perspective. In-cluding four copious empirical studies, it can only give an extract ofrelevant issues in context of human multitasking. When i started in2005, i shortly realized that investigating human multitasking and thenecessary, connected domains feels like opening Pandoras box1.

Figure 32.: Pandoras box, found on: www.icarusgirl.blogspot.com

For this work can never be complete by going into all details, threeissues are critically discussed:

1. The role of (prospective) memory in human multitasking (theo-retical aspect)

2. Domain (in-)dependence (practical aspect)

3. Need for a (computational) model of human multitasking (mod-eling aspect)

The following sections concentrate on these three issues.

1 In Greek mythology, Pandora was the first woman. Each god helped create her by givingher unique gifts. Zeus ordered Hephaestus to mold her out of Earth as part of thepunishment of mankind for Prometheus theft of the secret of fire, and all the gods joinedin offering this beautiful evil seductive gifts. According to the myth, pandora opened ajar in modern accounts referred to as Pandoras box, releasing all the evils of mankind(greed, vanity, slander, envy, pining) leaving only hope inside once she had closed itagain.


4.4.1 The role of memory in human multitasking

Each task interrupted becomes a prospective task?In study I-IV, design and complexity of the applied secondary task (i.e.

D2-Drive, developed by Kiefer et al. [2006]) have been systematicallyanalyzed and varied. However, in study I and II, the same main task(lane keeping on a rather monotonous street) was applied. To increasingcomplexity and cognitively enrich the main task, study III and IV focuson LCT as main task. In a recent study by Soyak, a modification ofLCT based on the hypotheses in this study throws some new light tothe scenario. Soyak points out the importance of prospective memory("remembering to remember", Winograd [1988]). In the area of taskinterruption, Dodhia and Dismukes claim that "a task interruptedbecomes a prospective task".

Figure 33.: Modification of LCT in a prospective task study

McDaniel and Einstein [2000] distinguish between two kinds ofprospective memory (PM), namely:

event-based pm Recalling an action or an intention triggered by astimulus ("event"), e.g. receiving a reminder-email ("cue") remindsto submit a paper ("intention")

time-based pm Recalling an action or an intention triggered by atime, e.g. watching the news in television at 8pm.

Based on McDaniel and Einstein [2000], Soyak investigated the impactof disruptions on prospective memory performance. Main task inher study was a modified version of LCT (see Fig.33: participantswere asked to keep in mind a verbally given city name (Hamburg,Berlin, München, Stuttgart, Köln, Leipzig) and change the lane at themoment the road sign with this name appears. Soyak showed a negativeinfluence of disruptions on successful prospective task performance.Delayed disruptions require longer reaction times on the target cue.In her study, Soyak used a more demanding, cognitively enriched maintask and concurrently the two versions of D2-Drive mentioned in studyIV of the previous chapter. Eye tracking data were recorded but notyet analyzed (purpose of her work was not on cognitive heuristicsunder multitasking). Nevertheless, it would be of interest to look

4.4 criticism and outlook 69

for participants processing in the two tasks individually and undermultitasking. We can assume the following:

1. Modified LCT require to many cognitive resources and heuristicsfor D2-Drive cannot be applied.

2. Participants use the "merge heuristic" in a reduced frequency.

3. Due to the new task configuration, participants develop newheuristic(s) adaptively.

In context of human multitasking, the author highlights the im-portance to take memory aspects under deeper consideration. Theimportance of the prospective task seems to be a core aspect in eachmultitasking scenario.

4.4.2 Domain independence

All of the four presented studies in this work include a driving task(as main task) in the multitasking scenario. The question, hence iswhether this multitasking behavior, the application of heuristics andinformation processing, can be transferred to other situations in reallife in which people use technical systems while doing a continuous"task" (e.g., walking on the street). Studies by Antti Oulasvirta from theHelsinki Institute of Technology (HIIT) (Oulasvirta and Blom [2008],Oulasvirta et al. [2007], Oulasvirta and Saariluoma [2006], Oulasvirtaet al. [2005], Oulasvirta and Saariluoma [2005]) show that in fact, in var-ious situations, an ongoing task needs to be interrupted and resumed.Oulasvirta provides both empirical data (eye tracking) as well as aqualitative analysis to explain how multitasking in these scenarios isexplained. Especially the area of mobile computing is a promising fieldfor further research. Modern technology for us is a constant challengeto develop fast and frugal (conscious or unconscious) "heuristics" toadapt to a dynamically changing environment. This direction in thearea of human machine interaction studies in this direction will becomeof increasing importance in future research.

4.4.3 Need for a computational model of human multitasking

In this chapter, a computational model of the applied secondary taskwas introduced and explained. However, a general model for humanmultitasking is still missing (though it was not meant to be part ofthis work). Saluvicci [2005] proposes an approach in which he aimsto incorporate human multitasking in cognitive modeling, namelywithin the ACT-R architecture (Anderson et al. [2004], Anderson [2005].Salvucci proposes an general executive which is

• an architectural mechanism

• dependent on time

• sensitive to goal representations

Fig34 represents core mechanisms within this idea. Main features ofthe general executive proposed by Saluvicci [2005] are:


Figure 34.: Overview of multitasking general executive proposed by Saluvicci[2005]

• A cognitive processor manages the concurrency of multiple goalsat the same time.

• As in earlier frameworks of ACT-R, only one goal can be executedat the same time.

• Goal switching is moderated by heuristics.

• Urgency defines when to switch to which goal.

Salvucci mentions "natural breaking points" necessary for interleav-ing tasks. He proposes two core heuristics to decide when to switchbetween goals, namely

the iterating heuristic Salvucci gives the example of a task witha duration of 100 sec, assuming 50msec per production rule, soin sum 2.000 rules to fire. When the model returns to a previousfired rule, task switching should be proposed at this point. Thenext iteration is initiated by a new goal. Especially for modelswith a long duration in terms of execution time, this heuristicbecomes plausible.

the blocking heuristic Salvucci mentions "significant time" andthe problem how to decide about that. He illustrates that for per-ceptual motor actions (PM) in particular, ACT-R has to wait untilan action is done. In such a case, the blocking heuristic proposesto create a new goal which gives permission to a secondary taskto intercede.

4.5 fmri studies on multitasking

The reported work did not consider functional magnetic resonanceimaging (fMRI) scans. However, some studies (e.g., Leber et al. [2008])have revealed that superior multitasking performance is correlated

4.6 popular stereotypes about multitasking 71

with higher basal ganglia, anterior cingulate cortex, prefrontal cortex,and parietal cortex activity. Philippe Peigneux, Professor of ClinicalNeuropsychology in Brussels, even talks of a multitasking mind (pub-lished in April 2006 on seedmagazine.com), meaning that even whenwe sleep, our mind works constantly, processing several tasks concur-rently. In future, neuro-scientific studies will dive deeper into the areaof multitasking.

4.6 popular stereotypes about multitasking

4.6.1 Multitasking and happiness

Multitasking has been criticized as a hindrance to completing tasks orfeeling happiness. Timothy Ferriss argues that one should rarely domultitasking and should instead devote full attention to completinga very small set of defined goals (taken from the interview "I receive500 to 1000 emails per day", published in The Economist on 2008-04-04).Barry Schwartz has noted that, given the media-rich landscape of theInternet era, it is tempting to get into a habit of dwelling in a constantsea of information with too many choices, which has been noted tohave a negative effect on human happiness.

B I B L I O G R A P H Y

E.M. Altmann and J.G. Trafton. Memory for goals: An activation-basedmodel. Cognitive Science, 26:39–83, 2002. (Cited on pages 24, 26, 29,and 35.)

J. R. Anderson. Human symbol manipulation within an integratedcognitive architecture. Cognitive Science, 29(3):313–341, 2005. (Citedon page 69.)

J. R. Anderson, D. Bothell, M. D. Byrne, S. Douglass, C. Lebiere, and Y .Qin. An integrated theory of the mind. Psychological Review, 111(4):1036–1060, 2004. (Cited on page 69.)

J.R. Anderson. Acquisition of cognitive skill. Psychological Review, 89:369–403, 1982. (Cited on page 19.)

J.R. Anderson. How Can the Human Mind Occur in the Physical Universe?Oxford University Press, New York, 2007. (Cited on pages 29 and 64.)

R. Borger. The refractory period and serial choice reactions. Quarterlyjournal of Experimental Psychology, 15:1–12, 1963. (Cited on page 18.)

R. Brickenkamp. Test D2, Aufmerksamkeits-Belastungs-Test. HogrefeVerlage, Schweiz, 1992. (Cited on pages 36, 37, 39, and 44.)

C. Brod. Technostress: The human cost of the computer revolution. Addison-Wesley Reading, MA, 1984. (Cited on page 8.)

D.P. Brumby, D.D. Saluvicci, and A. Howes. Dialing while driving?A bounded rational analysis of concurrent multi-task behavior. InProceedings of the 8th International Conference on Cognitive Modeling,Michigan, 2007. Ann Arbor, Michigan, USA. (Cited on pages 35

and 63.)

E. C. Cherry. Some experiments on the recognition of speech ears.Journal of the Acoustical Society of America, 25, 1953. (Cited on page 10.)

J.P. Chin, V.A. Diehl, and K.L. Norman. Development of a tool mea-suring user satisfaction of the human-computer interface. In ACMSIGCHI, volume 88, pages 213–218, 1988. (Cited on page 10.)

G. Gigerenzer and R. Selten. Bounded rationality: The adaptive toolbox.the MIT Press, 2002. (Cited on page 30.)

T. Gillie and D. Broadbent. What makes interruptions disruptive? astudy of length, similarity, and complexity. Psychological Research, 50:243–50, 1989. (Cited on page 23.)

P. Green. Estimating compliance with the 15-second rule for driver-interface usability and safety. In Proceedings of the Human Factors andErgonomics Society 43rd Annual Meeting, Santa Monica, 1999. HumanFactors and Ergonomics Society. (Cited on page 7.)

A. G. Greenwald. On doing two things at once: I. time-sharing as afunction of ideomotor compatibility. Psychological Review, 100:52–57,1972. (Cited on page 20.)

73

74 bibliography

A. G. Greenwald and H. Shulman. On doing two things at once:Ii. elimination of the psychological refractory period. PsychologicalReview, 101:70–76, 1973. (Cited on page 20.)

Nowlis H. H. The influence of success and failure on the resumption ofan interrupted task. Journal of Experimental Psychology: General, 28(4):304–325, 1941. (Cited on page 23.)

E. Hazeltine, D. Teague, and R. B. Ivry. Simultaneous dual-task per-formance reveals parallel response selection after practice. Journalof Experimental Psychology: Human Perception and Performance, 28(3):527–545, 2002. (Cited on page 20.)

A. Howes, R.L. Lewis, A. Vera, and J. Richardson. Information Re-quirements Grammar: A theory of the structure of competence forinteraction. In Proceedings of the 27 th Annual Meeting of the CognitiveScience Society, Stresa, Italy, 2005. (Cited on pages 26 and 29.)

T. Jersild. Mental set and shift. Archives of Psychology, 1927. (Cited onpages 16, 17, and 28.)

I. Juvina and N.A. Taatgen. Modeling control strategies in the N-backtask. In Proceedings of the eight International Conference on CognitiveModeling, pages 73–78, New York, 2007. Psychology Press. (Cited onpage 26.)

D. Kahneman. Attention and effort. NJ: Prentice-Hall, Englewood Cliffs,1992. (Cited on pages 26 and 27.)

D. Kahneman and A. Tversky. On the psychology of prediction. Psy-chological Review, 80:237–251, 1973. (Cited on page 30.)

J. Kiefer, M. Schulz, D. Schulze-Kissing, and L. Urbas. Multitasking-Strategien in der Mensch-Maschine-Interaktion. MMI-Interaktiv, 11:26–42, 2006. (Cited on pages 65 and 68.)

Y. Kushleyeva, D.D. Saluvicci, and F.J. Lee. Deciding when to switchtasks in time-critical multitasking. Cognitive Systems Research, 6:41–49,2005. (Cited on pages 25 and 26.)

A.B. Leber, N.B. Turk-Browne, and M.M. Chun. Neural predictors ofmoment-to-moment fluctuations in cognitive flexibility. Proceedingsof the National Academy of Sciences, 105(36):13592, 2008. (Cited onpage 70.)

F.J. Lee and N.A. Taatgen. Multi-tasking as Skill Acquisition. In Pro-ceedings of the twenty-fourth annual conference of the cognitive sciencesociety, Fairfax, VA, 2002. ahwah, NJ: Erlbaum. (Cited on pages 19,21, and 29.)

R.L. Lewis. Cognitive modeling, symbolic. MIT Press, Cambridge, MA,1999. (Cited on page 65.)

S. Mattes. The lane-change-task as a tool for driver distraction evalua-tion. In Proc. of IGfA, 2003. (Cited on pages 47 and 49.)

J.S. McCarley, M. Vais, H. Pringle, A.F. Kramer, D.E. Irwin, and D.L.Strayer. Conversation disrupts scanning and change detection incomplex visual scenes. Human Factors, 46, 2004. (Cited on page 7.)

bibliography 75

M.A. McDaniel and G.O. Einstein. Strategic and automatic processesin prospective memory retrieval: A multiprocess framework. AppliedCognitive Psychology, 14:127–144, 2000. (Cited on pages 25 and 68.)

B. McElree. Working memory and focal attention. Journal of ExperimentalPsychology: Learning, Memory and Cognition, 27:817–835, 2001. (Citedon page 26.)

D. C. McFarlane and K. A. Latorella. The scope and importance ofhuman interruption in human-computer interaction design. Human-Computer Interaction, 17(1):1–61, 2002. (Cited on page 23.)

D.C. McFarlane. Interruption of People in Human-Computer Interaction(Dissertation). George Washington University, Washington, 1998.(Cited on page 23.)

N. Meiran. Reconfiguration of processing mode prior to task per-formance. Journal of Experimental Psychology: Learning, Memory, andCognition, 22:1423–1442, 1996. (Cited on page 17.)

N. Meiran and A. Daichman. Advance task preparation reduces taskerror in the cuing task-switching paradigm. Memory and Cognition,33(7):1272–1288, 2005. (Cited on page 20.)

R.F.I. Meuter and A. Allport. Bilingual language switching in naming:Asymmetrical costs of language selection. Journal of Memory andLanguage, 40:25–40, 1999. (Cited on page 9.)

D. E. Meyer and D. E. Kieras. A computational theory of executivecognitive processes and multiple-task performance: Part 1. basicmechanisms. Psychological Review, 104:3–65, 1997a. (Cited on pages 18,20, and 23.)

D. E. Meyer and D. E. Kieras. A computational theory of executivecognitive processes and multiple-task performance: 2. accounts ofpsychological refractory-period phenomena. Psychological Review,104:749–791, 1997b. (Cited on pages 18, 20, and 23.)

S. Monsell. Control of mental processes, pages 93–148. Erlbaum (UK),Hove, E. Sussex, 1967. (Cited on page 20.)

D.A. Norman. Affordance, conventions, and design. interactions, 6(3):38–43, 1999. (Cited on page 10.)

A. Oulasvirta and J. Blom. Motivations in personalisation behavior.Interacting with Computers, 20(1):1–16, 2008. (Cited on page 69.)

A. Oulasvirta and P. Saariluoma. Long-term working memory andinterrupting messages in human-computer interaction. Behavior andInformation Technology, 23(1):53–64, 2005. (Cited on page 69.)

A. Oulasvirta and P. Saariluoma. Surviving task interruptions: In-vestigating the implications of long-term working memory theory.International Journal of Human-Computer Studies, 64(10):941–961, 2006.(Cited on page 69.)

A. Oulasvirta, L. Kärkkäinen, and J. Laarni. Expectations and memoryin link search. Computers in Human Behavior, 21(5):773–789, 2005.(Cited on page 69.)

76 bibliography

A. Oulasvirta, R. Petit, M. Raento, and S. Tiitta. Interpreting andacting on mobile awareness cues. Human-Computer Interaction, 22

(1,2):97–135, 2007. (Cited on page 69.)

M. Ovsiankina. Die wiederaufnahme unterbrochener handlungen.Psychologische Forschung, 11(1):302–379, 1928. (Cited on page 23.)

A. M. Owen, K. M. McMillan, A. R. Laird, and E. Bullmore. N-backworking memory paradigm: A meta-analysis of normative functionalneuroimaging studies. Human Brain Mapping, 25:46–591, 2005. (Citedon page 26.)

H. Pashler. Task switching and multitask performance. MA: MIT Press,Cambridge, 2000. (Cited on pages 17, 18, 20, 22, 28, 29, 35, and 63.)

H. Pashler. Processing stages in overlapping tasks: Evidence for a cen-tral bottleneck. Journal of Experimental Psychology: Human Perceptionand Performance, 10:358–377, 1984. (Cited on page 20.)

H. Pashler. Doing two things at the same time. American Scientist,Jan-Feb:47–56., 1993. (Cited on pages 20, 21, and 35.)

J. Rasmussen. Skills, rules, and knowledge; signals, signs and symbols,and other distinctions in human performance models. IEEE Transac-tions on Systems, Man, and Cybernetics, 13(3):257–266, 1983. (Cited onpage 19.)

R. Rogers and S. Monsell. The costs of a predictable switch betweensimple cognitive tasks. Journal of Experimental Psychology: General,124:207–231, 1995. (Cited on pages 5, 9, and 20.)

J. S. Rubinstein, D. E. Meyer, and J. E. Evans. Executive control ofcognitive processes in task switching. Journal of Experimental Psychol-ogy: Human Perception and Performance, 27(4):763–797, 2001. (Cited onpages 9 and 17.)

E. Ruthruff, J. C. Johnston, M. Van Selst, S. Whitsell, and R. Rem-ington. Vanishing dual-task interference after practice: Has thebottleneck been eliminated or is it merely latent? Journal of Exper-imental Psychology-Human Perception and Performance, 29(2):280–289,2003. (Cited on page 20.)

D.D. Saluvicci. A multitaksing general executive for compound contin-uous tasks. Cognitive Science, 29:457–492, 2005. (Cited on pages 6, 16,21, 22, 23, 28, 29, 35, 63, 69, and 70.)

D. D. Salvucci. An integrated model of eye movements and visualencoding. Cognitive Systems Research, 1(4):201–220, 2001. (Cited onpage 23.)

W. Schneider and R.M. Shiffrin. Controlled and automatic human infor-mation processing: I. detection, search, and attention. PsychologicalReview, 84(1):1–66, 1977. (Cited on pages 18, 19, and 44.)

E. H. Schumacher, T. L. Seymour, J. M. Glass, D. E. Fencsik, E. J.Lauber, D. E. Kieras, and D. E. Meyer. Virtually perfect time sharingin dual-task performance: Uncorking the central cognitive bottleneck.Psychological Science, 12(2):101–108, 2001. (Cited on pages 21, 28, 29,and 44.)

bibliography 77

R. E. Smith. The cost of remembering to remember in event-basedprospective memory: Investigating the capacity demands of delayedintention performance. Journal of Experimental Psychology: LearningMemory and Cognition, 29(3):347–361, 2003. (Cited on page 25.)

R. E. Smith and U. J. Bayen. A multinomial model of event-basedprospective memory. Journal of Experimental Psychology: LearningMemory and Cognition, 30(4):756–777, 2004. (Cited on page 25.)

M. H. Sohn and J. R. Anderson. Task preparation and task repetition:Two-component model of task switching. Journal of ExperimentalPsychology: General, 130:764–778, 2005. (Cited on page 20.)

A. Spector and I. Biederman. Mental set and mental shift revisited.American Journal of Psychology, 89:669–679, 1976. (Cited on page 17.)

P. Steel. The nature of procrastination: A meta-analytic and theoreticalreview of quintessential self-regulatory failure. Psychological Bulletin,133(1):65, 2007. (Cited on page 5.)

B.S. Stewart, C.F. Liaw, C.C. White III, U.S.W. Technol, and CO Denver.A bibliography of heuristic search research through 1992. IEEE Trans-actions on Systems, Man and Cybernetics, 24(2):268–293, 1994. (Citedon page 30.)

D. L. Strayer and W. A. Johnston. Driven to distraction: Dual-taskstudies of simulated driving and conversing on a cellular phone.Psychological Science, 12:462–466, 2001. (Cited on page 7.)

N. Taatgen. Learning without Limits: From Problem Solving towards aUnified Theory of Learning (PhD Thesis). Rijksuniversiteit Groningen,Groningen, 1999. (Cited on pages 64 and 65.)

N. A. Taatgen, D. Huss, D. Dickison, and J. R. Anderson. The acqui-sition of robust and flexible cognitive skills. Journal of ExperimentalPsychology: General. (Cited on page 18.)

N. A. Taatgen, D. Huss, and J. R. Anderson. How cognitive modelscan inform the design of instructions. British Journal of Psychology, 40:23–40, 2006. (Cited on page 18.)

N.A. Taatgen. Modeling parallelization and speed improvement in skillacquisition: from dual tasks to complex dynamic skills. CognitiveScience, 29:421–455, 2005. (Cited on pages 16, 28, 29, 39, 40, and 44.)

W. H. Tack. Wege zu einer differentiellen kognitiven Psychologie, volume 2,pages 172–185. Hogrefe, Göttingen, 1995. (Cited on page 64.)

C.W. Telford. The refractory phase of voluntary and associative re-sponses. Journal of Experimental Psychology, 14:1–36, 1931. (Cited onpages 17 and 28.)

J. G. Trafton, E. M. Altmann, D. P. Brock, and F. E. Mintz. Preparingto resume an interrupted task: Effects of prospective goal encodingand retrospective rehearsal. International Journal of Human-ComputerStudies, 58:583–603, 2003. (Cited on page 24.)

L. Urbas, S. Dzaack J. Schulze-Kissing, D.and Leuchter, J. Kiefer,and M. Heinath. [Programmbeschreibung D2-Drive-Aufmerksamkeitstest(Manual for D2-Drive Test of Attention). 2005.

78 bibliography

A. van Bergen. Task Interruption. North-Holland, Amsterdam, 1968.(Cited on page 23.)

M. Vince. Rapid response sequences and the psychological refrac-tory period. British Journal of Psychology, 40:23–40, 1949. (Cited onpage 17.)

A.T. Welford. The psychological refractory period and the timing ofhighspeed performance. British Journal of Psychology, 43:2–19, 1952.(Cited on pages 17 and 18.)

C.D. Wickens. Multiple resources and performance prediction. Theoreti-cal Issues in Ergonomics Science, 3(2):159–177, 2002. (Cited on pages 27

and 39.)

C.D. Wickens. Multiple Resource Time Sharing Mode, pages 77–105. CRCPress, London, 2004. (Cited on pages 27 and 28.)

C.D. Wickens. Multiple Resource Model of Human Performance: impli-cations for Display Design. In AGARD/NATO Proceedings, Williams-burg, VA., 1984. AGARD. (Cited on page 27.)

C.D. Wickens and Y. Liu. Codes and modalities in multiple resources: asuccess and a qualification. Human Factors, 30:599–616, 1988. (Citedon pages 27 and 28.)

E. Winograd. Some observations on prospective remembering, volume 2,pages 348–353. MA: MIT Press, Chichester: Wiley, 1988. (Cited onpages 25 and 68.)

N. Yeung and S. Monsell. Switching between tasks of unequal fa-miliarity: The role of stimulus-attribute and response-set selection.Journal of Experimental Psychology-Human Perception and Performance,29(2):455–469, 2003. (Cited on page 9.)

B. Zeigarnik. Das behalten erledigter und unerledigter handlungen.Psychologische Forschung, 9:1–85, 1927. (Cited on page 23.)

B. Zeigarnik. On finished and unfinished tasks. Humanities press, NewYork, 1967. (Cited on page 23.)

D E C L A R AT I O N

I hereby declare that:

• I autonomously carried out the PhD-thesis entitled "Multitaskingin HMI". All third party assistance has been enlisted.

• My submission as a whole is not substantially the same as anythat I have previously made or am currently making, whether inpublished or unpublished form, for a degree, diploma, or similarqualification at any university or similar institution.

The thesis has not been submitted elsewhere for an exam, as thesisor for evaluation in a similar context.

Berlin 2010D 83, Tag der wissenschaftlichen Aussprache: 05.10.2009

Dipl.-Psych. Jürgen Kiefer

AA P P E N D I X

a.1 appendix : structured interview

The following questions (qualitative interview) were put to the partici-pants after the experiment:

• Which of the two tasks did you perform with prioritization?

• Which of the two tasks did you experience as more difficult?

• Please explain how you proceeded the both tasks?

• On a scale from 1 to 5 (where 1 is the lowest and five is thehighest value), how much mental fatigue during the completeexperiment?

• Can you in detail describe how you performed the driving taskas a single task?

• Can you in detail describe how you performed the pattern taskas a single task?

• Can you in detail describe how you performed the combinationof both tasks?

In addition to these questions, for all of the four reported studies, averbal qualitative interview on the task processing of all participantswas applied.

81

A Study on the Behavior under Multitasking Conditions in a ...After introducing into the topic labeled as "human multitasking", embedded in situation of routine life, ... task (a test

Documents