Human Technology Interaction: Methods · TIFF (Uncompressed) decompressor are needed to see this picture. Cashpoint Interface QuickTime™ and a TIFF (Uncompressed) decompressor are

1

Human Technology Interaction: Methods

Cato A. Bjørkli, Associate professor, UIO

http://www.regjeringen.no/nb/dep/jd/dok/NOUer/2008/nou-2008-8.html?id=504445

«Undersøkelseskommisjonen skal foreta de undersøkelser den finner nødvendig for å kartleggehendelsesforløp og årsaksfaktorer, og utrede forhold avbetydning for å forebygge at en slik ulykke kan skje i fremtiden.

Kommisjonen har i egen regi foretatt en MTO (Man/Technology/Organisation)-analyse med gjennomgang av hendelsesforløpet fra fartøyet bleplanlagt bygget til forliset for å avdekkesikkerhetsbarrierer og brudd på slike.

p.127-133: Anbefalinger

En ulykkeshendelse er ofte et resultat av flere samvirkende årsaker –både direkte og indirekte årsaker, det vil si forhold som ikke direktehar medført at hendelsen har funnet sted, men har vært medvirkendetil hendelsen eller til at denne ikke ble avverget.

p.78-87: Hendelser forut og under forliset

Background for the lecture

• S.G.Charlton (1996) Mental Workload Test ...

• B.Nardi (1997) The Use of Ethnography Methods...

• (X.Faulkner (2000) Usability Engineering ...)

2

Art is ‘I’

Science is ‘we’

Claude Bernard

(why method?)

-- method --

a way of doing something, especially a systematic way; implies an orderly logical arrangement (usually in fixed steps)

-- science --

ability to produce solutions in some problem domain

approximate d(f)

The Science ofHuman Technology Interaction

(MMI)

What characterizes a good product?

What characterizes a safe system?

What characterizes a efficient system?

How do we know that something is what we designed it to be?

3

Field of Study

Product

Social

Individual

Systems

Your examples?

Life of Products

• How do a product come to life? • How does an artifact come in to existence?• When is a product completely designed?

CS60/65

Know the user

Know the taskKnow thecontext

Usability goalsetting

Know the existingproducts

Design

PrototypingEvaluation

Follow upstudies

ANALYSE DESIGN

COGNITIVEENGINEERING

USABILITYDESIGN

INTERACTIONDESIGN

Methods for all phases of product cycles

4

Evaluation & Redesign

Innovation & Design

Methods underlying all aspects of HTI

A way of gaining insight into the system studied

What insight is useful?

What insight is not useful?

Method is choices

How you approach the phenomenon is guiding for

the methods you chose to investigate it.

So, beyond the type of project, you must consider your perspective

to the system i question.

“... theoretical interpretations serve to provide much of the direction for the research questions and methodologies.”S.G.Charlton, 1996, p.181.

Traditional Information Processing

Embodiment

Social Matrix

Political

Ergonomics

?Engineering

Sociology

Ethical

5

Good news / Bad news

Fact:There are a number of ways for modeling human –technology interaction and equally many ways of measuring it.

Consequence:You must be able to provide arguments for your specific approach and make sure that it corresponds with your choice of measurement.

Considerations

• What system?– Complex– Product– Social

• What phase and focus of your project?– Evaluation of existing system– Redesign– Innovation & Development

• What theoretical perspective?– Traditional Infor Pro– Embodiment– Cognitive Engineering

Concepts and techniques

6

W O R K L O A DCharlton, S. (1996) Mental Workload Test and Evaluation

7

Workload

• Quantitative measurement of a specific transitory subjective state or temporal feature of mental processes describing human technology interaction.

• What is ‘workload’? – Definition? – No 1-to-1 coupling between performance and workload– Multifaceted?– Related to performance– Related to motivation– Related to strategies/timing– Situatedness / Contextuality / Lab versus world– No direct external manifestation

Workload is a multidimensional concept, and should not be estimated with single indicators, but with multivariate measures

NASA TLXSWAT

Yet, some univariateglobal measures yieldgood information

Strengths and weaknesses ?

How to describe a

house

Height? Weight?Shape?

Describe a house

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

8

1. Sensitivity: Can the test show changes in workload?

2. Intrusiveness: Do the test disturb the operator during work?

3. Diagnostical: Which aspect of several is the most important?

4. Convenience: How much logistics are necessary?

5. Relevance: How well do the measurement fit the work?

6. Operator acceptance: Is the test ok for the operator?

Criteria for good workload tests

Types of Tests: Overview

Projective tests• Timeline Analysis• Scheduling Theory• Pro-SWAT

Empirical tests• Modified Cooper-Harper (MCH)• SWAT• NASA TLX• Crew Status Survey

Projective Tests

9

PROJECTIVE TESTS

Time available

T-0 T-N

a b c

Time-Line Analysis and Scheduling

Task Time versus Time Available

Task Difficulty

Demanded skills

Frequencies

Percept/Motor channel

Criteria: task time < 75% of total time

PROJECTIVE TESTS

Pro-SWAT

Task Enactment w/prototype

Detailed Briefing

SWAT scores

Debriefing

(During prototyping)

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Cashpoint Interface

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Cellphone Interface

Know the user

Know the taskKnow thecontext

Usability goalsetting

Know the existingproducts

Design

PrototypingEvaluation

Follow upstudies

ANALYSE DESIGN

COGNITIVEENGINEERING

USABILITYDESIGN

INTERACTIONDESIGN

Where do projective tests fit in?

10

Empirical Tests

Empirical Techniques

“... collection of data from one or more subjects actually

performing the task(s) of interest.”

Q: What is relevant data?

Types of Data

• Primary task: Objective variables: – Speed, lateral position, observable

performance

• Secondary task: Additional tasks– Mental calculus, counting backwards

• PsychoPhys: Changes in patterns– Heart, eye, startle, skin conducance

• Subjective methods– Self-reports (NASA, SWAT, etc)

Workload by Inference

Workload by Measure

?

11

Examples from different projects

Example I: Adaptive Front Lights

The objectives of the CLARESCO project is to carry out an extensive assessment of advanced lighting technologies by studying advanced front-lighting systems for both carand truck lighting.

Example I: Adaptive Front Lights

15

20

25

30

35

40

45

50

55

-80 -70 -60 -50 -40 -30 -20 -10 0 10 20 30

Distance from object(m)

Spee

d(km

/h)

AFSNon-AFSDay

Objective variablesSpeed, position

PhysiologicalHeart, eye movement

Subjective variablesNASA TLX, Acceptance

Secondary TaskAlarm detection

12

Example II: Invehicle Information System

Objective variablesSpeed, position

Subjective variablesNASA TLX

GUI vs Speech Recognition

Example III: HSC Sailing

How can this be approached?

-Workload?

-Performance?

-Secondary task?

-Primary Task?Focus:

Design of electronic charts and other navigation equipment

Example III: HSC Sailing

PhysiologicalHeart, Skin Conduct

Subjective variablesNASA TLX, SWAT

Secondary TaskAdHoc Attention Task

Video RecordingLink Analysis

13

Know the user

Know the taskKnow thecontext

Usability goalsetting

Know the existingproducts

Design

PrototypingEvaluation

Follow upstudies

ANALYSE DESIGN

COGNITIVEENGINEERING

USABILITYDESIGN

INTERACTIONDESIGN

What kind of data do we need?

Specific empirical tests

Assumes a three dimensional workload construct

1. Time Load 2. Mental Effort 3. Psych Stress

Procedure:1) Normalize for subjects (scale development)2) Workload Scores (data collection)

Evaluation1) Subscales2) Total score in accordance to scale development

Empirical Technique:SWAT

14

• Simple dimension structure– Subjective fatigue– Workload estimate

• Seven point Likert-scale (1 to 7)– 1 = Fully Alert / Nothing to do– 7 = Completly exhausted / Unacceptable

• Compared to other system information– Length of shifts– Task Complexity– Multitasking

• Later elaborated with more dimensions in the ARWES Questionnaire

Empirical Technique:Crew Status Survey

Empirical Technique:NASA TLX

Procedure

1. Rate all six subscales from 1 to 1002. Make pairwise comparison of subscales3. Multiply rating with weigthing

Evaluation

A) Add all 6 subscales, divide by 15 (comparisons) for total scoreand/or

B) Evaluate workload profile of all 6 subscales

NASA TLX RATING SCALE DEFINITIONS

MENTAL DEMAND (Low/High) How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, remembering, etc.)? Was the task easy or demanding, simple or complex?

PHYSICAL DEMAND (Low/High) How much physical activity was required (e.g., pushing, pulling, turning, controlling, activating, etc.)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious?

TEMPORAL DEMAND (Low/High) How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred? Was the pace slow and leisurely or rapid and frantic?

EFFORT (Low/High) How hard did you have to work (mentally and physically) to accomplish your level of performance?

PERFORMANCE (Good/Poor)How successful do you think you were in accomplishing the goals of the task set by the experimenter? How satisfied were you with your performance in accomplishing these goals?

FRUSTRATION LEVEL (Low/High) How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

15

see example from excel and program

Empirical Technique:NASA TLX

One step back, again

Types of Tests: Overview

Projective tests• Timeline Analysis• Scheduling Theory• Pro-SWAT

Empirical tests• Modified Cooper-Harper (MCH)• SWAT• NASA TLX• Crew Status Survey

16

1. Sensitivity: Can the test show changes in workload?

2. Intrusiveness: Do the test disturb the operator during work?

3. Diagnostical: Which aspect of several is the most important?

4. Convenience: How much logistics are necessary?

5. Relevance: How well do the measurement fit the work?

6. Operator acceptance: Is the test ok for the operator?

Criteria for good workload tests

SWAT vs NASA TLX?

Alternatives?

Beyond quantitative methods ...

Nardi and Ethnography

• Sometimes you don’t know what you are looking at

• Cognition in the wild (Welcome to the Jungle)

• Golden Rule: Before measuring a phenomenon, you should understand it.

• Main ethnographic activities: – Interviews– Observation– Participation

17

Ethnography: The Choices

• How long should I study this?

• What role should I have?

• What enables me to access information?

• What tools do I have?

• What data do I collect? How? When?

• How do I report ethnographic data?

Ethnography, cont

Your role and your mission

• Goal: Reflect and state your goal for observation. What do I want to achieve? What is success?

• Method: How do can I get to the information and by what means? Tools? Methods?

• Conditions: What resources do I have available? Time and money? Equipment?

• Analysis: What will be the format of data? How do I make sense of experiences? What do I bring back?

Know the user

Know the taskKnow thecontext

Usability goalsetting

Know the existingproducts

Design

PrototypingEvaluation

Follow upstudies

ANALYSE DESIGN

COGNITIVEENGINEERING

USABILITYDESIGN

INTERACTIONDESIGN

What kind of data do we need?

18

Ethnography: How to report?

Analysis is difficult with qualitative data (show dvd)

The choice of methods ...... in accordance to what?

Questions?

19

PSY2403 Man Machine Interaction:Method I

Cato A. BjørkliPsychologist