Making sense out of recorded user-system interaction Dr Willem-Paul Brinkman Lecturer Department of Information Systems and Computing Brunel University.

Making sense out of Making sense out of recorded user-system recorded user-system

interactioninteraction

Dr Willem-Paul BrinkmanLecturer

Department of Information Systems and Computing

Brunel University([email protected])

TopicsTopics VIVID Research Centre Motivation - Component-Based Software

Engineering Experiment 1: Searching for a

component-specific measure Experiment 2: Validating a component-

specific measure New and future research

VIVID Research VIVID Research CentreCentre

Based in the Department of Information Systems and Computing, Brunel University (London)

Original focus on visualisation, but now also includes:

- Mobile technology

- Design for diverse user groups

- Novel input/output devices

11 academics member of staff, 13 PhD Students

disc.brunel.ac.uk/research/vivid/index.htm

MotivationMotivationStudying the usability of

a system

Work conducted together with Reinder Haakma (Philips), Don Bouwhuis (Eindhoven University of Technology)

MotivationMotivation

ExternalExternal ComparisonComparison relating difference in usability to differences in the systems

InternalInternal ComparisonComparison trying to link usability problems with parts of the systems

Component-Based Software Component-Based Software EngineeringEngineering

Create

S upport

Reuse

M anage

new components

components from repos ito ry

productsP roduct requirementsand exis ting so ftware

feedback

feedback

Multiple versions testing paradigm

Single version testing paradigm

Manage

Support

Re-use

Create

Re-use

MotivationMotivationPROBLEM

1. Only empirical analysis of the overall system such as (task time, keystrokes, questionnaires etc) - not powerful

2. Usability tests, heuristic evaluations, cognitive walkthroughs where experts problems – unreliable

SOLUTION

• Component-Specific usability measures: more powerful and reliable

Searching for a Searching for a component-specific component-specific

measuremeasureQuestions

1. What is a component?

2. What interaction data should be recorded?

3. How do we link interaction data with the usability of a component?

Layered Protocol Layered Protocol TheoryTheory

(Taylor, 1988)

Interaction layersInteraction layers

15 + 23 =

15+23=

01111

10111

Add

100110

38

Processor

Editor

Control results

Control equation

User Calculator

15

15

15 +

15 +

15 + 23

15 + 23

38

38

Experiment 1 Experiment 1 – Fictitious – Fictitious InterfaceInterface

User Task: Rotate User Task: Rotate the Trumpetthe Trumpet

Experiment 1 Experiment 1 - Architecture- Architecture

Other symbols

Rotator

MapSelector

Buttons

Bike Aeroplane

RotateChange

X

Rotate(x)

Experiment 1 Experiment 1 - Architecture- Architecture

Other symbols

Rotator

MapSelector

Buttons

Bike Aeroplane

RotateChange

X

Rotate(x)

Low

High Measures

Task time

#Rotate(T0),

#Rotate(T-1),

#Rotate(T-2)

#change, #rotate

#bike,#aeroplane, #other

#clicks

Experiment 1 Experiment 1 - Training- Training

Training Groups I II III IV V VI VII VIII Rotator 0 0 0 0 + + + + Map 0 0 + + 0 0 + + Selector 0 + 0 + 0 + 0 + + useful training, 0 dummy training. Each group exists out of ten subjects.

Experiment 1 : Experiment 1 : Test Test ProcedureProcedure

80 participants, all students of Eindhoven University of Technology

8 different trainings

After training participants were asked to rotate, as fast as possible, a specific music instrument

User interaction with the system was recorded in log file

Once a task was complete the recording stops

Experiment 1 - Experiment 1 - Low-level Low-level Effect of Selector trainingEffect of Selector training

Clicks on

Nu

mb

er

messag

es

0

5

10

15

20

bike aeroplane others

withheld

provided

Experiment 1 - Experiment 1 - High-level High-level Effect Rotator TrainingEffect Rotator Training

#Rotate (X)

Nu

mb

er

messag

es

0

1

2

3

4

5

target target-1 target-2

withheld

provided

Experiment 1 – Control LoopExperiment 1 – Control Loop

Reliability: how do we link interaction data with the usability of a component?

Evaluation

Component

User message

Feedback

Reference value

User

System

Each message is a cycle of the control loop Number of messages presents the user’s effort to control the component

Each message is a cycle of the control loop Number of messages presents the user’s effort to control the component

Experiment 1 - ConclusionExperiment 1 - Conclusion

1. What is a component?

An interaction component is a unit within a device that directly or indirectly receives signals from the user. These signals enable the user to change the physical state of the interaction component

2. What interaction data should be recorded?

Message exchange between the interaction components

Experiment 2 : ValidationExperiment 2 : Validation

80 users8 mobile telephones3 components were manipulated

according to Cognitive Complexity Theory (Kieras & Polson, 1985)

1. Function Selector 2. Keypad3. Short Text Messages

Architecture Mobile Architecture Mobile telephonetelephone

Send Text Message

Send Text Message Function

SelectorFunction Selector

KeypadKeypad

Experiment 2 Experiment 2 – Function – Function SelectorSelector

Versions:

Broad/shallow

Narrow/deep

Experiment 2 Experiment 2 – Keypad– Keypad

Versions

Repeated-Key Method

“L”

Modified-Model-Position method

“J”

Experiment 2 Experiment 2 – Send Text – Send Text MessageMessage

Versions

Simple

Complex

Statistical Tests Statistical Tests

p-value: probability of making type I, or , error, wrongly rejecting the hypothesis that underlying distribution is same.

Results Results – Function – Function SelectorSelector

Mean df Measure Broad Deep Hyp. Er. F p η2 Normal Joint measure — — 7 66 34.47 <0.001 0.80 Time in seconds 947 1394 1 72 29.56 <0.001 0.29 Number of keystrokes 461 686 1 72 37.72 <0.001 0.34 Number of messages received 67 265 1 72 155.34 <0.001 0.68 Ease of use mobile phone 5.5 4.8 1 72 11.86 0.001 0.14 Ease of use menu 5.6 4.5 1 72 22.33 <0.001 0.24 Satisfaction of mobile phone 4.4 3.8 1 72 4.25 0.043 0.06 Satisfaction of menu 4.6 3.5 1 72 15.96 <0.001 0.18 Correcteda Joint measure — — 2 71 60.96 <0.001 0.63 Number of keystrokes 437 602 1 72 20.27 <0.001 0.22 Number of messages received 52 190 1 72 75.36 <0.001 0.51

aCorrected for all a-priori differences between versions of the components.

Results of two multivariate analyses and related univariate analyses of variance with the version of the Function Selector as independent between-subjects variable.

Results Results – Keypad– Keypad

Results of multivariate and related univariate analyses of variance with the version of the Keypad as independent between-subjects variable.

Mean df Measure RK MMP Hyp. Er. F p η2 Normal Joint measure — — 7 66 4.05 0.001 0.30 Time in seconds 872 1083 1 72 9.44 0.003 0.12 Number of keystrokes 438 537 1 72 10.34 0.002 0.13 Number of messages received 233 271 1 72 13.92 <0.001 0.16 Ease of use mobile phone 5.3 5.0 1 72 1.07 0.305 0.02 Ease of use keyboard 5.6 4.9 1 72 11.13 0.001 0.13 Satisfaction of mobile phone 4.3 3.9 1 72 1.76 0.188 0.02 Satisfaction of keyboard 4.6 3.8 1 72 8.97 0.004 0.11

Results Results – Send Text – Send Text MessageMessage

Results of two multivariate analyses and related univariate analyses of variance with the version of the STM component as independent between-subjects variable

Mean df

Measure Simple Compl

ex Hyp. Er. F p η2

Normal Joint measure — — 7 66 18.16 <0.001 0.66 Time in seconds 523 672 1 72 8.15 0.006 0.10 Number of keystrokes 269 320 1 72 4.56 0.036 0.06 Number of messages received

12 49 1 72 74.18 <0.001 0.51

Ease of use mobile phone 5.0 5.3 1 72 1.15 0.288 0.02 Ease of use STM function 5.1 4.9 1 72 0.35 0.555 0.01 Satisfaction of mobile phone 3.9 4.2 1 72 0.93 0.339 0.01 Satisfaction of STM function 3.9 3.8 1 72 0.26 0.614 0.01 Correcteda Joint measure — — 2 71 20.85 <0.001 0.37 Number of keystrokes 249 289 1 72 2.30 0.134 0.03 Number of messages received

12 34 1 72 26.23 <0.001 0.27

aCorrected for all a-priori differences between versions of the components.

Power of number of messages as a Power of number of messages as a usability measureusability measure

Statistical Power: 1 - β

ResultsResults

Average probability that a measure finds a significant (α = 0.05) effect for the usability difference between the two versions of FS, STM, or the Keypad components

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80

Number of subjects

Po

wer

1. messages received

2. corrected messagesreceived

3. task duration

4. keystrokes

5. corrected keystrokes

6. comp.-spec. ease-of-use7. comp.-spec. satisfaction

8. overall eas-of-use

9. overall satisfaction

Component-Based Software Component-Based Software EngineeringEngineering

Create

S upport

Reuse

M anage

new components

components from repos ito ry

productsP roduct requirementsand exis ting so ftware

feedback

feedback

Multiple versions testing paradigm

Single version testing paradigm

Manage

Support

Re-use

Create

Re-use

Testing Different Testing Different ComponentsComponents

Component specific objective performance

measure:1. Messages received + Weight factor

A common currency

2. Compare with ideal userA common point of reference

Usability of individual components in a single device can be compared with each other and prioritized on potential improvements

Click <right>Click <left on Properties option>

{1}{1}

Click <left on Fill tab>Click <left on on colour red>Click <left on Outline tab>Click <left No Line button>Click <left no Ok button>

{1}{1}{1}{1}{1}

Call <>{2}

Set <Fill colour red, no border>{7}

Right MouseButton Menu

Properties

Assigning weight factors to represent theAssigning weight factors to represent the user’s effort in the case of ideal user user’s effort in the case of ideal user

Total effort valueTotal effort value

Total effort = MRi.W

• MRi.W : Message received. Weight factor

Click <right>

Click <left on Properties option>

{1}{1}

Click <left on Fill tab>Click <left on on colour red>Click <left on Outline tab>Click <left No Line button>Click <left no Ok button>

{1}{1}{1}{1}{1}

Call <>{2}

Right MouseButton Menu

Properties

5 2 = 7+2

Assigning weight factors in Assigning weight factors in case of real usercase of real user

Correction for inefficiency of higher and lower components

Visual Drawing Objects

Properties

Right MouseButton Menu

Assigning weight factors in Assigning weight factors in case of real usercase of real user

Assign weight factors as if lower components operate optimal

Visual Drawing Objects

Properties

Right MouseButton Menu

Inefficiency of lower level components: need more messages to pass on a message upwards than ideally required

Assigning weight factors in case of real Assigning weight factors in case of real useruser

Visual Drawing Objects

Properties

Right MouseButton Menu

Inefficiency of higher level components: more messages are requested than ideally required

UE : User effort

MRi.W : Message received. Weight factor

#MSUreal :Number of messages sent upward by real user

#MSUideal :Number of messages sent upward by ideal user

MRi.W

#MSU real

#MSU ideal

UE =

Ideal User versus Real UserIdeal User versus Real User

Extra User Effort = User Effort - Total effort

The total effort an ideal user would make

The total effort a real user made

The extra effort a real user made

Calculate for each component:

Prioritize

Experiment 2 Experiment 2 - Single - Single versionversion

40 users4 mobile telephones2 components were manipulated

(Keypad only Repeated-Key Method)

1. Function Selector 2. Short Text Messages

ResultsResults

010

20304050

6070

Broad & Simple

Narrow & Simple

Broad &Complex

Narrow &Complex

1 2 3 4

Function Selector

Send Text Message

Mobile phones

Ext

ra U

ser

Eff

ort

ResultsResults

Measure Function Selector

Send Text Message

Objective    

Extra keystrokes 0.64** 0.44**

Task duration 0.63** 0.39**

Perceived    

Overall ease-of-use -0.43** -0.26*

Overall satisfaction -0.25* -0.22

Component-specific ease-of-use -0.55** -0.34**

Component-specific satisfaction -0.41** -0.37**

Partial correlation between extra user effort regarding the two components and other usability measures

*p. < .05. **p. < .01.

TopicsTopics VIVID Research Centre Motivation - Component-Based Software

Engineering Experiment 1: Searching for a component-

specific measure Experiment 2: Validating a component-

specific measure New and future research

- Extending the analysis outside the lab

- Extending the analysis beyond only usability issues

New Projects New Projects - Field - Field usabilityusability

• CD player, which 10 users will use at home

• Record interaction: online assignment of weigh factors, both optimal and real user, to messages

• Correlated interaction data with other data (questionnaire, dairy, interview)

(Pui-Fong Man)

New Projects New Projects - PROSKIN- PROSKIN

• Exciting Interface designed for the average user. However, the average user does not exist.

• Developing skins for specific user groups could be a way forward

• Question:

• How to identify user groups?

• What do user groups want?

Work conducted together with Nick Fine

User profiling for skinnable domestic technology

New Projects New Projects - PROSKIN- PROSKIN

Possible solution

• Recording online interaction, Identifying user groups, Developing skins for these user groups

Question

• How to establish user groups that are relevant for designer?

• This time, how to make sense of the interaction data beyond usability? Work conducted

together with Nick Fine

User profiling for skinnable domestic technology

New Projects New Projects - PROSKIN- PROSKINApproachApproach

Interaction data

User metrics

User groups based on interaction data Design of

Skins

Online Validation

Conclusions and Final Conclusions and Final RemarksRemarks

Interaction data can be used to study the usability of interaction components

- External Comparison between different versions: More Powerful

- Internal Comparison: prioritized on potential improvements

Future questions - Usability analysis of everyday life interaction- Establishing new paradigms to understand

interaction data beyond usability issues

Questions

Thank you for your attention