Evaluation beyond the desktop

Human Computer Interaction & Usability Unit http://icts.sbg.ac.at

Thomas Grillmailto:[email protected]

17.06.2009

Evaluation beyond the desktop ...

Thursday, May 27, 2010

http://icts.sbg.ac.at

http://icts.sbg.ac.at

mailto:[email protected]


/ 41

...

● - HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5

What’s the difference?

2


●

● / 45- What - Where - How - Tools - Assignment

• The range of computerized systems is far greater than just desktop application.• In particular, unconventional input channels such as

Tangible user interfaces• Physical objects are representations and controls for digital objects

Gesture inputs, movement tracking Eye- and gaze tracking

• and environmental profiles such as Virtual environments Augmented reality

• are difficult or impossible to test using ordinary usability labs

Why is it different?

3


●

● / 45- What - Where - How - Tools - Assignment

• In UbiComp applications The feasibility rather than the usability is tested No established software usability culture for mobile and UbiComp

applications• Lack of trained specialists• Screenbased testing methods are often not feasible• Direct porting between different platforms is not feasible nor does it cover the particular platform requirements• Fast-paced software market environment

No time for long-lasting user testing?

General problems related to the usability of unconventional user interaction

4


/ 41- HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5

...

●

What to evaluate?

5


1

● - What - Where - How - Tools - Assignment / 45

What to design - what to evaluate?

6

GUIs

Screen Print

Non-GUIs

Objects Devices

Information Architecture

Interactions

Interactionwith

InterfacesThursday, May 27, 2010

• desktop applications WIMP environment Standard interaction devices (mouse/keyboard/...)

• mobile applications mobile environment mobile devices (smartphones, handhelds, ubicomp appliances, ...)

• real world applications real world environment (mobile+static) real world objects, ubicomp objects

1


Usability of ...

7

Usability Utility User ExperienceRequirements


1


Evaluation in a UCD process

8

User requirements

- Usability parameters- UI design requirements

Context of use

- User tasks, Task characteristics- User groups, User characteristics- environment

iterative formative evaluations of Lo/Fi

prototypes

summative usability evaluation

Product evaluations

- user experience analysis- usability evaluations

iterative formative evaluations of Hi/Fi

Prototypes and products

???


• desktop applications• mobile applications• ubicomp applications• real world applications

1


... identify and evaluate

9



1


• User requirements Which are the targeted users? Data for defining personas, user profiles

• Task requirements functionality, task what to design?

• Design requirements which information do I need for this is there an existing workflow? (reuse prior knowledge) Usage environment Type of information

• mobile vs. static• outside vs. indoor, desk vs. standing, ...• public places vs. private places• ...

10



1


• Evaluate according to usability heuristics http://www.useit.com/papers/heuristic/heuristic_list.html

• Task performance Task completion Delay

• Errors Error rate, error messages, error handling, ...

• Intuitiveness, Simplicity, Learnability, ... Simple and natural dialogues

• Consistent interface and interaction• Feedback

No states where the user does not know what to do.• Visibility of System State

11



1


• Utility refers to the design's functionality:

Does it do what users need?

Usefulness Is the functionality needed Is the functionality appropriate

• Usability and utility are equally important!

12



1


Usability User ExperienceRequirements Utility

http://sakshigupta.files.wordpress.com/2009/11/


http://sakshigupta.files.wordpress.com/2009/11/uxd11.jpg


1


Usability User ExperienceRequirements Utility

http://sakshigupta.files.wordpress.com/2009/11/

What constitutes a good user experience………….

1. useful2. functional3. intuitive4. reliable5. efficient6. effective7. usable8. innovative9. aesthetically pleasing

beautiful10. delightful, ‘aha’ moment

wow-factor

How to design for a good user experience………….

1. by understanding people needs, wants,behavior,constraints

2. based in social & cultural context3. exploring opportunities4. based on people’s past

experiences5. power to evoke emotions6. forgiving to errors7. simplicity8. optimized for most frequent

tasks9. informative & timely feedback10. story-telling11. human touch12. multiple iterations13. prototyping




/ 41

the field lab ...

●

Where to evaluate?

15


one way mirror

video camera

facilitator

evaluator

system

tasks

usability lab

video monitor

recording unit

audio

2

● - What - Where - How - Tools / 45

Evaluation environment

16

Usability laboratory Field evaluation

what to apply?


2


Select the appropriate evaluation environment

• What to evaluate?

• In the lab (in-Vitro) “Realistic” usage scenario can be setup/simulated in a usability

laboratory Usually in-door usage Usability lab, Smart Homes Evaluating concrete tasks, performing user tests lower costs

17


2




• In the field (in-Situ) Mobile task scenario Outdoor usage Observing users in real world scenarios higher costs

18


2




• In the real world (in-Vivo) Long time usability observations User experience analysis Indoor and outdoor usage Observing/Sensing/Logging of users in real world scenarios higher costs / higher effort

19


/ 41

...

●

How to evaluate?

20


3


Evaluate a product

21

http://www.ambientdevices.com/products/umbrella.html




3


Define the goals of the evaluation ...

• What do I want to find out? I have a design / idea.

• Does it make sense?• Is it useful?

Usability User Experience Utility General usage behaviour Which user group to address ...

22


3


Evaluate behaviour

• Observation Audio/Video recording note taking diary methods (e.g. photo diary, post-it notes, DRM - day

reconstruction method, ...)• Interviews

structured vs. narrative• Focus groups

A group of people are asked about their perceptions, opinions, beliefs and attitudes towards a product, service, concept, advertisement, idea, or packaging ...

23

tool


3


Evaluate behaviour

24


3


Evaluate usability

• Evaluate behaviour• Measure

task completion error rate critical incidents method

• Measuring Sensing Logging

• Important Time synchronization of all measured data!!! Analysis tools help here (ELAN, NVivo, MacEval, Morae, ...)

25


w

ord

of

m

outh

br

and

m

anag

emen

t

se

arch

eng

ine

st

rate

gy

resp

onse

tim

e

bro

wser

compa

tibility

stan

dard

complian

ce

WCAG-2

complian

colorsc and co

media use graphic

elements

placement

of elements

naming and

categorization

tone of voice

typography navigation

intuitiveness

structure

appropriate for

purpose

expected

information

no unexpected errors

conformity consistency

verifyability

satis

faction d

ifferen

tiatio

n

un

iquen

ess

conten separa

na

min

g

m

arke

ting

expected functionality

launch

content and fu

nctio

nalit

y

u

ser r

equi

rem

ents

str

ategic foundation

se

p

hase

ual

st

rate

gic

cust

omer

provider

experi

ence

user

VALUE

findability accessability desirability usability

cr

edib

ility

u

sefu

lnes

s

3


Evaluate user experience

• Observe user experience factors Usability, Utility Emotions Social distinction Stress Trust Privacy Safety ...

26

http://tpgblog.comQuick UX - Heuristics for User ExperienceThe Quick-UX evaluates the degree to which a product successfully addresses the following 3 questions:

• Can I use it? (Usability)

• Should I use it? (Usefulness)

• Do I want to use it? (Desirability)

http://userexperienceproject.blogspot.com/2007/04/user-experience-wheel.html


http://tpgblog.com

http://tpgblog.com

http://tpgblog.com/2008/03/31/can-i-use-it-evaluating-usability-through-quick-ux/

http://tpgblog.com/2008/03/31/can-i-use-it-evaluating-usability-through-quick-ux/

http://tpgblog.com/2008/04/07/should-i-use-it-evaluating-usefulness-through-quick-ux/

http://tpgblog.com/2008/04/07/should-i-use-it-evaluating-usefulness-through-quick-ux/

http://tpgblog.com/2008/04/14/do-i-want-to-use-it-evaluating-desirability-through-quick-ux/

http://tpgblog.com/2008/04/14/do-i-want-to-use-it-evaluating-desirability-through-quick-ux/



3


Evaluate user experience

• How?

• Evaluation methods based on cognitive and social sciences

Interviews Questionnaires long-term evaluations with diary method Critical Incident Method (ICT) ...

27


3


How to evaluate ...

• Select, create, or define an appropriate Evaluation Method Evaluation Environment The participants of the evaluation scenario

• experts• test users

Prepare • test scenarios• questionnaires• necessary forms (e.g. non-disclosure form, session checklists, ...)

• http://www.stcsig.org/usability/resources/toolkit/toolkit.html• A set of document templates useful during usability evaluations• NASA Usability toolkit (http://www.hq.nasa.gov/pao/portal/usability/resources/index.htm)

• ....

28

tool


http://www.stcsig.org/usability/resources/toolkit/toolkit.html




3


How to evaluate ...

• Conduct the evaluation• Analyze the data

depends on the type of data collected video/audio analysis text analysis questionnaire evaluation quantitative / qualitative methods

• Compile findings compile the identified issues prepare a report

29


3


Measuring usability

30

!"#$%&'()&*'#$%&*+',-$+,&%.&-,'/"0"12&345&


/ 41

... tools

●

Evaluation tools in the field ...

31


4


Questionnaires

32

• SUS - System Usability Scale "SUS: a "quick and dirty" usability scale"

http://www.usabilitynet.org/trump/documents/Suschapt.doc• AttrakDiff• SUMI

Addresses quality of use of software http://sumi.ucc.ie/

• QUIS• IsoMetrics

Operationalising the design principles of ISO 9241 Part 10 http://www.isometrics.uni-osnabrueck.de/

• ...


http://www.usabilitynet.org/trump/documents/Suschapt.doc




http://www.isometrics.uni-osnabrueck.de

http://www.isometrics.uni-osnabrueck.de

4


Traditional tools ...

• Paper & Pencil - Wizard of Oz Evaluation• Storyboarding

33

download video!


4


Recording tools

• Audio Mobile audio recorders

• Video Morae/Camtasia

• Excellent tools for capturing video, logging issues, and creating highlight tapes

• http://www.techsmith.com/ Noldus Observer

• http://www.noldus.com Silverback

• http://silverbackapp.com/ MacEval

• http://www.tomgrill.info/maceval OvoLogger

• http://www.ovostudios.com

34

cap with mobile camera

maybe show cap/scenario from gerhard leitner


http://www.noldus.com


http://www.tomgrill.info/maceval

http://www.tomgrill.info/maceval

http://www.ovostudios.com

http://www.ovostudios.com

4


Recording tools

• Text Notebooks Instrumented tools that log data Keylogger tools

• Transformed audio/video via annotation tools

NVivo Elan Interact

• http://www.mangold-international.com

35


http://www.mangold-international.com




4


Recording tools

• UTE - Usability Testing Environment Comprehensive tool for capturing and analyzing usability data Automatically calculates success rates, time-on-task, and many other

metrics Remote Usability testing http://www.mindd.com

• Data Logger Free Excel program used to collect and analyze usability test data. Records task success, time-on-task, survey questions, and

automatically generates charts. www.userfocus.co.uk/resources/datalogger .html

36


http://www.mindd.com

http://www.mindd.com

http://www.userfocus.co.uk/resources/datalogger

http://www.userfocus.co.uk/resources/datalogger

4


EyeTracking

• Record eye movement• Gaze• Focus of attention• Systems

eye tracking system on computer screen eye tracking in real world

SMI Tobii

37


4


Motion and position tracking

• Tracking motion• Tracking position• Finding information about movement• Crowd behaviour• ...

• Systems based on Optical: video, IR Accoustic RFID GPS

38

RFID

Optical


4


Recording tools

• The Observer XT from Noldus Sophisticated software for usability data

collection, analysis and presentation. Ability to integrate multiple video feeds,

analyze eye-tracking data, and other physiological measurements.

Allows to integrate eye-trackers http://www.noldus.com

• FaceReader FaceReader is a tool that is capable of

automatically analyzing facial expressions, providing users with an objective assessment of a person’s emotion.

39




4


Recording tools

40

Noldus mobile camera ...


4


Recording tools

41

Noldus observer

Usability lab - ICTS Salzburgobservation place

Receiver for mobile camera

Recording & Notetaking

Video system using regular and IP based moveable cameras


4


Recording tools

42

Noldus observer - recording modeUsability lab - ICTS Salzburg

Note taking and evaluation control interface

Video recorder - monitor



4


Analysis tools

• Video/Audio/Text annotation tools ELAN NVivo MacEval

• Statistical Analysis tools SPSS

• http://www.spss.com R

• http://www.r-project.org/ Microsoft Excel

• statistics add-in available: “Analyze-it”• http://analyse-it.com/

Stata• http://www.stata.com

44


* Contact

Human Computer Interaction & Usability Unit [email protected]

HCI & Usability UnitICT&S Center, University of SalzburgSigmund-Haffner-Gasse 185020 Salzburg, [email protected]

Dr. Thomas [email protected]




mailto:[email protected]?subject=

mailto:[email protected]?subject=

Evaluation beyond the desktop

Education

good user

usability

uniticts

assignment

grillsbg

evaluation

tools thursday

topic 5 thursday