Evaluation Methods April 20, 2005 Tara Matthews CS 160.

Evaluation Methods

April 20, 2005

Tara Matthews

CS 160

In 160 We’ve Covered…

• Task Analysis & Contextual Inquiry

• Cognitive Walkthrough

• Heuristic Evaluation

• WOZ usability study w/ paper prototypes

There are many more methods…

• Survey• Interview• Controlled-lab

experiment• In-lab observation• Controlled field

experiment• Field observation

study

• Automated observation user study

• Experimental simulation

• Claims analysis• GOMS• Computer simulation• Formal theory

How to chose a method?

• Stage of study– formative, iterative, summative

• Pros & cons

• Metrics– depends on what you want to measure

• Qualitative vs. quantitative

• Research perspective– CS vs. psychology vs. sociology

Pros & Cons

• Realism

• Precision

• Generalizability

• Time & cost

• Researcher expertise

Methods




study

• Automated observation user study

• Experimental simulation

• Claims analysis• GOMS• Computer simulation• Formal theory

Survey

• Online / paper questionnaires distributed to target audience

• Can be used to– tabulate quantitative data– gather qualitative feedback (opinions,

feelings, etc.)

• Useful at any time in study

Survey

• Pros– Easy to get a large number of responses.– Quick and easy to conduct.– Highly generalizable.

• Cons– Self-selection.– Participants often only offer enough information to

answer the question.– Can miss details.– Low in realism and precision.

Interview

• Evaluators formulate questions on the issues of interest.

• Interview representative users, asking them these questions in order to gather information desired.

• Interviewer reads questions to user, who replies verbally; interviewer records responses.

Interview

• Pros– Quick and easy to conduct.– Gives designer quick feedback on a range of ideas.– Can get a person’s initial reaction to an idea.– Can get detailed information from a person.

• Cons– Often takes place away from natural setting.– Question wording or interviewer “body language” can bias

answers.– High probability of false positives and missed problems (e.g.,

users may not have a clear idea of how an app will be used).– Can miss details if interviewer doesn’t know what issues to

draw out.

Controlled Lab Experiment

• In lab, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system.

• Example:– in lab, show users 4 versions of a website:

• blue, yellow, red, and black text

– measure time to find specific words– compare

Controlled Lab Experiment

• Pros– Provides precise, quantifiable data.– Easier to draw inferences from data.– Relatively quick.– Can get a medium-sized number of participants.

• Cons– Short duration of a lab experiment may not be enough

to allow users to become accustomed to an app.– Not a natural setting – interaction may not be normal.

In-lab Observation

• Participants come to lab to "use" an interface• Given sample tasks to complete with it• Evaluators observe and possibly audio- or

videotape• Participants may "think out loud"• Can use lo-fi prototype (for a project in the

design stage) to an almost-complete interface• Evaluators note participants’

– emotions, exclamations, facial expressions, and other "qualitative" data

– take note of quantitative data such as time to complete a task or number of errors

In-lab Observation

• Pros– Relatively quick.– Can get a medium-sized number of participants.

• Cons– Observations are subjective and error prone.– Short duration of lab observation is not enough time

for user to get accustomed to using the interface.– Not a natural setting – interaction may not be normal.

Controlled Field Experiment

• In natural setting, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system.

• Example:– Participants use 3 different input devices in

their own office: mouse with 1, 2, or 3 buttons– Perform a set of tasks– Measure differences

Controlled Field Experiment

• Pros– Less intrusive than most other evaluation methods.– Provides more precise data than field observation.– Can observe natural behavior of user (though some

part of the system will be controlled/unnatural).

• Cons– More intrusive than field observation.– Less natural than field observation.

Field Observation Study

• Evaluator makes direct observations of “natural” systems

• Takes care to not intrude on / disturb those systems

• A.K.A. “ethnography”

Field Observation Study

• Pros– Only way to observe natural behavior of user &

interaction between user & tools.

• Cons– Difficult and time consuming.– Hard to get permission to observe people.– Observations are subjective and error prone.– Cannot make strong interpretations from

observations.– Not very generalizable.

Heuristic Evaluation

• Pros– Quick and easy.

• Cons– Nielson’s heuristics may not be as relevant to

non-GUIs.– Results in false positives in missed problems,

especially when experts are not part of target audience.

Cognitive Walkthrough

• Pros– Quick and easy.

• Cons– Results in false positives and missed

problems when evaluator is different from target audience.

Automate Observation Study

• Techniques include– video or audio recording of user– pop-up screens– screen shots– time logging– log users actions (collecting statistics about

detailed system use)

Automate Observation Study

• Pros– Eases burden on observers for data collection

& analysis.

• Cons– Setup is often more time-consuming to

complete.– Harder to get approved if it involves analysis

of videotape or audiotape.– May miss nuanced/interpretive details.

Experimental Simulation

• In-lab experiment that is as much like some real situation as possible.

• Example:– ground-based flight simulator– behaves as closely as possible to a real flight– still under researcher control

Experimental Simulation

• Pros– Still fairly precise.– More realistic than in-lab experiment.

• Cons (same as lab exp.)– Short duration of a lab experiment may not be

enough to allow users to become accustomed to an app.

– Not a natural setting – interaction may not be normal.

Claims Analysis

• Claim = statement that a certain aspect (button, scrollbar) of a design has psychological implications reflected in how capable a user is in using that design

• UI artifacts are listed along with their design features & pros/cons

• Helps– select among alternative designs– clarify questions to be analyzed through user testing

by stating how the design should work (in claims)

GOMS

• A method to describe user tasks and how a user performs those tasks with a specific interface design

• Views humans as information processors– Small number of cognitive, perceptual, and motor operators

characterize user behavior• To apply GOMS:

– Analyze task to identify user goals (hierarchical)– Identify operators to achieve goals– Sum operator times to predict performance

• GOMS = – Goals: What a user wants to accomplish– Operators: Cognitive or physical actions that change the state of

the user or the system– Methods: Groups of goals and operators– Selection rules: Determine which method to apply

GOMS

• Pros– Predict human performance before committing to a

specific design in code or running user studies– Many studies have validated the model (it works)

• Cons– Assumes error-free, skilled user behavior– No formal recipe for how to perform analysis– Significant time investment

Computer Simulation

• Creating a complete & closed system that models the operation of the concrete system without users.

• Example:– geophysical process going on in connection

with the eruption of Mount St. Helens

Computer Simulation

• Pros– Supposedly high in realism (depends on

accuracy of data/system replication)

• Cons– Low in precision & generalizability

Formal Theory

• Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest.

• Pros– Relatively generalizable

• Cons– Not realistic or precise

How to chose a method?

• Stage of study• Pros & cons

– Realism– Precision– Generalizability– Time & cost

• Researcher expertise• Metrics• Qualitative vs. quantitative• Research perspective

Methods




study• Heuristic Evaluation

• Cognitive Walkthrough• Contextual Inquiry• Automated

observation user study• Experimental

simulation• Claims analysis• GOMS• Computer simulation• Formal theory

Early Stage








Early Stage








Iterative & Summative Stages








Iterative & Summative Stages








Realism








Realism








Precision








Precision








Generalizability








Generalizability








Time & Cost








Time & Cost








Researcher Perspective








Metrics: examples

• Traditional GUIs:– efficiency (time to complete task)– accuracy (# of errors)– simplicity

• Peripheral Displays:– awareness (recall)– distraction (dual-task behavior)– aesthetics

Peripheral Displays








Questions?

Evaluation Methods April 20, 2005 Tara Matthews CS 160.

Documents

study slide

lab observation participants

controlled lab experiment

sociology slide

interview pros quick

survey pros easy

methods survey interview

interview evaluators