© IDC, IIT Bombay 1
Dec 24, 2015
© IDC, IIT Bombay1
© IDC, IIT Bombay2
BadDesigns.com
© IDC, IIT Bombay3
© IDC, IIT Bombay4
© IDC, IIT Bombay5
© IDC, IIT Bombay6
© IDC, IIT Bombay7
© IDC, IIT Bombay8
© IDC, IIT Bombay9
© IDC, IIT Bombay10
© IDC, IIT Bombay11
© IDC, IIT Bombay12
© IDC, IIT Bombay13
© IDC, IIT Bombay14
© IDC, IIT Bombay15
© IDC, IIT Bombay16
© IDC, IIT Bombay17
© IDC, IIT Bombay18
19
World Is Full of Unusable Objects
Home assignment– Find 3 unique usability problems in the day-to-day objects
that you have observed or encountered– Describe them as 3 slides in a presentation
Usability Evaluation
© IDC, IIT Bombay21
Objective
To identify usability problems in an interface– To improve the design – formative evaluation– To compare competitive products or as acceptance test
– summative evaluation
22
Many Methods
Heuristic evaluation User Testing
Thinking Aloud Protocol Card sort Feedback from real life usage
– Observational field studies– Logging actual use– User Feedback– Questionnaires and Interviews– Focus groups
23
Usability Evaluation Methods
Heuristic evaluation– Jakob Nielsen, Usability Engineering, Morgan Kaufman,
1990
© IDC, IIT Bombay24
Task
For use by a shopkeeper in a kirana shop to make a bill– Find as many usability problems
as you can in 5 minutes in the Standard Windows calculator
For each problem – State it succinctly (if you
thought of a design idea, tell us what problem do you solve)
– Identify a (potential) user experience goal that was not met
– Rate it for severity– Identify layer of user
experience
© IDC, IIT Bombay25
Task
Analyse the alarm app shown in this video for use by a patient educated till class VII as reminder tool for his medication – Find as many usability problems as you can in 5
minutes
Analyse the app shown in this videohttp://www.youtube.com/watch?v=s94JxnVwJtI
26
© IDC, IIT Bombay27
Task
Analyse the alarm app shown in this video for use by a patient educated till class VII as reminder tool for his medication – Find as many usability problems as you can in 5
minutes For each problem
– State it succinctly (if you thought of a design idea, tell us what problem do you solve)
– Identify a (potential) user experience goal that was not met
– Rate it for severity– Identify layer of user experience where the problem
occurs
© IDC, IIT Bombay28
Heuristic Evaluation Technique
5-10 evaluators examine the product and identify usability problems
It helps to have a mix of usability experts, domain experts and users– Single / double experts– Same evaluators provide consistency across versions
and save time– New evaluators provide fresh eyes
© IDC, IIT Bombay29
Heuristic Evaluation Technique
Give assistance regarding the domain knowledge– Brief about users, personas, scenarios, primary goals,
and usability goals– Evaluators may be asked to focus on parts of the
product / specific personas Give a checklist of
– HCI heuristics and principles– Platform standards, common practices– Domain standards, common practices
© IDC, IIT Bombay30
Heuristic Evaluation Technique …
Each evaluator has the freedom to choose his/her method of evaluation– Each evaluator evaluates the product independently– Evaluators should go through product 2-3 times
First iteration– Look for problems novice might face– Analyze Surface and Skeleton issues
Second iteration– Use primary goals, usability goals, heuristics, domain
standards, platform standards to identify problems– Look for conceptual problems, analyze Structure
Third iteration– Look for what is being missed out – Question Scope and Strategy issues
© IDC, IIT Bombay31
Heuristic Evaluation Technique …
Sometimes, an observer notes the findings– Each of us will note for ourselves here
Observer may give hints about product use – These are evaluators, not users– Save time, capture the problem, and move on– Help only when evaluators ask, after noting problems
© IDC, IIT Bombay32
Heuristic Evaluation Technique …
After the evaluations, have a debriefing meeting to compile findings
The debriefing meeting also produces design suggestions– Make them sound like suggestions rather than decisions– Do not express only a design idea, point to the problem
first Typical debriefing sessions last 1-4 hours
– Produce preliminary report during the debriefing meeting
– Fine-tune later
© IDC, IIT Bombay33
Heuristic Evaluation Output
Output of heuristic evaluation is a list of usability problems
Report formats– Introduction, product evaluated, method used– Scenario based problems
~ Enter the cubicle, swipe the card, enter PIN, check balance, withdraw cash, collect cash, card and receipt, exit the cubicle
– Screen based problems in callouts~ Idle screen, test keyboard screen, login screen,
password screen, main menu Common Industry Format (CIF)
~ ISO/IEC 25062:2006
© IDC, IIT Bombay34
Heuristic Evaluation Some Numbers
5 evaluators find about 75% of the problems 10 evaluators find about 90% of the problems
– HE is prone to reporting false alarms– Don’t negotiate for too long unless it is a very important
issue– If someone thinks it is a problem, capture it
35
Usability Evaluation Methods
Heuristic evaluation User Testing
© IDC, IIT Bombay36
User Testing
A representative set of tasks Representative users try to perform these tasks An experimenter conducts the experiment
– Observes users– Captures, interprets and prioritizes problems
© IDC, IIT Bombay37
Planning a User Test
Determine the key question that the UT should answer~ Should we design an IVR? Can low-tech savvy users
use IVRs? ~ Is the design OK? Should we start the development?~ Can all voters use the new EVM? Can we use it in the
next election?– Subset of usability goals and persona goals
© IDC, IIT Bombay38
Planning a User Test
Determine the key question that the UT should answer Determine test tasks
– Determine training to be given prior to tasks– Provide a scenario description– Give users a goal to achieve, not instructions to execute– Avoid leading instructions / hints / questions– Set ‘error traps’ and planned disturbances– Try to provide early success experiences
© IDC, IIT Bombay39
Planning a User Test
Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria
– Determine the output of the UT– Identify performance measures
~ Quantitative~ Success, time, errors, efficiency, satisfaction~ Qualitative~ Ability to handle new situations~ Conceptual understandability of the product~ Questions asked by users~ Training procedure
>> Output of a user test
© IDC, IIT Bombay40
User Testing Output
Qualitative output~ What is the first reaction?~ What is confusing? What do users ask? Why?~ What do the users like / hate most?~ What features are users likely to use?~ What are user priorities?
– Good for formative evaluation
© IDC, IIT Bombay41
User Testing Output
Qualitative output Task success
~ Find the share price of Reliance~ Find a good insurance policy
– Levels of success ~ Yes / no~ Successful in first attempt / second attempt / with
help
© IDC, IIT Bombay42
User Testing Output
Qualitative output Task success Task time
– Tell the participant to be as fast as possible – But avoid declaring
~ “You took 6.3 seconds for that task. Great show.”– Decide when to stop the timer– To probe or not to probe?– Include unsuccessful tasks in timing?
© IDC, IIT Bombay43
User Testing Output
Qualitative output Task success Task time Errors
– Error “an action of the user that does not accomplish the desired goal”
– Severity of errors “the more difficult it is to recover from the error, more sever it is”
Few users face the problem / problem occurs infrequently
Many users face the problem / problem occurs frequently
Easy to recover Low Medium
Difficult to recover Medium High
© IDC, IIT Bombay44
User Testing Output
Qualitative output Task success Task time Errors Efficiency
– Efficiency = success + time– Number of mouse clicks– “Lostness” – Smith
© IDC, IIT Bombay45
User Testing Output
Qualitative output Task success Task time Errors Efficiency Satisfaction scores
– What to measure?~ Expectation, ease, satisfaction, time (no watch),
smiles / frowns – How to measure?
~ Likert scales, semantic differential scales– When to measure?
~ Pre-test, post-test, in the field– Try to make it look anonymous – online reduces bias
© IDC, IIT Bombay46
Planning a User Test
Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria
– This will be the output of the UT~ Qualitative~ Quantitative: success, time errors, efficiency,
satisfaction– Correctness criteria for each task– Things to measure– Things to watch out for– When to help users
© IDC, IIT Bombay47
Planning a User Test
Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria Invite / recruit appropriate users
– Matching personas, target markets– Covering independent variables– Professional recruiters– Screener– Telephonic screening
© IDC, IIT Bombay48
User Testing Screener
Make it short – Less than 20 questions that take about 5 minutes to run
through. Be clear and specific
– Avoid questions that can lead to ambiguous answers Avoid unnecessary questions
– A screener is not an interview– Each question should eliminate some users
Order questions from general to specific – Try to eliminate more people sooner
Communicate the format of the UT and the expected time commitments – Clearly communicate if the user needs to be prepared
in some way Plan on drop outs and no-shows
– Recruit 20% more users than you need
© IDC, IIT Bombay49
Planning a User Test
Determine the key question that the UT should answer Determine test tasks Determine evaluation criteria Invite / recruit appropriate users Prepare the infrastructure
– Carry out all test tasks to ensure readiness– Carry out pilot tests for practice
© IDC, IIT Bombay50
User Testing Process
Introduce the user to the topic and put him/her at ease– Explicitly say that it is the product that is being tested– Explain that the product is new and untested– Promise confidentiality, inform about any recordings– Avoid large crowds, conspicuous cameras, interruptions other
than the planned ones Give test tasks one at time Give hints to only if the user is stuck for a long time
– Or if they run into a known usability problem After the tests, debrief the user clarify any doubts Review test plan before repeating with another user
© IDC, IIT Bombay51
Interview Structure
Start– Break ice – factual questions– Ask 1-2 open ended questions
Middle– Give tasks, one at a time– Contextualize findings – concrete, detailed data– Keep track of test goals
End– Debrief, answer questions, discuss insights– Ask for suggestions
Let the user talk – Avoid interruptions– Try to build a partnership
© IDC, IIT Bombay52
Questions for Non-directed Interviewing
Don’t imply the answer~ “We put in a lot of effort to design this product. Do
you like it?”
© IDC, IIT Bombay53
Questions for Non-directed Interviewing
Don’t imply the answer Questions should be non-judgmental
~ “Don’t you think that calorie advice will be more useful as a graph?”
~ “What do you learn from this graph?” or “With this tool select an appropriate dessert for the family.” then observe and probe as appropriate
© IDC, IIT Bombay54
Questions for Non-directed Interviewing
Don’t imply the answer Questions should be non-judgmental Keep questions “open-ended”
~ “Which of these features is most important to you?” implies “At least some features should be important to you.”
~ “Rate the importance of these features on a scale 1-5”
© IDC, IIT Bombay55
Questions for Non-directed Interviewing
Don’t imply the answer Questions should be non-judgmental Keep questions open-ended Focus on experience, not extrapolation
~ “Do you think car pooling is a useful feature?” Implies “In the universe of all things, do you think that someone somewhere will find carpooling useful?”
~ “Given your lifestyle and the amount of commuting you do, do you think carpooling valuable to you right now?”
© IDC, IIT Bombay56
Questions for Non-directed Interviewing
Don’t imply the answer Questions should be non-judgmental Keep questions open-ended Focus on experience, not extrapolation Further, focus on immediate experience
~ “Is a in-car GPS interesting to you?” Implies “Will a in-car GPS facility be of interest to you at some time in your life?”
~ “If your car is equipped with this GPS facility, would you use it?” Then, if yes “How will you use it?”
© IDC, IIT Bombay57
Questions for Non-directed Interviewing
Don’t imply the answer Questions should be non-judgmental Keep questions open-ended Focus on experience, not extrapolation Further, focus on immediate experience Focus a question on a single topic
~ “How will you use the MP3 option at a party and while driving?”
© IDC, IIT Bombay58
Running a Non-directed Interview
Define ambiguous terms Restate answers to verify (= share interpretation)
– Listen to user’s questions carefully Use artifacts to ground discussions Don’t force opinions
– Never say that the user is wrong – you are here to find out what the user thinks, not to convince him / her about your opinions
Follow up with examples to probe– But wait for an undirected answer first
Review interview tapes– You will find your own ways of improvement
© IDC, IIT Bombay59
What can go wrong?
Reliability– “Can we get the result if we repeat the test”– 10x difference in individual computer usage speeds– Background of users
~ Access, age, familiarity with UTs
© IDC, IIT Bombay60
What can go wrong?
Reliability Validity
– “Does the test reflect the usability issues of interest”– Choice of users
~ Are they a representative sample of real users?– Choice of tasks
~ Are these tasks important? To whom?– Conclusions and their relation with the experiment
~ User took time in first attempt, but this is a frequent task
~ We tested a prototype, then built the whole system, but the prototype was too small, and the system is far too complex
~ We tested the new widget, but the real problem was finding it
~ Probing confusions and evaluating task time at the same time
© IDC, IIT Bombay61
What can go wrong?
Reliability Validity Experimental design (mostly more validity issues)
– Testing too many things~ The UT took 4 hours, but the user was exhausted~ You cannot text everything, you have to rely on the
abilities and the experience of the designers– Confusing independent & dependent variables
~ Measuring tech-savvyness from task success
© IDC, IIT Bombay62
What can go wrong?
Reliability Validity Experimental design Experimenter bias
– Experimenter likes one design– The way the task is given– What the experimenter chooses to see, probe
© IDC, IIT Bombay63
What can go wrong?
Reliability Validity Experimental design Experimenter bias Intervening variables
– Unforeseen circumstances that might interfere with the testing or performance~ Interruptions in task time~ User took three attempts to enter amount (Rs. 500) –
this gave him extra practice with the PIN
© IDC, IIT Bombay64
User Testing
Gives ‘better’ results than heuristic evaluation– More confidence about findings
Needs more time and money
© IDC, IIT Bombay65
Task: Find 2-3 UTs of Each Type
Qualitative output Task success Task time Errors Efficiency Satisfaction scores
66
Usability Evaluation Methods
Heuristic evaluation User Testing
Think Aloud Protocol
© IDC, IIT Bombay67
Think Aloud Protocol
A set of users use the product while continuously thinking out aloud– An evaluator notes all comments
Users need practice before real tasks– Use a short, familiar but unrelated task
Result is a summary of qualitative data– Excellent for formative evaluation– Can’t get performance data, particularly tasks times
Videotaping sessions helps capture nuances
© IDC, IIT Bombay68
Thinking Aloud Protocol Variants
Constructive Interaction– Two users use the system together
Retrospective Testing– Let users think aloud in a video-tape playback
Coaching method– Let an expert teach a novice
69
Usability Evaluation Methods
Heuristic evaluation User Testing
Think Aloud Protocol Card sort
– Louis Rosenfield, Peter Morville, Information Architecture for the World Wide Web
© IDC, IIT Bombay70
Categorise...
Brinjal
Brush
Camera
Cauliflower
Chicken
DVDs
Electronics
Easel
Eraser
Fruits
Food
Mango
Milk
Mobile phone
Non-veg
Notebook
Pen
Pencil
Potatoes
Raw Banana
Rice
Scale
Shampoo
Stationery
Sweet potatoes
Tomatoes
Toiletries
Toothpaste
Vegetables
Water colours
© IDC, IIT Bombay71
What did you use as categories? Did you make new categories?
Brinjal
Brush
Camera
Cauliflower
Chicken
DVDs
Electronics
Easel
Eraser
Fruits
Food
Mango
Milk
Mobile phone
Non-veg
Notebook
Pen
Pencil
Potatoes
Raw Banana
Rice
Scale
Shampoo
Stationery
Sweet potatoes
Tomatoes
Toiletries
Toothpaste
Vegetables
Water colours
© IDC, IIT Bombay72
Where did you put these?
Brinjal
Brush
Camera
Cauliflower
Chicken
DVDs
Electronics
Easel
Eraser
Fruits
Food
Mango
Milk
Mobile phone
Non-veg
Notebook
Pen
Pencil
Potatoes
Raw Banana
Rice
Scale
Shampoo
Stationery
Sweet potatoes
Tomatoes
Toiletries
Toothpaste
Vegetables
Water colours
© IDC, IIT Bombay73
Card Sort Technique
Make index cards– Categories– Sub-categories– Content– Number the cards (for later analysis)
© IDC, IIT Bombay74
Card Sort Technique
Make index cards Give a practice task
– People we know– Celebrities– Fruits and vegetables
© IDC, IIT Bombay75
Card Sort Technique
Make index cards Give a practice task Ask users to
– Make piles of cards– Label piles– One pile could be ‘I don’t care’ pile– Think aloud
Take notes while users work
© IDC, IIT Bombay76
Card Sort Variables
Open / Closed– Totally open – users write their own cards and
categories– In between – cards are given, users write category and
sub-category labels– Totally closed – pre-labelled cards and categories
– Early on – use open (discovery)– Later on – closed (validation)
© IDC, IIT Bombay77
Card Sort Variables
Open / Closed Phrasing
– Word, phrase, sentence~ Equipment, Having fun, Honesty is the best policy
– Category with sample sub-categories~ Equipment (Computers, Lathe machines, ACs)
– Picture, question, answer, topic, task
© IDC, IIT Bombay78
Card Sort Variables
Open / Closed Phrasing Granularity
– High-level or detailed– Paragraphs, pages, main pages, sub-sites– Focussed or generic– Focussed or random
© IDC, IIT Bombay79
Card Sort Variables
Open / Closed Phrasing Granularity Heterogeneity
– Early on – mix it up– Later on – high consistency
© IDC, IIT Bombay80
Card Sort Variables
Open / Closed Phrasing Granularity Heterogeneity Cross-listing
– Early on – avoid copies (structure)– Later on – allow copies (navigation)
© IDC, IIT Bombay81
Card Sort Variables
Open / Closed Phrasing Granularity Heterogeneity Cross-listing Qualitative / Quantitative
– Qualitative suitable for CI / mind-mapping / evaluating IA
– Quantitative suitable for summative evaluation, to do cluster analysis
© IDC, IIT Bombay82
Card Sort Analysis
Users’ terminologies Mind-maps
– Users’ conceptual models % times two cards go together % times a card goes in a category
83
Questions?
Heuristic evaluation User Testing
Thinking Aloud Protocol Card sort Feedback from real life usage
– Observational field studies– Logging actual use– User Feedback– Questionnaires and Interviews– Focus groups
84
Assignment 4
Pick a topic of your interest– Accessibility, HCI and robotics, e-learning,
Read 5 recent papers (2 years) on this topic Prepare a 10-minute presentation on this topic
– Submit a 2-page abstract with references Inform your topics to the Aakash and Abhishek by next class
– Presentations start from the class of 19-3– The presentation slots will be drawn randomly and
announced by 12-3– This assignment will be evaluated by your peers