Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003.
31
Embed
Usability and Human Factors Usability Evaluation Methods Lecture b This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Usability and Human Factors
Usability Evaluation Methods
Lecture b
This material (Comp15_Unit5b) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003.
Usability Evaluation MethodsLearning Objectives
2
• Conduct a cognitive walkthrough (Lecture b)
• Design appropriate tasks for a usability test (Lecture b)
• Describe the usability testing environment, required equipment, logistics, and materials (Lecture b)
Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Cognitive Walkthrough (Polson et al, 1992)
3Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
• Kind of cognitive task analysis• Assesses system usability• Criteria focuses on cognitive processes
needed to perform tasks• Identifying sequences of actions and
subgoals to successfully complete a task• Assigning causes to usability problems• Are the cues provided by the interface
sufficient to perform task?
Cognitive Task Analysis (CTA)
• Tools and techniques for describing knowledge & strategies required for task performance– Hierarchical decomposition of goals and
component tasks
• Objective: – Yield information about the knowledge,
thought processes, and goal structures that underlie observable task performance
4Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Why Do a CTA?
• Develop theory of competent performance• Understand invariant features of a task
– There are invariant performance characteristics of any class of tasks
• Understand process of skill acquisition• Training and instructional resources
– e.g. manuals and tutorials• Develop methods for usability testing
– Design– Coding scheme for data analysis
5Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Cognitive Walkthrough Step 1: Preparations
6Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Step 2: Walkthrough Process
7Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Step 3: Explicate Sources of Potential
Problems
8Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
ATMs: Goal Structure
9Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
A Partial Walkthrough: ATMGoal: Obtain $80 Cash from Checking
Account
10Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Cognitive Walkthrough: Measure Glucose
11Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
CW: Blood Glucose 2
12Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
CW: Blood Glucose 3
13Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
CW Glucose Results
14Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Usability Testing
• Gold standard for usability evaluation• Set of techniques to collect empirical data
– while observing representative end users using the system under study to perform representative tasks
• Video-recorded• Provide information that can lead to systems that:
– Easy to learn and use– Satisfying to use– Provide utility and functionality that are valued by the
15Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Think-Aloud Protocol
• Method broadly used in cognitive research and usability testing
• User verbalizes his/her thoughts while performing a task– Report the contents of working memory– Session is audio and/or video recorded
• Transcript of think aloud is coordinated with video analysis
16Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Selection of Representative Users
• Users may differ:– including age, education, gender, computer
experience, etc.
• Select subjects based on relevant criteria (e.g., age, education)
• Fully representative not possible
• Convenience sample is less desirable
17Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Development of Test Plan
• Outline task and procedure– Informed by objectives– Prior testing– Constrained by time and setting– Ethical and IRB issues
• Exploratory – Characterize potential problems
• Controlled Experiment– Comparing 2 Interfaces
18Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Role of Researcher
• Neutral Observer vs. Active Participant
• Researchers may play a more interactive role in field testing – Guide the subject as necessary
• Skilled user will need a minimum• Novice may need step-by-step instructions
– Autonomy Rule: No more guidance than necessary
19Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Field Usability Testing
• Hybrid Method: Lab and Ethnography/ Field Study
• Naturalistic setting– Numerous constraints
• Proscribed set of tasks– Quasi- Experiment
• Video analysis is key– Intrusive
20Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Video-Analytic Usability Testing on Location: Old School
Microsoft Clipart
21Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Software-based Video Analysis
• Provides a video of all screen activity • Captures user via a webcam• Logs a wide range of events and system
interactions including mouse clicks, text entries, web-page changes and windows dialogue events (e.g., saving a document, selecting among a set of choices).
• Morae state of the art usability testing software
22Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Morae Video Analytic Usability Software
Khan, S.A., Ancker, J.S., Li, J., Kaufman, D., Hutchinson, C., Cohall, A., Kukafka, R. (2009)
23Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Data Analysis: Transcript
• Working document for video analysis
• Verbatim and Time Stamped– Every 10 to 30 seconds
• Iteratively modified document as coding categories become refined
• Add field notes and observations to the transcript
24Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Video Analysis: Granularity
25Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Macro-AnalysisMarch 18, 2002
I: … You can choose not to answer any questions. May I ask how old are you?
P: 74
I: What is or was your professional occupation.
P: I have always worked in stores as a salesperson
I: How long have you lived in this country?
P: 48, 49 years
I: If I may ask what level of education do you have?
P: I finished high school, but in Colombia. Before I got married, then I came here after I got married, a couple of years afterwards.
I: How long have you known that you have diabetes.
P: A couple of years. About two years or three. I have never felt that I have diabetes.
Profile:
74 years of age
Female
Native Spanish speaker
Originally from Colombia
48-49 years in US
High school education
Worked as salesperson in stores
Has two sons, one in NJ
Never used computer prior to IDEATel
Diabetes History:
Had it for 2-3 years
Did not have monitoring device prior to IDEATel
1y.2months in program
Good health/Has asthma
A lot of fluctuation in glucose values
Does not look at them over a period of time, watches it one day to the other.
Generally good memory remembers values for previous days
1.1 Table: (Kaufman, et al., 2003).
26Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Observations: Sending Results
27Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Micro-AnalysisTime
Dialogue and Conversational Code
Action Code System Response
Comments
7:28 R: Can you explain what you see now?P: It is calling the Internet. There it turns off, and leaves me with the system.
Body Position Change (BPC): Patient moves head, looks from side to sideGesture: Points at screen with index fingerAction: Takes mouse and clicks security screen awayBPC: keeps hand on mouse for a couple of seconds, then offAction: Takes mouse and clicks identification screen away.
The task is to access the web and patient immediately assumes the control
7:54 R: So far, so good. Action: Takes hand off mouse, hunches over keyboard, and puts in password.Action: Takes mouse and clicks OK button and hand off mouse.Action: Takes mouse in hand immediately as DM page comes up.
Password screen transitions to Diabetes Manager Page
Patient performs all actions without any prompting.
1.2 Table: (Kaufman, et al., 2003).
28Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Triangulate
29Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Usability Evaluation Methods Summary
• The value of usability evaluations in healthcare contexts has been well established
• Wide range of methods which vary in terms of their advantages and disadvantages
• Lectures illustrated how to use someof these methods including the cognitive walkthrough and usability testing
30Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods
Lecture b
Usability Evaluation MethodsReferences – Lecture b
References
1. Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. John Wiley & Sons, New York, NY.
2. Polson, P., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 741–773.
3. Preece, J., Rogers, Y., & Sharp, H. (2007). Interaction Design: Beyond Human-Computer Interaction (2nd ed.). West Sussex, England: Wiley.
Images
Slide 21: Microsoft Clipart
Slide 23: Khan, S.A., Ancker, J.S., Li, J., Kaufman, D., Hutchinson, C., Cohall, A., Kukafka, R. (2009). GetHealthyHarlem.org: developing a web platform for health promotion and wellness driven by and for the Harlem community. AMIA Annu Symp Proc.317–21.
Charts, Tables & Figures
1.1 Table: Kaufman, D.R., Patel, V.L., Hilliman, C., Morin, P.C., Pevzner, J, Weinstock, Goland, R. Shea, S. & Starren, J. (2003). Usability in the real world: Assessing medical information technologies in patients’ homes. Journal of Biomedical Informatics, 36, 45-60.
1.2 Table: Kaufman, D.R., Patel, V.L., Hilliman, C., Morin, P.C., Pevzner, J, Weinstock, Goland, R. Shea, S. & Starren, J. (2003). Usability in the real world: Assessing medical information technologies in patients’ homes. Journal of Biomedical Informatics, 36, 45-60.
31Health IT Workforce Curriculum Version 3.0/Spring 2012
Usability & Human Factors Usability Evaluation Methods