TURF – A Unified Framework For EHR Usability Jiajie Zhang & Muhammad Walji National Center for Cognitive Informatics & Decision Making in Healthcare
TURF – A Unified Framework
For EHR Usability
Jiajie Zhang & Muhammad Walji
National Center for Cognitive Informatics & Decision Making
in Healthcare
Research Goals for TURF
• Language for
– defining usability
• Theory for
– describing, explaining, and predicting usability differences
• Methods for
– evaluating
– measuring
– designing usability
• Software tools for
– partially automating the methods
2
What is Usability?
• Usability refers to how
–useful,
–usable,
–satisfying
• a system is for the intended users to accomplish goals in the work domain by performing certain sequences of tasks
3
Zhang J, Walji MF. TURF: A Unified Framework of EHR Usability. Journal of Biomedical Informatics. 2011, In Press.
Electronic Health Record
System Usability
TURF Framework for EHR Usability
4
Functions
Users
Representations
Tasks
Useful
Usable
Satisfying
Intrinsic Complexity
Extrinsic Difficulty
Rapid Usability Assessment
Comprehensive Usability Assessment
Laboratory based Focus on usableness Usability Problem Identification Collaborate with GCREC & Vendors
Also includes: Usefulness Clinical settings Design and redesign
Laboratory based Focus on usableness Usability Problem Identification Collaborate with GCREC & Vendors
Rapid Usability Assessment
5
Stage 1 Meaningful Use Objectives
6
Core Set Objectives (selected from 15)
1. CPOE 2. E-prescribing 3. Demographics 4. Problem list 5. Medication list 6. Medication allergy list 7. Vital signs, BMI, growth chart 8. Smoking status 9. E-Copy of health information 10. Clinical summary
Usability Violations
Consistency & Visibility 1. Window shown is
called the Patient Select popup but the Alerts being shown are for the physician and not specific to a patient and there is no indication to the user that these are for the user logged on to the system.
Memory & Match 2. The user has to
remember that the Alerts are for all patients. The user has to know where to look for the Alerts
3. The icons do not match the expected use of these symbols.
Representation Analysis – Expert Review
7
1. Consistency 2. Visibility 3. Match 4. Memory 5. Feedback + Error 6. Undo 7. Document
Record CBC for PATIENT,CLINICAL F
Operator Sequence:
KLM
Code Time (s)
… … …
Find the "Orders" tab M 1.2
Point to "Orders" tab P 1.1
Click "Orders" tab B 0.1
Find the "Common Orders" M 1.2
Point to "Common Orders" P 1.1
Click "Common Orders" BB 0.2
Wait for the system to show "Common
Orders" pop-up window W 0.1
Find the "LAB ORDERS" M 1.2
Find the "Search All Labs" M 1.2
Click "Search All Labs" B 0.1
Wait for the system to show "Order a Lab
Test" pop-up window W 0.2
Find "Available Lab Tests" M 1.2
Point to "Available Lab Tests" field P 1.1
Click "Available Lab Tests" field B 0.1
Type "CBC" K(3) 0.84
Point to "CBC" on the lab test list P 1.1
Click "CBC" on the lab test list B 0.1
Point to "Order" button P 1.1
Click "Order" button B 0.1
Point to "Close" button P 1.1
Click "Close" to close the "Order a Lab
Test" pop-up window B 0.1
… … …
Total Time 39 35.06
Task Analysis - Keystroke Level Model
Findings from Expert Reviews
9
23 28 19 38 48
15 13 15
124
46 72
125
35 44
21
29
93
30 22 45
165
44
88
154
2.5 2.5
2.3 2.2
2.6 2.6 2.7 2.7
2.4 2.4 2.5 2.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0
50
100
150
200
250
300
350
Nu
mb
er o
f U
sab
ility
Pro
ble
ms
Minor Issues Major Issues Avg Severity
Findings from Expert Reviews
10
508
426
362
355
343
90
0 100 200 300 400 500 600
Match
Feedback + Error
Memory
Visibility
Document
Undo
Findings from Task Analysis - Time on Task (seconds)
11
27
44
28
59
129
208
127
22
546
118
127
349
0 100 200 300 400 500 600
Demographics
Vital Signs
Body Mass Index
Growth Chart
Problem List
Medication List
Medication Allergies
Smoking Status
Clinical Summary
e-Copy of Health Info
e-Prescribing
CPOE
Optimal Time after Redesign
Average User Time
Optimal Time
194 seconds
97 seconds
129 seconds
Comprehensive Usability Assessment
Moving from the lab to the field
12
Assess Usefulness and
Usableness Redesign Re-evaluate
13
TURF in Action: Redesigning EHR
Harrington, Wood, Breuer, Pinzon, Howell, Pednekar, Zhu, & Zhang (AMIA 2011)
– AMIA Session S85 @ 8:30am on Wed 10/26
75 50
81
41
91
14
97
10
21
15
21
31
187
79
199
82
0
50
100
150
200
250
OpenVista: Steps Prototype: Steps OpenVista: Timing Prototype: Timing
Record
Modify
Retrieve
Total
Time (seconds) # Steps
99
28 19
53
0
20
40
60
80
100
120
Overhead Functions Essential Functions
OpenVista Prototype
TURF Architecture
Data Layer
Mapping editor
Data Collection
Presentation layer
Business layer
Data access components
Usability Evaluator System administrator Developer Provider
Data Capturing
Representation data: Screenshot, video, widget
Interaction data: Keystroke & mouse movement
User data: Profiles
Function Data: Work domain ontology
Model-Driven Analysis
Usability Metrics
Usability Benchmarks
Usability & safety patterns
EHR domain ontology
Entity-relational database Ontology database
Modeling
Modeling engine
Data integration
Repository management
Populate TURF
models
Data utilities Service agents
Other usability test/analysis services (e.g., Cogtool, Ulog, Noldus, etc.)
Modeling Analysis & Report
Users Other Users
TURF – Video / Screen Shots here
This project was supported by :
1) ONC SHARP Grant for Patient-Centered Cognitive Support
2) ONC Texas Gulf Coast Regional Extension Center for Health IT
Acknowledgements
SHARPC Project 1 Personnel
16
Project Leaders:
Jiajie Zhang Keith Butler Muhammad Walji
Project Advisor:
Mark Haselkorn
Project Co-Is:
Amy Franklin Debora Simmons Brent King Yan Xiao Ali Mokdad Emeka Okafor Axel Roessler
Consultants:
Ali Bahrami Ellen Bass Chris Esposito David Kieras Mark Musen David Woods
Postdocs, GRAs, Programmers:
Min Zhu Thomas Kannampallil Louis Lee Yingliu (Meg) Gu
REC Personnel:
Zhen Zhang Krisanne Graves Craig Harrington Yuanyuan Li