Page 1
1
EHRUsabilityTestReportofExscribeEHR(Version6.0)ReportbasedonNISTIR7742CommonIndustryFormatforUsabilityTestReports
DateofUsabilityTest:February5thtoFebruary7th,2018DateofReport:February14,2018
ReportPreparedBy:
TheUsabilityPeople,LLC
4000LegatoRoad,Suite1100Fairfax,VA22033
www.TheUsabilityPeople.com
Page 2
2
TABLEOFCONTENTSExecutiveSummary..................................................................................................................................................................3Introduction.................................................................................................................................................................................4Method............................................................................................................................................................................................5Participants.............................................................................................................................................................................5StudyDesign...........................................................................................................................................................................8Tasks..........................................................................................................................................................................................9TestLocation.......................................................................................................................................................................10TestEnvironment..............................................................................................................................................................11TestFormsandTools......................................................................................................................................................12ParticipantInstructions..................................................................................................................................................13Procedure.............................................................................................................................................................................14UsabilityMetrics................................................................................................................................................................15DataScoring...................................................................................................................................................................16
Results.........................................................................................................................................................................................18DataAnalysisandReporting........................................................................................................................................18Reliability.........................................................................................................................................................................18EffectivenessandEfficiency....................................................................................................................................19Satisfaction......................................................................................................................................................................20
SpecificTaskResultDetails(Includingcriteriaevaluated)............................................................................25DiscussionofFindings..........................................................................................................................................................50
Effectiveness..................................................................................................................................................................50Efficiency..........................................................................................................................................................................50Satisfaction......................................................................................................................................................................51SummaryofMajorFindings....................................................................................................................................52RiskAnalysis..................................................................................................................................................................53AreasforImprovement.............................................................................................................................................54
Appendices................................................................................................................................................................................55AppendixA:RecruitingScreener...............................................................................................................................55AppendixB:InformedConsentForm.......................................................................................................................57AppendixC:ParticipantGuide.....................................................................................................................................58AppendixD:SystemUsabilityScaleQuestionnaire............................................................................................73AppendixE:ComputerSystemUsabilityQuestionnaire..................................................................................74AppendixF.References..................................................................................................................................................76
Page 3
3
ExecutiveSummary
OnFebruary5thtoFebruary7th,2017,TheUsabilityPeople,LLCconductedasummative
usabilitytestoftheExcribeEHR.ThetestwasconductedintheFairfax,VAofficeofTheUsability
Peopleoverremotetele-conferencingsessionsusingGotoMeeting.Thepurposewastotestand
validatetheusabilityofthecurrentuserinterfaceandprovideevidenceofusabilityoftheExcribe
EHRastheEHRUnderTest(EHRUT).Ten(10)healthcareprovidersmatchingthetarget
demographiccriteriaparticipatedintheusabilitytestusingtheEHRUTinsimulated,but
representativetasks.
Thestudyfocusedonmeasuringtheeffectivenessof,efficiencyof,andsatisfaction(ISO
9241-11)withExscribeamongasampleofparticipantsrepresentingpotentialusersofthesystem.
Performancedatawascollectedontwenty-five(25)taskstypicallyconductedonanEHR.Tasks
createdwerebaseduponthecriteriaspecifiedwithinthetestprocedurestructureforevaluating
conformanceofElectronicHealthRecord(EHR)technologytothecertificationcriteriadefinedin
certificationcriteriaidentifiedin45CFRPart170SubpartCoftheHealthInformationTechnology:
2015EditionHealthInformationTechnology(HealthIT)CertificationCriteria.
ResultsofthestudyindicatedthattheExscribeEHRsystemwasquitesatisfactorywith
regardstoeffectivenessandefficiencyandthattheparticipantsweresatisfiedwiththesystem.
Page 4
4
Introduction
TheElectronicHealthRecordSystemUnderTest(EHRUT)testedforthisstudy,theExcribe
EHR(V6.0),wasspecificallydesignedtopresentmedicalinformationtohealthcareproviderson
desktopcomputersinstandardambulatoryhealthcaresettings.Thisstudytestedandvalidatedthe
usabilityoftheExscribeEHRsoftwareuserinterfaceandprovidesevidenceoftheusabilityofthe
ExscribeEHRwithrepresentativeexercisesandinrealisticuserconditions.Tothisend,measures
ofeffectivenessandefficiency,suchastimeontask,numberoferrorsmade,andcompletionrates
werecapturedduringusabilitytesting.Satisfactionwasassessed,andusercommentscollected
usingtwoindustry-standardquestionnaires:TheSystemUsabilityScale(SUS)andtheComputer
SystemUsabilityQuestionnaire(CSUQ).
Page 5
5
MethodParticipants
Ten(10)individuals(7womenand3men)participatedintheEHRUT(s)usingtheExscribe
EHR.Participantswerephysicians,nurses,medicalassistants,and/orotherhealthcare/HealthIT
practitioners.Participantswererecruitedfromadatabaseofpotentialparticipantsmaintainedby
TheUsabilityPeople,LLC.Thecontactscontainedwithinthisdatabaseweregeneratedfrompast
participants,andviapotentialparticipantsresponsestopostingsinInternetandsocialmediasites,
andalinkatthebottomofTheUsabilityPeoplewebsite.Thosewhorespondedtotheinvitationto
takepartinthestudyweredirectedtoanonlinequestionnairethatservedastheparticipant
screener.(ThescreeningquestionnaireisprovidedasAppendix1.)Participantsmeetingthe
criteriaforparticipationinthestudywerecontactedandscheduledviatelephoneandemail.
ParticipantsintheusabilitytestoftheExscribeEHRhadavarietyofhealthcare
backgroundsanddemographiccharacteristics.
Table1presentsparticipantcharacteristics,includingdemographics,professional
experience,computingexperience,andpreviousEHRexperience.Participantcharacteristicsreflect
theaudienceofcurrentandfutureusersandmeetthecriteriadesignatedinthe2015Edition
CertificationCompanionGuideforSafety-enhanceddesign-45CFR170.315(g)(3).Noneofthe
participantswerefromthevendororganization(Exscribe)thatproducedandsuppliedthe
evaluatedsystemnordidanyparticipanthaveanydirectconnectiontothetestingorganization
(TheUsabilityPeople,LLC).Allparticipantswerecompensatedfortheirtime.
Page 6
6
Table1.ParticipantCharacteristics
PartID Gender Age Education Role/Title
ProfessionalExperience(Months)
ComputerExperience(Months)
Experiencewith
Exscribe(Months)
AssistiveTechNeeds
P01 Male 50to59 Doctor MD 312 96 0 None
P02 Female 20to29
Tradetechnicalvocationaltraining MedicalAssistant 60 60 0 None
P03 Male 40to49Bachelor’sdegree RegisteredNurse 180 84 0 None
P04 Female 20to29
Tradetechnicalvocationaltraining LPN 36 36 0 None
P05 Female 50to59 Master’sdegreeInformatics
NurseSpecialist 72 96 0 None
P06 Female 40to49Bachelor’sdegree ClinicalAnalyst 252 192 0 None
P07 Male 40to49 AssociatedegreeChiropracticAssistant 144 36 0 None
P08 Female 40to49Bachelor’sdegree RegisteredNurse 36 36 0 None
P09 Female 50to59 Master’sdegree RegisteredNurse 72 60 0 None
P10 Female 30to39Bachelor’sdegree RegisteredNurse 144 180 0 None
Page 7
7
SummaryofParticipantCharacteristics:Participantshadexperiencewiththeoccupationandexpertisethatalignswiththecapabilityundertesting.Thecohortofuserswhoareselectedasparticipantswasvariedwiththeproductanditsintendedusersandwasnotlimitedtoclinicians.Thedemographiccharacteristicsofthetestparticipantcharacteristicsreflectedtheaudienceofcurrentandfutureusers.Gender Male 3Female 7 AgeRange 20to29 230to39 140to49 450to59 360to69 070to79 0 Education Somecollegecredit,nodegree 0Tradetechnicalvocationaltraining 2Associatedegree 1Bachelor’sdegree 4Master’sdegree 2DoctorateDegree 1 YearsofExperiencewithExscribeNone 10Upto3years 03to5years 05to10years 0Morethan10years 0
Page 8
8
StudyDesign
TheoverallobjectiveofthisusabilitytestwastouncoverareaswheretheExscribeEHR
systemperformedwell–thatis,effectively,efficiently,andwithsatisfaction–andareaswherethe
systemfailedtoservetheclinicaldocumentationandworkflowneedsofusers.Datafromthistest
maybeusedasabaselineforfuturetestsofupdatedversionsofExscribeand/orforcomparing
ExscribesoftwarewithotherEHRspresentingthesametasks.Inshort,thistestingservesasbotha
meanstorecordorbenchmarkcurrentusabilityandtoidentifyareaswhereimprovementsmustbe
made.
ParticipantshadarangeofexperiencewithEHRsingeneral,butnonehadanyprior
experiencewiththeExscribesystem.ParticipantscompletedtheExscribeusabilitystudyduring
individual60-90-minuteGotoMeetingsessions.Duringthetest,eachparticipantinteractedwith
variouscomponentsoftheExscribesoftwaresystem.Eachparticipantwasprovidedwiththesame
instructions.
TheExscribeEHRwasevaluatedforeffectiveness,efficiencyandsatisfactionasdefinedby
thefollowingmeasurescollectedandanalyzedforeachparticipant:
• Numberoftaskssuccessfullycompletedwithoutassistance
• Timetocompletethetasks
• Numberanddescriptionoferrors
• Pathdeviations
• Participant’sverbalizations(comments)
• Participant’ssatisfactionratingsofthesystem
Page 9
9
TasksAtotaloftwenty-five(25)taskswereconstructedbyTheUsabilityPeople,LLC.(inclose
collaborationwithExscribestaff)toberealisticandrepresentativeoftheactivitiesausermight
engageinwhileusingExscribeinactualmedicalsettings.Thetwenty-five(25)taskswerecreated
baseduponthecriteriaspecifiedwithinthetestprocedurestructureforevaluatingconformanceof
ElectronicHealthRecord(EHR)technologytothecertificationcriteriaasdefinedin45CFRPart
170SubpartCoftheHealthInformationTechnology:Standards,ImplementationSpecifications,
andCertificationCriteriaforElectronicHealthRecordTechnology.Thetasksfocusedonthetwelve
2015EditioncertificationcriteriaspecifiedbyONC,specifically:
• Section170.315(a)(1)Computerizedproviderorderentry–medications
• Section170.315(a)(2)Computerizedproviderorderentry–laboratory
• Section170.315(a)(3)Computerizedproviderorderentry–diagnosticimaging
• Section170.315(a)(4)Drug-drug,drug-allergyinteractionchecks
• Section170.315(a)(5)Demographics
• Section170.315(a)(6)Problemlist
• Section170.315(a)(7)Medicationlist
• Section170.315(a)(8)Medicationallergylist
• Section170.315(a)(9)Clinicaldecisionsupport
• Section170.315(a)(14)Implantabledevicelist
• Section170.315(b)(2)Clinicalinformationreconciliationandincorporation
• Section170.315(b)(3)Electronicprescribing
AcopyofthetaskspresentedtoparticipantsintheusabilitytestoftheExscribeEHRcanbefound
inAppendixC.
Page 10
10
TestLocation
AllparticipantsweretestedontheExscribeEHRsystemduringremoteconferencing
sessionsusingGotoMeeting.Eachparticipantwasrequestedinadvancetosecureaquietroom
withminimaldistractionsandadesktoporlaptopcomputerthatcouldconnecttotheInternetwith
aGotoMeetingsession.Althoughthetypeofcomputer,operatingsystemanddisplayresolutionof
theremoteparticipantsystemwasunknown,thesystemthatwasusedbythetestadministrator
andcontrolledbytheremoteparticipantwasaDellInspironLaptoprunningtheWindows10
professionaloperatingsystemataresolutionof1366x768pixels.DuringagivenGotoMeeting
session,onlythetestadministratorandparticipantcommunicatedwithoneanother.
TheGotoMeetingusabilitytestsessionwasconductedbyatestadministratorfromthe
testingorganization(TheUsabilityPeople,LLC)workingfromasmallconferenceroomatThe
UsabilityPeople’sFairfax,VAlocation.Adataloggerfromthetestingorganizationalsotook
detailednotesoneachsession,includingusercommentsandotherratingsfollowingeachtask.
Duringasessionboththetestadministratorandthedataloggercouldseeonlytheparticipant’s
screenandheartheparticipant’scomments,questions,andresponses.
Page 11
11
TestEnvironment
WhiletheEHRUTtypicallywouldbeusedinahealthcareoffice,orambulatorycenter
facility,testingoftheExscribeEHRsystemwasconductedviaremoteconnectionduringindividual
GotoMeetingsessions.EachparticipantcalledintoaGotoMeetingsessionandwasconnectedby
thetestadministratortotheapplication.
TheExscribesystemitselfonaWindows™clientplatformonaLANconnectionusinga
sampledatabasethatwassetupspecificallyforthetest.Participantsusedamouseandkeyboard
wheninteractingwiththeEHRUTandweregivenremotecontroloftheadministrator’s
workstationtoperformthetasks.
Page 12
12
TestFormsandTools
Aspartoftheusabilitytest,severaldocumentsandinstrumentswereused.Examplesofthe
documentsusedduringtheusabilitytest,includinganinformedconsentform,thetasks,andpost-
testquestionnaires,canbefoundinAppendicesBtoE,respectively.
Participants’interactionwiththeExscribeEHRsoftwarewascapturedandrecordeddigitally
usingtheMoraescreencapturesoftwarerunningonthetestadministrator’sworkstation.Verbal
responseswererecordedthrougheitherthemicrophoneintegratedintotheparticipant’scomputer
orthroughatelephoneconnection.Thisinformationwaselectronicallytransmittedtothe
administratorandtothedataloggerduringeachtestsession.
Page 13
13
ParticipantInstructions
Theadministratorreadthefollowinginstructionsaloudtoeachparticipant:
Thankyouforparticipatinginthisstudy.Yourinputisveryimportant.Oursessiontodaywilllastabout60-90minutes.Duringthattime,youwilluseaninstanceofanelectronichealthrecord.Iwillaskyoutocompleteafewtasksusingthissystemandanswersomequestions.
Pleasenotethatwearenottestingyou;wearetestingthesystem.Therefore,ifyouhaveanydifficultythismaymeanthatsomethingneedstobeimprovedinthesystem.Iwillbehereincaseyouneedspecifichelp,butIamnotabletoinstructyouorprovidehelpinhowtousetheapplication.
Overall,weareinterestedinhoweasy(orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Ididnothaveanyinvolvementinitscreation,sopleasebehonestwithyouropinions.Alloftheinformationthatyouprovidewillbekeptconfidentialandyournamewillnotbeassociatedwithyourcommentsatanytime.Shouldyoufeelitnecessaryyouareabletowithdrawatanytimeduringthetesting.Participantswerethengiventaskstocomplete.
Page 14
14
Procedure
Uponconnectiontotheonlinemeetingtool(GotoMeeting),eachparticipantwasgreeted,
hisorheridentityverified,andmatchedtoanameontheparticipantschedule.Participant
demographicinformationwasalsoverified,includinginformationregardingoverallcomputer
experienceandparticipantexperiencewiththeEHUT.Participantnameswerereplacedwith
participantIDssothatagivenindividual’sdatacannotbelinkedtohis/heridentity.Priorto
beginningtesting,eachparticipantreviewedandsignedaninformedconsentform(SeeAppendix
B).
StaffmembersoftheUsabilityPeople,ausabilitytestadministratoradministeredthetest.
Theadministratormoderatedthesessionbyprovidingbothverbalandwritteninstructionsforthe
overallusabilitytestandforeachofthetaskscomprisingthetest.Theadministratoralso
monitoredtasksuccess,pathdeviations,numberanddescriptionoferrors,andaudio-recorded
participantverbalcomments.Adataloggerloggedtasktimes,obtainedpost-taskratingdata,and
tooknotesonparticipantcommentsandadministratorfeedback.
Foreachofthetwenty-five(25)tasks,participantswereprovidedwritteninstructionsto
theircomputers.Followingtheadministrator’sinstructions,eachparticipantperformedeachtask
byfirstreadingthetaskthenstatinginhisorherownwordshisorherinterpretationofthetask
requirements.Whentheparticipant’sinterpretationmatchedtheactualgoalofthetask,the
administratorinstructedtheparticipanttobeginandtasktimingbegan.Tasktimewasstopped
andrecordedwhenthetestadministratorobservedontheirworkstationthattheparticipanthad
successfullycompletedthetask.Ifaparticipantfailedtocompleteataskbeforetheexpected
amountoftimeforeachtask,thattaskwasmarkedas“TimedOut.”Aftereachtask,thetest
administratoraskedtheparticipant,“Onascalefrom1to5,where1is‘VeryDifficultand5is‘Very
Easy,’howsatisfiedwereyouwiththeeaseofuseforthistask?”Thissameprocedurewas
conductedforeachofthetwenty-five(25)tasks.
Page 15
15
Followingcompletionofthetwenty-five(25)EHRtasks,theadministratorelectronically
presentedtotheparticipanttwopost-testquestionnaires(SystemUsabilityScale(SUS),see
AppendixDandComputerSystemUsabilityQuestionnaire(CSUQ),seeAppendixE).Afterthe
participantcompletedbothquestionnaires,theadministratorthankedeachparticipantforhisor
hertimeandallowedtheparticipanttomakeanycommentsonoraskanyquestionsaboutthe
systemand/orthetaskspresented.
Foreachsession,theparticipant’sschedule,demographicinformation,tasksuccessrate,
timeontask,errors,deviations,verbalresponses,andpost-testquestionnaireweredigitally
recorded.Thesystemwasthenresettopropertestconditionsforthenextparticipant.
UsabilityMetrics
AccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronic
HealthRecords(NISTIR7741,November2010)EHRsshouldsupportaprocessthatprovidesahigh
levelofusabilityforallusers.Thegoalisforuserstointeractwiththesystemeffectively,efficiently,
andwithanacceptablelevelofsatisfaction.Tothisend,metricsforeffectiveness,efficiencyand
usersatisfactionwerecapturedduringtheusabilitytesting.Thegoalsofthetestweretoassess:
• EffectivenessoftheExscribeEHRsoftwarebymeasuringparticipantsuccessratesand
errors.
• EfficiencyoftheExscribeEHRsoftwareSoftwarebymeasuringtheaveragetasktimeand
pathdeviations.
• SatisfactionwiththeExscribeEHRsoftwarebymeasuringease-of-useratings.
Page 16
16
DataScoringTable2detailshowtaskswerescored,errorsevaluated,andthetimedataanalyzed.Table2.ScoringProtocolsforEffectiveness,Efficiency,andSatisfaction
Measures RationaleandScoring
Effectiveness:• TaskSuccess
A task was counted as “Success” if the participant was able to achieve thecorrect outcome,without assistance,within the time allotted on a per taskbasis.ThetotalnumberofSuccesseswascalculatedforeachtaskandthendividedbythetotalnumberoftimesthattaskwasattempted.Resultsareprovidedasapercentage.
Effectiveness:• TaskFailures
If the participant abandoned the task, did not reach the correct answer orperformed it incorrectly, or reached the end of the allotted time beforesuccessfulcompletion,thetaskwascountedas“Fail.”Notasktimesweretakenforfailedattempts.Thetotalnumberoferrorswascalculatedforeachtaskanddividedbythetotalnumberoftimesthattaskwasattempted.Resultsarepresentedastheaverageerrorrate.Note:Notalldeviationsarecountedaserrors
Effectiveness:• Prompted
Successes
Becausesometasksaredependentuponthesuccessfulcompletionofprevioustasks,participantsmayreceivealimitednumberof“prompts”tohelppreparethesystemdataforthepre-requisitesforsubsequenttasks.Whenaparticipantwasable tocomplete thedataentryona taskwith3orfewer prompts, the taskwas counted as an “Assisted” completion. No tasktimeswererecordedforAssistedcompletions.
Efficiency:• Task
Deviations
The participant’s path (i.e., steps) through the application was recorded.Deviations occur if for example, the participant navigated to an incorrectscreen, clicked on an incorrect menu item, followed an incorrect link, orinteractedincorrectlywithanon-screencontrol.
Efficiency:• TaskTime
Each task was timed from the administrator’s prompt “Begin” until said,“Done.” If the participant failed to say, “Done,” timing stopped when theparticipantstoppedperformingthetask.Onlytasktimesfortasksthatweresuccessfullycompletedwereincludedintheaveragetasktimeanalysis.Averagetimepertaskwascalculatedforeachtask.
Page 17
17
Satisfaction:• EaseofUse
ratings• System
Satisfaction
Participant’ssubjective impressionof theeaseofuseof theapplicationwasmeasured by administering both a single post-task question aswell as twopost-sessionquestionnaires.Aftereachtask,theparticipantdeterminedonascaleof1to5theirsubjectivesatisfaction with performance on the task. These data are averaged acrossparticipants.Tomeasureparticipants’confidenceinandlikeabilityoftheEHRoverall,thetesting team administered electronic versions of the SystemUsability Scale(SUS)andtheComputerSystemUsabilityQuestionnaire(CSUQ).SeetheSUSquestionnaireasAppendixD.,andtheCSUQasAppendixE.
Page 18
18
Results
DataAnalysisandReporting
TheresultsoftheusabilitytestoftheExscribeEHRsoftwaresystemwereanalyzed
accordingtothemethodsdescribedintheUsabilityMetricssectionaboveandaredetailedbelow.
Notethattheresultsshouldbeevaluatedrelativetothestudyobjectivesandgoals,asoutlinedin
thestudydesignsectionabove.Thedatashouldyieldactionableresultsthat,ifcorrected,yield
material,positiveimpactonuserperformance.
Reliability
Duringthedatacollectionphase,itwasobservedthatthesystemprovidedausually
consistentandreliableinterfacetoeachparticipantastheycompletedtheirtasks.Aseach
participantcompletedtheirassignedtasks,thesystemprovidedthesameinformationand
respondedtotheirinputwiththesameverbiageandusingthesamemodeofcommunication(e.g.
Pop-upmessage,orembeddedassistance).
Page 19
19
EffectivenessandEfficiency
Table3presentsasummaryofoveralltaskperformanceshowingtask,meantimeontask,
taskcompletionrates,meanpathdeviationsandmeantasksatisfaction:
Table3.UsabilityTestResults
TaskMeanTaskTime
SD
CompletionRate(%)
Mean#Path
Deviations
SD
MeanTaskSatisfaction
SD
Task1-Demographics 2:26 1:29 80% 2.30 2.10 3.70 0.64Task2-Demographics-Change 1:30 0:45 100% 0.70 1.00 4.10 0.54Task3-ClinicalInfoRecon.Incorp. 3:49 1:33 90% 2.00 1.41 4.20 0.40Task4-AddanAllergy 2:06 1:08 100% 1.90 1.14 3.90 0.54Task5-ViewAllergyList-ChangeAllergy
0:27 0:09 100% 0.00 0.00 4.80 0.40
Task6-ImplantableDeviceList 0:56 0:19 100% 0.10 0.30 4.70 0.46Task7-ImplantableDevice-Change 0:27 0:22 100% 0.40 0.66 4.80 0.40Task8-ProblemList–Add 0:55 0:10 100% 0.80 0.60 4.50 0.50Task9-ProblemList-Change 0:23 0:26 100% 0.10 0.30 4.60 0.66Task10-ProblemList-Views 0:28 0:12 100% 0.10 0.30 4.20 0.60Task11-MedicationList-AddMedication
0:50 0:12 100% 1.20 0.40 4.10 0.54
Task12-MedicationList-ChangeMedication
0:42 0:11 100% 0.40 0.66 4.50 0.50
Task13-DiagnosticImaging-AddOrder
1:07 0:15 100% 0.60 0.66 4.40 0.66
Task14-ChangeDiagnosticImagingOrder
0:17 0:05 100% 0.10 0.30 4.80 0.40
Task15-LabOrder-Add 1:11 0:39 90% 1.10 1.70 3.90 0.83Task16-LabOrder–Change 0:42 0:08 100% 0.10 0.30 4.20 0.75Task17-Drug-DrugInteractionCheck
2:17 1:00 80% 2.40 2.15 2.60 0.66
Task18-DrugAllergyInteractionCheck
1:20 0:52 90% 1.90 1.87 2.90 0.94
Task19-ElectronicRx-NewRx 1:37 0:30 90% 1.50 1.75 3.60 0.66Task20-ElectronicRx-ChangeRx 1:41 0:56 80% 1.70 2.24 3.60 0.66Task21-ElectronicRx-RefillRx 0:41 0:16 100% 1.10 0.94 4.20 0.60Task22-ElectronicRx-Views 0:37 0:27 100% 0.30 0.90 4.00 0.77Task23-ElectronicRx-CancelRx 0:31 0:16 100% 0.10 0.30 4.20 0.87Task24-CDS-EnableRule-ViewAttributes
1:18 0:25 100% 0.80 0.98 4.20 0.60
Task25-CCDAViewReconciledData
1:57 0:38 100% 0.70 0.64 4.70 0.46
Page 20
20
AsTable3shows,relativetooptimalperformancestandardsasdefinedbyExscribeand
TheUsabilityPeople,participantperformanceintheExscribeEHRsoftwareusabilitytestwasquite
satisfactory.Theoverallaveragetaskcompletionratewasninety-six(96)percent.
Satisfaction
IndividualTaskSatisfaction
Participantsverballyindicatedtheirsatisfactionwiththeeaseofuseforeachtaskusinga
scaleof“1”(“VeryDifficult”)to“5,”(“VeryEasy”).AsFigure1showsindividualtasksatisfaction
rangedfromalowof2.6outof5onTask17(Drug-Druginteractioncheck)toahighof4.8outof5
onTasks5,7,and14(Viewallergylist-ChangeAllergy,ImplantableDevice–Change,andChange
DiagnosticImagingOrder).
Page 21
21
Figure1.SatisfactionRatingsofIndividualTasks
0.00 1.00 2.00 3.00 4.00 5.00
Task1-Demographics
Task2-Demographics-Change
Task3-ClinicalInfoRecon.Incorp.
Task4-AddanAllergy
Task5-ViewAllergyList-ChangeAllergy
Task6-ImplantableDeviceList
Task7-ImplantableDevice-Change
Task8-ProblemList- Add
Task9-ProblemList- Change
Task10-ProblemList- Views
Task11-MedicationList-AddMedication
Task12-MedicationList-ChangeMedication
Task13-DiagnosticImaging-AddOrder
Task14-ChangeDiagnosticImagingOrder
Task15-LabOrder-Add
Task16-LabOrder- Change
Task17-Drug-DrugInteractionCheck
Task18-DrugAllergyInteractionCheck
Task19-ElectronicRx- NewRx
Task20-ElectronicRx-ChangeRx
Task21-ElectronicRx-RefillRx
Task22-ElectronicRx-Views
Task23-ElectronicRx- CancelRx
Task24-CDS-EnableRule-ViewAttributes
Task25-CCDAViewReconciledData
AverageTaskSatisfaction
Page 22
22
IndividualParticipantSatisfaction
Ingeneral,theparticipantsweresatisfiedwiththeeaseofuseoftheExscribeEHRsoftwaresystem.Thefollowingchartdisplaysoverallsatisfactionforeachparticipant:
Theaverageoveralltasksatisfactionratewas4.14outof5.
3.64
4.20
4.16
4.80
4.08
4.24
4.12
4.04
3.80
4.28
1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
P01
P02
P03
P04
P05
P06
P07
P08
P09
P10
OverallSatisfaction(average)
Page 23
23
SystemUsabilityScale
TheSystemUsabilityScale(SUS)isasimple,10-itemLikert-typeattitudescaleprovidinga
globalsubjectiveassessmentofusabilityfromtheuser’sperspective(JohnBrookeatDigital
EquipmentCompanydevelopedtheSUSin1986).TheSUSscaleisscoredfrom0to100;scores
under60representsystemswithlessthanoptimalusability,scoresover80areconsideredbetter
thanaverage.SeeAppendixDforacopyoftheSUS.
ThemeantotalSUSscoretheExscribeEHRwassixty-one(61)andrangedfromalowof
forty-five(45)andahighofseventy-five(75).Overall,participant-usersratedtheirsatisfaction
withtheExscribesoftwaresystemtobewithintherangeofausableandsatisfyingEHR.
ThefollowingchartshowstheSUSscorebyeachparticipant:
0 10 20 30 40 50 60 70 80
Participant1
Participant2
Participant3
Participant4
Participant5
Participant6
Participant7
Participant8
Participant9
Participant10
55
72.5
75
62.5
45
65
50
70
60
57.5
SUSScorebyParticipant
Page 24
24
ComputerSystemUsabilityQuestionnaire
UsingtheComputerSystemUsabilityQuestionnaire(CSUQ;Lewis,J.R.(1995).(See:IBM
ComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationandInstructionsforUse.
InternationalJournalofHuman-ComputerInteraction,7:1,57-78).),participantsratedeachof19
itemsoftheCSUQquestionnaireonascalefrom1to7,witharatingof7beingmostinagreement
withthepositively-wordeditem.Responsesforeachitemweresummedandaveragedtofour
scales–InterfaceQuality,InformationQuality,SystemUsefulness-andanoverallscale.See
AppendixEforacopyoftheCSUQ.
Figure2displaysCUSQratingsforeachofthefourscales.Ingeneral,participantsinthe
Exscribestudyratedsystemusabilitytobehigh.OnInterfaceQualitytheaveragescoreforthe
participantswas5.47/7;onInformationQualitytheaveragescore4.86/7;onSystemUsefulness
theaveragescorewas4.90/7;andtheoverallaverageCUSQscorewas4.96/7.
3.00 3.50 4.00 4.50 5.00 5.50 6.00 6.50 7.00
Overall Score
System Usefulness
Information Quality
Interface Quality
4.96
4.90
4.86
5.47
User Rating (Out of 7)
Computer System Usability Questionnaire
Page 25
25
SpecificTaskResultDetails
ParticipantNumber
Task1-Demographics
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 4:06 Fail 6 2P02 1:09 Success 1 4P03 5:37 Success 2 4P04 3:28 Success 1 4P05 1:10 Success 0 4P06 1:18 Success 1 4P07 1:31 Success 1 4P08 3:31 Success 4 4P09 3:51 Fail 6 3P10 1:43 Success 1 4
ExpectedTimeonTask 2:00 (SD) AverageTimeonTask 2:26 1:29 AverageTaskSatisfaction 3.70 0.64 Average#PathDeviations 2.30 2.10 PercentSuccess 80%
Page 26
26
ParticipantNumber
Task2-Demographics-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 3:32Success 3 3P02 0:51Success 0 4P03 1:54Success 0 4P04 1:05Success 0 5P05 1:24Success 1 4P06 1:01Success 1 4P07 1:18Success 0 4P08 1:22Success 0 4P09 1:43Success 2 4P10 0:53Success 0 5
ExpectedTimeonTask 0:30 (SD)AverageTimeonTask 1:30 0:45AverageTaskSatisfaction 4.10 0.54Average#PathDeviations 0.70 1.00PercentSuccess 100%
Page 27
27
ParticipantNumber
Task3-ClinicalInfoRecon.Incorp.
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 6:25Success 3 5P02 3:34Success 1 4P03 7:35TimeOut 4 4P04 2:41Success 0 5P05 2:51Success 0 4P06 2:53Success 2 4P07 3:02Success 2 4P08 4:08Success 3 4P09 4:33Success 4 4P10 4:11Success 1 4
ExpectedTimeonTask 4:00 (SD)AverageTimeonTask 3:49 1:33AverageTaskSatisfaction 4.20 0.40Average#PathDeviations 2.00 1.41PercentSuccess 90%
Page 28
28
ParticipantNumber
Task4-AddanAllergy
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 4:38 Success 4 3P02 1:10 Success 2 3P03 2:56 Success 1 4P04 2:13 Success 2 4P05 1:11 Success 1 5P06 1:22 Success 2 4P07 1:43 Success 1 4P08 1:32 Success 1 4P09 3:19 Success 4 4P10 0:54 Success 1 4
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 2:06 1:08 AverageTaskSatisfaction 3.90 0.54 Average#PathDeviations 1.90 1.14 PercentSuccess 100%
Page 29
29
ParticipantNumber
Task5-ViewAllergyList-ChangeAllergy
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:27 Success 0 4P02 0:18 Success 0 5P03 0:21 Success 0 5P04 0:51 Success 0 5P05 0:20 Success 0 5P06 0:22 Success 0 5P07 0:23 Success 0 5P08 0:33 Success 0 5P09 0:31 Success 0 4P10 0:20 Success 0 5
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:27 0:09 AverageTaskSatisfaction 4.80 0.40 Average#PathDeviations 0.00 0.00 PercentSuccess 100%
Page 30
30
ParticipantNumber
Task6-ImplantableDeviceList
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:39 Success 0 4P02 0:40 Success 0 4P03 1:45 Success 0 4P04 0:51 Success 0 5P05 1:13 Success 1 5P06 0:51 Success 0 5P07 0:49 Success 0 5P08 0:55 Success 0 5P09 0:48 Success 0 5P10 0:51 Success 0 5
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 0:56 0:19 AverageTaskSatisfaction 4.70 0.46 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
Page 31
31
ParticipantNumber
Task7-ImplantableDevice-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:28 Success 2 4P02 0:16 Success 0 5P03 0:34 Success 1 5P04 0:12 Success 0 5P05 0:13 Success 0 5P06 0:16 Success 0 5P07 0:15 Success 0 5P08 0:18 Success 0 5P09 0:41 Success 1 4P10 0:18 Success 0 5
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:27 0:22 AverageTaskSatisfaction 4.80 0.40 Average#PathDeviations 0.40 0.66 PercentSuccess 100%
Page 32
32
ParticipantNumber
Task8-ProblemList-Add
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:47 Success 0 5P02 0:49 Success 0 5P03 1:19 Success 0 5P04 0:47 Success 1 5P05 1:02 Success 1 4P06 0:51 Success 1 5P07 0:48 Success 1 4P08 0:44 Success 1 4P09 0:58 Success 2 4P10 1:03 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 0:55 0:10 AverageTaskSatisfaction 4.50 0.50 Average#PathDeviations 0.80 0.60 PercentSuccess 100%
Page 33
33
ParticipantNumber
Task9-ProblemList-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:33 Success 0 4P02 0:14 Success 0 5P03 1:38 Success 1 3P04 0:13 Success 0 5P05 0:08 Success 0 5P06 0:09 Success 0 5P07 0:10 Success 0 5P08 0:11 Success 0 5P09 0:18 Success 0 4P10 0:11 Success 0 5
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:23 0:26 AverageTaskSatisfaction 4.6 0.66 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
Page 34
34
ParticipantNumber
Task10-ProblemList-Views
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:01 Success 1 5P02 0:19 Success 0 4P03 0:18 Success 0 5P04 0:26 Success 0 5P05 0:35 Success 0 4P06 0:26 Success 0 4P07 0:22 Success 0 4P08 0:20 Success 0 4P09 0:31 Success 0 3P10 0:25 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:28 0:12 AverageTaskSatisfaction 4.2 0.60 Average#PathDeviations 0.1 0.30 PercentSuccess 100%
Page 35
35
ParticipantNumber
Task11-MedicationList-AddMedication
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:42 Success 1 4P02 0:55 Success 1 4P03 1:13 Success 1 5P04 1:05 Success 1 5P05 0:38 Success 1 3P06 0:44 Success 1 4P07 0:33 Success 2 4P08 0:45 Success 1 4P09 0:55 Success 2 4P10 0:53 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 0:50 0:12 AverageTaskSatisfaction 4.10 0.54 Average#PathDeviations 1.20 0.40 PercentSuccess 100%
Page 36
36
ParticipantNumber
Task12-MedicationList-ChangeMedication
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:11 Success 2 4P02 0:47 Success 0 5P03 0:33 Success 0 5P04 0:44 Success 1 5P05 0:40 Success 0 4P06 0:31 Success 0 4P07 0:42 Success 1 4P08 0:33 Success 0 4P09 0:35 Success 0 5P10 0:44 Success 0 5
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 0:42 0:11 AverageTaskSatisfaction 4.50 0.50 Average#PathDeviations 0.40 0.66 PercentSuccess 100%
Page 37
37
ParticipantNumber
Task13-DiagnosticImaging-AddOrder
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:35 Success 1 3P02 1:14 Success 1 4P03 0:56 Success 0 4P04 0:53 Success 0 5P05 0:47 Success 0 5P06 0:58 Success 1 5P07 1:03 Success 0 4P08 1:18 Success 0 5P09 1:26 Success 1 5P10 1:01 Success 2 4
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 1:07 0:15 AverageTaskSatisfaction 4.40 0.66 Average#PathDeviations 0.60 0.66 PercentSuccess 100%
Page 38
38
ParticipantNumber
Task14-ChangeDiagnosticImagingOrder
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:23 Success 0 4P02 0:20 Success 1 4P03 0:16 Success 0 5P04 0:18 Success 0 5P05 0:08 Success 0 5P06 0:10 Success 0 5P07 0:15 Success 0 5P08 0:22 Success 0 5P09 0:14 Success 0 5P10 0:19 Success 0 5
ExpectedTimeonTask 0:30 (SD) AverageTimeonTask 0:17 0:05 AverageTaskSatisfaction 4.80 0.40 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
Page 39
39
ParticipantNumber
Task15-LabOrder-Add
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:35 Success 1 3P02 0:41 Success 0 4P03 1:07 Success 0 5P04 0:44 Success 0 5P05 1:25 Success 1 3P06 1:13 Success 1 3P07 1:24 Success 1 4P08 1:17 Success 1 4P09 3:11 Fail 6 3P10 1:15 Success 0 5
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 1:11 0:39 AverageTaskSatisfaction 3.90 0.83 Average#PathDeviations 1.10 1.70 PercentSuccess 90%
Page 40
40
ParticipantNumber
Task16-LabOrder-Change
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:38 Success 0 3P02 0:24 Success 0 4P03 0:51 Success 1 5P04 0:45 Success 0 5P05 0:46 Success 0 3P06 0:38 Success 0 4P07 0:41 Success 0 4P08 0:55 Success 0 4P09 0:44 Success 0 5P10 0:39 Success 0 5
ExpectedTimeonTask 1:00 (SD)
AverageTimeonTask 0:42 0:08 AverageTaskSatisfaction 4.20 0.75 Average#PathDeviations 0.10 0.30 PercentSuccess 100%
Page 41
41
ParticipantNumber
Task17-Drug-DrugInteractionCheck
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:33 Success 1 3P02 1:53 Success 2 3P03 2:40 Success 2 2P04 2:01 Success 0 4P05 3:02 Success 1 2P06 2:52 Success 1 2P07 2:28 Success 2 3P08 4:23 Fail 7 2P09 4:33 Fail 6 2P10 1:44 Success 2 3
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 2:17 1:00 AverageTaskSatisfaction 2.60 0.66 Average#PathDeviations 2.40 2.15 PercentSuccess 80%
Page 42
42
ParticipantNumber
Task18-DrugAllergyInteractionCheck
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:12 Success 2 2P02 0:32 Success 0 4P03 1:22 Success 2 2P04 1:04 Success 0 5P05 0:51 Success 0 3P06 0:46 Success 1 3P07 0:57 Success 1 3P08 2:46 Success 4 2P09 3:13 Fail 6 2P10 1:31 Success 3 3
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 1:20 0:52 AverageTaskSatisfaction 2.90 0.94 Average#PathDeviations 1.90 1.87 PercentSuccess 90%
Page 43
43
ParticipantNumber
Task19-ElectronicRx-NewRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:12 Success 1 4P02 1:14 Success 0 4P03 2:32 Success 2 4P04 1:22 Success 0 4P05 1:01 Success 0 3P06 1:25 Success 1 4P07 1:19 Success 1 4P08 1:58 Success 3 3P09 2:19 Fail 6 2P10 1:34 Success 1 4
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 1:37 0:30 AverageTaskSatisfaction 3.60 0.66 Average#PathDeviations 1.50 1.75 PercentSuccess 90%
Page 44
44
ParticipantNumber
Task20-ElectronicRx-ChangeRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:14 Success 1 3P02 0:58 Success 0 5P03 3:25 Fail 6 3P04 2:23 Success 1 4P05 2:04 Success 0 3P06 0:45 Success 0 4P07 0:54 Success 0 4P08 3:30 Fail 6 3P09 1:39 Success 2 3P10 2:34 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 1:41 0:56 AverageTaskSatisfaction 3.60 0.66 Average#PathDeviations 1.70 2.24 PercentSuccess 80%
Page 45
45
ParticipantNumber
Task21-ElectronicRx-RefillRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:15 Success 1 3P02 0:35 Success 0 4P03 0:20 Success 0 5P04 1:04 Success 2 5P05 0:43 Success 1 5P06 0:27 Success 1 4P07 0:34 Success 1 4P08 0:37 Success 2 4P09 0:41 Success 3 4P10 0:38 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:41 0:16 AverageTaskSatisfaction 4.20 0.60 Average#PathDeviations 1.10 0.94 PercentSuccess 100%
Page 46
46
ParticipantNumber
Task22-ElectronicRx-Views
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 0:30 Success 0 5P02 0:21 Success 0 4P03 1:46 Success 3 2P04 0:23 Success 0 5P05 0:19 Success 0 4P06 0:22 Success 0 4P07 0:12 Success 0 4P08 1:04 Success 0 4P09 0:33 Success 0 4P10 0:41 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:37 0:27 AverageTaskSatisfaction 4.0 0.77 Average#PathDeviations 0.30 0.90 PercentSuccess 100%
Page 47
47
ParticipantNumber
Task23-ElectronicRx-CancelRx
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:02 Success 0 2P02 0:09 Success 0 5P03 0:12 Success 0 5P04 0:20 Success 0 5P05 0:32 Success 0 4P06 0:18 Success 0 5P07 0:22 Success 0 4P08 0:43 Success 1 4P09 0:44 Success 0 4P10 0:45 Success 0 4
ExpectedTimeonTask 0:30 (SD)
AverageTimeonTask 0:31 0:16 AverageTaskSatisfaction 4.2 0.87 Average#PathDeviations 0.1 0.30 PercentSuccess 100%
Page 48
48
ParticipantNumber
Task24-CDS-EnableRule-ViewAttributes
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 1:05 Success 0 4P02 0:51 Success 1 4P03 1:34 Success 0 4P04 1:01 Success 0 5P05 0:48 Success 0 5P06 0:53 Success 0 5P07 1:23 Success 1 4P08 1:44 Success 2 4P09 2:03 Success 3 3P10 1:42 Success 1 4
ExpectedTimeonTask 2:00 (SD)
AverageTimeonTask 1:18 0:25 AverageTaskSatisfaction 4.20 0.60 Average#PathDeviations 0.80 0.98 PercentSuccess 100%
Page 49
49
ParticipantNumber
Task25-CCDAViewReconciledData
TaskTime Outcome#Path
DeviationsTask
SatisfactionP01 2:11 Success 0 5P02 2:35 Success 0 4P03 1:46 Success 0 5P04 1:23 Success 0 5P05 0:58 Success 1 5P06 0:55 Success 1 5P07 2:01 Success 1 4P08 2:47 Success 1 5P09 2:38 Success 2 5P10 2:15 Success 1 4
ExpectedTimeonTask 1:30 (SD)
AverageTimeonTask 1:57 0:38 AverageTaskSatisfaction 4.70 0.46 Average#PathDeviations 0.70 0.64 PercentSuccess 100%
Page 50
50
DiscussionofFindings
Ingeneral,theparticipantsperformedverywellandweresatisfiedwithExscribeEHR
softwaresystem.Afewoftheparticipantsstruggledwithsomeportionsofafewtasksbutin
generalmostwereabletosuccessfullycompletealargemajorityofthetaskswithlittleorno
difficulty.Giventhatnoparticipantshadanypriorexperiencewiththesystem,thisshowsthatthe
ExcribeEHRisaneasy-to-useandaneasy-to-learnEHR.Participantsweremostlyabletoperform
alltaskssuccessfullyontheirownwithnoassistanceorexternaldocumentation.Theparticipant
averageperformanceratewashigh.TheExscribesystemappearstobeausableEHR.
Effectiveness
Ofthetwenty-five(25)taskspresented,alargemajorityofthetasksweresuccessfully
completedbyalloftheparticipants.Overallofparticipants,themeansuccessfultaskcompetition
ratewashighwithanoverallaveragerateofninety-six(96)percentindicatingthatingeneralthe
participantshadlittleornodifficultycompletingthetasks.
TheamountofpriorexperiencewithEHRsystemswasonlyslightlyrelatedtosuccessful
taskperformanceanderrorrates;participantswithmorepriorexperienceweremoreslightly
likelytosuccessfullycompletetaskswithlesserrorsthanthosewithlesspriorexperience.
Efficiency
Participantswhosuccessfullycompletedtasksgenerallycompletedthosetaskswithinan
acceptabletime.Sometaskswerecompletedmorequicklythanthecalculatedoptimaltime,while
severaltaskstookslightlylongerthanexpected.Thetasksthattookthelongestrequiredthe
participantstonavigatetoanunfamiliarportionofapage,interactwithaworkflow,locateand
selectspecificactions.Thosetasksmaybeperformedmorequicklywithaminorsetofupdatesto
theuserinterfaceand/ortheuserexperience.
Page 51
51
Someparticipantsmadeerrorswhenattemptingtonavigatetowardsolvingtheirassigned
tasks.Theseerrorsmaybeassociatedwiththoseparticipantsnotbeingfamiliarwithfeaturesand
notunderstandingthepresentedworkflowoftheExscribesoftwaresystem.Asnotedabove,prior
experiencewithEHRsystemswasrelatedtosuccessfultaskcompletion.
Satisfaction
ParticipantsweresatisfiedwiththeExscribeSoftwaresystem;ratingsontheSUS(mean=
61outofapossible100)andtheCSUQ(Overallscore=4.96outofapossible7.0)demonstrated
satisfactionwiththesystem.
OntheCSUQ,participantsrankedthescale“(InterfaceQuality)”highestofthethreescales,
suggestingthatthesystemprovidedaneffectiveandefficientuserinterface.Individualtask
satisfactionratingswererelatedtoindividualuserperformance.Thoseparticipantswhowereable
tosuccessfullycompletetaskswerealsomorelikelytorankthosetasksassatisfying,whilethose
participantswhodidpoorlyorwerenotabletocompleteataskrankedthosetasksasless
satisfying.
Page 52
52
SummaryofMajorFindings
ThisevaluationdemonstratedthattheExscribeEHRsystemisausablesystemwitha
relativelyshortlearningcurve.Participantswithnoexperienceusingthesystemexperiencedvery
littledifficultyunderstandingthenavigationandinformationarchitecture.Participantswithmore
overallEHRexperiencewereslightlymoreabletosolvemosttaskswithoutdifficultyorerror.
Page 53
53
RiskAnalysisThefollowingtablepresentsalistoftaskspresentedwiththeriskoferrorasobservedduringthetesting.Table5.RiskAnalysis
Task DescriptionPercentComplete
RiskStatus
Task1-Demographics Verifyandmodifypatientdemographics
80% Moderate
Task2-Demographics-Change Modifypatientdemographics 100% None
Task3-ClinicalInfoRecon.Incorp. Access,reconcileandincorporateC-CDAInformation
90% Low
Task4-AddanAllergy AddanallergytoallergyList 100% None
Task5-ViewAllergyList-ChangeAllergy Changeanallergyintheallergylist 100% None
Task6-ImplantableDeviceList EnterimplantabledeviceintoEHR 100% None
Task7-ImplantableDevice-Change Modifyimplantabledeviceinformation
100% None
Task8-ProblemList-Add AddanDiagnosistotheproblemlist 100% None
Task9-ProblemList-Change Modifyaproblemintheproblemlist 100% None
Task10-ProblemList-Views Viewhistoricalprobleminformation 100% None
Task11-MedicationList-AddMedication Addamedicationtomedicationlist 100% None
Task12-MedicationList-ChangeMedication
Modifymedicationinthemedicationlist
100% None
Task13-DiagnosticImaging-AddOrder Createadiagnosticimagingorder 100% None
Task14-ChangeDiagnosticImagingOrder Modifyimagingorder 100% None
Task15-LabOrder-Add Createalaboratoryorder 90% Low
Task16-LabOrder-Change Modifyalaboratoryorder 100% None
Task17-Drug-DrugInteractionCheck Demonstratedrug-druginteractionwarning
80% Moderate
Task18-DrugAllergyInteractionCheck Demonstratedrug-allergyinteractionwarning
90% Low
Task19-ElectronicRx-NewRx Electronicallyprescribeanewdrug 90% Low
Task20-ElectronicRx-ChangeRx Modifyanelectronicprescription 80% Moderate
Task21-ElectronicRx-RefillRx Refillaprescriptionelectronically 100% None
Task22-ElectronicRx-Views Viewhistoricalprescriptiondata 100% None
Task23-ElectronicRx-CancelRx Cancelanelectronicprescription 100% None
Task24-CDS-EnableRule-ViewAttributes MakeadministrativechangetoCDSsystemtoenablerule;Viewextendedattributes
100% None
Task25-CCDAViewReconciledData GenerateandviewanupdatedCCDAdocument
100% None
Page 54
54
AreasforImprovement
Thefollowingisapartiallistofpotentialareasforimprovement.Makingtheseandotherminor
enhancementswillimprovetheoveralluserexperienceoftheExscribesystemandincreasethe
effectiveness,efficiency,andsatisfactionforbothexperiencedandnoviceusers.
• IndicationofRequiredFields
o Afrequenterrorwascausedwhenparticipantsattemptedtosubmitaform
withintheinterfacethatcontainedrequiredfieldsthatwerenotcompleted.
Thisislikelybecausethesystemdidnotprovideaclearindicationofwhich
fieldsarerequired.Addingaconsistentvisualindicationofrequiredfields
wouldlikelyeliminatemanyoftheseerrorsobserved.
• Inconsistent“Add,“Search”andotherselectfunctions
o Manyerrorsoccurredwhenusersattemptedtoperforman“Add”or“Update”or
“Search”function.Thesystemprovidedseveraldifferentuserinteraction
patternsacrossseveraldifferentfunctionalareas.Providingaconsistentuser
interfaceanduserexperienceacrosstheentireapplicationmayhelptoreduce
usererrors.
• InconsistentUserInterfaceandUserExperience
o Severalerrorsoccurredwhentheusersattemptedtoperformanactionusinga
userexperiencepatternthatexistedinonefunctionalarea,butthispatterndid
notworkinthecurrentfunctionalarea.CreatingaUserExperienceandauser
interfacethatisconsistentacrossallfunctionalareaswouldhelptoimprovethe
overalluserexperienceofthesystem.
Page 55
55
Appendices
AppendixA:RecruitingScreener1.Areyoumaleorfemale? 2.Haveyouparticipatedinafocusgrouporusabilitytestinthepast6months?3.Doyou,ordoesanyoneinyourhomeworkinmarketingresearch,usabilityresearch,and/orwebdesign?4.Doyou,ordoesanyoneinyourhome,haveacommercialorresearchinterestinanelectronichealthrecordsoftwareorconsultingcompany?5.Whichofthefollowingbestdescribesyourage?
20-29,30-39,40-49,50-59,60-69,70-79,80-89,90-99,100andolder.6.Whichofthefollowingbestdescribesyoureducationlevel?
• Nohighschooldiploma• Highschoolgraduate,diplomaortheequivalent• Somecollegecredit,nodegree• Tradetechnicalvocationaltraining• Associatedegree• Bachelor’sdegree• Master’sdegree• Doctoratedegree(e.g.,MD,DNP,DMD,PhD)
7.Doyourequireanyassistivetechnologiestouseacomputer?8.Pleasedescribeyourmedicalornursingcredentials9.Whatisyourcurrentjobtitle?10.Howlonghaveyouheldthisposition?(numberofyears):11.Whattypeoffacilitydoyouworkinandwhatisyourrolethere?12.Howaremedicalrecordshandledatyour(main)workplace?
_____AllPaper _____SomePaper/SomeElectronic ___AllElectronic13.HowmanyEHRsdoyouuseorhaveyouworkedwith?14.Howmanyyearshaveyouusedanelectronichealthrecord?15.HowmanyyearshaveyouusedtheExscribesystem?
Page 56
56
16.Howmanyyearshaveyouusedcomputers?17Abouthowmanyhoursperweekdoyouspendusingacomputer?18.Whatcomputerplatform(s)doyouusuallyuse?19.Inthelastmonth,abouthowoftenhaveyouusedanelectronichealthrecord?
_____Didnotuselastmonth ___Everyday _____Afewtimesaweek.
Page 57
57
AppendixB:InformedConsentFormTheUsabilityPeoplewouldliketothankyouforparticipatinginthisstudy.Thepurposeofthisstudyistoevaluateanelectronichealthrecordssystem.Ifyoudecidetoparticipate,youwillbeaskedtoperform several tasks using the prototype and give your feedback. The studywill last about 60minutes.AgreementI understand and agree that as a voluntary participant in the present study conducted by TheUsabilityPeople.Iamfreetowithdrawconsentordiscontinueparticipationatanytime.IunderstandandagreetoparticipateinthestudyconductedandrecordedbyTheUsabilityPeople.Iunderstandandconsenttotheuseandreleaseof thevideorecordingbyTheUsabilityPeople. Iunderstandthattheinformationandvideoisforresearchpurposesonlyandthatmynameandimagewill not be used for any purpose other than research. I relinquish any rights to the video andunderstand thevideorecordingmaybecopiedandusedbyTheUsabilityPeoplewithout furtherpermission.Iunderstandandagreethatthepurposeofthisstudyistomakesoftwareapplicationsmoreusefulandusableinthefuture.IunderstandandagreethatthedatacollectedfromthisstudymaybesharedoutsideofTheUsabilityPeople.Iunderstandandagreethatdataconfidentialityisassured,becauseonlyde-identifieddata–i.e.,identificationnumbersnotnames–willbeusedinanalysisandreportingoftheresults.I agree to immediately raise any concerns or areas of discomfortwith the study administrator. IunderstandthatIcanleaveatanytime.Pleasecheckoneofthefollowing:____YES,Ihavereadtheabovestatementandagreetobeaparticipant.____NO,Ichoosenottoparticipateinthisstudy.Signature:_____________________________________Date_____________________
Page 58
58
AppendixC:ParticipantGuideORIENTATIONandINTRODUCTIONThankyouforparticipatinginthisstudy.Thesessiontodaywilllastapproximately60-90minutes.Duringthissession,youwilllookataspecialversionoftheExscribeEHR.Theproductyouwillbeusingtodaymaynotbeexactlylikethesoftwarethatyoumaybeusedto.Someofthedatamaynotmakesense,asitismerelyplaceholderdata.Thepurposeofthisstudyistogaugetheusabilityofthesoftware.Weareinterestedinhoweasy(orhowdifficult)thisversionistouse,andlearningaboutwhichofthefeaturesandbenefitswouldbemostusefultoyou.Wealsowanttoknowhowwecouldimproveit.Themoderatorwillguideyouthroughthetestingprocess.Pleaseusethesoftwareasyouwouldinarealclinicalsetting.Youwillbeaskedtosaveyourcommentsuntiltheendofataskortheendofthesessiontodiscussyourthoughtsfreely.Herearesomethingsyoushouldknowaboutyourparticipationinthissession
• TheModeratorwillguideyouthrougheachtask.• Pleasedonotworkahead.• Ifyounoticeanerror,justcanceloutandcontinuetothetask.Ifyoufeellostor
havedifficultycompletingthescenario,pleaseinformtheModerator.• TheTestingsessionwillbeaudiorecordedtheaudioforfurtherstudy.• Allinformationwillbekeptconfidential.Yournamewillnotbeassociatedwith
yourcommentsatanytime.Fictitiouspatientscenarioshavebeencreatedandpre-loadedinthesystemsoftware.TheModeratorwillaskyoutocompleteseveraltasksusingthesystem.Youwillbeaskedtoanswersomequestionsandtocompletesometasksonyourown.Trytocompletetasksasquicklyaspossible,withthefewestpossibleerrorsordeviations.Pleasedonotdoanythingmorethanasked.
Page 59
59
DemographicInformationYourpatientCandiceTaylorhasarrivedfortheirscheduledappointment.TheyhavebeenreferredtoourorganizationbyanoutsideprimarycarefacilityTask1(a.5)DemographicsBeforebeginningtheirappointment,youneedtoverifyorentersomeofthedemographicinformationthatisstoredwithinthesystemPreferredLanguage ‘FrenchPatientRace ‘Asian’
PatientEthnicity ‘NotHispanicorLatino’
BirthSex MaleSexualOrientation ‘Heterosexual’
GenderIdentity ‘IdentifiesasMale'
Switchtotheeditviewandverifyand/orenterandsavethisinformationintotheEHR.
Verifythatthesechangeshavebeensaved.
Page 60
60
Task2(a.5)Demographics–ChangesDuringyourconsultationwiththepatient,theyaskyoutomakethefollowingchangestotheirdemographicrecords:PreferredLanguage ‘EnglishPatientRace ‘Asian’
PatientEthnicity ‘NotHispanicorLatino’
DateofBirth 06/17/1950BirthSex MaleSexualOrientation ‘Unknown
GenderIdentity ‘IdentifiesasNon-ConformingGender
Switchtotheeditviewandverifyand/orenterandsavethisinformationintotheEHR.
Verifythatthechangeshavebeensaved.
Page 61
61
Task3(b.2)ClinicalInformationReconciliationandIncorporationYourpatienthascometotheirappointmentwitha“ThumbDrive”withClinicalinformationfromanotherprovider.Forthepurposesoftesting,wehaveplacedaHealthExchangeDocumentontheWindowsDesktop.ReconcileandincorporatetheMedicalInfoincluding:Medications,DiagnosesandAllergiesthatarepresentintheCCDAfilewithpatient’sexistingrecords
Medication
• AddtheAccupriltotheFinalReconciledMedications.• AddtheKeflextotheFinalReconciledMedications.• AddtheFlexeriltotheFinalReconciledMedications.
Allergy• AddtheSulfaAllergytotheFinalReconciledAllergyList.
Problems• AddHypertensiontotheFinalReconciledProblemList.• AddHeartDiseasetotheFinalReconciledProblemList.• AddHypercholseteromiatotheFinalReconciledProblemList.• AddArthritistotheFinalReconciledProblemList.
Confirmthatthereconciledlisthasbeenincorporatedintothecurrentpatientrecord.VerballyreportanyClinicalDecisionSupportmessagesthatmayappear.
Page 62
62
Task4(a.8)AddanallergyThepatientisreportingsomehivesaftertakingPenicillin.AddPenicillinasaMilddrugallergywithactivestatusandthesymptomofHives.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task5(a.8)ViewAllergyListandmakeachangeNoticethatthecurrentallergylistnowhastwoitems,ThePenicillinthatwasjustaddedandanallergytoSulfaDrugs.YounoticethatthereisnoinformationabouttheseverityandreactiontotheSulfadrug.ChangetheSulfaAllergysothatitisrecordedasanActiveMildDrugAllergywithareactionofRashVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Page 63
63
Task6(a.14)ImplantableDeviceListTheclienthashadanhistoricalsurgeryinwhichamedicaldevicewasimplanted.ThepatienthasprovidedyouwiththefollowingUniqueDeviceIdentifier(UDI):(01)10884521062856(11)141231(17)150707(10)A213B1(21)1234 (ThisUDInumbershouldbeinyour‘clipboard’andhasbeensavedasadesktopfile)PleaserecordthisMedicalInfointotheImplantableDevices(DeviceImplantHistory)list
Verifyand/orEnterandsavethisinformationintotheEHR.Verballystatethedevicedescription,identifiers,andotherattributes(brandetc.)ofthisdevice.Task7(a.14)ImplantableDeviceList-ChangeTheclientreportstheimplantabledevicewasremoved.ChangethedevicestatustoanInactivestatus.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved
Page 64
64
Task8(a.6)ProblemList.Duringtheevaluationofthepatient’sleftelbow,youchoosetoaddanotheritemtotheirlistofMedicalConditions.EnteranOrthopaedictypediagnosisofOsteoarthritisVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task9(a6)ProblemList-ChangeUsethesystemtoupdatethelistofmedicalconditionsothattheMedicalConditionthatwasjustenteredhasaStatussettoNEW
Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task10(a6)ProblemList–ViewsPleaseviewtheactiveproblemlistandconfirmthatitnowindicates:HypertensionHeartDiseaseHypercholesterolemiaArthritisAndOsteoarthritisModifytheviewsothatthesystemdisplaysonlyResolveditems.Viewtheproblemlistandverifythatthehistoricalproblemlisthasbeendisplayed.
Page 65
65
Task11(a.1)RecordmedicationYourpatientreportstakingTylenolforsomepaintheyhavebeenfeelingintheirleftelbow.Viewthepatient’scurrentmedslistandaddthefollowingmedication:Tylenol325mgtakenonce-a-dayevery4hours.Verifyand/orEnterandsavethisinformationintotheEHR.
Task12(a.1)ChangemedicationUponfurtherconversationwiththepatient,theyinformyouthattheyarenotinfacttakingTylenolbuthavebeentakingAleve.ChangethemedicationlistsothattheactualdrugtheyaretakingislistedasAleve(Naproxen)220mgcapsuleonceevery6to8hoursVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Page 66
66
Task13(a.3)DiagnosticImagingorderYouwanttofollowuponthepainthatyourpatienthasreportedintheirleftelbow.Usingthe“Superbill”functionsoftheEHR,Selectthepatient’sleftelbowandorderthefollowingprocedure:Orderanimagingstudy:73070ElbowLeft–2ViewsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
UponsavingtheimagingorderwithintheSuperbillfeature–theClinicalDecisionSupport(CDS)systemwilldisplayimportantinformation.VerballyidentifytheCDSitemthatappearsandoptionallyviewtheadditionalinformationthatisavailablethatsupplementstheinformationalmessage.
Task14(a.3)ChangeDiagnosticImagingorderOnsecondthought,youdecidethatyouwanttodoaslightlymorecomprehensiveimagingstudy.Changetheimagingorderthatwasjustenteredto:Orderanimagingstudy:73080ElbowLeft–3ViewsVerifyand/orEnterandsavethisinformationintotheEHR.
Uponsavingtheupdatedimagingorder-theClinicalDecisionSupport(CDS)systemwilldisplayimportantinformation.VerballyidentifytheCDSitemthatappearsandviewtheadditionalinformationthatisavailablethatsupplementstheinformationalmessage.
Page 67
67
Task15(a.2)RecordaLaborderBecauseyourpatienthasseveraldiagnosesassociatedwiththeiroverallhealth,youdecidethatyouwouldalsoliketoordersomelabwork.Again,usingthe“SuperBill”“PlanItems”feature,completeaLabRequestformforthefollowinglabs:Selectthefollowinglabtests:
• Glucose2hrpp• Creatinine
Verifyand/orEnterandsavethislabrequestintotheEHR.Verifythatanychangeshavebeensaved.
Task16(a.2)ChangeLaborderUponreviewingthelaborderjustentered,yourealizeyouwouldalsoliketorunafullUrinalysisMakethenecessarychangessuchthataUrinalysishadbeenaddedtothelaborder.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Page 68
68
Task17(a.4)Drug-drug,drug-allergyinteractionchecksYourpatienthashypertension.YouwanttoprescribeadrugusingtheePrescribefeaturetohelptreatthem.PrescribeLisinopril10mg,onceadayfor30daysDuringtheCompleteRXReviewthesystemwilldisplayamoderatewarningmessageregardingthisnewmedication.ReviewthewarningdetailsbutdoNOTcontinuewiththeselection/prescriptionofthisdrug.RemovethisRXfromthelistof“Pending”drugsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Task18(a.4)Drug-drug,drug-allergyinteractionThepatientpresentswithinfectedsoresontheirleftforearm.Addthefollowingmedication:PenicillinVpotassium125mg/5mLoralsolution3timesperdayfortheinfection.Thesystemwilldisplayawarningmessageregardingthisnewmedication.Reviewthewarningandcanceltheorderofthisdrug.Additionally,removethisRXfromthelistof“Pending”drugsVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Page 69
69
Task19(b.3)e-Prescribing-NewRXYouneedtocreateaprescriptionforyourpatientforsomethingthattheyarenotallergicto.CreatenewprescriptionforDoxycycline100mgwithaSIG:Take1capsulebyoralrouteonetimeperdayfor5days.Includeaquantityof5pillsand0refills.Finalizeandreviewthecompleteprescriptionandthenelectronicallysend(Using“TestPharmacyat100FakeAddress”astheselectedpharmacy.)tothepharmacyTask20(b.3)e-Prescribing-ChangeprescriptionUponfurtherthoughtandreviewoftheprescriptionabove,youhavedecidedthata150mgcapsuleisbetterforthispatient.Youwillneedtomakeachangetotheelectronicprescriptionthatyouhavejustordered.ViewtheirupdatedPrescriptionlistandMakethenecessarychangessothat:Doxycycline150MGwithaSIG:Take1capsulebyoralrouteonetimeperdayfor5days.Includeaquantityof5pillsand0refills.isinsteadprescribed.Verifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved
Page 70
70
Task21(b.3)eRX-RefillprescriptionYourpatienthasrequestedthattheirpharmacyrefillsomeoftheirmedications.AccessAlloftheirprescriptionsandrefilltheDoxycyclineprescriptionforanadditional2refillsVerifyand/orEnterandsavethisinformationintotheEHR.VerifythattherefillhasbeensavedTask22(b.3)eRX–ViewprescriptioninformationYoualsowanttoexaminethispatient’shistoricalmedications.FiltertheviewofthemedicationslisttoseethefullhistoryofprescribedmedicationsVerballyidentifyandhistoricalmedicationsthataredisplayedTask23(b.3)eRxCancelprescriptionDiscontinueorcanceltheDoxycyclineprescription.VerifythattheDoxycyclinehasbeenremovedfromthepatientcurrentprescriptions.
Page 71
71
Task24(a.4)CDSAdjustmentYouwanttomakeachangetothewarningmessagesand/orclinicalinformationsupportthatappearsinthesystem.AccesstheadministrativefunctionsforClinicalrulesandperformthefollowing:LocateandopenineditmodetheClinicalrulethathasbeencreatedfor“PQRS#39OsteoporosisforWomen“EnsurethatthisruleissettoActiveNavigatetoviewtheattributesofthisruleandverballystatethe:BibliographiclinkDeveloperFundingSourceVerifyand/orEnterandsavethisinformationintotheEHR.Verifythatthechangeshavebeensaved.
Page 72
72
Task25(b.2)CCDAViewreconcileddataThepatientisdonewiththeirappointment.Thepatientgoestothecheckoutdesk.Whilemakingtheirnextappointment,thepatientaskstohaveafullyup-to-dateContinuityofCareDocument(CCD)thatincludesthechangesmadeasaresultoftoday'svisit.YouneedtogenerateaCCDHealthExchangeDocumentforyourclient.Viewtheclient’sContinuityofCareDocumentForthepurposesofthistesting
• settheDocumentTypetoContinuityofCareDocument”• andselecttheoptiontoDisplaythedocumenttothescreen.
ViewtheCCDdocumentthatcanbesharedwiththepatient.
Page 73
73
AppendixD:SystemUsabilityScaleQuestionnaire
Page 74
74
AppendixE:ComputerSystemUsabilityQuestionnairePleaseprovideyourimpressionoftheusabilityofthesystembyansweringeachofthequestionsbelow:1.Overall,IamsatisfiedwithhoweasyitistousethissystemStrongly 1234567 NA StronglyDisagree Agree2.ItwassimpletousethissystemStrongly 1234567 NA StronglyDisagree Agree3.IcaneffectivelycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree4.IamabletocompletemyworkquicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree5.IamabletoefficientlycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree6.IfeelcomfortableusingthissystemStrongly 1234567 NA StronglyDisagree Agree7.ItwaseasytolearntousethissystemStrongly 1234567 NA StronglyDisagree Agree8.IbelieveIbecameproductivequicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree9.ThesystemgiveserrormessagesthatclearlytellmehowtofixproblemsStrongly 1234567 NA StronglyDisagree Agree10.WheneverImakeamistakeusingthesystem,IrecovereasilyandquicklyStrongly 1234567 NA StronglyDisagree Agree11.Theinformation(suchasonlinehelp,on-screenmessages,andotherdocumentation)providedwiththissystemisclearStrongly1 234567 NA StronglyDisagree Agree12.ItiseasytofindtheinformationIneeded
Page 75
75
Strongly 1234567 NA StronglyDisagree Agree13.TheinformationprovidedforthesystemiseasytounderstandStrongly 1234567 NA StronglyDisagree Agree14.TheinformationiseffectiveinhelpingmecompletethetasksandscenariosStrongly 1234567 NA StronglyDisagree Agree15.TheorganizationofinformationonthesystemscreensisclearStrongly 1234567 NA StronglyDisagree Agree16.TheinterfaceofthissystemispleasantStrongly 1234567 NA StronglyDisagree Agree17.IlikeusingtheinterfaceofthissystemStrongly 1234567 NA StronglyDisagree Agree18.ThissystemhasallthefunctionsandcapabilitiesIexpectittohaveStrongly 1234567 NA StronglyDisagree Agree19.Overall,IamsatisfiedwiththissystemStrongly 1234567 NA StronglyDisagree Agree
Page 76
76
AppendixF.ReferencesNISTIR7741-NISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronicHealthRecords,RobertM.Schumacher,UserCentric,Inc.,SvetlanaZ.Lowry,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Nov.2010.NISTIR7742-NISTCustomizedCommonIndustryFormatTemplateforElectronicHealthRecordUsabilityTesting,RobertM.Schumacher,UserCentric,Inc.,SvetlanaZ.Lowry,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Nov.2010.NISTIR7804-TechnicalEvaluation,Testing,andValidationoftheUsabilityofElectronicHealthRecords,SvetlanaZ.Lowry,MatthewT.Quinn,MalaRamaiah,RobertM.Schumacher,Emilys.Patterson,RobertNorth,,InformationAccessdivision,InformationTechnologyLaboratory,NationalInstituteofStandardsandTechnology,U.S.DepartmentofCommerce,NationalInstituteofStandardsandTechnology,Version0.2,15-Feb.2012.ONC-2015EditionCertificationCompanionGuideSafety-enhanceddesign-45CFR170.315(g)(3b)3/30/2016ONC-2015EditionCertification-TestProcedurefor§170.315(g)(3)SafetyEnhancedDesignCSUQ-Lewis,J.R.(1995).IBMComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationandInstructionsforUse.InternationalJournalofHuman-ComputerInteraction,7,57-78.SUSBrooke,J.(1996)SUS:a"quickanddirty"usabilityscale.InP.W.Jordan,B.Thomas,B.A.Weerdmeester&A.L.McClelland(eds.)UsabilityEvaluationinIndustry.London:TaylorandFrancis.