Clinical Information System Evaluation

8/12/2019 Clinical Information System Evaluation

http://slidepdf.com/reader/full/clinical-information-system-evaluation 1/16

Clinical Information System Evaluation

A Framework for Estimating CIS Value andIdentifying Opportunities for Improvement



Why Evaluate?

Three purposes of the evaluation process:

1. To compare the results with the goals and expected effects of the system

2. To direct work towards the expected result with the help of evaluation during thedevelopment of the system

3. To use the findings and outcomes as an experience base for the next project

(Peterson & Jelger, 19




Framework

Does IT make a difference?Literature Review

• The earliest studies were done at the Regenstrief Institute in the late 1970s, evaluating theffectiveness of CIS on adherence to influenza vaccination. Since then, there have beenhundreds of articles seeking to evaluate CIS in various domains: quality, efficiency, andcosts.

• Neumann, Parente, and Paramore (1996) reviewed eleven studies and presented aconsolidated analysis of each. They found that fully automating administrative functionscould save between $5 and $8 billion annually.

• The “Most Wired” report (Solovy, 2001) by Hospital and Health Networks and Deloitte

Consulting shows that “most wired” hospitals have better control of expenses and higher

productivity.




Framework

Does IT make a difference?Literature Review

• Chaudhry et al. conducted a systematic review on 257 articles evaluating healthinformation technology.

– Quality improvement: increased adherence to guidelines, enhanced disease surveillance, anddecreased medication errors.

– Major efficiency benefit: decreased utilization of care

– Effect on time utilization is mixed

– Empirically measured cost data is limited and inconclusive

– Major limitation of the literature is its generalizability

Chaudhry B, Ann Intern Med. 2006;144:E-12-E-



Two Approaches to CIS Evaluation

Objectivist ApproachQuantitative

Subjectivist ApproachQualitative

Important system attributes that can bemeasured and interpreted

Observation results are dependent oncontext and observer; different

individuals or groups may hold adifferent opinion about a systems value

(Burkle et. Al,



Two Approaches to CIS Evaluation

Has seldom produced positive evaluation results when applied to complex clinicalinformation systems

– Consumed large amounts of resources

– Expected and measured objective parameters, such as time saving, did not exceed those of t

control

Very often, the influence of such a system will not manifest in direct time savings, bu

in: – Improved cooperation between departments

– Increased quality of documentation

– Better patient care

Subjective Approach

Objective Approach

(Burkle et. al, 2001)



Objectives of the CIS Evaluation Framework

Internal

• To provide a method for assessing CIS systems in order toquantify value and identify opportunities for improvement

External

• To contribute to the broader healthcare community bydesigning a generalizable methodology for value-rating CISsystem technologies in order to facilitate decisions on

technology implementations




Framework

Does it makea difference?

Extent of

technologyfacilitation ofoutcomes ?

Is the CISused?



Benefits Realization Number

• Quantifies the benefits of the Clinical Information System

• Based on three questions: – Is the CIS being used?

– Does it make a difference in quality of care, clinician workflow, or business

decision making? – Is the change related to the technology implemented?

• Ranking value product of 3 numbers multiplied (highest 27) – Use 1-3

– Difference 1-3 – Sharing 1-3

Individual projects can be evaluated comparatively

for benefit realization



Ranking Scheme – Benefits Realization

BRN = Population Rank x Outcomes Rank x CIS Rank

Rank Population Outcomes

CIS contribution

to outcomes

1Limited use impacting< 25% of population

No harm averted,minor efficiency or

cost savings achieved

Small CIScontribution

2

Moderate use

impacting 50% ofpopulation

Minor harm averted,

moderate efficiency orcost efficiency

CIS plus otherapproaches

3Maximum use

impacting 95 - 100%

of population

Transferability toother centers, seriousharm averted, major

efficiencyimprovement, major

cost savings

Entirely related to CIS



Benefits of CIS Improvements (08)

Benefit for Patient

Care

Problem

Addressed

Technology

Applied

Outcomes BRN 1-27

Safe & EffectiveCare

•72% of pregnancyscreeningdocumentation iscomplete

•90% of eligiblepatients tested pre-operatively

May 06

Screeningquestionsstandardized andspecified throughconditional logicwith automaticordering of a test asappropriate

• 97% completescreeningdocumentation(35% improvement)

•1% improvement(91%) in eligiblepatients beingtested

April/May 08

2-3-2

12


75-80% compliancewith discharge

teaching for CHFpatients

Easy to use andmandatory

documentation ofdefined elementsfor education

100% compliancesince Feb 08

2-2-3

12



Benefits of CIS Improvements

Benefit for Patient

Care

Problem

Addressed

Technology

Applied

Outcomes BRN 1-27


Future ordersavailable formistaken activationon subsequent

admission

CCL script tocancel future orders2 weeks after orderplace (discharged

pts only)

Negligible futureorders available forinadvertentactivation on

readmission

3-3-3

27


Ordered carepotentially notcarried out becauseof un-activatedfuture orders

Remove futurestatus from orderset orders wherepossible

65% reduction inchance for careomissions

2-2-2

8

Safe & Effective

Care

73% compliance

with pneumococcalvaccine

Mandatory screen

and automatedorder

100% compliance

with appropriatevaccination

3-2-3

18



Benefits of CIS Improvements

Benefit for Patient

Care

Problem

Addressed

Technology

Applied

Outcomes BRN


Inappropriate dosesordered based onage, weight orkidney function

Dose rangechecking for singleand daily doses

28% of orderschange or advertedsecondary to alert

3-3-3

27


Increased risk forerror with manual

calculation ofmg/Kg doses forneonates

Dose calculatorand 48 new order

sets and almost700 ordersentences to guideappropriateprescribing

tbd 3-3-3

27


3.08 falls/1000patient days

95% compliance

with care process

Enhancedassessment tools,prompts for

interventions, alertsto physicians

2.2 falls/1000patient days

98.8 compliance

with care process

3-2-2

12



The Benefits of CIS Improvements

Benefit Category Problem

Addressed

Technology

Applied

Outcomes BRN


Less than 95%compliance withskin assessment

10% prevalence

New forms fordocumentation,rules forinterventions, MDnotification of issueelectronically

99% compliancewith skinassessment

7.8 % prevalence

3-2-2

12


4520 free textorders per monthcausing potentialomissions in care

Order set withdiscreet orders forcommonly usedorders

53% reduction inuse of free textorders (2320);subjectiveimprovement incliniciansatisfaction withorders

3-1-3

9





Addressed

Technology

Applied

Outcomes BRN 1-27

Coordinated Quality of shift

hand-offs

Automated reportthat abstracts EMRdata – physiciansign out andnursing SBAR pilot

40% improvementin perceivedefficiency of nursingreport and 8 foldimprovement inquality of reports

received.

1-3-2

6

Coordinated 1524/month

discharge

summaries

dictated byresidents

Electronic note 87% reduction indictated d/csummaries (July203) Annual costsavings $191,650

2-2-3

12

Timely & Efficient Radiology report

turn-around-time

Radiology voicerecognition

20% reduction inturn around time (2

days to 38 hrs. 7mins) – adoptiononly at 83% of

Attendings and44% of reports

2-1-3

6





Addressed

Technology

Applied

Outcomes BRN 1-27

Coordinated .6 preventable

codes per 100patient discharges

Mews report toguide RRT onproactiveevaluation ofpatients who might

deteriorate

.09 preventablecodes per 100discharges

3-3-2

18

Coordinated Lack of direct wayto communicatebetween patients &providers

Relay Health(patient access tochart) for InternalMedicine

30% patientsenrolled; patientsatisfaction results:

1-2-2

4


Lack ofmeasurement onclinical decision

support

performance relative tobenchmark

Dose Range anddrug/druginteractioncheckingimplemented

Passing score onthe Leapfrogclinical decisionsupport test

27

Clinical Information System Evaluation

Documents