Dr Melissa Baysari, Research Fellow, Australian Institute of Health Innovation, University of New South Wales- Optimising Decision Support within Electronic Medication Management Systems

OPTIMISING DECISION

SUPPORT WITHIN EMMS

AN ONGOING CHALLENGE

MELISSA BAYSARI

+ JOHANNA WESTBROOK, RIC DAY, LING LI, KATE

RICHARDSON, ELIN LEHNBOM & MANY MORE

EMMS EFFECTIVENESS

Error rates fell by 66% at hospital A and 59.7% at hospital B

But most of the improvement was seen in “procedural” errors – e.g.

fewer incomplete orders

Little change in clinical errors (e.g. wrong doses) – those targeted by

decision support

DECISION SUPPORT (DS)

Can mean different things to different people

Computerised alerts

Pre-written orders

Reference material

What about:

Drop down lists?

Notes or instructions?

Calculators?

DS EFFECTIVENESS

Literature tells us that alerts can result in substantial

changes in prescribing behaviour

BUT

Most studies evaluate an alert for a specific condition or

problem

e.g. alerts designed to reduce the use of contraindicated drugs in

patients with renal failure drop in proportion of patients

receiving a contraindicated medication from 89% to 47%

Less evidence for the effectiveness of basic decision support

alerts within eMMS

e.g. few studies showing that DDI alerts lead to reductions in DDIs

aJAMIA 2005 12:269-74

ALERT FATIGUE

A consequence of too many alerts being presented

Main barrier to prescriber acceptance of computerised alerts

A significant problem for hospitals because it

results in user frustration & annoyance

leads to prescribers learning to ignore all alerts, even those that

present useful & sometimes safety critical information

Alert fatigue affects most doctors in most

organisations

most alerts are overridden

ALERTS @ SVH (MEDCHART)

Allergy

Therapeutic duplication

Dose range

Local messages

Pregnancy

50% alerts are for information only

10% prescribers must enter an

override reason

7 alerts do not allow prescriber to

continue

ALERT FATIGUE A PROBLEM?

1. Observations 2. Interviews 3. Chart audit

OPINIONS OF ALERTS

Registrar: It’s certainly helpful in, like I say, avoiding errors and

mistakes but I don’t think it really helps in deciding say what

antibiotic or what antihypertensive or whatever because that’s a

clinical decision

Registrar: The decision to prescribe something is based on your

clinical knowledge…by the time you type it in and prescribe it

you’ve already made that decision

Resident: I guess less words and more point forms would be

easier because then we wouldn’t have to scroll through

paragraphs and sentences of text

OPINIONS OF ALERTS

Registrar: It pops up so often which can be a very bad thing

because you’re dismissing it so often that you develop this sort of

mechanism so it can be bad in a sense that sometimes you might

miss some important things

Registrar: I at least scan them and work out what it is that they’re

trying to tell me. Often it’s saying you’ve just prescribed, do you

want to prescribe it again, and I’m like well yes, I do

Resident: I don’t have a problem with all the alerts because I

know what they say now before they even come up

CHART AUDIT

No reporting function in MedChart to allow us to extract alert

information - had to conduct a detailed audit of electronic

charts to identify alerts

Pharmacist randomly selected patients each day from a list

of all inpatients

180 medication charts reviewed (6 weeks)

The following info was recorded:

Patient info (MRN, age, sex)

# total active orders, # orders with 1 or more alerts

For orders with an alert: prescriber, med name, med schedule

(e.g. PRN), alert type

ONLY OVERRIDDEN ALERTS

Limitation: Only alerts that were overridden were visible on

charts for review

But our observational work showed that the proportion of

orders abandoned or changed is small (0-5%)

Organisations implementing eMMS should specify the

logging and reporting of alert data by vendors

RESULTS – PATIENTS & ORDERS

Mean patient age: 63.7 yrs (20-100 yrs)

58% patients were male

2209 orders were active

Mean: 12.3 orders/patient

96.8% of orders were initiated by junior doctors

RESULTS – ALERTS

600/2209 orders had 1 or more computerised alerts

27.2% of orders

934 alerts in total, mean 1.6 alerts/alerted order

Alert type # (% of total alerts)

Duplication 572 (61.2)

Local messages 232 (24.8)

Pregnancy 100 (10.7)

Allergy 21 (2.3)

Dose range 9 (1.0)

Total 934

PREGNANCY ALERTS

20 patients met the criteria (female, aged 12-55 yrs)

Of 119 meds ordered for these patients, 43.3% triggered a

pregnancy alert

Prescribers received on average 5 pregnancy alerts per

eligible patient (range 1-10 alerts)

½ the alerts for these eligible patients were pregnancy alerts

LOCAL MESSAGES

¼ alerts we found were local messages

Most offer prescribers advice rather than warning about a

safety critical event

Could these be removed and presented in a non-interruptive

fashion?

DUPLICATION ALERTS

Most frequent trigger = different drug, same therapeutic class

(40% of duplication alerts)

½ duplication alerts were triggered because a medication

was prescribed that was identical to, or in the same class as

a drug that had been ceased within the previous 24 h

During observations we noticed that a

number of duplication alerts were

being triggered because prescribers

were not using all the eMMS functions

PEOPLE USE SYSTEMS IN

UNEXPECTED WAYS

Most users of applications utilize only a sub-set of system

features

(Think of all the functions you DON’T use in Excel, on your

iPhone, on your washing machine…)

People use applications in less optimal ways to manage

problematic or poorly designed IT

E.g. users of an EHR used free-text boxes instead of the

appropriate functions because the functions were hard to find

STUDY AIM

To examine how the use of eMMS functions by prescribers can

influence alert generation

That is, to identify the proportion of duplication alerts triggered as

a result of prescribers not utilizing eMMS functions

SHORT-CUT IN EMMS

THEN, AND, OR = allow similar sequential, concurrent or

alternative orders for the same medication to be prescribed

together

e.g. Frusemide in the morning AND midday

+

These would save the prescriber up to 11 mouse clicks

During training, all Drs are shown where to find the short-

cuts and complete case scenarios using them

Drs are encouraged to use short-cuts as they save time

SHORT-CUT IN EMMS (2)

To make a change to an order on a patient’s chart, a doctor

should click on the order and edit the parameter (e.g. change

the dose), instead of ceasing and re-ordering the medication

PROCEDURE

For all orders where at least 1 alert was triggered we asked:

Could the use of a different system function (THEN, AND, OR,

or MODIFY) have prevented the alert from firing?

Yes = technically preventable

No = not technically preventable

PREVENTABLE ALERTS

189 alerts were technically preventable

= 1/3 of duplication alerts

= 20% of all alerts

Prescribers did not use the eMMS functions as intended,

despite the functions’ potential to improve efficiency of work

JAMIA 2012, 19: 1003-1010

WHY?

The efficient strategies are not known to users

and/or

The strategies are known but system design features are

poor

and/or

The strategies are not viewed as beneficial or consistent with

preferred prescribing practice

CONSISTENCY OR

EFFICIENCY?

There is a tension between designing systems which

replicate paper-based processes and integrate quickly into

clinical practice

vs.

Harnessing the advantage of technology to allow tasks to be

completed in more efficient ways, but which require a change

in work & cognitive processes

REDUCING ALERT FATIGUE

Following the discovery that too many alerts are being

presented, how do we decide what alerts to remove from the

system?

Previous study1:

Interviewed doctors & pharmacists

Found no alert types that all clinicians agreed could be turned off

Found specialties differed in the number and types of alerts they

thought could be safely turned off

1Van der Sijs et al. JAMIA. 2008;15(4):439-48.

THE DELPHI TECHNIQUE

Group facilitation technique used to obtain consensus

among experts in a systematic way

Consensus is reached by allowing participants to consider

their responses in light of the overall groups’ responses

Delphi previously used to:

Identify appropriate information to include in alerts

Determine what information about the user and context is helpful

in prioritizing and presenting alerts

STUDY AIM

To reach consensus among prescribers of different

specialties and with various levels of experience on

appropriate strategies for reducing alerts within eMMS

No previous studies have used Delphi for this purpose

Previous Delphi research has included recruitment of experts

in CPOE or decision support implementation, not users of

the system

SURVEY DEVELOPMENT

10-question web-based survey

Input was sought from prescribers, pharmacists & clinical

information system staff

In the survey, doctors were asked:

What alert types they found useful/not useful

What alert types, if any, they would remove from the system

To rate each alert type on a Likert scale of usefulness

Whether or not they believed 2 potential strategies for reducing

alerts numbers would compromise patient safety:

POTENTIAL STRATEGIES

Identified in our previous work on alert fatigue:

1. Modifying most local messages so that they were

presented as hyperlinks on the prescribing screen, rather

than interruptive alerts

2. Modifying therapeutic duplication alerts so that they fired

only when the initial order was active on a patient’s chart,

not when it was ceased within 24 hours

PROCEDURE

To recruit prescribers, an ad (with a link to the survey) was

posted in the weekly JMO bulletin sent to all JMOs at the site

In round 2, doctors were sent a personalized email

containing a link to their round 2 survey

Feedback about round 1 responses were incorporated

into each question in round 2:

SAMPLE ROUND 2 QUESTION

The percentages beside each option below indicate the proportion

of doctors who selected that option in round 1.

Q2. If you could remove only one alert type from the current alert

set in MedChart, which type would you remove?

In round 1, you selected ‘Pregnancy’.

☐ Allergy & intolerances (2%)

☐ Pregnancy (34%)

☐ Therapeutic duplication (28%)

☐ Local rule (13%)

☐ None, I’d not remove any alert type (23%)

CONSENSUS

Consensus was defined as 80% agreement between

participants on questions requiring a single response

Although consensus was not reached after 2 rounds, response

stability was apparent, making it unlikely that participants would

change views during a 3rd round

RESPONDENTS

Round 1: 47 prescribers, Round 2: 21 prescribers

Various specialties and levels of experience

Round 1

Alcohol and drug

Anesthetics

Cardiology

Clin Pharm

Dermatology

ED

Gastroenterology

Geriatrics

Surgery

Hematology

Immunology

ICU

Medical oncology

Nephrology

Neurology

Palliative care

Psychiatry

Rehabilitation

Respiratory

Urology

Night

shift/seconded

Round 2

Alcohol and drug

Cardiology

Clin Pharm

ED

Geriatrics

Surgery

Hematology

Immunology

ICU

Medical oncology

Neurology

Palliative care

Psychiatry

Rehabilitation

Night

shift/seconded

AREAS WHERE CONSENSUS

WAS REACHED

Prescribers agreed on what alert type should be retained

81% rated Allergy & intolerance alerts as the most useful alert

type

No participant believed this alert type should be removed

All participants rated this alert type as ‘often’ or ‘sometimes’ useful

Prescribers agreed that our suggested strategies would work

95% thought that changing local messages so they appeared as

hyperlinks on the prescribing screen would be safe

91% thought that changing duplication warnings so they only fired

when the initial order was active would be safe

AREAS WHERE NO

CONSENSUS WAS REACHED

0

10

20

30

40

50

Pro

port

ion o

f pre

scribers

Prescriber

responses to the

question ‘If you

could remove

one alert type

from the current

alert set in

MedChart, which

type would you

remove?’

ALERT USEFULNESS

0

10

20

30

40

50

60

Never Rarely Sometimes Often

Pro

port

ion o

f pre

scribers

Allergy

Pregnancy

Duplication

Local

Prescriber responses

to the question ‘How

useful is each alert

type in warning you

about prescribing

something potentially

dangerous for your

patients?’

STUDY CONCLUSIONS

We identified some strategies that users viewed as

appropriate for reducing alert numbers

1. Present local messages as hyperlinks

Not unexpected because many messages provide low

priority information

2. Ensure duplication alerts trigger when initial order is

active – this would eliminate more than ½ of these alerts

24 h time-frame is only useful for a small number of

medications (e.g. colchicine)

RESEARCH TRANSLATION

Based on observations, interviews and chart audit

Pregnancy alerts were removed

Many of the local messages were replaced

with corresponding pre-written orders

Next step – assess clinical impact of altering duplication

alerts so they fire only when the initial order is active

OTHER STRATEGIES

(not as easy to implement as they sound)

Tier alerts according to severity

Include only high severity alerts

Apply human factors principles in designing alerts

Customize alerts for doctors

CONCLUSIONS

Getting alerts right is a challenge

Most sensible approach: include only a few alert types and

provide alternative forms of DS to prescribers (e.g. pre-

written orders)

Continuously evaluate DS!

Quantitative and qualitative methods allow us to determine if DS

is working and why

Seeking input and feedback from users is invaluable

THANK YOU

Contact: [email protected]

This research is supported by NH&MRC Program grant #

568612