Combating Alert Fatigue: New Best Practices...Level Seven (HL7) Clinical Decision Support (CDS) Working Group, which develops CDS standards in areas such as InfoButtons, order sets,

Today’s webinar is sponsored by

Wolters Kluwer Health Drug Information Solutions

Medi-Span® and Lexicomp®

Who we are:

Medi-Span- Embedded drug databases used in 60 EMRs, 1,000+ hospitals, and

37,000 retail pharmacies

Lexicomp- Referential drug information used in nearly 2,000 hospitals and by

80,000 mobile users

Dedicated to building innovative and trusted resources in multiple formats for use in any setting,

WKH Drug Information Solutions aims to lessen system-wide costs while helping reduce errors,

assist in improving patient safety and creating practical drug information solutions to increase

workflow efficiency throughout the continuum of care.

Combating Alert Fatigue:

New Best Practices

Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health


Howard Strasberg Vice President, Medical Informatics

Wolters Kluwer Health – Clinical Solutions

• Actively involved in standards development as a co-chair of the Health

Level Seven (HL7) Clinical Decision Support (CDS) Working Group, which

develops CDS standards in areas such as InfoButtons, order sets, and

decision support services.

• Prior to joining Wolters Kluwer Health in 2003, Howard was CEO of

Skolar, Inc., an online provider of clinical information and "in context"

continuing medical education (CME) for medical professionals.

• Howard received his MD degree from the University of Western Ontario

and his MS degree in Medical Information Sciences from Stanford

University. He is board certified in Family Medicine.

Our Speaker


2

Important tool to help reduce adverse drug events caused by

medication errors

Types of alerts (examples):

• Drug-Drug Interactions

• Drug-Allergy Conflicts

• Dosing

• Pregnancy/Lactation/Age/Gender Precautions

• Drug-Gene Interactions

Medication Safety Screening


3

2014 EHR Certification Criteria § 170.314 (a)(2)

• (i) Before a medication order is completed and acted upon during

computerized provider order entry (CPOE), interventions must

automatically and electronically indicate to a user drug-drug and drug-

allergy contraindications based on a patient’s medication list and

medication allergy list.

• (ii) Adjustments.

• (A) Enable the severity level of interventions provided for drug-drug

interaction checks to be adjusted.

• (B) Limit the ability to adjust severity levels to an identified set of

users or available as a system administrative function.

2014 EHR Certification Criteria


4

Alerts occur frequently.

Dutch study[1] – 9.6% of medication orders resulted in a drug-

drug interaction alert.

Major EMR using Medi-Span data:

• 0.12 drug-drug interaction alerts per order

• 0.095 dose alert per order

• 0.05 allergy alerts per order

• …

• Overall – 0.53 alerts per order

Medication Alerts

[1] Zwart-van Rijkom JE, Uijtendaal EV, ten Berg MJ, van Solinge WW, Egberts AC. Frequency and nature of drug-drug

interactions in a Dutch university hospital. Br J Clin Pharmacol. 2009 Aug;68(2):187-93. doi: 10.1111/j.1365-

2125.2009.03443.x. PubMed PMID: 19694737; PubMed Central PMCID: PMC2767281.


5

•Alerts can be conceptually divided:

• Clinically significant (signal)

• Not clinically significant (noise)

• Alert fatigue occurs when too many alerts are noise and

providers start to ignore the alerts

• A systematic review found that drug safety alerts are

ignored in 49%-96% of cases (Van Der Sijs. JAMIA 2006 [2])

• Partners study found that 53% of medication-related CDS

alerts were overridden (Nanji et al. JAMIA 2013 [3])

Alert Fatigue

[2] van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med

Inform Assoc. 2006 Mar-Apr;13(2):138-47. Epub 2005 Dec 15. PubMed PMID: 16357358; PubMed Central PMCID: PMC1447540.

[3] Nanji KC, Slight SP, Seger DL, Cho I, Fiskio JM, Redden LM, Volk LA, Bates DW. Overrides of medication-related clinical

decision support alerts in outpatients. J Am Med Inform Assoc. 2013 Oct 28. doi: 10.1136/amiajnl-2013-001813. [Epub ahead

of print] PubMed PMID: 24166725.


6

Alert Fatigue

Signal

Noise

Question 1:

Where do you draw the line?


7

Alert Fatigue

Signal

Noise

Precision = 18/28 = 0.64

Recall = 18/32 = 0.56


8

Alert Fatigue

Signal

Noise

Precision = 6/7 = 0.86

Recall = 6/32 = 0.19


9

Alert Fatigue

Signal

Noise

Precision = 30/63 = 0.48

Recall = 30/32 = 0.94


10

Alert Fatigue

Signal

Noise

Question 2:

How should we evaluate provider

responses to alerts?


11

• McCoy et al provided a framework for evaluating CDS alerts

and responses (JAMIA 2012; 19:346-352 [4])

• Overriding an alert may or may not be an appropriate

provider response; cannot just look at the override rate

Evaluation Framework

Provider

Response

Appropriate

Provider

Response

Inappropriate

Alert Display

Appropriate

(Signal)

Successful alerts Provider non-

adherence

(inappropriate

overrides)

Alert Display

Inappropriate

(Noise)

Justifiable

overrides

Unintended

adverse

consequences

[4] McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, Peterson JF. A framework for evaluating the

appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc. 2012 May-Jun;19(3):346-52.


12

• Nanji study in McCoy framework (assuming all alerts that

were not overridden were successful)

• Nanji: 53% of alerts were overridden AND 53% of alert

overrides were appropriate (depicted above as

28%/(28%+25%))


Provider

Response

Appropriate

Provider

Response

Inappropriate

Alert Display

Appropriate

(Signal)

47%

Successful (not

overridden)

25%

Inappropriate

overrides

Alert Display

Inappropriate

(Noise)

28%

Justifiable

overrides


13

• Evaluation of provider response shouldn’t be limited to the

ordering session; need to establish a time window

• Junior clinicians may initially override an alert, but after discussion

with senior clinicians, they may return to implement the alert’s

recommendation

• Similarly, clinicians may discuss the options with other members of the

team, with specialists and/or consult reference material before

making a management decision

• Clinicians may also override an alert but still implement its

recommendation (e.g. tell the patient not to take Drug A

within 4 hours of Drug B, but proceed with the order)



14

Alert Fatigue

Signal

Noise

Question 3:

Do people agree on which coins

(alerts) should be gold-colored?

Are some of my gold alerts your

grey alerts, and vice versa?


15

• Is it possible to achieve consensus on which alerts are clinically

significant?

• ONC/RAND list identified 15 critical DDIs (Phansalkar et al; JAMIA

2012;19:735-743 [5])

• Separately, Pharmacy Quality Alliance (PQA) developed a list of 14

critical DDIs for use in evaluating health plans

• These two lists overlapped only for MAO-I interactions

• Warfarin interactions were on PQA list but not ONC/RAND list

• In our own research, we have found poor agreement among

generalist physicians on which DDIs are clinically significant

• Universal consensus on what constitutes a clinically significant

alert remains elusive

Reducing Alert Fatigue

[5] Phansalkar S, Desai AA, Bell D, Yoshida E, Doole J, Czochanski M, Middleton B, Bates DW. High-priority drug-drug

interactions for use in electronic health records. J Am Med Inform Assoc. 2012 Sep-Oct;19(5):735-43. doi: 10.1136/amiajnl-

2011-000612. Epub 2012 Apr 26. PubMed PMID: 22539083; PubMed Central PMCID: PMC3422823.


16

• In the absence of a universal consensus, systems must be

tailored to the needs of individual hospitals, departments and

users

• User level

• Don’t show me this alert again

• Don’t show me this alert again for this patient

• Snooze for a period of time

• Hospital level

• Always suppress this alert

• Suppress this alert for physicians, but display it to pharmacists



17

Alert Fatigue

Signal

Noise

Question 4:

Does the color assignment depend

on the patient and other context

factors?

Is an alert gold if the patient has

kidney disease but grey otherwise?


18

• Lack of granularity in CDS knowledge

• Failing to account for other EMR data (patient

demographics, other medications, lab results)

• Relevant information buried in free text without any NLP

capability to extract it (but clinician can read the free text

and conclude that the alert is inappropriate)

Causes of inappropriate alerts


19

• Consider additional contextual factors

• Top contextual factors identified by subjects in Riedmann et al in an

international Delphi study (JAMIA 2011; 18:760-766 [6])

• Severity of the effect

• Clinical status of the patient

• Probability of the occurrence

• Risk factors of the patient

• Strength of evidence

• For many of these factors, data are limited

• We often don’t have data to provide the probability of occurrence for a

given set of risk factors

• Even if we did, below what level of probability of occurrence is it

considered safe to suppress an alert?

Contextual Factors

[6] Riedmann D, Jung M, Hackl WO,

Ammenwerth E. How to improve the delivery

of medication alerts within computerized

physician order entry systems: an

international Delphi study. J Am Med Inform

Assoc. 2011 Nov-Dec;18(6):760-6.


20

Alert Fatigue

Signal

Noise

Question 5:

What's the best way to show you

the coins (alerts) in your workflow?


21

• Seidling et al looked at factors influencing alert acceptance

[taking appropriate action] (JAMIA 2011; 18:479-484 [7])

• Display of the alert (OR 4.75)

• Setting [inpatient/outpatient] (OR 2.63)

• Level of the alert (OR 1.74)

• Display of the alert refers to such characteristics as how

alerts are grouped, where alert appears relative to

medication order, visibility, legibility, color, shape and icon

• In other words, the most important factor in alert

acceptance was the design of the alerting system


[7] Seidling HM, Phansalkar S, Seger DL, Paterno MD, Shaykevich S, Haefeli WE, Bates DW. Factors influencing alert

acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc. 2011 Jul-

Aug;18(4):479-84.


22

Horsky et al [8] published some design recommendations:

• Color (e.g. reserve red for severe situations)

• Size (e.g. variable depending on content length)

• Layout (e.g. simple geometry)

• Font (e.g. draw attention to drug names)

• Language (clear and concise, with links to additional information)

• Interruptive vs Non-Interruptive

• Use Non-Interruptive for low severity alerts

Design of Decision Support Interventions

[8] Design of decision support interventions for medication prescribing Jan Horsky, Shobha Phansalkar, Amrita Desai, Douglas Bell, Blackford

Middleton International journal of medical informatics 14 March 2013 (Article in Press DOI: 10.1016/j.ijmedinf.2013.02.003)


23

Russ et al[9] tested a redesign of medication alerts in the VA

CPRS system using a simulation study.

Changes included:

• Streamlining the information

• Reducing textual information overload

Redesigned alerts had improved usability scores for:

• Overall satisfaction

• Information quality (!)

• Interface quality

Applying Human Factors Principles

[9] Russ AL, Zillich AJ, Melton B, Spina J, Weiner M, Russell SA, McManus MS, Kobylinski A, Doebbeling BN, Hawsey J, Puleo A, Johnson E, Saleem JJ.

Applying Human Factors Principles to Improve Medication Alerts. AMIA Annual Symposium; 2012 Nov 3; Chicago, Illinois.


24

Duke, Li and Dexter[10]

• Enhance alert display with patient context

• Intervention group – display most recent potassium and creatinine

along with potassium-related alerts

• Control group – standard alerts

• Hypothesis: Adherence rate would be higher in high risk patients

(elevated baseline potassium and/or creatinine)

• Results: No significant difference in adherence rates

• Possible explanations:

• Physicians ignored the alerts entirely

• Physicians did not gauge patient risk accurately

• High risk patients were carefully monitored, so less cause for concern

Context Display

[10] Duke JD, Li X, Dexter P. Adherence to drug-drug interaction alerts in high-risk patients: a trial of context-enhanced alerting. J Am Med Inform

Assoc. 2013 May 1;20(3):494-8. doi: 10.1136/amiajnl-2012-001073. Epub 2012 Nov 17. PubMed PMID: 23161895; PubMed Central PMCID: PMC3628050.


25

Alert Fatigue

Signal

Noise

Question 6:

What actually happens when coins

(alerts) are shown to users? What

can we learn from the data?


26

Institutions should study their alert logs to identify areas for

improvement.

Health system - 1354 pregnancy alerts in one week.

• Through simulation, identified a new filter setting that would cut down

the number of alerts by 60%.

• OR rate would still be high, but alerts per order would be reduced.

• EMR issue: precautions for Placenta Previa were triggered for all

pregnant patients, leading to unnecessary alerts.

• Solution: EMR updated its software.

• Workflow issue:

• MMR vaccine – orders were given immediately post-partum, but

nobody told the EMR that the patient was no longer pregnant.

Alert Logs


27

Pharmacy - 70% of alerts were duplicate therapy alerts.

• Problem: Patient comes in with a new prescription for a drug with 0

refills remaining, but the old prescription is still active generates

duplicate therapy alert.

• Solution: Implement a days allowance (e.g. if patient comes in within

14 days of the end of the old prescription period, suppress the alert).

• Problem: Numerous alerts for patients taking three anti-hypertensive

drugs.

• Solution: Change the allowance for this class of medications (e.g. only

alert if four or more anti-hypertensive drugs).

Alert Logs


28

Health system

• 89,426 Drug-Disease precaution alerts in 6 months

• 60,600 (68%) were below our recommended filter threshold

• Solution: Implement our recommended filter settings

Alert Logs


29

Alert Fatigue

Signal

Noise

Question 7:

Is there an alternative to showing

you a bunch of coins (alerts) within

your workflow?


30

• Don’t show an alert at all, but instead institute surveillance

monitoring for adverse effects

• Alert a provider at the first sign of an adverse effect

• Monitoring could involve vital signs, lab test and other

observations about the patient, some of which may require

NLP analysis of free text notes


Question 1: Where do you draw the line?

• Strike a balance between precision and recall, but don't ignore

recall. Patient safety is at stake.

• Our research suggest that both precision and recall can

(simultaneously) be above 0.65.

• Don't go for 0.90 precision at the expense of a huge drop in recall

(e.g. recall of 0.20).

Recommendations

31 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health

Question 2: How should we evaluate provider responses to

alerts?

• Look at the override rate, but realize that it doesn't tell the whole

story.

• Override alert – probably appropriate

• Override alert

• May still have implemented its recommendations (e.g. implement

monitoring) [overridden literally but not functionally]

• May be inappropriate

Recommendations


Question 3: Do people agree on which coins (alerts) should

be gold-colored?

• Make some decisions at the institutional level.

• Allow end users to make some of their own decisions.

• Consider guideposts (e.g. end user cannot suppress highest

severity alerts).

• Don't show me this alert again [for 6 months].

• Don't show me this alert again for this patient [for 6 months].

Recommendations


Question 4: Does the color assignment depend on the patient

and other context factors?

• Use context whenever possible.

• Dose checking – include the indication, patient age, patient weight,

and patient renal function.

• Interaction checking – include the route, patient age, patient gender,

and patient's problem list.

Recommendations


Question 5: What's the best way to show you the coins

(alerts) in your workflow?

• Pay attention to the design of the alerts (color, size, layout, font,

language)

• Distinguish between interruptive (more severe; interrupts workflow)

and informative alerts (less severe; appears on screen but does not

interrupt workflow).

• Consider displaying context information (e.g. recent lab results).

Recommendations


Question 6: What actually happens when coins (alerts) are

shown to users? What can we learn from the data?

• Study your institution's alert data.

• Measure and improve.

• What are the most common alerts being overridden? Why?

Recommendations


Question 7: Is there an alternative to showing you a bunch of

coins (alerts) within your workflow?

• For rare adverse drug events, consider surveillance monitoring

(e.g. Sentri7 from Pharmacy OneSource – also part of Wolters Kluwer

Health) in lieu of prospective alerting.

• Create rules such as:

• If (Drug X AND Drug Y) AND

(AST > 35 OR ALT > 40) THEN

Send Alert to Provider

Recommendations


Q & A



Heather Ciranna

Marketing Production & Tradeshow Manager

Wolters Kluwer Health

Lexicomp | Medi-Span

[email protected]

For More Information

Follow @Medispan on

Twitter to learn about

upcoming webinars!

mailto:[email protected]


40

Notice

This presentation may not be copied, reproduced, redistributed or otherwise transmitted in any

form or by any means, in whole or in part, without prior written consent from Wolters Kluwer

Health. Any unauthorized use of copyrighted material incorporated within this presentation will

be subject to legal action. The content in this presentation has been prepared strictly for

informational purposes only, and is not intended or offered as legal advice. Although the

information provided is believed to be reliable and accurate, all information is without

warranties of any kind, either express or implied, including but not limited to warranties of the

accuracy, currency, completeness of information, or the suitability of the information for any

particular purpose. Wolters Kluwer Health shall not be liable for any direct, indirect, incidental,

special, consequential, punitive or any other damages arising out of the use of any information

discussed in this presentation. In deciding your organization’s specific technology, business and

legal requirements, you must not rely on the information in this presentation as a substitute

to advice from your own legal counsel.

Combating Alert Fatigue: New Best Practices...Level Seven (HL7) Clinical Decision Support (CDS) Working Group, which develops CDS standards in areas such as InfoButtons, order sets,

Documents