Teaching Evidence Based Medicine
What’s
Ahead?
• Practicing EBM in the
Trenches
– Staying up to date
– Answering questions
• Teaching EBM to
beginners
• Teaching EBM to
advanced learners
– Tough topics
• Making EBM practical
Barriers to EBM
• Time
• Evidence base poor
• Lack of Internet (or resources)
• Understanding EBM
• Teachers attitude
Med Educ. 2004;38:987-97. Acad Med. 2003;78:1183-90. Fam Med. 2001
Sep;33(8):602-6. Med Educ. 2004 Sep;38(9):987-97. BMJ 1998; 316: 361-5. J Gen
Intern Med 1999; 14: 236-42. The Cochrane Database of Systematic Reviews 2001,
Issue 3 Art. No.: CD 001270. Intern Med J. 2005;35(1):9-17.
Are we Evidence-based?
• So do doctors understand EBM:
– Our residents tested poorly (50%) and use
Evidence based sources for 18% of their
questions.
– Multiple studies1 find doctors struggle with EBM
concepts (e.g. RR and AR), even when they think
they understand.
Ann Int Med 1992; 117: 916-21. Lancet 1994; 343: 1209-11. Am J Card. 2001; 87: 203-
7. Br J Gen Pract 1996; 46: 661-3. JAMA 2002; 287: 2813-4. BMJ 1998; 316: 361-
5. J Eval Clin Pract 2001; 7: 201-10.
Staying Up-To-Date: Is it Possible?
• Reading all relevant journals to General
Practice?
• There are 341 journals, 90,000 articles/yr
• Even if narrowed down to the top 85, still
would review 16,000 articles/yr.
• If trained in epidemiology, still would take us
627.5 hours/month, or 21 hours a day
Alper et al. J Med Libr Assoc 2004;902(4):429-37.
Answering Daily Questions: Is it Possible?
• We have ~1 “uncertainties” (Questions) for every 2
patient encounters1
• If we did a search (30-60 minutes2) and appraised a
paper (~30-60 minutes)
– 30 patients = 15 questions
– Assuming 1.5 hours per question = 22.5 hours/day
• In truth, Doctors1,3
– Spend 2-3 minutes getting answers to their questions
– Search pubmed (or embase) for <1% of their question
– Do critical appraisals < 0.1% of their questions
1. JAMA Intern Med. 2014;174(5):710-718. 2. J Fam Pract. 1996; 43:140-4. Bull Med
Libr Assoc 1994; 82: 140-146 3. BMJ 1999; 319: 358-61.
So, Should We Quit Now?
• There are many barriers to EBM
• Most clinicians struggle understanding EBM
• Staying up-to-date is overwhelming
• Answer questions in practice by critical
appraisal is impossible.
• We NEED to Rethink EBM education and
Practice
A New Approach to EBM
• Teaching:
– For >90%: Understand Basics and Terms.
• Needed for Practice.
– For <10%: Advanced understanding.
• Needed for people presenting or writing on Evidence
• Staying up-to-date requires
– Pre-Filtered & Pre-Appraised:
• Answering Clinical Questions requires
– Quick Search, Thorough and Unbiased
Staying Up-to-Date: Options
• Studies – InfoPOEMS: Regular 1 pg summary up-dates of recent
articles. ~1/day (5 Minutes) US. Cost
– ACP Journal Club or Evidence Based Medicine: 30 articles
q2 months, 1 p summary (methods, findings & commentary)
(ACP = Internal Med, EBM = Family Med). US. Cost
– Goodfellow Gems: q 2 weeks, NZ. The briefest. FREE
• Questions – Tools for Practice: Focused Clinical Question with Bottom-
Line, Evidence, Context. ~300 words. Family Doctor
written. q 2 weeks FREE
Intensive lipid lowering with statins unnecessary
with stable CAD (IDEAL) • Clinical question: Is intensive lowering of serum lipids with statin drugs
beneficial in patients with stable coronary artery disease?
• Bottom line: The intensive reduction of low-density lipoprotein (LDL) levels to well below 100 mg/dL (2.5 mmol/L) did not result in a significant reduction in the recurrence of major coronary events or all-cause mortality among patients with stable coronary artery disease. Intensive lowering is associated with an increased risk of discontinuing medication because of adverse events and significant drug costs. Aiming for an LDL of approximately 100 mg/dL (2.5 mmol/L) seems optimal for the majority of patients with stable disease. (LOE = 1b-)
• Reference: Pedersen TR, Faergeman O, Kastelein JJ, et al, for the Incremental Decrease in End Points Through Aggressive Lipid Lowering (IDEAL) Study Group. High-dose atorvastatin vs usual-dose simvastatin for secondary prevention after myocardial infarction. The IDEAL study: A randomized controlled trial. JAMA 2005; 294:2437-45.
• Study design: Randomized controlled trial (single-blinded)
• Funding: Industry
• Allocation: Uncertain
• Setting: Outpatient (specialty)
• Synopsis: Intensive,… (250 Words)
Staying Up-to-Date: Other Options
• Journal Watch: Summaries of Articles (less
structured) - Cost
• Get 1-2 Journals Table of Contents emailed to you,
– Example: NEJM. Abstracts for Free
– BMJ excellent choice (more Primary Care): Free
P
r
e
v
i
o
u
s
N
e
x
t
P
r
e
v
i
o
u
s
N
e
x
t
Example:
NEJM
Abstract
Answering Clinical Questions:
• Resources like Tools for Practice (if answer there)
1.UpToDate (Cost (big)): Most complete, most
expensive. Very much like a medical textbook.
2.DynaMed (Cost): more Evidence-based, slightly
less cumbersome. More point form. Less
comprehensive.
Answering Clinical Questions:
3. TRIP database (Free): Huge Search Engine, can
limit but often like PubMed (too broad). Answers
there, time inefficient.
4. Clinical Evidence (Cost): Quite Evidence-based,
but not very comprehensive (sometimes within
topics too) some old ones free via Pubmed.
5. BMJ Best Practice (Cost): Relatively evidence-
based but not as comprehensive & a bit
cumbersome with sub-categories
UpToDate: Plus and Minus
• Positives:
– Full summary (pathophysio to f/u)
– Left Margin topics & Everything linked (pictures, drugs, etc)
– Links quick, good search
– Everything cited
• Negatives:
– Cost !
– Some Bias (single authors in field)
– Not Completely “Up-To-Date” (but very good)
– Can be cumbersome
Primary Goals
• To help residents become competent self-
directed life-long learners with the skills to
effectively and efficiently keep up-to-date.
• To learn how to use Evidence Based Medicine
(EBM) to rapidly (as possible) solve problems
encountered in daily practice.
Secondary Goals
• Review EBM components within the family medicine residency
curriculum, maximizing educational opportunities,
– Eliminating or modifying components with poor evaluation or performance.
– Enhancing other areas to promote EBM.
– Adding education time only to achieve the core features of life-long learning
skills.
• Promote the development of the skills in Residents needed to
critique educational opportunities.
• Promote and encourage residents seeking additional skills, training
or experience in EBM, knowledge transfer and/or research.
Learning Objectives:
Knowledge Component
• To understand the rationale and benefits of EBM
• Provide a strong foundation in the basic principles of
EBM including
– Recognizing and formulating clinical questions
– Finding and Accessing information
– Interpreting information
– Applying information
• Provide and identify online resources, educational tools
and web-links
Learning Objectives:
Skill Components
• Learn to identify problems/questions encountered in
practice and seek solutions based upon the best evidence.
(Via EBM “assignment” to promote rapidly answering
clinical questions).
• Build skills and comfort in knowledge transfer (primarily
through oral presentation skills).
Learning Objectives:
Attitude Components
• To facilitate appreciation and enthusiasm for
– the judicious use of current evidence to optimize
patient care
– maintenance of skills and knowledge through life long
self-directed learning
Some Background Research • Critical Appraisal based curriculum are often less
successful.1
• Workshops (3 day) have been shown to improve knowledge.2
• Classroom EBM teaching alone only improve knowledge while Integration improves skills, attitudes and behavior.3
• Web-based Clinical Questions & Answers helpful getting resident participation in EBM.4
1. Acad Med. 1999;74(6):686-94. & CMAJ 1998;158(2):177-81. 2. BMJ. 2002
7;325(7376):1338-41. & Acad Med. 2000;75(12):1212-4. 3. Acad Med. 2002;77(7):
741-2. & BMJ. 2004;329(7473):1017. 4. Acad Med. 2003 Mar;78(3):270-4.
EBM Curriculum Components
• Family Medicine Evidence Based Medicine Workshop
– 2 day mixed lecture & small group work + computer lab
• Informatics/Evidence based resources
– UpToDate, DynaMed, Clinical Evidence, others
• Brief Evidence-based Assessment of Research (BEARs)
– 1 page assignment, ≥4 during Family block (1st yr)
• Resident Journal Club
– Selected by EBM faculty, rapid review summary articles in teams, shared presentations, focus on clinical application.
• Other components
– q 2 monthly Lit review lecture
Our Curriculum Research 1
• IMG residents (vs Canadian grads) reported little
or poor quality EBM training & indicated
computer skills were a barrier1
• After introducing the curriculum, stat. sign.
improvement of resident comfort, self-reported
skills & attitudes towards EBM.2
– E.g. EBM curriculum improved my comfort with
evidence-based practice (1-5 scale: 2.76 before; 3.86
after, p<0.001)
1. Allan GM, Manca DP, Szafran O, Korownyk C. EBM a Challenge for International Medical
Graduates. Fam Med 2007; 39(3): 612. 2. Allan GM, Korownyk C, Tan A, Hindle H, Kung L,
Manca D. Developing an Integrated Evidence-Based Medicine Curriculum for Family Medicine
Residency at the University of Alberta. Acad Med 2008; 83: 581-587.
Our Curriculum Research 2
• How do residents address questions in clinics
– 38 residents: 325 questions x114 clinical ½ days (420 pts)
– 0.8 questions/pts, 83% answered with one resource (1-6)
– Attempts: colleagues 65.5% (94% were preceptor), electronic resources 20.7% and paper resources 13.8%.
– Colleagues least likely need second resource (ss).
– Utility of answers: Colleagues (ss) superior to electronic
– EBM workshop training did not influence electronic resource use but semi-independence did: 17% to 51% (ss).
• Studied practicing graduates and presently analyzing
Can Fam Physician. 2012 Jun;58(6):e344-51.
Curriculum Research: yet unpublished
• Our EBM Workshop (unpublished)
– Pre-workshop survey / post-workshop eval
• No diff year to year, Strongly positive feedback
– Pre-Post Workshop Quiz
• 15 questions, no difference year to year
• Mean score: pre 50% and post 75%
• BEARs Study1:
– Reviewed 317 BEARs first two years.
– Search: Pubmed (38%); Summary (22%); Prefiltered (19%)
– 48% from review, CPG, summary & filtered sites
– 19% large change, 50% small change, 24% reassured, 8% no help
Med Teach. 2013 Oct;35(10):e1546-50.
Summary
• Curriculum involves training physicians to be
good evidence
• While successful on many levels, not perfect.
– Still need additional factors (e.g. independence)
• When you make an education program,
– Evaluate it (Study it).
Advanced Education Components
• Tough topics
– Allocation Concealment
– Intention to Treat
– Numbers
– Funding
– P-values (hypothesis testing)
• Advanced Workshop
What is in the DM trials?
Number Mean Age Duration of
DM
Past CVD Weight in
2nd Meta
UKPDS 33
(1998)
3,867
(4209*) 53.2 New 2%
23.9 UKPDS 34
(1998)
753
(4209*) 53.2 New 2%
ACCORD
(2008) 10,251 62.2 8.3 35% 21.5
ADVANCE
(2008) 11,140 66 7.5 32% 28.2
VADT (2009) 1791 60.4 11.5 40% 11.9
UGDP
(1978) 414 52.7 New 6% 11.8
* The n used in the BMJ meta for UKPDS combined
What is in those Vitamin D trials
Study Country Endpoint Age
Camargo (2012) Mongolia Acute RI 10
Jorde (2012) Norway Influenza 63
Laaksi (2010) Finland URTI 18-28
Li-Ng (2009) USA URI* 59
Manaseki-Holland (2010) Afghanistan pneumonia 0-3
Manaseki-Holland (2012) Afghanistan pneumonia 0-1
Murdoch (2012) New Zealand Colds 47
Urashima (2010) Japan Flu 10.2
URI* = 2 or more of fever, cough, productive sputum or change in sputum
color and quantity, muscle aches, nausea or vomiting
What is in those Vitamin D trials
• Let’s look specifically at Urashima (Am J
Clin Nutr 2010;91:1255-60)
Vitamin D Placebo Relative Risk*
n (167) % n (167) %
Influenza A 18 11% 31 19% 0.58 (0.34-0.996)
Influenza B 39 23% 28 17% 1.39 (0.90-2.15)
Influenza Like
Illness 8 5% 9 5% -
Total 65 38.9% 68 40.7% 0.96 (0.73-1.24)
* Using http://www.hutchon.net/ConfidRR.htm for RR (& CI) estimation
EBM TP: Cornerstones of Quality
Randomization, Allocation
Concealment & Blinding
• 1) Hypothesis
• 2) Randomize
• 3) Allocation to groups
• 4) Blind
(Do the Study)
• 5) Report (Baseline Characteristics)
EBM TP: Cornerstones of Quality
Randomization & Blinding
• Many quality scores weight randomization and blinding heavily
– JADAD - 4/5 points (5th for follow-up)
• Randomization: ensures compared groups are equivalent.
• Blinding: prevents study personnel & pts from determining the group which pts are assigned
– cannot always be implemented
• Inadequate blinding favors treatment by about 17%1
1. JAMA. 1995;273:408-12
EBM TP: Cornerstones of Quality
• BUT Randomization & Blinding mean little
without their forgotten cousin,…
EBM TP: Cornerstones of Quality
Allocation concealment
• Allocation concealment prevents clinicians from
circumventing the randomization process
– AC can always be implemented
– Reporting AC poor (only 7% - 55%)1
• Poor Allocation concealment gives bias,2
– If Unclear AC = 33%
– If Inadequate AC = 41%
– Overall (unclear/inadequate) AC = 37%
1. BMJ 2004 3;328:22-4. ACP J Club. 2000 Mar-Apr;132:A11 .
2. JAMA 1995; 273: 408-412. Lancet. 1998;352:609-13.
EBM TP: Cornerstones of Quality
Allocation concealment Examples
• If a trial sets it’s treatment effect at 25-50%,
– Doesn’t Conceal allocation,
– Any effect or prove of benefit may be entirely the
fault of poor AC (41%)
Reporting Baseline Characteristics:
The Final Check
• Authors must report (usually Table 1) the
characteristics of the pts in each group (so we
can compare them)
• Ex Post-MI trial.
– ex if one group is older & has 10% higher risk
of MI, analysis must account for known > risk
EBM TP: “What is Intention to Treat”
• “Analysis was by intention to treat”
• Simply: Subjects were analyzed by what group they were
randomized to, not by what happened to them.
• The Analysis is by how the investigator intended to treat
them.
Intention to Treat: Who Cares?
• The alternatives are: Per Protocol Analysis,
Explanatory Analysis, Analysis by Treatment
Administered, Efficacy Analysis
• Using these, you can hide outcomes or
select the best patients
• Example 1: Quiters (also known as drop-
outs)
Intention to Treat: Example
Imagine a Stroke Study
Assume 10% have strokes a year
Assume it takes 6 months to get to surgery
Intention to Treat: Example
• Intention to Treat
• 10% CER
• 10% EER
• = 0% AR
• = 0% RR
• NNT = infinite
• No difference
• Per Protocol
• 10% CER
• 5% EER (approx)
• = 5% AR
• = 50% RR
• NNT = 20
• publishable
Intention to Treat: Summary
• Definition: Subjects are analyzed by the
group they were randomized to (aka: How
you Intended to Treat them), regardless of
what actually happened to them.
• The only non-bias way to analyze study
results (others bias trials up to 51%)
• Only 7-48% of articles do ITT (& less
properly)
BMJ. 1999; 319: 670-4. J Fam Pract. 2002; 51: 969-71. BMJ 2003;326: 117-5. J Clin
Epidemiol. 2003; 56: 833-42
RR, AR and NNT
e.g. Treating Hypertension for MI Risk (Framingham data)
• AR = CER – EER
• RR= EER / CER
• NNT = 1 / AR
AR= 20% - 10% = 10%
RR= 10% / 20% = 50%
NNT= 1 / 0.1 = 10
CER: 60 y.o. M, HDL 1.0, BP>160 = 20%
EER: 60 y.o. M, HDL 1.0, BP<130 = 10%
RR when event unlikely &
RR vs AR (NNT)
Common Diseases
• RR = 50%
• CER = 20%
• EER = 10%
• AR = 10%
• NNT = 10
Uncommon Diseases
• RR = 50%
• CER = 1%
• EER = 0.5%
• AR = 0.5%
• NNT = 200
The Media: Women’s Health Initiative
• TV Evening News
(September 24, 2003)
• 41% increase in stroke
• 29% increase in MI
• Double rate of VTE
• 26% in Breast Ca
• No benefits mentioned
• Actual Risk per 10,000
(JAMA 288(3):321-33)
• 8 strokes in 10,000 p. y.
• 7 MI’s in 10,000 p. y.
• 18 VTE in 10,000 p. y.
• 8 Breast Ca in 10,000
• 6 Colorectal Ca & 5 Hip
Fractures Less/10,000
Bottom Line
• Absolute Excess Adverse Events in the
Global Index are 19 in 10,000 patient years
OR
• 100 women on HRT for 5 years = 1 additional
adverse event
EBM TP: Funding Bias
“He who pays, Calls the Tune”
Some Facts (& Myths)
• Is industry research more likely to be published
• No
• Is industry research comparatively poor quality
• No (?)
• Does Funding Influence Outcomes &
Recommendations
• Yes
Funding Influence
• Funding gives an OR of 4-5.3 that,
– Study outcomes favor the drug being studied
– Drug will be recommended as Treatment of
choice
• Resources – Lexchin et al, BMJ, 2003; 326: 1167-70.
– Als-Nielsen et al, JAMA, 2003; 290(7): 921-8
– Melander et al, BMJ, 2003; 326: 1171-73
How does it Happen?
• 1) “Pick your Battles.” Fund studies likely to be +ve (& don’t fund harm trials!)
• 2) “Pick your enemies.” Comparators should be poor drug, too low dose, not absorbed, etc.
• 3) “Give them what they want” Good Methods but ITT, reporting & others
• 4) “Let Sleeping Dogs Lie.” Don’t publish trials with bad outcomes; And Get good mileage out of good results.
• 5) “Buy what you need” Give Docs $12,000/pt + $30,000 bonus after 6.
Selective Publication and Reporting - Circles: trials done by industry (red = favorable
outcome, blue = no better than placebo)
- Green Diamonds: publications off one trial.
-Yellow Boxes: publications off more than one trial
-Three Trials find their way into 15 publications (5 each)
Selective Publication and Reporting
Rare cross referencing
Changing authors &
Definitions
Publications from
Single Trails:
- if trial +ve = 90%
- if trial –ve = 29%
Bottom Line
• Don’t Blame Industry Entirely
– The authors of those papers are Doctors!
• Be Skeptical (Don’t “buy” in to advertising)
• Always “Cheque” Funding Source
• Then, Check Methods, Including ITT
• Then, Check that Recommendation Matches
Outcome & Treatment Effect
EBM TP: Alpha Errors, P-Values
and Confusion Intervals
• Can an X-ray have the appearance of a
fracture even when there isn’t one?
• Any investigation, whether a test or study,
can suggest something falsely.
• In research, they change false positive
to,…
EBM TP:
Alpha error
• Alpha error (also Type I error): The null hypothesis is considered false when it is actually true (probability of error = p)
• In English: The study shows there is difference but,… it is a fluke and there really is no difference.
• This leads to,…
P-value
• P-value 0.05 = There is 5% we will falsely reject the null hypothesis.
• English: If a study is showing there is a difference,… there is a 5% chance that it’s wrong (and there is no difference).
• When the p-value gets smaller (0.01 or 0.001, etc), so does the chance of making the mistake (i.e. assuming there is a difference when there is not).
Shortfalls of P-Values
• Using P-value alone to examine a study turns the results
into a yes or no answer
• It also doesn’t give you an idea of the range of
possibilities (ie the effect could be is high or low as this)
• Therefore we need,…
Hypothetical Example:
Beer Damaged Neurons
• Imagine a university with 1000 students
• If I randomly selected 5 and used them to estimate the
mean # of damaged neurons, how confident should I be
that the next person will have a similar #.
• If I randomly selected 250, how confident would I be.
CI: What does it tell us?
• Is the intervention helpful
• Is it statistically and clinically significant
• What is the degree of “precision”
Summary
• Alpha Error is when a study shows there is
difference but,… it is a fluke and there really is
no difference
• The probability of that error is the P-value
• A 0.05 P-Value means that if the study shows
a difference there is a 5% chance that it’s just
luck (and there is no difference).
• Confidence Intervals show the range of values
reflecting the precision of the estimate