Systematic Reviews-intro level Phil Wiffen Director of Training UK Cochrane Centre Editor, Cochrane Collaborative Review Group in Pain, Palliative & Supportive.

Systematic Reviews-intro level

Phil WiffenDirector of Training UK Cochrane CentreEditor, Cochrane Collaborative Review Groupin Pain, Palliative & Supportive Care

The rationale

•Evidence based medicine

•What are systematic reviews ?

•How do we find, understand and evaluate systematic reviews?

•Tools to present data

561882563770

4401941305

300946

127279816716506196322831167442649

257169183

308636100

762222898726186

2 4 6 8 10 12 14 16 18

Number needed to treat with 95% Confidence Intervals

Ibuprofen 800 mgIbuprofen 600 mgIbuprofen 400 mgIbuprofen 200 mgIbuprofen 100 mg

Diclofenac 100 mgDiclofenac 50 mgDiclofenac 25 mgNaproxen 440 mgNaproxen 550 mg

Naproxen 220/250 mgParacetamol 1000 mg/Codeine 60 mg

Aspirin 1200 mgParacetamol 600 or 650 mg/Codeine 60 mg

Aspirin 1000 mgAspirin 600/650 mg

Dextropropoxyphene HCl 65mg/Paracetamol 650 mgParacetamol 1000 mg

Paracetamol 600/650 mgParacetamol 300 mg/Codeine 30 mg

Paracetamol 500 mgPethidine 100 mg (intramuscular)

Morphine 10 mg (intramuscular)Dextropropoxyphene HCl 65mg

Dihydrocodeine 30 mgCodeine 60 mg

Tramadol 150 mgTramadol 100 mg

Tramadol 75 mgTramadol 50 mg

Total Numberin Comparison

League table of NNTs to produce at least 50% pain reliefover 4-6 hours compared to placebo in pain of

moderate or severe intensity

What evidence-based medicine is:

Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.

Sackett (BMJ 1996; 312: 71-2)

What evidence-based medicine is:

The practice of EBM requires the integration of

• individual clinical expertise with the

• best available external clinical evidence from systematic research.

Another definition of EBM

Evidence based medicine is an approach to health care that promotes the collection,

interpretation and integration of valid, important and applicable patient reported, clinician observed and research derived evidence. The best available evidence,

moderated by patient circumstances and preferences, is applied to improve the

quality of clinical judgements.

McKibbon KA et al ‘The medical literature as a resource for Evidence Based Care’http://hiru.mcmaster.ca/hiru/medline/mdl-ebc.htm

“There are perhaps 30000 biomedical journals in the world, and they have grown steadily by 7% a year since the seventeenth century.

Yet about 15% of medical interventions are supported by solid scientific evidence...

...only 1% of the articles in medical journals are scientifically sound”

R. Smith quoting Prof. D. Eddy, BMJ 1991; 303: 798-99

More than 20 000 RCTs have been

published in pain relief research since

1950

“...approximately 17000 new biomedical books are published annually.”

Lowe and Barnett, JAMA 1994; 271: 1103-8

The size of the task

How many biomedical papers are there ?

•Medline 17 million records, 5000 journals, 80 countries

•Embase 14 million records, 7000 journals, 70 countries

•CINAHL 1 million records 2900 journals. 13 languages

•Others: e.g. LILACS ????

April 2009

enthusiastUNBIASED‘good’ RCT

numbers / inclusion / exclusionblindingpower

open ‘biased’ study

local use‘expert’ clinical practice

+systematic reviews

Tools not Rules

I Strong evidence from at least 1 systematic review of multiple well-designed randomised controlled trials

II Strong evidence from at least 1 properly designed randomised controlled trial of appropriate size

III Evidence from well designed trials without randomisation, single group pre-post, cohort, time series or matched case-controlled studies

IV Evidence from well-designed non experimental studies from more than 1 centre or research group

V Opinions of respected authorities, based on clinical evidence, descriptive studies or reports of expert committees

Type & Strength of Evidence

What is a systematic review ?

•Filing Cabinets

•Friends

•Foreigners ?

•The world literature on a subject

Systematic Reviews

“Clinical review articles should be as scientific as the articles they review”

Haynes, BMJ 1992; 304: 330-1

“Clinical review articles should be as scientific as the articles they review”

Haynes, BMJ 1992; 304: 330-1

“ The fundamental difference between a review and a primary study is the unit of analysis, not the scientific principles that apply”

Oxman & Guyatt, CMAJ 1988; 138: 697-703

“ The fundamental difference between a review and a primary study is the unit of analysis, not the scientific principles that apply”

Oxman & Guyatt, CMAJ 1988; 138: 697-703

0. Frame question

1. Search for trials

2. Score for quality

3. Validity of trial results

4. Vote-counting 4. Extract data

qualitative quantitative

5. Meta-analysis

narrative review (overview)

systematic review ± meta-analysis

Finding systematic reviews

• Medline has a filter

Finding systematic reviews

• Cochrane Library

• DARE database on Cochrane Library

Understanding systematic reviews

• Meta-analysis (forest plot)

• NNTs

• L’Abbe plots

• Assessment of bias in the included studies

Ibuprofen 400 mg vs. paracetamol 1000 mg for acute postoperative pain

0

1

2

3

4

5

6

7

-20 -10 0 10 20 30

Cooper et al, 1984

Cooper, 1984

Cooper et al, 1989Schachtel al, 1989

Mehlisch et al, 1990

Overall weighted difference

mean differences and 95% CI( % of the maximum possible TOTPAR value)

favours ibuprofenfavours paracetamol

no difference

difference between the mean effects within the trial and 95% CI

A closer look at a forest plot (the meta-analysis)

Cochrane Database of Systematic ReviewsPublished by John Wiley & Sons, Ltd

There is a label to tellyou what the comparisonis and what the outcomeof interest is

At the bottom there’sa horizontal line. This is the scale measuringthe treatment effect.Here the outcome is ……..

Take care to read whatthe labels say – things tothe left do not always mean the treatment is better than the control.

The vertical line in themiddle is where thetreatment and control have the same effect – there is no differencebetween the two

For each study there is an ID

The data foreach trial are here, divided into the treatment and control groups

This is the % weightgiven to this study in the pooled analysis

Each study is given a blob, placed where the data measure the effect.

The size of the blob is proportional to the % weight The horizontal line is called a confidence interval and is a measure of how we think the result of this study might vary with the play of chance.

The wider the horizontal line is, the less confident we are of the observed effect.

The label above the graph tells you what statistic has been used

The data shown in the graph are also given numerically

The pooled analysis is given a diamond shapewhere the widest bit in the middle is located at the calculated best guess (point estimate), and the horizontal width is the confidence interval

** Note on interpretation **

If the confidence interval crosses the line of no effect, this is equivalent to saying that we have found no statistically significant difference in the effects of the two interventions

Cochrane Database of Systematic ReviewsPublished by John Wiley & Sons, Ltd

Numbers needed to treat (NNTs)

The Number of people who have to be treatedfor ONE to benefit

Number-needed-to-treat (NNT)

Number of patients

Improved = Clinical end point

Actives

Nact

Impact

Controls

Ncon

Impcon

1

Impcon

Ncon

Impact

Nact

-NNT =

Number-needed-to-treat (NNT)

1

0

100

100

100 -NNT =

NNT is treatment specific -takes into account the event rate in controls:• may be a placebo effect• may be the effect of another treatment

active control

improved 80 20

N 100 100

Relative Risk (RR) = (Impact/Nact) / (Impcon/Ncon)

Relative Risk Reduction (RRR) = (1-RR) / 100

Absolute Risk (AR) = (Impact/Nact) - (Impcon/Ncon)

Number Needed to Treat (NNT) = 1/AR

RR = 4; AR = 0.6; NNT = 1.7 (best 1.25)

0

25

50

75

100

0 25 50 75 100

L'Abbé plot for treatment

equal

ity

Treatment betterthan

control

Control betterthan treatment

Proportion improved

with treatment

Proportion improved with control

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

0

100

200

600/650

500

1000

At least 50% pain relief with placebo

At least 50% pain relief with

paracetamol

World literature on paracetamol

Moore et al Pain 1997;70:193

‘Risk of bias’ assessment in Cochrane reviews

Risk of bias summary

• Here ‘Blinding’ and ‘Incomplete outcomes data’ have been assessed for two sets of outcomes

Ad

eq

ua

te s

eq

ue

nce

ge

ne

ratio

n

Barry 1988 +

Baylis 1989 +

Cooper 1987 +

Dodd 1985 +

Goodwin 1986 +

Sanders 1983 +

Allo

ca

tio

n c

on

ce

alm

en

t

-

+

?

?

+

+

Blin

din

g (

Pa

tie

nt-

rep

ort

ed

ou

tco

me

s)

+

+

-

+

+

-

Blin

din

g (

Mo

rta

lity

)

+

+

?

+

+

?

Inco

mp

lete

ou

tco

me

da

ta a

dd

resse

d (

Sh

ort

-te

rm o

utc

om

es (

2-6

wks))

-

?

-

+

+

-

Inco

mp

lete

ou

tco

me

da

ta a

dd

resse

d (

Lo

ng

er-

term

ou

tco

me

s (

> 6

ks))

-

?

-

-

+

-

Fre

e o

f se

lective

re

po

rtin

g

-

+

+

?

+

-

Fre

e o

f o

the

r b

ias

-

?

?

?

+

-

Evaluating systematic reviews

• Critical appraisal skill programme (CASP)

• 10 questions to make sense of a review

Ten questions to make sense of a review Adapted from Oxman AD et al Users Guide to the Medical Literature VI How to use an overview. JAMA 1994; 272 (17): 1367-71 For each question answer : YES, NO or DON’T KNOW A. Are the results of the review valid ? 1. Did the review address a clearly focused issue ?

e.g. the population , intervention and or outcomes  2. Did the authors look for the appropriate sort of papers ?

Did they deal with the issues and have appropriate study design ? Is it worth continuing ?? 3. Do you think the important relevant studies were included ? Look for search methods, reference list use, unpublished studies and non English language 4. Did the authors do enough to assess the quality of included studies ? 5. If the results of studies have been combined, was it reasonable to do so ? 

B. What are the results ? 6. What is the overall result of the review ?

Is there a clear numerical expression ? 7. How precise are the results ?

Confidence intervals ? C. Will the results help my local situation ? 8. Can the results be applied locally ? 9. Were all important outcomes considered ? 10. Are the benefits worth the harms and costs ?   Phil Wiffen Sep2011 ([email protected])

Conclusions

• Evidence based medicine has emphasised the importance of systematic reviews as evidence for care

• Need to know how find systematic reviews

• Need to know how to read a meta-analysis