RSS 2013 - A re-analysis of the Cochrane Library data]

BackgroundMethodsResults

Summary

A re-analysis of the Cochrane Library datathe dangers of unobserved heterogeneity in meta-analyses

Evan Kontopantelis12 David Springate12 DavidReeves12

1NIHR School for Primary Care Research, University of Manchester

2Centre for Biostatistics, Institute of Population Health, University of Manchester

RSS Newcastle, 3 Sep 2013

Kontopantelis A re-analysis of the Cochrane Library data


Summary

Outline

1 Background

2 MethodsDataAnalyses

3 ResultsMethod performanceCochrane data

4 Summary



Summary

Meta-analysis

Synthesising existing evidence to answer clinical questionsRelatively young and dymanic field of researchActivity reflects the importance of MA and potential toprovide conclusive answersIndividual Patient Data meta-analysis is the best option,but considerable cost and access to patient data requiredWhen original data unavailable, evidence combined in atwo stage process

retrieving the relevant summary effect statisticsusing MA model to calculate the overall effect estimate µ̂



Summary

Heterogeneity estimateor between-study variance estimate τ̂ 2

Model selection depends on the heterogeneity estimateIf present usually a random-effects approach is selectedBut a fixed-effects model may be chosen for theoretical orpractical reasonsDifferent approaches for combining study results

Inverse varianceMantel-HaenszelPeto



Summary

Meta-analysis methods

Inverse variance: fixed- or random-effects & continuous ordichotomous outcome

DerSimonian-Laird, moment based estimatorAlso: ML, REML, PL, Biggerstaff-Tweedie,Follmann-Proschan, Sidik-Jonkman

Mantel-Haenszel: fixed-effect & dichotomous outcomeodds ratio, risk ratio or risk differencedifferent weighting schemelow events numbers or small studies

Peto: fixed-effect & dichotomous outcomePeto odds ratiosmall intervention effects or very rare events

if τ̂2 > 0 only modelled through inverse variance weighting



Summary

Random-effects (RE) models

Accurate τ̂2 important performance driverLarge τ̂2 leads to wider CIsZero τ̂2 reduces all methods to fixed-effectThree main approaches to estimating:

DerSimonian-Laird (τ̂2DL)

Maximum Likelihood (τ̂2ML)

Restricted Maximum Likelihood (τ̂2REML)

All other RE methods use one of these but vary inestimating µIn practice, τ̂2

DL computed and heterogeneity quantified andreported using Cochran’s Q, I2 or H2



Summary

Random or fixed?two ‘schools’ of thought

Fixed-effect (FE)‘what is the average result of trials conducted to date’?assumption-free

Random-effects (RE)‘what is the true treatment effect’?various assumptions

normally distributed trial effectsvarying treatment effect across populations although findingslimited since based on observed studies only

more conservative; findings potentially more generalisable

Researchers reassured when τ̂2 = 0FE often used when low heterogeneity detected



Summary

Cochrane Database for Systematic Reviews

Richest resource of meta-analyses in the worldFifty-four active groups responsible for organising, advisingon and publishing systematic reviewsAuthors obliged to use RevMan and submit the data andanalyses file along with the review, contributing to thecreation of a vast data resourceRevMan offers quite a few fixed-effect choices but only theDerSimonian-Laird random-effects method has beenimplemented to quantify and account for heterogeneity



Summary

The questions

Investigate the potential bias when assuming τ̂2 = 0Compare the performance of τ2 estimators in variousscenariosPresent the distribution of τ̂2 derived from allmeta-analyses in the Cochrane LibraryPresent details on the number of meta-analysed studies,model selection and zero τ̂2

Assess the sensitivity of results and conclusions usingalternative models



Summary

DataAnalyses

‘Real’ DataCochrane Database for Systematic Reviews

Python code to crawl Wiley website for RevMan filesDownloaded 3,845 relevant RevMan files (of 3,984available in Aug 2012) and imported in StataEach file a systematic review (e.g. olanzapine forschizophrenia)Within each file, various research questions might havebeen posed (e.g. vs placebo, vs typical antipsychotics)

investigated across various relevant outcomes? (e.g. noclinical response, adverse events)

variability in intervention or outcome? (e.g. drug dosage,types of adverse events)



Summary

DataAnalyses

Simulated Data

Generated effect size Yi and within study varianceestimates σ̂2

i for each simulated meta-analysis studyDistribution for σ̂2

i based on the χ21 distribution

For Yi (where Yi = θi + ei )assumed ei ∼ N(0, σ̂2

i )various distributional scenarios for θi : normal, moderateand extreme skew-normal, uniform, bimodalthree τ2 values to capture low (I2 = 15.1%), medium(I2 = 34.9%) and large (I2 = 64.1%) heterogeneity

For each distributional assumption and τ2 value, 10,000meta-analysis cases simulated



Summary

DataAnalyses

Between-study variance estimatorsfrequentist, more or less

DerSimonian-Lairdone-step (τ̂2

DL)two-step (τ̂2

DL2)non-parametric bootstrap (τ̂2

DLb)minimum τ̂2

DL = 0.01 assumed (τ̂2DLi )

Variance componentsone-step (τ̂2

VC)two-step (τ̂2

VC2)Iterative

Maximum likelihood (τ̂2ML)

Restricted maximum likelihood (τ̂2REML)

Profile likelihood (τ̂2PL)



Summary

DataAnalyses

Between-study variance estimatorsBayesian

Sidik and Jonkman model error variancecrude ratio estimates used as a-priori values (τ̂2

MV )VC estimator used to inform a-priori values with minimumvalue of 0.01 (τ̂2

MVb)Rukhin

prior between-study variance zero (τ̂2B0)

prior between-study variance non-zero and fixed (τ̂2BP)



Summary

DataAnalyses

Assessment criteriain the 10,000 meta-analysis cases for each simulation scenario

Average bias & average absolute bias in τ̂2

Percentage of zero τ̂2

Coverage probability for the effect estimateType I errorproportion of 95% CIs for the overall effect estimate thatcontain the true overall effect θi

Error-interval estimation for the effectquantifies accuracy of estimation of the error-intervalaround the point estimateratio of estimated confidence interval for the effect,compared to the interval based on the true τ2



Summary

Method performanceCochrane data

Which method?

Performance not affected much by effects’ distributionAbsolute bias

B0 (k ≤ 3) and MLCoverage

MVa-BP (k ≤ 3) and DLbError-interval estimation and detecting

DLbDLb seems best method overall, especially in detectingheterogeneity

appears to be a big problem: DL failed to detect high τ2 forover 50% of small meta-analyses

Bayesian methods did well for very small MAs



Summary


Meta-analyses numbers

Of the 3,845 files 2,801 had identified relevant studies andcontained any data98,615 analyses extracted 57,397 of which meta-analyses

32,005 were overall meta-analyses25,392 were subgroup meta-analyses

Estimation of an overall effectPeto method in 4,340 (7.6%)Mantel-Haenszel in 33,184 (57.8%)Inverse variance in 19,873 (34.6%)random-effects more prevalent in inverse variance methodsand larger meta-analyses

34% of meta-analyses on 2 studies (53% k ≤ 3)!



Summary


Meta-analyses by Cochrane group

22

Figures Figure 1: All meta-analyses, including single-study and subgroup meta-analyses

0

2000

4000

6000

8000

10000

12000

14000

Preg

nanc

y an

d Ch

ildbi

rth

Schi

zoph

reni

aN

eona

tal

Men

stru

al D

isord

ers a

nd S

ubfe

rtili

tyDe

pres

sion

Anxi

ety

and

Neu

rosis

Airw

ays

Hepa

to-B

iliar

yFe

rtili

ty R

egul

atio

nM

uscu

losk

elet

alSt

roke

Acut

e Re

spira

tory

Infe

ctio

nsRe

nal

Dem

entia

and

Cog

nitiv

e Im

prov

emen

tPa

in P

allia

tive

and

Supp

ortiv

e Ca

reIn

fect

ious

Dise

ases

Hear

tBo

ne Jo

int a

nd M

uscl

e Tr

aum

aM

etab

olic

and

End

ocrin

e Di

sord

ers

Gyna

ecol

ogic

al C

ance

rDe

velo

pmen

tal P

sych

osoc

ial a

nd L

earn

ing…

Colo

rect

al C

ance

rHy

pert

ensio

nAn

aest

hesia

Haem

atol

ogic

al M

alig

nanc

ies

Drug

s and

Alc

ohol

Inco

ntin

ence

Infla

mm

ator

y Bo

wel

Dise

ase

and

Func

tiona

l…M

ovem

ent D

isord

ers

Neu

rom

uscu

lar D

iseas

eO

ral H

ealth

Perip

hera

l Vas

cula

r Dise

ases

Brea

st C

ance

rTo

bacc

o Ad

dict

ion

Cyst

ic F

ibro

sis a

nd G

enet

ic D

isord

ers

Back

Skin

HIV/

AIDS

Inju

ries

Eyes

and

Visi

onW

ound

sEa

r Nos

e an

d Th

roat

Diso

rder

sEp

ileps

yU

pper

Gas

troi

ntes

tinal

and

Pan

crea

tic D

iseas

esEf

fect

ive

Prac

tice

and

Org

anisa

tion

of C

are

Pros

tatic

Dise

ases

and

Uro

logi

c Ca

ncer

sM

ultip

le S

cler

osis

and

Rare

Dise

ases

of t

he…

Mul

tiple

Scl

eros

isCo

nsum

ers a

nd C

omm

unic

atio

nLu

ng C

ance

rSe

xual

ly T

rans

mitt

ed D

iseas

esCh

ildho

od C

ance

rO

ccup

atio

nal S

afet

y an

d He

alth

Sexu

ally

Tra

nsm

itted

Infe

ctio

nsPu

blic

Hea

lth

Single Study Fixed-effect model (by choice or necessity) Random-effects model



Summary


Meta-analyses by method choice

23

Figure 2: Model selection by number of available studies (and % of random-effects meta-analyses)*

*note that in many case fixed-effect models were used when heterogeneity was detected

Figure 3: Comparison of zero between-study variance estimates rates in the Cochrane library data and in simulations, using the DerSimonian-Laird method*

*Normal distribution of the effects assumed in the simulations (more extreme distributions produced similar results).

21%

27%

31% 37%

41% 51%

15%

19%

22%

22%

27% 30%

0

2000

4000

6000

8000

10000

12000

2 3 4 5 6-9 10+Number of Studies in meta-analysis

Peto (FE) Inverse Variance (FE) Inverse Variance (RE) Mantel-Haenszel (FE) Mantel-Haenszel (RE)

0

10

20

30

40

50

60

70

80

90

100

2 3 4 5 10 20

% o

f zer

o τ^

2 es

timat

es w

ith D

erSi

mon

ian-

Laird

Number of studies in meta-analyis

Observed

true τ^2=0.01

true τ^2=0.03

true τ^2=0.10



Summary


Comparing Cochrane data with simulated

To assess the validity of a homogeneity assumption wecompared the percentage of zero τ̂2

DL, in real andsimulated dataCalculated τ̂2

DL for all Cochrane meta-analysesPercentage of zero τ̂2

DL was lower in the real data than inthe low and moderate heterogeneity simulated dataSuggests that mean true between-study variance is higherthan generally assumed but fails to be detected; especiallyfor small meta-analyses



Summary


Comparing Cochrane data with simulated

23

Figure 2: Model selection by number of available studies (and % of random-effects meta-analyses)*

*note that in many case fixed-effect models were used when heterogeneity was detected

Figure 3: Comparison of zero between-study variance estimates rates in the Cochrane library data and in simulations, using the DerSimonian-Laird method*

*Normal distribution of the effects assumed in the simulations (more extreme distributions produced similar results).

21%

27%

31% 37%

41% 51%

15%

19%

22%

22%

27% 30%

0

2000

4000

6000

8000

10000

12000

2 3 4 5 6-9 10+Number of Studies in meta-analysis

Peto (FE) Inverse Variance (FE) Inverse Variance (RE) Mantel-Haenszel (FE) Mantel-Haenszel (RE)

0

10

20

30

40

50

60

70

80

90

100

2 3 4 5 10 20

% o

f zer

o τ^

2 es

timat

es w

ith D

erSi

mon

ian-

Laird

Number of studies in meta-analyis

Observed

true τ^2=0.01

true τ^2=0.03

true τ^2=0.10



Summary


Reanalysing the Cochrane data

We applied all methods to all 57,397 meta-analyses toassess τ̂2 distributions and the sensitivity of the resultsand conclusionsFor simplicity discuss differences between standardmethods and DLb; not a perfect method but one thatperformed well overallAs in simulations, DLb identifies more heterogeneousmeta-analyses; τ̂2

DL = 0 for 50.5% & τ̂2DLb = 0 for 31.2%

Distributions of τ̂2 agree with the hypothesised χ21



Summary


Distributions for τ̂2

050

010

0015

0020

00

# of

met

a-an

alys

es

0 .1 .2 .3 .4 .5

t2 estimate

Zero est(%): DL=44.9, DLb=29.6, VC=48.9 REML=45.4 ML=62.2, B0=49.2, VC2=44.3, DL2=45.3Non-convergence(%): ML=0.7, REML=1.4.

Inverse Variance

010

0020

0030

0040

0050

00

# of

met

a-an

alys

es

0 .1 .2 .3 .4 .5

t2 estimate


Mantel-Haenszel

020

040

060

0

# of

met

a-an

alys

es

0 .1 .2 .3 .4 .5

t2 estimate


Peto & O-E

020

0040

0060

0080

00

# of

met

a-an

alys

es

0 .1 .2 .3 .4 .5

t2 estimate


all methods

non-zero estimates only

DL DLb VC ML

REML B0 VC2 DL2



Summary


Changes in results and conclusions

Inverse variance with DLbwhen τ̂2

DL = 0, conclusions change for 0.9% of analyseswhen τ̂2

DL > 0 and not ignored, conclusions change for2.4% of analyseswhen τ̂2

DL > 0 but ignored, conclusions change for 19.1% ofanalysesin overwhelming majority of changes (19.0%, 0.8%, 2.3%),effects stopped being statistically significant

Findings were similar for Mantel-Haenszel and Petomethods, although the validity of the inverse varianceweighting in these (which is a prerequisite for the use orrandom-effects models) warrants further investigation



Summary

Findings

Methods often fail to detect τ2 in small MAEven when τ̂2 > 0, often ignoredMean true heterogeneity higher than assumed orestimated; but standard method fails to detect itNon-parametric DerSimonian-Laird bootstrap seems bestmethod overall, especially in detecting heterogeneityBayesian estimators MVa (Sidik-Jonkman) and BP(Ruhkin) performed very well when k ≤ 319-21% of statistical conclusions change, when τ̂2

DL > 0but ignored



Summary

Conclusions

Detecting and accurately estimating τ̂2 in a small MA isvery difficult; yet for 53% of Cochrane MAs, k ≤ 3τ̂2 = 0 assumed to lead to a more reliable meta-analysisand high τ̂2 is alarming and potentially prohibitiveEstimates of zero heterogeneity should also be a concernsince heterogeneity is likely present but undetectedBootstrapped DL leads to a small improvement butproblem largely remains, especially for very small MAsCaution against ignoring heterogeneity when detectedFor full generalisability, random-effects essential?


Appendix Thank you!

Heal

th S

cien

ces P

rimar

y Ca

re R

esea

rch

Grou

p

[Poster title]

ABSTRACT TITLE: [Add text here.]

BACKGROUND: [Add text here.]

OBJECTIVE: [Add text here.]

METHODS: [Add text here.]

RESULTS: [Add text here.]

CONCLUSIONS: [Add text here.]

BACKGROUND [Add title, if necessary.]

Label One

Label Two

Label Three

Label Four

[Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]

OBJECTIVE

[Repeat objective from above.]

METHODS

[Add title, if necessary.]

[Add key point.] [Add description of key point.]




[Add key point.] [Sub-bullet] [Sub-bullet]

[Add key point.]

[Sub-bullet] [Sub-bullet]


[Add key point.]


[Add key point.]


RESULTS


[Add key point.] [Add key point.] [Add key point.] [Add key point.] [Add key point.]

CONCLUSIONS

[Add text as bulleted list or a paragraph.] [Add key point.]

[Add key point.]

[Add key point.]

[Add key point.]

[Add key point.]

[Graphic title]

RESULTS

[Graphic title] [Graphic title]

0% 20% 40% 60% 80% 100%

1st Qtr

2nd Qtr

3rd Qtr

4th Qtr

0% 20% 40% 60% 80% 100%

1st Qtr

2nd Qtr

3rd Qtr

4th Qtr

0% 20% 40% 60% 80% 100%

1st Qtr

2nd Qtr

3rd Qtr

4th Qtr

[Replace, move, resize, or delete graphic, as necessary.]

[Replace, move, resize, or delete graphic, as necessary.]

[Replace, move, resize, or delete graphic, as necessary.] [Replace, move, resize, or delete graphic, as necessary.]

For additional information please contact: [Name] [Department] [Institution or organization] [E-mail address]

Excepteur Sint Lkl

(n=212) Controls

(n=27)

Lorum Wt (kg) 18 (SD 10) 29 (SD 07)

Ipsum (wk) 31 (SD 5) 37 (SD 2)

Irure: B W H HB O

Unknown

79 (373%) 121 (571%)

2 (09%) 0

1 (05%) 9 (42%)

7 (259%) 18 (667%)

0 1 (37%) 1 (37%)

0

Proident F

Lorem

106 (50%) 101 (476%)

5 (24%)

17 (63%) 10 (37%)

Nostrud: N Y

Unknown

172 (811%) 22 (104%) 18 (85%)

26 (963%) 0

1 (37%)


[Add key point.] [Sub-bullet] [Sub-bullet]

[Add key point.]



[Add key point.]


[Add key point.]


Kontopantelis E, Springate D, Reeves D. A re-analysis of the CochraneLibrary data: the dangers of unobserved heterogeneity inmeta-analyses. PLoS ONE, 2013 July; 8(7): e69930.doi:10.1371/journal.pone.0069930

This project is supported by the School for Primary Care Researchwhich is funded by the National Institute for Health Research (NIHR).The views expressed are those of the author(s) and not necessarilythose of the NHS, the NIHR or the Department of Health.

Comments, suggestions: [email protected]


RSS 2013 - A re-analysis of the Cochrane Library data]

Science

primary care

restricted

cochrane library

add key point

key point

inverse variance

add description

add title