Funnel Plot PPT IMP

, , ,

Publication Bias Funnel Plots Examples References

Publication Bias and Funnel Plots

Julia Braun

University of Zurich

Zurich, March 31, 2009

Julia Braun University of Zurich


, , ,


Outline

1 Publication Bias

2 Funnel Plots

3 Examples



, , ,


Definition

Publication Bias (Dickersin, 1990):

Publication bias occurs when the publication of research resultsdepends on their nature and direction.

Other kinds of bias: language bias

availability bias

cost bias

familiarity bias

outcome bias

...



, , ,


Sources of publication bias

Explorative studies:

Experimental: submission of similar manuscripts with positiveand negative findings, comparison of recommendations topublish

Cohort: pursuit of all studies accepted by a local ethicscommittee

Main factors:

Failure to publish due to negative or null findings (mostly onthe side of the researchers, not editors/journals)

Selective reporting of outcomes (e.g. due to changes inresearch plan)

Selective citation of positive results



, , ,


Impact of publication bias

Comparison of the results that are reported and the best guess(informed by the results of publication bias analyses).

Three levels:

minimal: essentially similar estimates of the effect size

modest: effect size changes substantially, but key finding(e.g. treatment is or is not effective) remains the same

severe: basic conclusion of the analysis changes

=⇒ danger of false conclusions and patient harm



, , ,


Solutions?

Registration and consequent tracking of clinical trials⇒ Mandatory? One general registry?

Encouraging the publication of all studies,e.g. Journal of Negative Results in Biomedicine⇒ Acceptance? Practicability?

Required:

Every meta-analysis should include an analysis of publication bias,and this analysis should be reported as a standard part of theresults.

=⇒ Techniques for the assessment of publication bias



, , ,


Funnel Plot

−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5

0.6

0.4

0.2

0.0

log OR

Sta

ndar

d er

ror

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

● ●●

●

●

●

●

●

●

●

●

●●

−1.0 −0.5 0.0 0.5 1.0

02

46

810

1214

log OR

Inve

rse

stan

dard

err

or

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●●●

●

●

●

●

●

●

●

●

● ●

Fundamental assumption: In the absence of bias, a plot fromstudies which estimate the same intervention effect will besymmetrical and bear some resemblance to a funnel(German translation: funnel = Kamin, Schacht, Trichter).



, , ,


Definition

Funnel Plot:

A funnel plot is a scatter plot of the treatment effects estimatedfrom individual studies against a measure of study size.

Properties:

The smaller the study size, the wider the spread of thetreatment effects and vice versa

If there is no bias: form of a funnel, symmetrical

If there is bias: asymmetrical

Explorative tool, no quantitative information on the amountor the source of the bias

No empirical investigations on funnel plots for continuousoutcomes so far



, , ,


Composition of a funnel plot

Decisions:

Which direction (horizontal vs. vertical, ascendingvs. descending order)?

Which measure for the treatment effect (log OR, log RR,RD)?

Which measure for the study size?

Inclusion of summary estimate?

Which confidence intervals (pointwise vs. lines around thesummary estimate)?

Mirror option?

Choice of these elements can have a substantial influence on thevisual impression!



, , ,


Measures for the treatment effect

Risk difference:

Easy interpretation

Possibly increased heterogeneitycan cause additional asymmetry

−0.4 −0.2 0.0 0.2 0.4

0.20

0.15

0.10

0.05

0.00

Risk difference

Sta

ndar

d er

ror

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●

⇒ not recommended



, , ,



Log risk ratio:

Easy interpretation

Possible heterogeneity if eventrate is high

Conclusions may depend onoutcome definition

−3 −2 −1 0 1 2 3

1.5

1.0

0.5

0.0

Log risk ratio

Sta

ndar

d er

ror

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

⇒ not recommended if event rate is high



, , ,



Log odds ratio:

Interpretation more complicated

Shape of the plot not dependenton outcome definition

−3 −2 −1 0 1 2 3

1.5

1.0

0.5

0.0

Log odds ratio

Sta

ndar

d er

ror

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

⇒ recommended in most cases



, , ,



−3 −2 −1 0 1 2 3

1.5

1.0

0.5

0.0

Log odds ratio

Sta

ndar

d er

ror

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

−3 −2 −1 0 1 2 3

1.5

1.0

0.5

0.0

Log risk ratio

Sta

ndar

d er

ror

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

−0.4 −0.2 0.0 0.2 0.4

0.20

0.15

0.10

0.05

0.00

Risk difference

Sta

ndar

d er

ror

●

●

●

●

● ●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●



, , ,


Measures for the study size

Study size/logarithmic study size:

−2 −1 0 1

050

0010

000

1500

0

Log odds ratio

Stu

dy s

ize

● ● ● ●

●●●

●● ● ●●

●● ● ●

●●

●●

●●

●●

●● ●● ● ●● ●●●

●

●

●

● ●● ● ●● ●● ● ●● ●●● ● ●●●

●

●●●

●

●

● ●●

●

●● ● ●●

−2 −1 0 13

45

67

89

10Log odds ratio

Log

stud

y si

ze

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

Expected shape of the plot in the case of no bias can not bederived⇒ not recommended!



, , ,



Standard error:

Characteristic funnel shape

Straight confidence lines

Emphasis on smaller studies(bias more likely)

Axis often inverted

−3 −2 −1 0 1 2 3

1.5

1.0

0.5

0.0

Log odds ratio

Sta

ndar

d er

ror

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

⇒ recommended in most cases



, , ,



Inverse standard error:

No funnel shape

Curved confidence lines

Emphasis on larger studies(small studies compressed at thebottom)

Probably useful for thecomparison of meta-analyses ofsmall and large studies

−3 −2 −1 0 1 2 3

05

1015

20

Log odds ratio

Inve

rse

of s

tand

ard

erro

r

●

●●

●

●

●●

●

●

●●●

●

●●

●

●●

●

●

●

●

●

●

●

● ●●

●

●

●●

●●

●

●

●

●

●

● ● ●

●

●●● ●●

●●●●

●

●

●

●

●

●●

●

●

●●●

●

●●

●

●

●

⇒ recommended for special tasks



, , ,



Variance:

No funnel shape


Emphasis on smaller studies(bias more likely)

Axis often inverted

−3 −2 −1 0 1 2 3

2.5

2.0

1.5

1.0

0.5

0.0

Log odds ratio

Var

ianc

e

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

⇒ not recommended



, , ,



Inverse variance:

No funnel shape


Emphasis on larger studies(small studies compressed at thebottom)

Probably useful for thecomparison of meta-analyses ofsmall and large studies

−3 −2 −1 0 1 2 3

010

020

030

040

0

Log odds ratio

Inve

rse

of v

aria

nce

● ● ● ●

●

●●●

●● ●●

●●● ●

●●●

●

●

●●

●●● ●● ● ●● ●●●

●

●

●

● ●● ● ●● ●● ● ●● ●●● ● ●●●●

●●●

●

●

● ●●

●

●●● ●●

⇒ sometimes useful, but inverse standard error should be preferred



, , ,



−3 −2 −1 0 1 2 3

1.5

1.0

0.5

0.0

Log odds ratio

Sta

ndar

d er

ror

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

−3 −2 −1 0 1 2 3

2.5

2.0

1.5

1.0

0.5

0.0

Log odds ratioV

aria

nce

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

−2 −1 0 1

050

0010

000

1500

0

Log odds ratio

Stu

dy s

ize

● ● ● ●●

●●●● ● ●● ●●● ●

●●●●

●●●●●● ●● ● ●● ●●●

●

●

●

● ●● ● ●● ●● ● ●● ●●● ● ●●●

●●●●

●

●

● ●●

●

●● ● ●●

−3 −2 −1 0 1 2 3

05

1015

20

Log odds ratio

Inve

rse

of s

tand

ard

erro

r

●● ●

●

●

●●●

●

●●●

●

●●

●

●●

●

●

●

●

●

●●

● ●●

●

●

●●●●

●

●

●

●

●

● ● ●●

●● ● ●●●●● ● ●●

●

●

●

●●

●

●

● ●●

●

●●

●●

●

−3 −2 −1 0 1 2 3

010

020

030

040

0

Log odds ratio

Inve

rse

of v

aria

nce

● ● ● ●

●

●●●●● ●●

●●● ●

●●●

●

●

●●

●●● ●● ● ●● ●●●

●

●

●

● ●● ● ●● ●● ● ●●●●● ● ●●●●

●●●

●

●

● ●●

●

●● ●●●

−2 −1 0 13

45

67

89

10

Log odds ratio

Log

stud

y si

ze●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●



, , ,


Confidence intervals

Alternative to confidence lines: Pointwise confidence intervals

●

●

●

●

●

●

●

● ●

●

−2.0 −1.5 −1.0 −0.5

01

23

4

Log odds ratio

Inve

rse

stan

dard

err

or

●

●

●

●

●

●

●

● ●

●

−3 −2 −1 0

01

23

4

Log odds ratio

Inve

rse

stan

dard

err

or



, , ,


Mirror option

●

●

●

●

●

●

●

● ●

●

−4 −3 −2 −1 0

01

23

4

Log odds ratio

Inve

rse

stan

dard

err

or

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

−4 −3 −2 −1 0 1 2

01

23

4

Log odds ratio

Inve

rse

stan

dard

err

or

●

●

●

●

●

●

●

●●

●



, , ,


Use of funnel plots

Problems:

No clear recommendations for non-binary outcomes

Primary visual tool, no quantification of bias

Asymmetry not necessarily due to bias (e.g. heterogeneity inthe underlying effects can also lead to asymmetry)

No information on the type of bias

Interpretation difficult if only few studies in meta-analysis

Funnel plots and tools for meta-analysis in R:packages meta and rmeta



, , ,


Example 1: Environmental tobacco smoke

Meta-analysis by Hackshaw et al. (1997):

Effect of environmental tobacco smoke on risk of lung cancer

Comparison of spouses of smokers and non-smokers

37 studies

Controversy: Affected by publication bias?



, , ,


Funnel plot of Hackshaw et al. (1997)

−1.0 −0.5 0.0 0.5 1.0 1.5

0.6

0.4

0.2

0.0

Log odds ratio

Sta

ndar

d er

ror

●●

●●●

●●● ●●●

●

●●●

●●●● ●● ●

●

●●

●●●

●●●

●

●●

●

●

●

Smaller studies tend to show greater adverse effects⇒ asymmetry

Not clear if real bias or chance

Visual inspection not enough



, , ,


Example 2: Antibacterial catheter coating

Meta-analysis by Veenstra et al. (1998):

Effectiveness of a special coating on venuous catheters

Two endpoints: Bacterial colonisation of the catheter andbloodstream infection

16 controlled trials



, , ,


Funnel plot of Veenstra et al. (1998)

Bacterial colonisation:

−4 −3 −2 −1 0 1 2

1.5

1.0

0.5

0.0

Log odds ratio

Sta

ndar

d er

ror

●

●

●

●

●

●

●

●

●

●

● ●

●

⇒ hints for publication bias



, , ,


Funnel plot of Veenstra et al. (1998)

Bloodstream infection:

−3 −2 −1 0 1 2

1.5

1.0

0.5

0.0

Log odds ratio

Sta

ndar

d er

ror

●

●

● ●

●

●

●

●

●

●

●

●

⇒ hints for publication bias



, , ,


Summary

Visual tool for investigation of (publication) bias inmeta-analysis

Scatter plots of treatment effect versus study size

For binary outcomes: odds ratio and standard errors bestchoice

In the case of bias: asymmetry

Other causes of asymmetry possible

Interpretation difficult if number of studies small



, , ,


References

Dickersin, K. (1990). The existence of publication bias and risk factors for itsoccurrence, Journal of the American Medical Association263(10): 1385–1395.

Hackshaw, A., Law, M. and Wald, N. (1997). The accumulated evidence onlung cancer and environmental tobacco smoke, British Medical Journal315: 980–988.

Rothstein, H., Sutton, A. and Borenstein, M. (eds) (2005). Publication Bias inMeta-Analysis, Wiley.

Veenstra, D., Saint, S., Saha, S., Lumley, T. and Sullivan, S. (1998). Efficacyof antiseptic impregnated central venous catheters in preventing nosocomialinfections: a meta analysis, Journal of the American Medical Association281: 261–267.



Funnel Plot PPT IMP

Documents

environmental

log odds ratio

log study

inverse standard

small studies

study size

standard error

qq qq