evaluation of excess Statistical Significance in Meta-analyses of 98 Biomarker Associations with Cancer Risk

DOI:10.1093/jnci/djs437Advance Access publication October 22, 2012

JNCI | Articles 1867

© The Author 2012. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: [email protected].

jnci.oxfordjournals.org

Review

evaluation of excess Statistical Significance in Meta-analyses of 98 Biomarker Associations with Cancer RiskKonstantinos K. Tsilidis, Stefania I. Papatheodorou, Evangelos Evangelou, John P. A. Ioannidis

Manuscript received March 6, 2012; revised August 13, 2012; accepted August 20, 2012.

Correspondence to: John P.A. Ioannidis, MD, DSc, Stanford Prevention Research Center, 1265 Welch Rd, MSOB X306, Stanford, CA 94350. (e-mail: [email protected]).

Background Numerous biomarkers have been associated with cancer risk. We assessed whether there is evidence for excess statistical significance in results of cancer biomarker studies, suggesting biases.

Methods We systematically searched PubMed for meta-analyses of nongenetic biomarkers and cancer risk. The number of observed studies with statistically significant results was compared with the expected number, based on the statistical power of each study under different assumptions for the plausible effect size. We also evaluated small-study effects using asymmetry tests. All statistical tests were two-sided.

Results We included 98 meta-analyses with 847 studies. Forty-three meta-analyses (44%) found nominally statistically significant summary effects (random effects). The proportion of meta-analyses with statistically significant effects was highest for infectious agents (86%), inflammatory (67%), and insulin-like growth factor (IGF)/insulin sys-tem (52%) biomarkers. Overall, 269 (32%) individual studies observed nominally statistically significant results. A statistically significant excess of the observed over the expected number of studies with statistically significant results was seen in 20 meta-analyses. An excess of observed vs expected was observed in studies of IGF/insulin (P ≤ .04) and inflammation systems (P ≤ .02). Only 12 meta-analyses (12%) had a statistically significant summary effect size, more than 1000 case patients, and no hints of small-study effects or excess statistical significance; only four of them had large effect sizes, three of which pertained to infectious agents (Helicobacter pylori, hepatitis and human papilloma viruses).

Conclusions Most well-documented biomarkers of cancer risk without evidence of bias pertain to infectious agents. Conversely, an excess of statistically significant findings was observed in studies of IGF/insulin and inflammation systems, suggesting reporting biases.

J Natl Cancer Inst 2012;104:1867–1878

Since the 1980s, biomarkers have frequently been used to refine exposure assessment and evaluate more accurately potential asso-ciations with cancer incidence. Biomarkers may also further our understanding of the mechanisms of carcinogenesis. However, biomarker-based epidemiologic studies are not free from biases. Empirical evidence from diverse fields suggests that the literature of biomarker studies may sometimes highlight strong effects that are irreproducible or found to be smaller when larger studies are performed (1,2). One particular concern is that the literature of biomarkers seems to suffer from selective reporting biases favor-ing statistically significant (“positive”) results (3,4). Studies for biomarkers of cancer prognosis almost always report some statisti-cally significant result (5). However, it is unclear whether this also equally applies to studies that try to identify associations of bio-markers with cancer risk rather than prognosis.

Bias in favor of positive results may be generated with several different mechanisms (6). First, bias against the publication of neg-ative results or publication of those results after considerable delay

may exist (7). Second, selective analysis and outcome reporting bias may emerge when there are many analyses that can be performed (using, for example, different outcome definitions, different adjust-ments for confounders, or models with different statistical terms for exposures and confounders), but only the analysis with the “best” results is presented (8–11). Third, in theory, positive results may be totally faked, although fraud and data fabrication is unlikely to be nearly as common as the other mechanisms. All of these mecha-nisms eventually end up producing a literature where an inflated proportion of published studies with positive results exists.

Detecting these biases is not a straightforward process. There are several statistical methods that try to probe for publication bias in studies included in meta-analyses, the most popular of which are asymmetry tests evaluating whether small studies give different results than larger ones (12). However, these methods may not be very sensitive or specific for detecting such biases, especially when a limited number of studies is included in a meta-analysis (12–14). An alternative approach is to examine whether there are too many

by guest on June 12, 2016http://jnci.oxfordjournals.org/

Dow

nloaded from

mailto:[email protected]

http://jnci.oxfordjournals.org/

Vol. 104, Issue 24 | December 19, 20121868 Articles | JNCI

reported statistically significant results in single studies based on what would be expected under different assumptions about the plausible effect size of each association (15). An added advantage of this excess statistical significance test is that it can be applied not only to a single meta-analysis but also to many meta-analyses across a given field. Thus power is optimized to detect biases that pertain to larger fields and disciplines rather than just single associations. This test has been applied before and has found an excess of statistically significant findings in the fields of randomized trials for neuroleptic drugs, genetic association studies for Alzheimer’s disease, and brain volume abnormalities (15–17).

The literature on biomarkers and risk of cancer is rapidly expanding and is expected to grow even more in the future with the incorporation of transcriptomics, proteomics, and metabo-lomics (18). It is important to understand the extent of potential biases in this field as multiple associations accumulate in its litera-ture. Therefore, in this paper, we probed whether there is evidence for excess statistical significance in studies of biomarkers and risk of cancer, and we tried to evaluate how many of the previously studied associations that have been synthesized with meta-analyses have robust evidence for their presence.

MethodsStudy IdentificationWe systematically searched PubMed from 1966 through December 20, 2010, to identify meta-analyses of epidemiologic studies for the association between any biomarker and cancer risk. The search strategy used the key words “cancer or carcinoma or tumor or neo-plasm or neoplasia or maligna*.” The search was limited to meta-analyses of human studies published in the English language, and the above keywords had to appear in the title of the papers. The full text of potentially eligible articles was scrutinized independently by two investigators (S. I. Papatheodorou and K. K. Tsilidis). We excluded meta-analyses that investigated the association between genetic markers and risk of cancer and also studies where biomark-ers were used for screening, diagnostic, or prognostic purposes. Articles were retained if they included at least one meta-analysis in which information was provided per included study on a measure of association and its standard error between the biomarker and cancer risk and on the number of cancer cases and controls. When more than one meta-analysis on the same research question was eligible, the meta-analysis with the largest number of component studies was retained for the main analysis. We also performed a sensitivity analysis where we compared the summary effects and 95% confi-dence intervals between older and newer meta-analyses on the same question. We examined how many of these associations changed nominal statistical significance (statistically significant becoming statistically nonsignificant and vice versa) or retained nominal sta-tistical significance but their effect was more than halved or more than doubled in the new vs the old meta-analysis.

Estimation of Summary Effect and HeterogeneityFor each meta-analysis, we estimated the summary effect size and its confidence intervals using both fixed and random effects models (19). We consistently used the odds ratio for all calculations, except for one meta-analysis that provided a standardized mean difference as a

measure of association, and this was transformed to an odds ratio using an established formula (20,21). We also tested for between-study het-erogeneity, and we reported the P value of the χ2-based Cochran Q test, and the I2 metric of inconsistency. The Q test is obtained from the weighted sum of the squared differences of the observed effect in each study minus the fixed summary effect (22). The I2 metric ranges between 0 and 100% and is the ratio of between-study variance over the sum of the within- and between-study variances (23). Its 95% confidence intervals were calculated as per Ioannidis et al. (24).

Asymmetry Tests for Small-Study EffectsWe evaluated whether there is evidence for small-study effects (ie, whether smaller studies tend to give substantially larger estimates of effect size compared with larger studies). Small-study effects may offer a hint for publication and other selective reporting biases, but they may also reflect genuine heterogeneity, chance, or other rea-sons for differences between small and large studies (12). We used the regression asymmetry test proposed by Egger et al (25). When odds ratios are derived from 2 × 2 tables, this test may be modestly biased, and modified variants of the test have been proposed (26). However, the 2 × 2 table data were not reported by individual study in many meta-analyses to allow using the modified regression tests, and also most meta-analyses used confounder-adjusted estimates of effect in single studies rather than unadjusted odds ratios calculated from 2 × 2 tables; therefore, the chosen asymmetry test would be appropriate. We used the modified variant of the Egger test in six meta-analyses with available 2 × 2 table data for each component study that had a P value for the Egger test less than .25 (27–32). A P value less than .10 with more conservative effect in larger studies was considered evidence for small-study effects.

Evaluation of Excess Statistical SignificanceWe applied the excess statistical significance test, which evaluates whether the observed number of studies with nominally statisti-cally significant results (positive studies, P < .05) differs from their expected number. We used a binomial test, as previously presented in detail (15). This test evaluates whether the number of positive studies among those in a meta-analysis is too large based on the power that these studies have to detect plausible effects at α equal to 0.05. The observed vs expected comparison is performed sep-arately for each meta-analysis, and it is also extended to groups including many meta-analyses after summing the observed and expected from each meta-analysis.

The expected number of studies with statistically significant results is calculated in each meta-analysis by the sum of the statistical power estimates for each component study. The sum of the power estimates gives the expected number of positive studies. The estimated power of each component study depends on the plausible effect size for the tested biomarker–cancer association. The true effect size for any meta-analysis is not known, but it can be assumed to be the observed summary effect size in the absence of bias. We conducted sensitivity analyses using various plausible effect sizes: the fixed effects summary, the random effects summary, the effect size of the largest study (smallest standard error) in a meta-analysis, and the most conservative of these three estimates. In the presence of bias, the summary effects are likely to be larger than the true effect, and this situation may arise also for the effect


Dow

nloaded from


JNCI | Articles 1869jnci.oxfordjournals.org

of the largest study. Therefore, all these assumptions tend to be very conservative in testing for excess statistical significance.

The power of each study was calculated with an algorithm using a noncentral t distribution. When we repeated the power calcula-tions in a subset of our sample using logistic regression models in freely available software (33), we observed very similar estimates. Excess statistical significance for single meta-analyses was claimed at P less than .10 [one-sided P < .05, with observed > expected as previously proposed (15)].

We classified biomarkers into the following categories based on biological pathways or types of exposures involved: insulin-like growth factor (IGF)/insulin system, sex hormones, diet, inflamma-tion, infectious agents, and environment. We assessed excess statisti-cal significance separately in each of these categories because selective reporting bias may affect to a different extent different domains of research. These domains may also have different typical magnitudes of effect sizes and different analytical biases, even if research is some-times conducted by the same teams across various domains.

We also performed subgroup analyses. The excess statistical significance test was performed separately for meta-analyses with I2 values less than or equal to 50% and greater than 50% because values exceeding 50% are typically considered evidence of large heterogeneity beyond chance (34). Analyses were also performed by evidence or not of small-study effects, by whether the eligible study was a meta-analysis or an analysis of shared data through a

consortium, and by whether the summary effect of the meta-analy-sis was nominally statistically significant or not per random effects calculations. All statistical tests were two-sided.

ResultsDescription of Eligible Meta-analysesThe search identified 2840 items, of which 2775 were excluded after the title and abstract review (Figure 1). Of the remaining 65 articles that entered the full-text review, 16 pertained to associa-tions where a more recent meta-analysis was available, 11 did not report sufficient information for the calculation of the excess sta-tistical significance, and one did not use a biomarker. Therefore, 37 articles were selected (20,27–32,35–64), which included data on 98 meta-analyses (comparisons) in six broad areas of biomarkers for cancer risk (IGF/insulin system [n = 21 comparisons], sex hor-mones [n = 13 comparisons], diet [n = 31 comparisons], inflamma-tion [n = 3 comparisons], infectious agents [n = 22 comparisons] and environment [n = 8 comparisons]).

Supplementary Table 1 (available online) summarizes these 98 meta-analyses that included 847 studies. There were 2 to 42 stud-ies per meta-analysis, with a median of seven studies. The median number of case and control subjects in each study was 120 and 229, respectively, whereas the median number of case and control subjects in each meta-analysis was 1157 and 2119, respectively.

PubMed Search (n = 2,840)

Abstract Review (n = 147)

Article Review (n = 65)

37 Articles Selected (with data on 98 meta-analyses)

References excluded (n = 82) Studies of genes and cancer (n = 3) Studies of other risk factors and cancer (n = 6) Studies of efficacy of cancer treatments (n = 2) Studies of cancer prognosis (n = 5) Studies of cancer diagnosis/screening (n = 2) No biomarker used (n = 47) No outcome of cancer (n = 8) Not a meta-analysis (n = 9)

References excluded (n = 28) No biomarker used (n = 1)

Duplicate data/more recent study available (n = 16) Appropriate information not available (n = 11)

References excluded (n = 2,693) Studies of genes and cancer (n = 326) Studies of other risk factors and cancer (n = 629)Studies of efficacy of cancer treatments (n = 887)Studies of cancer prognosis (n = 296) Studies of cancer diagnosis/screening (n = 177) No biomarker used (n = 6) No outcome of cancer (n = 263) Not a meta-analysis (n = 109)

Title Review (n = 2,840)

Figure 1. Flow diagram of the study selection process.


Dow

nloaded from

http://jnci.oxfordjournals.org/lookup/suppl/doi:10.1093/jnci/djs437/-/DC1



Fifty-seven meta-analyses included more than 1000 case patients. Overall, 269 (32%) individual studies observed nominally statisti-cally significant results.

Summary Effect Sizes. Of the 98 meta-analyses, 52 (53%) had nominally statistically significant findings using the fixed effects method, of which 37 reported increased risks and 15 showed decreased risks of cancer. A total of 43 (44%) meta-analyses reported statistically significant findings using the random effects method, of which 34 showed elevated risks. Table 1 shows the summary effect sizes per fixed and random effects and the results of the largest study for these 43 meta-analyses. Supplementary Table 1 (available online) provides this information for all 98 meta-analyses. Only 30 of the 98 associations were nominally statistically significant, based on the largest study estimate. Figure 2 juxtaposes the estimates of the largest studies against the random effects meta-analysis estimates. As shown, among the 41 meta-analyses with summary estimates suggesting decreased risk estimates, 24 showed smaller decreased risk estimates or even increased risk estimates in the largest stud-ies; among the 57 meta-analyses with summary estimates suggesting increased risk, 38 showed smaller increased risk estimates or even decreased risk estimates in the largest studies. Thus, the largest stud-ies were more conservative than the summary effects of the meta-analysis in 62 (63%) of the 98 meta-analyses. Most of the differences in effect size between the largest study and the meta-analyses were modest: among the 43 associations of Table 1, there were nine for which the ratio of the odds ratios of meta-analyses vs largest studies exceeded 1.5 in either direction (five for which the result was more conservative in the meta-analysis and four for which the result was more conservative in the largest study).

There were marked differences in the proportion of associa-tions that had nominally statistically significant summary effects across the six categories of biomarkers. Based on random effects

calculations, 86%, 67%, and 52% of the meta-analyses on infec-tious agents, inflammation, and IGF/insulin systems, respectively, found nominally statistically significant summary effects, whereas this was seen only in 31%, 19%, and 13% of the meta-analyses on sex hormones, diet, and environment, respectively.

When we compared the summary effects in the 24 meta-analyses (derived from 16 older articles) that were excluded because a more recent meta-analysis was available to the effects of the more recent respective meta-analyses, we found that in five instances the associa-tion was statistically significant in the oldest available meta-analysis but became nonstatistically significant in the new meta-analysis, in two instances the opposite happened, and in another two instances the effect was nominally statistically significant and remained so, but its size decreased to less than half in the new vs the old meta-analysis.

Between-Study HeterogeneityThere was nominally statistically significant heterogeneity in 40 of the 98 meta-analyses (Supplementary Table 1, available online). The highest proportion of statistically significant heterogeneity was observed in meta-analyses on biomarkers of inflammation (67%) and IGF/insulin (57%). Values of I2 exceeding 50% were noted in 35 of 98 meta-analyses (36%), and 16 of those had values exceeding 75%. However, many of the 95% confidence intervals of this heterogeneity metric were large, especially when there was only a limited number of studies.

Small-Study EffectsEvidence for statistically significant small-study effects was noted in 9 of 98 meta-analyses (Supplementary Table 1, available online). This included four meta-analyses on the IGF/insulin system (IGF-1 and pre-menopausal breast cancer, IGF binding protein 3 [IGFBP-3] and prostate cancer, IGFBP-2 and prostate cancer, and C-peptide and colorectal cancer), three on infectious agents

Figure 2. Odds ratio in each meta-analysis (random effects summary estimate) and in the largest study on each association. The size of each point is proportional to the weight of the evidence (inverse of the variance) of the largest study in each meta-analysis.


Dow

nloaded from






Tab

le 1

. D

escr

ipti

on

of

the

43 m

eta-

anal

yses

wit

h n

om

inal

ly s

tati

stic

ally

sig

nif

ican

t ef

fect

acc

ord

ing

to

ran

do

m e

ffec

ts c

alcu

lati

on

s*

Su

mm

ary

od

ds

rati

o (

95%

co

nfi

den

ce in

terv

al)

Met

a-an

alys

esC

om

par

iso

nN

o. o

f ca

se/c

on

tro

l su

bje

cts

Fixe

d e

ffec

tsR

and

om

eff

ects

Larg

est

stu

dy†

Rin

aldi

et

al, 2

010

(35)

IGF-

1 an

d C

RC

2862

/496

61.

07 (1

.01

to 1

.14)

1.07

(1.0

1 to

1.1

4)1.

04 (0

.94

to 1

.14)

Mor

ris e

t al

, 200

6 (3

6)IG

F-2

and

CR

C38

4/13

011.

95 (1

.26

to 3

.00)

1.95

(1.2

6 to

3.0

0)2.

09 (1

.14

to 3

.82)

Key

et a

l, 20

10 (3

7)IG

F-1

and

post

men

opau

sal B

rCA

2853

/533

21.

30 (1

.13

to 1

.49)

1.30

(1.1

3 to

1.4

9)1.

48 (1

.11

to 1

.97)

Key

et a

l, 20

10 (3

7)IG

FBP-

3 an

d po

stm

enop

ausa

l BrC

A28

16/5

196

1.21

(1.0

4 to

1.4

1)1.

22 (1

.01

to 1

.49)

1.12

(0.7

6 to

1.6

5)R

owla

nds

et a

l, 20

09 (3

9)IG

F-1

and

PrC

A74

81/1

1 86

61.

18 (1

.14

to 1

.23)

1.21

(1.0

7 to

1.3

6)1.

05 (0

.92

to 1

.19)

Row

land

s et

al,

2009

(39)

IGFB

P-3

and

PrC

A66

76/1

0 48

40.

97 (0

.93

to 1

.01)

0.88

(0.7

9 to

0.9

8)1.

23 (1

.07

to 1

.42)

Pis

ani,

2008

(40)

C-p

eptid

e an

d C

RC

1309

/423

31.

36 (1

.15

to 1

.62)

1.51

(1.1

4 to

1.9

9)1.

37 (1

.00

to 1

.88)

Pis

ani,

2008

(40)

C-p

eptid

e an

d B

rCA

1403

/211

41.

26 (1

.07

to 1

.48)

1.35

(1.0

1 to

1.8

1)1.

07 (0

.80

to 1

.44)

Pis

ani,

2008

(40)

C-p

eptid

e an

d pa

ncre

atic

CA

209/

483

1.70

(1.1

1 to

2.6

1)1.

70 (1

.11

to 2

.61)

1.52

(0.8

7 to

2.6

4)P

isan

i, 20

08 (4

0)G

luco

se a

nd C

RC

1741

/1 3

80 0

001.

19 (1

.07

to 1

.32)

1.28

(1.0

6 to

1.5

4)1.

13 (0

.98

to 1

.30)

Pis

ani,

2008

(40)

Glu

cose

and

pan

crea

tic C

A29

8/1

330

000

1.98

(1.6

7 to

2.3

5)1.

98 (1

.67

to 2

.35)

2.09

(1.7

0 to

2.5

8)R

odda

m e

t al

, 200

8 (4

1)S

HB

G a

nd P

rCA

3704

/599

80.

86 (0

.76

to 0

.97)

0.86

(0.7

6 to

0.9

7)0.

81 (0

.61

to 1

.08)

Key

et a

l, 20

02 (4

2)E

2 an

d po

stm

enop

ausa

l BrC

A65

6/17

091.

29 (1

.14

to 1

.45)

1.26

(1.0

7 to

1.4

9)1.

16 (0

.90

to 1

.48)

Bar

ba e

t al

, 200

9 (4

3)16

a-O

HE

1 an

d Pr

CA

122/

414

1.82

(1.0

8 to

3.0

5)1.

82 (1

.08

to 3

.05)

1.73

(0.9

6 to

3.1

2)B

arba

et

al, 2

009

(43)

2OH

E1/

16a-

OH

E1

and

PrC

A12

2/41

40.

52 (0

.31

to 0

.89)

0.52

(0.3

1 to

0.8

9)0.

54 (0

.30

to 0

.99)

Che

n et

al,

2010

(46)

25(O

H) v

itam

in D

and

BrC

A54

89/5

841

0.58

(0.5

1 to

0.6

6)0.

55 (0

.38

to 0

.80)

0.31

(0.2

3 to

0.4

1)G

allic

chio

et

al, 2

008

(47)

Lyco

pene

and

lung

CA

727/

4567

0.71

(0.5

1 to

0.9

9)0.

71 (0

.51

to 0

.99)

0.86

(0.5

2 to

1.4

3)S

aada

tian-

Ela

hi e

t al

, 200

4 (4

9)E

icos

apen

tano

ic a

cid

and

BrC

A93

1/13

600.

91 (0

.87

to 0

.95)

0.91

(0.8

7 to

0.9

5)0.

91 (0

.87

to 0

.95)

Sim

on e

t al

, 200

9 (5

1)A

-lino

leni

c ac

id a

nd P

rCA

1091

/127

01.

51 (1

.17

to 1

.94)

1.54

(1.1

6 to

2.0

6)1.

31 (0

.89

to 1

.95)

Col

lin e

t al

, 201

0 (5

3)V

itam

in B

12 a

nd P

rCA

2906

/649

51.

09 (1

.03

to 1

.14)

1.10

(1.0

1 to

1.1

9)1.

07 (0

.99

to 1

.15)

Lars

son

et a

l, 20

10 (5

4)V

itam

in B

6 an

d C

RC

883/

1424

0.52

(0.3

8 to

0.7

1)0.

52 (0

.38

to 0

.71)

0.52

(0.2

9 to

0.9

2)Ts

ilidi

s et

al,

2008

(55)

C-r

eact

ive

prot

ein

and

CR

C11

59/3

7 98

61.

10 (1

.02

to 1

.18)

1.12

(1.0

1 to

1.2

5)1.

11 (0

.95

to 1

.30)

Hei

kkila

et

al, 2

009

(56)

C-r

eact

ive

prot

ein

and

CA

4438

/70

107

1.09

(1.0

5 to

1.1

3)1.

10 (1

.02

to 1

.18)

1.00

(0.9

4 to

1.0

7)W

ang

et a

l, 20

07 (2

7)H

. pyl

ori a

nd e

arly

gas

tric

CA

2722

/13

976

4.83

(4.2

7 to

5.4

8)3.

38 (2

.15

to 5

.32)

6.38

(5.5

0 to

7.4

1)H

uang

et

al, 2

003

(57)

H. p

ylor

i and

gas

tric

CA

2284

/277

02.

05 (1

.79

to 2

.35)

2.29

(1.7

1 to

3.0

5)2.

07 (1

.44

to 2

.98)

Hua

ng e

t al

, 200

3 (5

7)ca

gA a

nd g

astr

ic C

A17

07/2

124

2.65

(2.2

9 to

3.0

5)2.

87 (1

.95

to 4

.22)

4.12

(2.9

7 to

5.7

2)Zh

ao e

t al

, 200

8 (5

8)H

. pyl

ori a

nd C

RC

1709

/187

21.

41 (1

.22

to 1

.65)

1.49

(1.1

6 to

1.9

0)1.

02 (0

.69

to 1

.50)

Zhuo

et

al, 2

008

(28)

H. p

ylor

i and

Lar

ynge

al C

A10

8/24

92.

02 (1

.27

to 3

.23)

2.02

(1.2

7 to

3.2

3)1.

74 (0

.84

to 3

.58)

Isla

mi a

nd K

aman

gar,

2008

(59)

H. p

ylor

i and

eso

phag

eal a

deno

CA

840/

2890

0.56

(0.4

8 to

0.6

7)0.

57 (0

.47

to 0

.69)

0.48

(0.3

3 to

0.6

9)Is

lam

i and

Kam

anga

r, 20

08 (5

9)ca

gA a

nd e

soph

agea

l ade

no C

A27

5/11

970.

41 (0

.29

to 0

.59)

0.41

(0.2

8 to

0.6

2)0.

26 (0

.14

to 0

.49)

Zhuo

et

al, 2

009

(29)

H. p

ylor

i and

lung

CA

199/

231

2.31

(1.4

6 to

3.6

5)3.

24 (1

.11

to 9

.41)

1.24

(0.6

3 to

2.4

3)G

utie

rrez

et

al, 2

006

(60)

HP

V (D

NA

) and

bla

dder

CA

478/

179

2.29

(1.3

7 to

3.8

4)2.

30 (1

.33

to 4

.00)

2.44

(0.9

4 to

6.3

3)G

utie

rrez

et

al, 2

006

(60)

HP

V (n

o D

NA

) and

bla

dder

CA

176/

203

2.98

(1.6

5 to

5.4

0)2.

98 (1

.65

to 5

.40)

2.59

(1.3

3 to

4.8

7)H

obbs

et

al, 2

006

(31)

HP

V a

nd o

ral C

A16

41/2

335

1.68

(1.3

6 to

2.0

8)1.

99 (1

.17

to 3

.38)

1.50

(1.2

0 to

2.0

0)H

obbs

et

al, 2

006

(31)

HP

V a

nd o

roph

aryn

x C

A38

3/18

163.

01 (2

.11

to 4

.30)

4.31

(2.0

7 to

8.9

5)2.

50 (1

.60

to 3

.80)

Hob

bs e

t al

, 200

6 (3

1)H

PV

and

ton

sil C

A21

7/16

315

.1 (6

.78

to 3

3.4)

15.1

(6.7

8 to

33.

4)10

.2 (2

.80

to 3

7.0)

Tayl

or e

t al

, 200

5 (3

2)H

PV

and

PrC

A22

42/2

622

1.37

(1.1

1 to

1.6

9)1.

52 (1

.12

to 2

.06)

1.07

(0.7

5 to

1.5

4)M

ande

lbla

tt e

t al

, 199

9 (6

1)H

PV

and

cer

vica

l CA

589/

3068

8.07

(6.4

9 to

10.

0)8.

08 (6

.04

to 1

0.8)

8.10

(5.5

0 to

11.

2)D

onat

o et

al,

1998

(30)

HB

V (H

CV-

) and

live

r C

A28

81/6

318

17.9

(15.

7 to

20.

5)21

.9 (1

4.9

to 3

2.3)

37.8

(26.

2 to

54.

5)D

onat

o et

al,

1998

(30)

HC

V (H

BV-

) and

live

r C

A20

79/5

615

16.8

(14.

1 to

20.

0)20

.3 (1

2.2

to 3

3.7)

2.40

(1.7

0 to

4.8

0)D

onat

o et

al,

1998

(30)

HC

V+

HB

V a

nd li

ver

CA

567/

1870

65.0

(35.

0 to

121

)61

.2 (2

7.0

to 1

39)

40.1

(12.

6 to

128

)Zh

ang

and

Beg

g, 1

994

(62)

T. v

agin

alis

and

cer

vica

l CA

642/

6512

21.

88 (1

.29

to 2

.74)

1.88

(1.2

9 to

2.7

4)2.

10 (1

.30

to 3

.40)

Vegl

ia e

t al

, 200

8 (2

0)D

NA

add

ucts

and

CA

(cur

rent

sm

oker

s)50

9/40

73.

88 (3

.31

to 4

.54)

3.76

(1.7

5 to

8.0

5)7.

46 (5

.89

to 9

.43)

* B

rCA

, bre

ast

canc

er; C

A, c

ance

r; C

RC

, col

orec

tal c

ance

r; E

1, e

stro

ne; E

2, e

stra

diol

; H. p

ylor

i, H

elic

obac

ter

pylo

ri; H

BV,

hep

atiti

s B

viru

s; H

CV,

hep

atiti

s C

viru

s; H

PV,

hum

an p

apill

omav

irus;

IGF,

insu

lin-li

ke g

row

th

fact

or; I

GFB

P, in

sulin

-like

gro

wth

fact

or b

indi

ng p

rote

in; P

rCA

, pro

stat

e ca

ncer

; SH

BG

, sex

hor

mon

e bi

ndin

g gl

obul

in; T

. vag

inal

is, T

richo

mon

as v

agin

alis

.

† O

dds

ratio

and

95%

con

fiden

ce in

terv

al o

f th

e la

rges

t st

udy

(sm

alle

st s

tand

ard

erro

r) in

eac

h m

eta-

anal

ysis

.


Dow

nloaded from



(H. pylori and gastric and laryngeal cancer, and hepatitis B virus and liver cancer), one on a dietary exposure (docosahexanoic acid and breast cancer), and one on an inflammatory biomarker (C-reactive protein [CRP] and colorectal cancer).

Test of Excess Statistical SignificanceFifteen, 10, or 16 meta-analyses had evidence of a statistically sig-nificant excess of positive studies when the plausible effect was assumed to be equal to the fixed effects summary, the random effects summary, or the result of the largest study, respectively. Table 2 shows the results of the excess statistical significance test for the 20 meta-analyses with statistically significant excess of posi-tive studies under any of these three assumptions for the plausible effect size. Eight of them pertained to the IGF/insulin system, four pertained to diet, four pertained to infectious agents, two pertained to inflammation, one pertained to sex hormones, and one pertained to an environmental association.

Table 3 shows aggregate data from all the meta-analyses and according to category of biomarkers. There was no evidence that the overall observed number of positive studies was greater than the expected according to the different assumptions about the plausible effect size. However, there were different patterns across the different categories of biomarkers. An excess of statistically sig-nificant studies was observed in meta-analyses that investigated the association between IGF/insulin concentrations and cancer risk (P ≤ .04 regardless of assumptions) and in studies of inflammatory biomarkers and risk (P ≤ .02 regardless of assumptions). However, the excess of statistically significant studies in the field of IGF/insu-lin and cancer risk was primarily driven by meta-analyses of the IGF system concentrations and risk of prostate cancer (Table 2). There was no evidence for excess statistical significance in studies of other types of biomarkers.

Table 4 shows the observed and expected number of positive studies in different subgroups. Overall, the excess in positive results was driven by meta-analyses with large estimates of heterogeneity. The results were not different by whether the meta-analyses had statistically significant summary effects or not. There were only 10 meta-analyses with evidence of small-study effects and 14 articles with consortia-based analyses, thus limiting inferences regarding these factors.

Biomarkers With Strong Evidence of AssociationOf the 98 meta-analyses, only 30 had nominally statistically sig-nificant summary associations per random effects calculations and had neither evidence of small-study effects nor evidence for excess statistical significance (Supplementary Table 1, available online). Of those, only 12 (12%) had also compiled evidence on more than 1000 case patients: four pertained to infectious agents (cagA strains of H. pylori and gastric cancer, human papilloma virus [HPV] and oral cancer, HPV and prostate cancer, and hepatitis C virus and liver cancer), four pertained to IGF/insulin system concentrations (IGF-1 and colorectal cancer, IGF-1 and postmenopausal breast cancer, IGFBP-3 and postmenopausal breast cancer, and glucose and colorectal cancer), and four pertained to other biomarkers (sex hormone binding globulin and prostate cancer, vitamin D and breast cancer, a-linolenic acid and prostate cancer, and vitamin B12 and prostate cancer). Across these 12 associations, the effect

sizes were small or moderate (odds ratio = 0.86 to 1.54) for eight of them; larger effect sizes were seen only for three associations of infectious agents (cagA strains of H. pylori and gastric cancer, HPV and oral cancer, and hepatitis C virus and liver cancer), and for the association of vitamin D and breast cancer.

DiscussionThis empirical evaluation of the literature on associations between nongenetic biomarkers and risk of cancer examined 98 meta-anal-yses with 847 studies. We showed that only a minority of these biomarker associations have statistically significant results and no suggestion of bias, as can be inferred by small-study effects and excess statistical significance testing. Most of these associations pertain to infectious agents such as H. pylori, hepatitis virus, and human papilloma virus; the associations of these infectious agents with the risk of specific malignancies are very strong and uncon-testable. Conversely, we found a statistically significant excess of positive studies among articles that investigated the association between biomarkers in the IGF/insulin or inflammation systems and cancer risk. This suggests the presence of potentially major selective reporting biases in these fields. Finally, biomarkers of dietary factors, environmental factors, or sex hormones usually had statistically nonsignificant associations with cancer risk in the majority of the meta-analyses where they were involved.

The excess of studies with positive results was driven predomi-nantly by meta-analyses that had large between-study heteroge-neity. This has been seen also in empirical evaluations of other research disciplines [eg, brain volume studies (16)]. Heterogeneity may often be a manifestation of bias in some studies of a meta-analysis, and this may be difficult to dissect and differentiate from genuine differences across studies (65). Some cancer fields had a clustering of several heterogeneous meta-analyses with evidence for excess statistical significance. Specific biomarkers and/or spe-cific cancer associations may be affected more than others by biases even within the same field.

For example, in the IGF/insulin system field, six of the eight meta-analyses with statistically significant excess of positive stud-ies came from the same article and pertained to prostate cancer risk (39). That article reported an elevated risk between IGF-1 concen-trations and total prostate cancer and an inverse association between IGFBP-3 and risk, both of which were statistically significant only in case–control studies, and null but highly heterogeneous results for IGF-2, IGFBP-1, and IGFBP-2 (39). Therefore, it seems that the evidence for an excess of statistically significant findings in the literature of IGF pathway biomarkers and prostate cancer risk is probably mostly due to biases in small primarily case–control inves-tigations. An empirical survey showed that when the results of the IGF-1 meta-analysis on prostate cancer were sent to the authors of the primary studies and to prominent methodologists, the former group claimed that there was stronger evidence for the presence of an association compared with the latter (66). Methodologists con-sistently argued that the data are consistent, with no effect at all or a small effect, whereas primary authors were more favorable for the presence of an association only when they had authored studies with statistically significant results themselves. This does not necessarily mean that these biomarkers are not associated with cancer. In fact,


Dow

nloaded from




Tab

le 2

. O

bse

rved

an

d e

xpec

ted

nu

mb

er o

f p

osi

tive

stu

die

s in

th

e 20

met

a-an

alys

es w

ith

a s

tati

stic

ally

sig

nif

ican

t ex

cess

of

po

siti

ve s

tud

ies

un

der

any

ass

um

pti

on

fo

r th

e p

lau

sib

le

effe

ct s

ize*

Met

a-an

alys

isC

om

par

iso

nN

o. o

f st

ud

ies

Ob

serv

ed

po

siti

ve

Exp

ecte

d

po

siti

ve

(fix

ed)†

P (f

ixed

)‡E

xpec

ted

po

siti

ve

(ran

do

m)§

P (r

and

om

)‡

Exp

ecte

d

po

siti

ve

(lar

gest

)‖P

(larg

est)

‡

Mor

ris e

t al

, 200

6 (3

6)IG

FBP-

3 an

d C

RC

72

0.4

.04

0.4

.05

1.5

.64

Row

land

s et

al,

2009

(39)

IGF-

1 an

d Pr

CA

4217

6.3

5.6

× 1

0−5

7.3

3.5

× 1

0−4

2.4

<1

× 1

0−9

Row

land

s et

al,

2009

(39)

IGF-

2 an

d Pr

CA

104

1.4

.04

1.0

.01

1.7

.08

Row

land

s et

al,

2009

(39)

IGFB

P-3

and

PrC

A29

71.

67.

3 ×

10−

43.

7.0

97.

21.

00R

owla

nds

et a

l, 20

09 (3

9)IG

F-1/

BP-

3 an

d Pr

CA

113

0.9

.06

1.2

.12

1.0

.07

Row

land

s et

al,

2009

(39)

IGFB

P-1

and

PrC

A3

20.

2.0

10.

5.0

90.

3.0

2R

owla

nds

et a

l, 20

09 (3

9)IG

FBP-

2 an

d Pr

CA

52

0.3

.04

0.8

.17

0.3

.03

Pis

ani,

2008

(40)

C-p

eptid

e an

d B

rCA

113

2.1

.45

3.1

1.00

0.7

.03

Key

et a

l, 20

02 (4

2)E

2 an

d po

stm

enop

ausa

l BrC

A9

41.

5.0

51.

4.0

40.

8.0

1S

aada

tian-

Ela

hi e

t al

, 200

4 (4

9)O

leic

aci

d an

d B

rCA

93

1.5

.18

0.5

.01

5.6

.09

Saa

datia

n-E

lahi

et

al, 2

004

(49)

Lino

leic

aci

d an

d B

rCA

83

0.8

.04

1.1

.08

0.7

.02

Buc

k et

al,

2010

(50)

Ent

erol

acto

ne a

nd B

rCA

126

2.6

.03

4.3

.37

6.4

1.00

Col

lin e

t al

, 201

0 (5

3)Fo

late

and

PrC

A7

20.

5.0

71.

0.2

70.

4.0

5Ts

ilidi

s et

al,

2008

(55)

C-r

eact

ive

prot

ein

and

CR

C8

30.

7.0

20.

9.0

40.

8.0

3H

eikk

ila e

t al

, 200

9 (5

6)C

-rea

ctiv

e pr

otei

n an

d C

A14

41.

6.0

71.

9.1

10.

7.0

04Zh

ao e

t al

, 200

8 (5

8)H

. pyl

ori a

nd C

RC

145

4.2

.57

5.1

1.00

0.7

4.6

× 1

0−4

Isla

mi a

nd K

aman

gar,

2008

(59)

H. p

ylor

i and

ES

CC

93

0.6

.02

0.7

.02

0.7

.03

Zhuo

et

al, 2

009

(29)

H. p

ylor

i and

lung

CA

42

2.5

.64

3.4

.10

0.4

.04

Hob

bs e

t al

, 200

6 (3

1)H

PV

and

lary

nx C

A8

22.

61.

003.

7.3

00.

5.0

8Ve

glia

et

al, 2

008

(20)

DN

A a

dduc

ts a

nd C

A (n

ever

sm

oker

s)9

30.

6.0

21.

5.1

80.

9.0

5

* B

rCA

, bre

ast

canc

er; C

A, c

ance

r; C

RC

, col

orec

tal c

ance

r; E

2, e

stra

diol

; ES

CC

, eso

phag

eal s

quam

ous

cell

carc

inom

a; H

. pyl

ori,

Hel

icob

acte

r py

lori;

HP

V, h

uman

pap

illom

aviru

s; IG

F, in

sulin

-like

gro

wth

fact

or; I

GFB

P,

insu

lin-li

ke g

row

th fa

ctor

bin

ding

pro

tein

; PrC

A, p

rost

ate

canc

er.

† E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

sum

mar

y fix

ed e

ffect

s es

timat

e of

eac

h m

eta-

anal

ysis

as

the

plau

sibl

e ef

fect

siz

e.

‡ P

val

ue o

f th

e ex

cess

sta

tistic

al s

igni

fican

ce t

est.

All

stat

istic

al t

ests

wer

e tw

o-si

ded.

§ E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

sum

mar

y ra

ndom

effe

cts

estim

ate

of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.

‖ E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

effe

ct o

f th

e la

rges

t st

udy

of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.


Dow

nloaded from



Tab

le 3

. O

bse

rved

an

d e

xpec

ted

nu

mb

er o

f p

osi

tive

stu

die

s b

y ty

pe

of

bio

mar

ker*

Are

aN

o. o

f st

ud

ies

Ob

serv

ed

po

siti

ve

Exp

ecte

d

po

siti

ve

(fix

ed)†

P‡

(fix

ed)

Exp

ecte

d

po

siti

ve

(ran

do

m)§

P‡

(ran

do

m)

Exp

ecte

d

po

siti

ve

(lar

gest

)‖P

‡(la

rges

t)

Exp

ecte

d

po

siti

ve

(co

mp

osi

te)¶

P‡

(co

mp

osi

te)

All

847

269

274.

9N

P29

5.1

NP

288.

7N

P27

4.9

NP

IGF/

insu

lin s

yste

m23

063

38.9

6.6

× 1

0−5

47.2

.01

49.8

.04

38.9

6.6

× 1

0−5

Sex

hor

mon

es10

69

14.1

NP

14.7

NP

17.9

NP

14.1

NP

Die

t19

542

47.0

NP

51.6

NP

49.1

NP

47.0

NP

Infla

mm

atio

n26

72.

5.0

12.

9.0

21.

7.0

011.

7.0

01In

fect

ious

age

nts

223

134

159.

0N

P16

5.0

NP

144.

5N

P14

4.5

NP

Env

ironm

ent

6714

13.5

.88

13.7

.88

25.8

NP

13.5

.88

* IG

F, in

sulin

-like

gro

wth

fact

or; N

P, n

ot p

ertin

ent,

beca

use

the

estim

ated

is la

rger

than

the

obse

rved

, and

ther

e is

no

evid

ence

of e

xces

s st

atis

tical

sig

nific

ance

bas

ed o

n th

e as

sum

ptio

n m

ade

for t

he p

laus

ible

effe

ct s

ize.

† E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

sum

mar

y fix

ed e

ffect

s es

timat

e of

eac

h m

eta-

anal

ysis

as

the

plau

sibl

e ef

fect

siz

e.

‡ P

val

ue o

f th

e ex

cess

sta

tistic

al s

igni

fican

ce t

est.

All

stat

istic

al t

ests

wer

e tw

o-si

ded.

§ E

xpec

ted

num

ber

of s

t atis

tical

ly s

igni

fican

t st

udie

s us

ing

the

sum

mar

y ra

ndom

effe

cts

estim

ate

of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.

‖ E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

effe

ct o

f th

e la

rges

t st

udy

of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.

¶

Exp

ecte

d nu

mbe

r of s

tatis

tical

ly s

igni

fican

t stu

dies

usi

ng th

e m

ost c

onse

rvat

ive

of th

e th

ree

estim

ates

(fix

ed e

ffect

s su

mm

ary,

rand

om e

ffect

s su

mm

ary,

larg

est s

tudy

) of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.


Dow

nloaded from



Tab

le 4

. O

bse

rved

an

d e

xpec

ted

nu

mb

er o

f p

osi

tive

stu

die

s in

su

bg

rou

ps*

Are

aN

o. o

f st

ud

ies

Ob

serv

ed

po

siti

ve

Exp

ecte

d

po

siti

ve

(fix

ed)†

P (

fixe

d)‡

Exp

ecte

d

po

siti

ve

(ran

do

m)§

P (

ran

do

m)‡

Exp

ecte

d

po

siti

ve

(lar

gest

)‖P

(la

rges

t)‡

Exp

ecte

d

po

siti

ve

(co

mp

osi

te)¶

P (c

om

po

site

)‡

All

Con

sort

ia15

216

21.7

NP

23.0

NP

27.5

NP

21.7

NP

Non

cons

ortia

695

253

253.

3N

P27

2.1

NP

261.

2N

P25

3.3

NP

I2 ≤5

046

987

124.

7N

P13

1.2

NP

132.

6N

P12

4.7

NP

I2 >

5037

818

215

0.2

.001

164.

0 .0

615

6.1

.01

150.

2.0

01 S

mal

l-stu

dy e

ffect

s99

3930

.8.0

838

.71.

0031

.2.1

030

.8.0

8 N

o sm

all-s

tudy

effe

cts

736

226

237.

9N

P24

9.9

NP

251.

3N

P23

7.9

NP

Met

a-an

alys

is s

tatis

tical

ly s

igni

fican

t#44

820

422

0.3

NP

233.

2N

P20

6.6

NP

206.

6N

P M

eta-

anal

ysis

not

sta

tistic

ally

sig

nific

ant#

399

6554

.7.1

462

.0.6

882

.1N

P54

.7.1

4

* N

P, n

ot p

ertin

ent,

bec

ause

the

est

imat

ed is

larg

er t

han

the

obse

rved

, and

the

re is

no

evid

ence

of

exce

ss s

tatis

tical

sig

nific

ance

bas

ed o

n th

e as

sum

ptio

n m

ade

for

the

plau

sibl

e ef

fect

siz

e.

† E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

sum

mar

y fix

ed e

ffect

s es

timat

e of

eac

h m

eta-

anal

ysis

as

the

plau

sibl

e ef

fect

siz

e.

‡ P

val

ue o

f th

e ex

cess

sta

tistic

al s

igni

fican

ce t

est.

All

stat

istic

al t

ests

wer

e tw

o-si

ded.

§ E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

sum

mar

y ra

ndom

effe

cts

estim

ate

of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.

‖E

xpec

ted

num

ber

of s

tatis

tical

ly s

igni

fican

t st

udie

s us

ing

the

effe

ct o

f th

e la

rges

t st

udy

of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.

¶

Exp

ecte

d nu

mbe

r of s

tatis

tical

ly s

igni

fican

t stu

dies

usi

ng th

e m

ost c

onse

rvat

ive

of th

e th

ree

estim

ates

(fix

ed e

ffect

s su

mm

ary,

rand

om e

ffect

s su

mm

ary,

larg

est s

tudy

) of e

ach

met

a-an

alys

is a

s th

e pl

ausi

ble

effe

ct s

ize.

# A

ccor

ding

to

rand

om e

ffect

s ca

lcul

atio

ns.


Dow

nloaded from



we found several IGF system associations that had statistically sig-nificant results with colorectal and breast cancer without any hints of bias. However, the effect sizes in these associations were consistently of small magnitude; similar small magnitude effects may also exist for prostate cancer, and they may be slightly stronger for advanced vs indolent disease as some (39), but not all reports (67), have shown.

We also observed an excess of statistically significant studies in the literature of inflammatory biomarkers, and these excesses pertained specifically to meta-analyses of CRP (55, 56). CRP has been proposed as a biomarker for a vast array of diseases and out-comes beyond cancer. The overall evidence for an association with cancer risk seems weak, and evidence for other previously con-sidered major associations, such as cardiovascular events, has also weakened over time (68). Eight studies investigated the association between circulating concentrations of CRP and risk of colorectal cancer; three observed nominally statistically significant findings, and the meta-analysis showed a weak but statistically significantly elevated risk (55). However, data from the Women’s Health Study (69), a randomized clinical trial of low-dose aspirin and vitamin E, showed null results, with, if anything, a trend in the opposite direc-tion. There is evidence suggesting that chronic low-grade colonic inflammation may be responsible for the development of colorectal neoplasia (70–72), but it is unclear whether circulating CRP corre-lates well with colonic inflammation. Further evidence suggesting a lack of a causal link between CRP and colorectal neoplasia comes from genetic association studies and Mendelian randomization analyses that have generally reported null results (73–75). It is thus probable that the statistically significant cancer studies in this field are subject to selective reporting or other biases (eg, varying degree of laboratory error in measuring certain biomarkers).

We did not observe any evidence of excess of statistically significant studies in the fields of sex hormones, dietary biomarkers, and environmental factors. Overall, the large majority of meta-analyses in these fields provide no evidence for statistically significant associations. In most of the remaining meta-analyses with nominally statistically significant summary results, there are no large-enough studies yet, and the total evidence is based on less than 1000 cancer cases. Thus even these few positive results may be spurious. Several large prospective cohort studies and randomized trials have generally found no or little evidence to support previously held nutritional associations (76). Consortia-based analyses of sex hormones also generally find mostly null associations with cancer (41). Finally, there was only one statistically significant meta-analysis for environmental factors, which reported a strong association between DNA adducts and cancer risk in current smokers (20), but this evidence is still limited, and no large-scale study has been published to date.

In contrast with the above, there are several very strong associ-ations in the literature of infectious agents and cancer risk, such as the association between cagA strains of H. pylori and gastric cancer, HPV and oral cancer, or hepatitis C virus and liver cancer. These associations are uncontestable; they have a clear causal hypoth-esis, and their effect sizes are typically much larger than the more uncertain effects described for other types of biomarkers. However, even in the field of infectious agents, we found some specific can-cer associations that had evidence of excess statistical significance. These were the meta-analyses suggesting increased risk of colo-rectal cancer and lung cancer and decreased risk of esophageal

squamous-cell carcinoma with H. pylori and increased risk of laryn-geal cancer with HPV. The summary effects for these more spuri-ous associations were not as strong as those of the associations of H. pylori with gastric cancer or HPV with oral cancer. It is possible that this situation may reflect a type of bias, where once a strong risk factor has been documented for one type of cancer, a collateral literature of lesser credibility is built for other types of cancer.

Several limitations and caveats should be considered in the interpretation of our findings. First, both asymmetry and excess statistical significance tests offer hints of bias, not definitive proof thereof. The frequency of meta-analyses with small-study asymme-try effects was not high (9%), and this rate is commensurate with chance. Our finding of an excess of statistically significant find-ings in studies of the IGF/insulin and inflammation systems may be challenged by the fact that in none of these meta-analyses was the summary random effect dramatically larger than that of the largest study in the meta-analysis. However, when the effects are mostly small, dramatic differences are unlikely to be seen even in biased evidence. Not surprisingly, we did not observe many situa-tions where the differences in the effect sizes in the largest sizes vs the meta-analyses exceeded 1.5-fold.

Second, most individual studies were relatively small, with approximately 100 cancer cases and 200 matched control subjects, and the median number of included studies in each meta-analysis was only seven. Therefore, the interpretation of the excess sta-tistical significance test for the results of a single meta-analysis, especially one with few studies, should be very cautious, because a negative test does not exclude the potential for bias (15). The excess statistical significance test is better suited to provide an overall impression about the average level of reporting bias affecting larger fields with several meta-analyses. However, different meta-analyses in the field may not be equally affected by bias to the same extent.

Third, the results of studies included in a meta-analysis may have already been standardized to some extent (eg, cleaned or made to follow consistent definitions and adjustments) compared with the results presented in each study’s original paper. However, such standardization efforts are likely to reduce, if anything, inconsist-ency and selective reporting bias. Selective reporting may be more prominent in the primary study reports.

Finally, the exact estimation of excess statistical significance is influenced by the choice of plausible effect size and/or the miscal-culation of power. We performed sensitivity analyses using differ-ent plausible effect sizes, which yielded similar findings. If anything, it is likely that our estimates of the plausible effect size are often larger than the true effect size because bias tends to inflate the esti-mates of the summary effect. Effect inflation may affect even the results of the largest studies because often these studies were not necessarily very large and/or may have had inherent biases them-selves. Thus, our estimates of the extent of excess statistical signifi-cance are conservative and the problem may be more severe. We also calculated power estimates under different assumptions and using different software programs but again observed very similar estimates.

Acknowledging these caveats, our evaluation maps the status of the evidence on 98 associations between nongenetic biomarkers and cancer risk. There is substantial diversity in the strength of the evidence across different fields, ranging from highly credible


Dow

nloaded from



uncontestable associations for several infectious agents to poten-tially spurious associations for several other popular biomarkers. Meta-analyses may help understand this diversity, but they are also subject to biases. It is possible that evidence eventually cor-rects itself, but data from more studies are needed to better support this statement. A substantial number of cancer biomarkers may be genuine but have small effect sizes, as has been documented recently for well-validated genetic biomarkers (77), and this means that great care needs to be taken to separate genuine small effects from the noise of bias. There are several ways to improve this evidence in the future. First, many of the biases in the scientific literature may be substantially lessened if studies were more com-pletely and transparently reported according to published guide-lines, such as the Strengthening the Reporting of Observational Studies in Epidemiology statement and its extension for Molecular Epidemiology (78–80). Second, statistical significance testing should not be used as a criterion for publication of biomarker stud-ies. Third, large prospective and multicenter studies and collabo-rative consortia should be encouraged for biomarker associations that currently have limited evidence. The use of standardized defi-nitions and protocols for exposures, outcomes, and statistical analy-ses may diminish the threat of biases and improve the reliability of this important literature.

References 1. Ioannidis JP, Panagiotou OA. Comparison of effect sizes associated with

biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA. 2011;305(21):2200–2210.

2. Bossuyt PM. The thin line between hope and hype in biomarker research. JAMA. 2011;305(21):2229–2230.

3. Rifai N, Altman DG, Bossuyt PM. Reporting bias in diagnostic and prog-nostic studies: time for action. Clin Chem. 2008;54(7):1101–1103.

4. Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognos-tic research: application and impact of prognostic models in clinical prac-tice. BMJ. 2009;338:b606.

5. Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer. 2007;43(17):2559–2579.

6. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124.

7. Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3(8):e3081.

8. Ioannidis JP. Why most discovered true associations are inflated. Epidemiology. 2008;19(5):640–648.

9. Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005;330(7494):753.

10. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291 (20):2457–2465.

11. Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171(7):735–740.

12. Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.

13. Ioannidis JP, Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ. 2007;176(8):1091–1096.

14. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the mis-leading funnel plot. BMJ. 2006;333(7568):597–600.

15. Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials. 2007;4(3):245–253.

16. Ioannidis JP. Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry. 2011;68(8):773–780.

17. Kavvoura FK, McQueen MB, Khoury MJ, Tanzi RE, Bertram L, Ioannidis JP. Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer’s disease. Am J Epidemiol. 2008;168(8):855–865.

18. Spitz MR, Bondy ML. The evolving discipline of molecular epidemiology of cancer. Carcinogenesis. 2010;31(1):127–134.

19. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–188.

20. Veglia F, Loft S, Matullo G, et al. DNA adducts and cancer risk in pro-spective studies: a pooled analysis and a meta-analysis. Carcinogenesis. 2008;29(5):932–936.

21. Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med. 2000;19(22):3127–3131.

22. Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10:101–129.

23. Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–1558.

24. Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses. BMJ. 2007;335(7626):914–916.

25. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–634.

26. Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443–3457.

27. Wang C, Yuan Y, Hunt RH. The association between Helicobacter pylori infection and early gastric cancer: a meta-analysis. Am J Gastroenterol. 2007;102(8):1789–1798.

28. Zhuo XL, Wang Y, Zhuo WL, Zhang XY. Possible association of Helicobacter pylori infection with laryngeal cancer risk: an evidence-based meta-analysis. Arch Med Res. 2008;39(6):625–628.

29. Zhuo WL, Zhu B, Xiang ZL, Zhuo XL, Cai L, Chen ZT. Assessment of the relationship between Helicobacter pylori and lung cancer: a meta-analysis. Arch Med Res. 2009;40(5):406–410.

30. Donato F, Boffetta P, Puoti M. A meta-analysis of epidemiological studies on the combined effect of hepatitis B and C virus infections in causing hepatocellular carcinoma. Int J Cancer. 1998;75(3):347–354.

31. Hobbs CG, Sterne JA, Bailey M, Heyderman RS, Birchall MA, Thomas SJ. Human papillomavirus and head and neck cancer: a systematic review and meta-analysis. Clin Otolaryngol. 2006;31(4):259–266.

32. Taylor ML, Mainous AG 3rd, Wells BJ. Prostate cancer and sexually trans-mitted diseases: a meta-analysis. Fam Med. 2005;37(7):506–512.

33. Lubin JH, Gail MH. On power and sample size for studying features of the relative odds of disease. Am J Epidemiol. 1990;131(3):552–566.

34. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsist-ency in meta-analyses. BMJ. 2003;327(7414):557–560.

35. Rinaldi S, Cleveland R, Norat T, et al. Serum levels of IGF-I, IGFBP-3 and colorectal cancer risk: results from the EPIC cohort, plus a meta-analysis of prospective studies. Int J Cancer. 2010;126(7):1702–1715.

36. Morris JK, George LM, Wu T, Wald NJ. Insulin-like growth fac-tors and cancer: no role in screening. Evidence from the BUPA study and meta-analysis of prospective epidemiological studies. Br J Cancer. 2006;95(1):112–117.

37. Key TJ, Appleby PN, Reeves GK, Roddam AW. Insulin-like growth fac-tor 1 (IGF1), IGF binding protein 3 (IGFBP3), and breast cancer risk: pooled individual data analysis of 17 prospective studies. Lancet Oncol. 2010;11(6):530–542.

38. Chen B, Liu S, Xu W, Wang X, Zhao W, Wu J. IGF-I and IGFBP-3 and the risk of lung cancer: a meta-analysis based on nested case-control studies. J Exp Clin Cancer Res. 2009;28(1):89.

39. Rowlands MA, Gunnell D, Harris R, Vatten LJ, Holly JM, Martin RM. Circulating insulin-like growth factor peptides and prostate cancer risk: a systematic review and meta-analysis. Int J Cancer. 2009;124(10):2416–2429.

40. Pisani P. Hyper-insulinaemia and cancer, meta-analyses of epidemiological studies. Arch Physiol Biochem. 2008;114(1):63–70.


Dow

nloaded from



41. Roddam AW, Allen NE, Appleby P, Key TJ. Endogenous sex hormones and prostate cancer: a collaborative analysis of 18 prospective studies. J Natl Cancer Inst. 2008;100(3):170–183.

42. Key T, Appleby P, Barnes I, Reeves G. Endogenous sex hormones and breast cancer in postmenopausal women: reanalysis of nine prospective studies. J Natl Cancer Inst. 2002;94(8):606–616.

43. Barba M, Yang L, Schunemann HJ, et al. Urinary estrogen metabolites and prostate cancer: a case-control study and meta-analysis. J Exp Clin Cancer Res. 2009;28(1):135.

44. Yin L, Grandi N, Raum E, Haug U, Arndt V, Brenner H. Meta-analysis: longitudinal studies of serum vitamin D and colorectal cancer risk. Aliment Pharmacol Ther. 2009;30(2):113–125.

45. Yin L, Raum E, Haug U, Arndt V, Brenner H. Meta-analysis of longitu-dinal studies: serum vitamin D and prostate cancer risk. Cancer Epidemiol. 2009;33(6):435–445.

46. Chen P, Hu P, Xie D, Qin Y, Wang F, Wang H. Meta-analysis of vitamin D, calcium and the prevention of breast cancer. Breast Cancer Res Treat. 2010;121(2):469–477.

47. Gallicchio L, Boyd K, Matanoski G, et al. Carotenoids and the risk of developing lung cancer: a systematic review. Am J Clin Nutr. 2008;88(2): 372–383.

48. Zhuo H, Smith AH, Steinmaus C. Selenium and lung cancer: a quantitative analysis of heterogeneity in the current epidemiological literature. Cancer Epidemiol Biomarkers Prev. 2004;13(5):771–778.

49. Saadatian-Elahi M, Norat T, Goudable J, Riboli E. Biomarkers of dietary fatty acid intake and the risk of breast cancer: a meta-analysis. Int J Cancer. 2004;111(4):584–591.

50. Buck K, Zaineddin AK, Vrieling A, Linseisen J, Chang-Claude J. Meta-analyses of lignans and enterolignans in relation to breast cancer risk. Am J Clin Nutr. 2010;92(1):141–153.

51. Simon JA, Chen YH, Bent S. The relation of alpha-linolenic acid to the risk of prostate cancer: a systematic review and meta-analysis. Am J Clin Nutr. 2009;89(5):1558S–1564S.

52. Larsson SC, Giovannucci E, Wolk A. Folate and risk of breast cancer: a meta-analysis. J Natl Cancer Inst. 2007;99(1):64–76.

53. Collin SM, Metcalfe C, Refsum H, et al. Circulating folate, vitamin B12, homocysteine, vitamin B12 transport proteins, and risk of prostate cancer: a case-control study, systematic review, and meta-analysis. Cancer Epidemiol Biomarkers Prev. 2010;19(6):1632–1642.

54. Larsson SC, Orsini N, Wolk A. Vitamin B6 and risk of colorectal cancer: a meta-analysis of prospective studies. JAMA. 2010;303(11):1077–1083.

55. Tsilidis KK, Branchini C, Guallar E, Helzlsouer KJ, Erlinger TP, Platz EA. C-reactive protein and colorectal cancer risk: a systematic review of pro-spective studies. Int J Cancer. 2008;123(5):1133–1140.

56. Heikkila K, Harris R, Lowe G, et al. Associations of circulating C-reactive protein and interleukin-6 with cancer risk: findings from two prospective cohorts and a meta-analysis. Cancer Causes Control. 2009;20(1):15–26.

57. Huang JQ, Zheng GF, Sumanac K, Irvine EJ, Hunt RH. Meta-analysis of the relationship between cagA seropositivity and gastric cancer. Gastroenterology. 2003;125(6):1636–1644.

58. Zhao YS, Wang F, Chang D, Han B, You DY. Meta-analysis of different test indicators: Helicobacter pylori infection and the risk of colorectal cancer. Int J Colorectal Dis. 2008;23(9):875–882.

59. Islami F, Kamangar F. Helicobacter pylori and esophageal cancer risk: a meta-analysis. Cancer Prev Res (Phila). 2008;1(5):329–338.

60. Gutierrez J, Jimenez A, de Dios Luna J, Soto MJ, Sorlozano A. Meta-analysis of studies analyzing the relationship between bladder cancer and infection by human papillomavirus. J Urol. 2006;176(6, pt 1):2474–2481.

61. Mandelblatt JS, Kanetsky P, Eggert L, Gold K. Is HIV infection a cofac-tor for cervical squamous cell neoplasia? Cancer Epidemiol Biomarkers Prev. 1999;8(1):97–106.

62. Zhang ZF, Begg CB. Is Trichomonas vaginalis a cause of cervical neo-plasia? Results from a combined analysis of 24 studies. Int J Epidemiol. 1994;23(4):682–690.

63. Khanjani N, Hoving JL, Forbes AB, Sim MR. Systematic review and meta-analysis of cyclodiene insecticides and breast cancer. J Environ Sci Health C Environ Carcinog Ecotoxicol Rev. 2007;25(1):23–52.

64. Lopez-Cervantes M, Torres-Sanchez L, Tobias A, Lopez-Carrillo L. Dichlorodiphenyldichloroethane burden and breast cancer risk: a meta-analysis of the epidemiologic evidence. Environ Health Perspect. 2004;112(2):207–214.

65. Ioannidis JP. Interpretation of tests of heterogeneity and bias in meta-anal-ysis. J Eval Clin Pract. 2008;14(5):951–957.

66. Panagiotou OA, Ioannidis JP. Primary study authors and methodologists differ in their interpretations of heterogeneous meta-analysis results. J Clin Epidemiol. 2012; 65(7):740–747.

67. Roddam AW, Allen NE, Appleby P, et al. Insulin-like growth factors, their binding proteins, and prostate cancer risk: analysis of individual patient data from 12 prospective studies. Ann Intern Med. 2008;149(7):461–471.

68. Ioannidis JP, Tzoulaki I. Minimal and null predictive effects for the most popular blood biomarkers of cardiovascular disease. Circ Res. 2012;110(5):658–662.

69. Zhang SM, Buring JE, Lee IM, Cook NR, Ridker PM. C-reactive protein levels are not associated with increased risk for colorectal cancer in women. Ann Intern Med. 2005;142(6):425–432.

70. Bertagnolli MM, Eagle CJ, Zauber AG, et al. Celecoxib for the prevention of sporadic colorectal adenomas. N Engl J Med. 2006;355(9):873–884.

71. Corpet DE, Pierre F. Point: from animal models to prevention of colon cancer. Systematic review of chemoprevention in min mice and choice of the model system. Cancer Epidemiol Biomarkers Prev. 2003;12(5):391–400.

72. Itzkowitz SH, Yio X. Inflammation and cancer IV. Colorectal cancer in inflammatory bowel disease: the role of inflammation. Am J Physiol Gastrointest Liver Physiol. 2004;287(1):G7–17.

73. Heikkila K, Silander K, Salomaa V, et al. C-reactive protein-associated genetic variants and cancer risk: findings from FINRISK 1992, FINRISK 1997 and Health 2000 studies. Eur J Cancer. 2011;47(3):404–412.

74. Poole EM, Bigler J, Whitton J, Sibert JG, Potter JD, Ulrich CM. C-reactive protein genotypes and haplotypes, polymorphisms in NSAID-metabolizing enzymes, and risk of colorectal polyps. Pharmacogenet Genomics. 2009;19(2):113–120.

75. Shinohara RT, Frangakis CE, Platz EA, Tsilidis K. Designs combining instrumental variables with case–control: estimating principal strata causal effects. Int J Biostat. 2012;8(1):1–21.

76. Boyle P, Boffetta P, Autier P. Diet, nutrition and cancer: public, media and scientific confusion. Ann Oncol.2008;19(10):1665–1667.

77. Ioannidis JP, Castaldi P, Evangelou E. A compendium of genome-wide associations for cancer: critical synopsis and reappraisal. J Natl Cancer Inst. 2010;102(12):846–858.

78. Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR: reporting guidelines for health research. Lancet. 2008;371(9619):1149–1150.

79. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296.

80. Gallo V, Egger M, McCormack V, et al. STrengthening the Reporting of OBservational studies in Epidemiology–Molecular Epidemiology (STROBE-ME): an extension of the STROBE Statement. PLoS Med. 2011;8(10):e1001117.

FundingThis work was supported by the Seventh Framework Programme of the European Union (PIEF-GA-2010–276017 to KKT) and by an unrestricted gift from Sue and Bob O’Donnell to the Stanford Prevention Research Center.

NotesThe study sponsors had no role in the design of the study; the collection, analy-sis, and interpretation of the data; the writing of the manuscript; and the decision to submit the systematic review for publication.

Affiliations of authors: Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece (KKT, SIP, EE, JPAI); Stanford Prevention Research Center, Department of Medicine, and Department of Health Research and Policy, Stanford University School of Medicine, and Department of Statistics, Stanford University, Stanford, CA (JPAI).


Dow

nloaded from


evaluation of excess Statistical Significance in Meta-analyses of 98 Biomarker Associations with Cancer Risk

Documents