Aims and Objectives - Discovering Statistics

11

Meta-AnalysisMetaMeta--AnalysisAnalysis

Dr. Andy FieldDr. Andy FieldDr. Andy Field

7-Mar-03 Andy Field Slide 2

Aims and ObjectivesAims and ObjectivesAims and Objectives

When and Why?

Theory• Fixed Effects Models

• Random Effects Models

Can meta-analysis be trusted?• Practical problems

• Choice of methods

When and Why?When and Why?

TheoryTheory

•• Fixed Effects ModelsFixed Effects Models

•• Random Effects ModelsRandom Effects Models

Can metaCan meta--analysis be trusted?analysis be trusted?

•• Practical problemsPractical problems

•• Choice of methodsChoice of methods


When and Why?When and Why?When and Why?To assimilate information from independent studies

Discursive Literature Reviews• Selective inclusion of studies

• Subjective weighting of studies

• Biased interpretation of evidence

Meta-Analysis• Objective assimilation of studies?

To assimilate information from To assimilate information from independent studiesindependent studies

Discursive Literature ReviewsDiscursive Literature Reviews•• Selective inclusion of studiesSelective inclusion of studies

•• Subjective weighting of studiesSubjective weighting of studies

•• Biased interpretation of evidenceBiased interpretation of evidence

MetaMeta--AnalysisAnalysis•• Objective assimilation of studies?Objective assimilation of studies?


Use of MetaUse of Meta--Analysis (1981Analysis (1981--2001)2001)

Year

81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2000 2001

Num

ber o

f Pub

licat

ions

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

In Social Science JournalsIn Science Journals


What do scientists do?What do scientists do?What do scientists do?Interested in knowing a ‘true’ effect size in our population of interest.Use samples to estimate this true effectExpress the effect:• Glass’s ∆• Cohen’s d• Hedge’s g• Pearson’s r• Odds ratios/risk rates

Discover that others have also tried to find this effectAssimilate these different studies.

Interested in knowing a ‘true’ effect size in our Interested in knowing a ‘true’ effect size in our population of interest.population of interest.Use samples to estimate this true effectUse samples to estimate this true effectExpress the effect:Express the effect:•• Glass’s Glass’s ∆∆•• Cohen’s Cohen’s dd•• Hedge’s Hedge’s gg•• Pearson’s Pearson’s rr•• Odds ratios/risk ratesOdds ratios/risk rates

Discover that others have also tried to find this effectDiscover that others have also tried to find this effectAssimilate these different studies.Assimilate these different studies.


Meta-Analysis: BasicsMetaMeta--Analysis: BasicsAnalysis: BasicsCalculate effect sizes for each study• Convert to a common metric

Weight each effect size by the sampling precision• Sample size (or some function of it)

Calculate the mean effect size across studies

Express this mean as a Z-score• Calculate significance/confidence intervals

Moderator Analysis?

Calculate effect sizes for each studyCalculate effect sizes for each study•• Convert to a common metricConvert to a common metric

Weight each effect size by the sampling Weight each effect size by the sampling precisionprecision•• Sample size (or some function of it)Sample size (or some function of it)

Calculate the mean effect size across studiesCalculate the mean effect size across studies

Express this mean as a ZExpress this mean as a Z--scorescore•• Calculate significance/confidence intervalsCalculate significance/confidence intervals

Moderator Analysis?Moderator Analysis?

22


r = 0.29 r = 0.30 r = 0.42

Fixed Effects ModelsFixed Effects Modelsρ = 0.30

r = 0.22


ρ = 0.25 ρ = 0.30

r = 0.29 r = 0.30r = 0.42

Random Effects Random Effects ModelsModels Superpopulation

ρ = 0.30

r = 0.22

ρ = 0.27 ρ = 0.36


Hedges et al.’s Fixed MethodHedges Hedges et alet al.’s Fixed Method.’s Fixed Method

( )i

i

i rr

erz −+= 11

21 log

∑

∑=

=

=k

ii

k

iiri

w

zw

rz1

1

iviw 1=

31−=

iniv

3−=∴ ii nw

( )∑

∑=

=

=

−

−

k

ii

k

iiri

n

zn

rz1

1

3

)3(

Mean Effect Size:Mean Effect Size:Mean Effect Size:( )( ) 1

12

2

+−=

iz

iz

ee

ir


Significance of Mean Effect Size:Significance of Mean Effect Size:Significance of Mean Effect Size:

( )∑

==

k

iiw

rzSE1

1 ( )( )∑

==

−k

iin

rzSE1

3

1

( )r

rzSE

zZ =


Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:

( )( )∑=

−−=k

irri zznQ

i1

23


Hedges et al.’s Random MethodHedges Hedges et alet al.’s Random Method.’s Random Method

∑

∑=

=

=k

ii

k

iiri

w

zw

rz1

*

1

*2

1*

τ+=

iviw

31−=

iniv

12*

31

−

+

−=∴ τ

ii nw

∑

∑=

=

−=

−

+

−

+

−k

i i

k

iir

i

n

zn

rz1

12

1

12

31

31

τ

τ

Mean Effect Size:Mean Effect Size:Mean Effect Size:

33


Estimating Between-Study VarianceEstimating BetweenEstimating Between--Study VarianceStudy Variance

( )ckQ 12 −−=τ

( )

∑

∑−=

=

=∑=

k

ii

k

ii

w

wk

iiwc

1

1

2

1

( )( )

( )∑

∑−−=

=

=

−

−

=∑ k

ii

k

ii

n

nk

iinc

1

1

2

3

3

1

3


Hunter-Schmidt MethodHunterHunter--Schmidt MethodSchmidt Method

Mean Effect Size:Mean Effect Size:Mean Effect Size:

∑

∑=

=

=k

ii

k

iii

n

rn

r1

1


Significance of Mean Effect Size:Significance of Mean Effect Size:Significance of Mean Effect Size:

( )

∑

∑=

=

=

−

k

ii

k

iii

n

rrn

rSD1

1

2

kSD

rrSE =

rSErZ =


Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:

( )( )( )∑

=−

−−=k

ir

rrn ii

1 1

1222

2

χ


Can meta-analysis be trusted?Can metaCan meta--analysis be trusted?analysis be trusted?

Publication bias: the ‘file drawer’ problem• Significant findings more often published

• 97% of published psychology articles report significant results (Sterling, 1959)

• Meta-analysis will overestimate the mean effect size

Artefacts• Is all research equally good?

Publication bias: the ‘file drawer’ problemPublication bias: the ‘file drawer’ problem

•• Significant findings more often publishedSignificant findings more often published

•• 97% of published psychology articles report 97% of published psychology articles report significant results (Sterling, 1959)significant results (Sterling, 1959)

•• MetaMeta--analysis will overestimate the mean effect analysis will overestimate the mean effect sizesize

ArtefactsArtefacts

•• Is all research equally good?Is all research equally good?


Which method: Fixed or Random?

Which method: Fixed or Which method: Fixed or Random?Random?

Real data are likely to have heterogeneous effect sizes:

However, Fixed methods are more commonly applied:• Hunter & Schmidt (2001)

– No random effects meta-analyses in Psychological Bulletin

• Field (2003): Bias when fixed effects methods are used on heterogeneous effect sizes.

Real data are likely to have heterogeneous Real data are likely to have heterogeneous effect sizes:effect sizes:

However, Fixed methods are more commonly However, Fixed methods are more commonly applied:applied:

•• Hunter & Schmidt (2001)Hunter & Schmidt (2001)

–– No random effects metaNo random effects meta--analyses in Psychological Bulletinanalyses in Psychological Bulletin

•• Field (2003): Bias when fixed effects methods are Field (2003): Bias when fixed effects methods are used on heterogeneous effect sizes.used on heterogeneous effect sizes.

44


Field (2003)Field (2003)Field (2003)

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

202020202020404040404040808080808080

16016016016016016051015202530

51015202530

51015202530

51015202530

Typ

e I

Err

or

Rate

Sample Size

Studies in Meta-Analysis

Fixed-Effects Models Applied toHeterogeneous Effect Sizes

Hedges & Colleagues

Rosenthal-Rubin


Which Method: Hedges or Hunter-Schmidt?

Which Method: Hedges or Which Method: Hedges or HunterHunter--Schmidt? Schmidt?

Transforming r-to-z

• Corrects for bias in sampling distribution of r

• Transforming r is a good thing (Silver & Dunlap (1987)

• Transforming r makes no difference (Strube, 1988; Hunter et al., 1996)

Transforming Transforming rr--toto--zz

•• Corrects for bias in sampling distribution Corrects for bias in sampling distribution of of rr

•• Transforming Transforming rr is a good thing (Silver & is a good thing (Silver & Dunlap (1987)Dunlap (1987)

•• Transforming Transforming rr makes no difference makes no difference ((StrubeStrube, 1988; Hunter , 1988; Hunter et alet al., 1996)., 1996)


Estimates of Standard ErrorEstimates of Standard ErrorEstimates of Standard Error

Hedges & Vevea (1998)• Hunter-Schmidt use suboptimal weights

• Hence, when between study variance exists H-S will underestimate SE and overestimate Z (Type I errors).

Field (2001):• Hedges & Vevea’s method uses truncated

estimates of between-study variance.

• Hence biased when the number of studies in the meta-analysis are small.

Hedges & Hedges & VeveaVevea (1998)(1998)•• HunterHunter--Schmidt use suboptimal weightsSchmidt use suboptimal weights

•• Hence, when between study variance exists HHence, when between study variance exists H--S S will underestimate SE and overestimate Z (Type I will underestimate SE and overestimate Z (Type I errors).errors).

Field (2001):Field (2001):•• Hedges & Hedges & Vevea’sVevea’s method uses truncated method uses truncated

estimates of betweenestimates of between--study variance.study variance.

•• Hence biased when the number of studies in the Hence biased when the number of studies in the metameta--analysis are small.analysis are small.


Comparing MethodsComparing MethodsComparing MethodsJohnson et al. (1995)• Compared Hedges (fixed) method with H-S by

manipulating a single data set.

• H-S provided conservative estimates of significance

Field (2001):• Johnson et al.’s study is pants:

– Results could be due to the data set used

– They got the H-S equations wrong

– Used Hedges d method and converted to r

– Only looked at fixed-effect data

Johnson et al. (1995)Johnson et al. (1995)

•• Compared Hedges (fixed) method with HCompared Hedges (fixed) method with H--S by S by manipulating a single data set.manipulating a single data set.

•• HH--S provided conservative estimates of significanceS provided conservative estimates of significance

Field (2001):Field (2001):

•• Johnson et al.’s study is pants:Johnson et al.’s study is pants:

–– Results could be due to the data set usedResults could be due to the data set used

–– They got the HThey got the H--S equations wrongS equations wrong

–– Used Hedges Used Hedges dd method and converted to method and converted to rr

–– Only looked at fixedOnly looked at fixed--effect dataeffect data


Field (2001)Field (2001)Field (2001)Monte Carlo Study• Effect sizes sampled from a population with a fixed

or variable mean effect size.

• For each: 100,000 repetitions

Variables:• Population effect size (0, .1, .3, .5, .8)

• Mean sample size of study (20, 40, 80, 160)

• Number of studies in meta-analysis (5, 10, 15, 20, 25, 30)

• Variability in population = 0.16

Monte Carlo StudyMonte Carlo Study•• Effect sizes sampled from a population with a fixed Effect sizes sampled from a population with a fixed

or variable mean effect size.or variable mean effect size.

•• For each: 100,000 repetitionsFor each: 100,000 repetitions

Variables:Variables:•• Population effect size (0, .1, .3, .5, .8)Population effect size (0, .1, .3, .5, .8)

•• Mean sample size of study (20, 40, 80, 160)Mean sample size of study (20, 40, 80, 160)

•• Number of studies in metaNumber of studies in meta--analysis (5, 10, 15, 20, analysis (5, 10, 15, 20, 25, 30)25, 30)

•• Variability in population = 0.16Variability in population = 0.16


Hedges Effect Size Estimates (Fixed)Hedges Effect Size Estimates (Fixed)ρ = 0.0

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.1

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.3

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.5

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.8

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

-0.010-0.005ρ+0.005+0.010

55


HunterHunter--Schmidt Effect Size Estimates (Fixed)Schmidt Effect Size Estimates (Fixed)

ρ = 0.0

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.1

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160S

tud

ies

in M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.3

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.5

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.8

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

-0.010-0.005ρ+0.005+0.010


Hedges Effect Size Estimates (Random)Hedges Effect Size Estimates (Random)ρ = 0.0

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.1

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.3

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.5

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.8

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s in

Meta

-An

aly

sis

5

10

15

20

25

30

-0.20-0.15-0.10-0.05ρ+0.05+0.10+0.15


HunterHunter--Schmidt Effect Size Estimates Schmidt Effect Size Estimates (Random)(Random)

ρ = 0.0

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.3

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.5

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.8

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

ρ = 0.0

Mean Sample Size

202020202020 404040404040 808080808080 160160160160160160

Stu

die

s i

n M

eta

-An

aly

sis

5

10

15

20

25

30

-0.20-0.15-0.10-0.05ρ+0.05+0.10+0.15


0.050

0.075

0.100

0.125

0.150

0.175

202020202020404040404040808080808080

16016016016016016051015202530

51015202530

51015202530

51015202530

Typ

e I

Err

or

Rate

Sample Size


Homogeneous Case

0.050

0.075

0.100

0.125

0.150

0.175

202020202020404040404040808080808080

160160160160160160

1015202530

1015202530

1015202530

1015202530

Typ

e I

Err

or

Rate

Sample Size


Heterogeneous Case

Hedges & Colleagues

Hunter-Schmidt

Effect Sizes: Type I ErrorEffect Sizes: Type I Error


Field (Submitted)Field (Submitted)Field (Submitted)Monte Carlo Study• Effect sizes sampled from a population with a

variable mean effect size.

• For each: 100,000 repetitions

Variables:• Population effect size (0, .1, .3, .5, .8)

• Mean sample size of study (20, 40, 80, 160)

• Number of studies in meta-analysis (5, 10, 20, 40, 80, 160)

• Variability in population effect sizes = 0.04, 0.08, 0.16, 0.32, 0.64

Monte Carlo StudyMonte Carlo Study•• Effect sizes sampled from a population with a Effect sizes sampled from a population with a

variable mean effect size.variable mean effect size.

•• For each: 100,000 repetitionsFor each: 100,000 repetitions

Variables:Variables:•• Population effect size (0, .1, .3, .5, .8)Population effect size (0, .1, .3, .5, .8)

•• Mean sample size of study (20, 40, 80, 160)Mean sample size of study (20, 40, 80, 160)

•• Number of studies in metaNumber of studies in meta--analysis (5, 10, 20, 40, analysis (5, 10, 20, 40, 80, 160)80, 160)

•• Variability in population effect sizes = 0.04, 0.08, Variability in population effect sizes = 0.04, 0.08, 0.16, 0.32, 0.640.16, 0.32, 0.64


0.050

0.075

0.100

0.125

0.150

0.175

202020202020404040404040808080808080

160160160160160160510204080

160

510204080160

510204080160

510204080160

Type I

Err

or

Rate

Sample Size


Population variance = .04

0.050

0.075

0.100

0.125

0.150

0.175

202020202020404040404040808080808080

160160160160160160510204080

160

510204080160

510204080160

510204080160

Typ

e I

Err

or

Rate

Sample Size



Hedges & Colleagues

Hunter-Schmidt

Field (Submitted)Field (Submitted)Field (Submitted)

66


0.050

0.075

0.100

0.125

0.150

0.175

202020202020404040404040808080808080

160160160160160160510204080

160

510204080160

510204080160

510204080160

Typ

e I

Err

or

Rate

Sample Size



0.050

0.075

0.100

0.125

0.150

0.175

202020202020404040404040808080808080

160160160160160160510204080

160

510204080160

510204080160

510204080160

Type I

Err

or

Rate

Sample Size



Hedges & Colleagues

Hunter-Schmidt




0.050

0.075

0.100

0.125

0.150

0.175

202020202020404040404040808080808080

160160160160160160510204080

160

510204080160

510204080160

510204080160

Typ

e I

Err

or

Rate

Sample Size




ConclusionsConclusionsConclusionsCan meta-analysis be trusted?• Not if fixed effects methods are used when population

effect sizes vary (significance tests of most of the ones you read are likely to be type I errors)

• Hunter-Schmidt generally gives better estimates

• Hedges generally controls the Type I error better than H-S, but both methods OK when combining 80 or more studies

But…• Are we ever interested in significance tests anyway?

• Are moderator variables more interesting?

Can metaCan meta--analysis be trusted?analysis be trusted?•• Not if fixed effects methods are used when population Not if fixed effects methods are used when population

effect sizes vary (significance tests of most of the ones effect sizes vary (significance tests of most of the ones you read are likely to be type I errors) you read are likely to be type I errors)

•• HunterHunter--Schmidt generally gives better estimatesSchmidt generally gives better estimates

•• Hedges generally controls the Type I error better than Hedges generally controls the Type I error better than HH--S, but both methods OK when combining 80 or more S, but both methods OK when combining 80 or more studiesstudies

But…But…•• Are we ever interested in significance tests anyway?Are we ever interested in significance tests anyway?

•• Are moderator variables more interesting?Are moderator variables more interesting?

Aims and Objectives - Discovering Statistics

Documents