Quarter Century on, Where are we? Tom Stanley, Hendrix College T.D. Stanley, Hendrix College MAER-Net September 12, 2014 (MRA) is at once a framework in which to organize and interpret exact and inexact replications, to review more objectively the literature and explain its disparities, and to engage in the self- analysis of investigating the socioeconomic phenomenon of social scientific research itself– Stanley & Jarrell (1989, p. 168). Stanley, T.D. and S.B. Jarrell (1989) Meta- regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 3: 161-70
28
Embed
A Quarter Century on, Where are we? Tom Stanley, Hendrix College T.D. Stanley, Hendrix College MAER-Net September 12, 2014 (MRA) is at once a framework.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A Quarter Century on, Where are we?
Tom Stanley, Hendrix College
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
(MRA) is at once a framework in which to organize and interpret exact and inexact replications, to review more objectively the literature and explain its disparities, and to engage in the self-analysis of investigating the socioeconomic phenomenon of social scientific research itself– Stanley & Jarrell (1989, p. 168).
Stanley, T.D. and S.B. Jarrell (1989) Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 3: 161-70
Exponential @
18%/year
Where are we going?
Has MRA’s promise been realized?
More objective reviews of economic research Explanation of disparate research findings¿ Investigation of the socio-economics of
economics research?
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Investigation of the socio-economics of economics research?
• Female economists find less wage discrimination against women than do male researchers (S&J, 1998; J&S, 2004, Weichselbaumer & Winter-Ebmer, 2005)
• Publication bias is the result of professional incentives and the pressure to publish.
• Researcher ideology affects reported results (Doucouliagos and Paldam, 2006).
• The Research Cycle: Reported findings generally confirm a novel hypothesis; later, the incentive for rejection increases (S, J & D, 2008).
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Has MRA’s promise been realized?
More objective reviews of economic research Explanation of disparate research findings Investigation of the socio-economics of
economics research, . . . but much more to do.¿ Framework to organize and interpret exact and
inexact replications?
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Framework to organize and interpret exact and inexact replications?
How about S&J’s (1989) exact MRA model? Does anyone still use it?
bi = b + SbkZik +ei (3)
Where: “bi is the reported estimate of b from the ith study. . . Zik the meta-independent variable which measures relevant characteristics of an empirical study and explains its systematic variation” (p. 164)
• Zik might include:
1. Dummy variables for omitted variables.
2. Specification variables
3. Sample Size
4. Author characteristics
5. Data characteristics
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Due to obvious Heteroskedasticity, S&J 89 recommended the WLS version of (3):
ti= bi/Sbi = b (1/Sbi
) + Sbk (Zik/Sbi) +ei /Sbi
(4)
• t-value, ti, is the dependent variable and
precision (1/Sbi) is an independent variable.
• WLS is neither fixed- nor random-effects, in practical application, WLS is better than both (Stanley and Doucouliagos, 2013a&b, Deakin SWP).
• We never wanted to use fixed- or random-effects, in spite of citing Hedges and Olkin (1985).
• Had we included the intercept, we would have fully anticipated current practice {FAT-PET-MRA}.
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Neither Fixed
• Fixed-effect MRA: same as our WLS regression, but divides SEs by square root of MSE (H&O, 85)• causes SEs & CIs to be too small & too narrow. • assumes that policy makers wish to make
inferences to a population that is identical, in every respect, to past research. Like that happens!
• So why divide by root MSE???• WLS already correct the SEs for both excess
heterogeneity and heteroskedasticity• Fixed-effect MRA is never relevant in economics.
T.D.
Sta
nley
, He
n d
rix C
olle
ge
MAE
R-N
et S
epte
mbe
r 12,
201
4
Nor Random
• Random-effects MRA adds a second error term, ni , to the conventional meta-regression model,
bi = b + SbkZik +ni + ei (1)
Where ni is assumed to be normal and independent of the sampling errors, ei, and the moderators, Zik.
• Problems:• In economics, excess heterogeneity is systematic!
• Typically, ni will be the result of omitting relevant variables; thus, it introduces bias.
• In the 1980s, we saw no reason to use FE or RE• Now, we know that we should never use FE or RE!
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Problems and Issues—2014
Multiple Meta-Regression (MRA)• Should we divide meta-regression SEs by the
square root of MSE? No! (Hedges and Olkin, 1985)
• Does Random-Effects MRA become biased with publication bias? YES! (Stanley and Doucouliagos, 2013b)
• MRAs should never be estimated by OLS, because there is much variation among the reported SEs of bi or effecti
• Enter WLS: =(MtW-1M)-1MtW-1b (2)
Where:
=
β
2
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
The Gauss-Markov Theorem{Proportional Heterogeneity Invariance}
• As long as W in (2) is known up to some unknown proportion, s2, WLS { } is the Best {Minimum Variance} Linear Unbiased Est.
• Invariance to proportional excess heterogeneity is a robustness property of the Gauss-Markov Theorem and WLS.
• It is not an assumption.
β
T.D.
Sta
nley
, Hen
drix
Col
lege
SRS
M
July
2, 2
014
Traditional, Unrestricted WLS
replaces with:
= (5)
and is estimated by the WLS residuals, automatically
2S
2
22
21
00
00
00
LSE
SE
SE
2S
T.D.
Sta
nley
, Hen
drix
Col
lege
SRS
M
July
2, 2
014
Simulation Design• Generate Yj and estimate b from:
Yj = 100 + b X1j +a2 X2j +a3i X3j + ej (6)
• Half the studies omit the relevant variable X2i
• a3i ~N(0, ) adds excess random heterogeneity by always omitting relevant variable X3i
• When b = 1, r between Y and X1 is .27
• n= {62, 125, 250, 500, 1000} in primary regressions
• X1j , X2j, X3j are generated randomly, but X2j & X3j are forced to be correlated with X1j .
• Fixed- or random-effects model is always true.
2h
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Simulation Design—Cont. • Experiment 1: 10,000 WLS, RE and FE-MRAs are
calculated with one moderator variable, Mi = {0,1}, reflecting whether the original study omitted X2i, or not, (Tables 1 & 2)
bi = b0 + b1 Mi + ui (7)
Where: bi is the ith primary study’s estimate of b
• Experiment 2: Experiment 1 plus 50% of the studies select statistically significant results, and either MRA (7), above, or a multiple FAT-PET-MRA is used. (Tables 3 & 4)
Simulation Results I:“A house divided by itself cannot stand”—A. Lincoln
• FE-MRA: unacceptable SEs in most actual applications. Thus, Do Not Divide by root MSE!• When there is no heterogeneity and FE-MRA is
true, WLS-MRA is equivalent to FE-MRA.
• RE-MRA: When RE-MRA’s model is true, WLS-MRA provides acceptable and comparable coverage, and its bias and MSE are a bit better.• Irony: WLS-MRA dominates RE-MRA in those exact
circumstances for which RE-MRA is designed—large additive, excess random heterogeneity
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Table 3: Bias and MSE with 50% Publication Bias MRA
Simulation Results IIWhen there is publication bias:
• WLS-MRA dominates RE-MRA. • WLS-MRA always has less bias and, on
average, substantially lower MSE
• Irony: WLS dominates RE in those exact circumstances for which RE-MRA is designed.
• When RE is better, it is not much better, and those cases cannot be identified, in practice.
• Thus, there is No Reason to use Random-Effects Meta-Regression. . . . . . Ever!
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
Why WLS works so well when there is high heterogeneity?
For very large heterogeneity, the random heterogeneity term, , will dominate sampling error, , and its variation, making the overall variance of the estimate, , roughly proportional to .
2j
2j2j
2j
0
1
2
3
4
5
6
7
8
9
SE
-squ
ared
0 50 100 150 200 250
Heterogeneity Variance
Unlike S & J, S & J’s MRA Model
Still works after all these Years
Multiple MRA
• Should we divide MRA SEs by √MSE? Never!
• Is RE-MRA biased with publication bias? Yes!• WLS-MRA dominates RE-MRA with or
without Correcting for Publication Bias. • WLS also dominates REE and FEE
weighted averages when combining Cohen’s d from RCTs.
T.D.
Sta
nley
, Hen
drix
Col
lege
M
AER-
Net
Sep
tem
ber 1
2, 2
014
The Task Ahead
• We need more realistic simulation studies• Alternative modeling strategies {general-