Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014.

Psychology 290

Special Topics Study Course: Advanced Meta-analysis

January 27, 2014

Overview

• Discussion of the Hedges / Hanushek exchange.

• Investigation of vote counting as an inferential method.

Education finance exchange

• Larry Hedges was then a professor of educational statistics at the University of Chicago.

• Eric Hanushek was an economist at the University of Rochester.

• Now at Northwestern and Stanford, respectively.

Background of the paper

• Hanushek at the time was extremely active as an expert witness in educational equity lawsuits.

• Paper grew out of a student project in Hedges’ meta-analysis class.

• Discussion.

Vote counting as inference

• To understand vote counting as an inferential method, we need to understand the probability that an individual study will reject the null hypothesis.

• Statisticians have a name for that idea.• Power.

What does power depend on?

• Lots of things: –characteristics of population –choices about how to do inference–characteristics of the sample.

Characteristics of the population

• How strong is the effect? • How much unmodeled variability exists?

Choices about how to do inference

• Alpha level.• One- vs. two-tailed tests.

Characteristics of the sample

• Sample size.

Back to vote counting

• To understand vote counting, we need to understand power.

• We’ve just seen that power is a complex function of lots of factors.

• If we want to understand something that is too complex, what can we do?

• Simplify.

Simplifying

• All of those issues that were characteristics of the population can be simplified by, for the moment, confining our interests to the fixed-effects context.

• In that case, we are assuming that all of the studies are samples from the same population.

Simplifying

• All of the issues that were characteristics of how we do inference are under our control.

• For example, we can simply say that a vote is positive if the null hypothesis is rejected using a two-tailed test with an alpha level of .05.

Simplifying

• The remaining issue that effects power is the sample size of the individual study.

• Obviously, in the real world, different studies will have different N.

• Simplify by assuming that all studies have the same N.

Simplifying

• With these simplifying assumptions, we can treat power (i.e., the probability of a positive vote) as a constant.

Distribution of votes

• We can now think of our studies as a series of independent attempts to vote YES.

• For each attempt, the probability of a YES vote is the same (power).

• We are interested in the total number of YES votes.

• This should sound vaguely familiar.

Distribution of votes

• Suppose that instead of studies and YES votes, I were talking about coin tosses and HEADS outcomes.

• We would be looking at a series of independent coin tosses with a constant probability of success, and would be interested in the probability of a particular number of successes.

Distribution of votes

• With the simplifying assumptions we have made, the number of YES votes follows a binomial distribution.

What should we assume for power?

• Given that Hanushek is arguing that there is no effect, we should be justified in considering it to be “small.”

• Empirical studies of power.

Cohen 1962

• The statistical power of abnormal-social psychological research, Journal of Abnormal and Social Psychology, 65, 145-153.

• Finding: 100% of studies of small effects in that field had power of < .50.

Brewer 1973

• A note on the power of statistical tests in the Journal of Educational Measurement, Journal of Educational Measurement, 10, 71-73.

• Very much the same finding as Cohen.

Understanding vote counting

• What happens with vote counting as the number of studies becomes large?

• (Another digression in R.)• Using the normal approximation to the

binomial distribution.