8/13/2019 PK PM Jan 2014 Power Handout
1/14
Power and Sample Size
Sameem Iqbal Siddiqui
Center for Economic Research Pakistan
8/13/2019 PK PM Jan 2014 Power Handout
2/14
Objectives
Introduce the concept
Figure out why we care
Factors affecting power: Understanding theIntuition
Possible ways to calculate power
8/13/2019 PK PM Jan 2014 Power Handout
3/14
Set up
In an experiment in Punjab, we randomly assignself-employed individuals not involved inagriculture to business training. We would like to
see whether this increases their income. From our baseline, we know that mean income for
self-employed individuals is about PKR 11,000.And, we expect the treatment to have an effect of
20% increase on income. Given this setup, what is the sample sizewe would
need to detect the 20% effect?
8/13/2019 PK PM Jan 2014 Power Handout
4/14
What are power calculations?
Power calculations will give you the sample sizerequired to be able to detect the treatment effect. In ourcase, 20%.
Definition of Power: Given a treatment, theprobability of being able to detect an effect.
Studies generally use 80% as a benchmark for power.That is, 80% of the time you will be able to detect the
treatment effect. So, in practice, generally, you will fix the power and see
what sample sizes are needed to achieve that power
8/13/2019 PK PM Jan 2014 Power Handout
5/14
Why do we care?
Being sure ex ante that we are able to detectexpected treatment effects
8/13/2019 PK PM Jan 2014 Power Handout
6/14
Design of Intervention
Main decisions:
1) Level of randomization: Individual or groupo Individual (IND): Assign treatment at the school levelo Group (G): Assign treatment at the village level
2) No of treatments
o Simple T/C oroAlter intensity of treatment oro Combine treatments
3) Compliance100% or lower.
Most important thing to think about is what effects you areinterested in capturing and how the design can help youcapture that while achieving the highest power possible.
8/13/2019 PK PM Jan 2014 Power Handout
7/14
Factors affecting power for IND
Increase in:
Sample size increases power
Size of the effect increases power
Standard deviation decreases power
Significance level decreases power
8/13/2019 PK PM Jan 2014 Power Handout
8/14
Sample Size Calculations for IND
Whats needed Where to get it
Mean of the outcome variable From pilot, baseline or other studies
Standard deviation of the outcome variable From pilot, baseline or other studies
Significance level Usually 5%
Effect size Expected, think about cost-benefitanalysis of program
Power Usually 80%
Level of randomization Depends on study design
Note 1: For group-design, you would need an intracluster coefficient, which tells you the proportion of varianceexplained by the group relative to the overall variance.Note 2: For compliance lower than 100%, we would need to adjust effect size.
8/13/2019 PK PM Jan 2014 Power Handout
9/14
Ways to Calculate Power
Stata commands: sampsi, sampclus
Stata simulations
Stata MDE formula dofile
Excel: MDE formula, set up parameters
Optimal Design software
8/13/2019 PK PM Jan 2014 Power Handout
10/14
Example: Sampsi
Help file: sampsi n1 n2, options
sampsi treatment mean control mean, p(0.8)sd(standard deviation)
This will give you the N needed in each group toachieve 80% power given mean and standarddeviation of outcome variable.
8/13/2019 PK PM Jan 2014 Power Handout
11/14
Sampsi output
Estimated sample size for two-sample comparison of means
Test Ho: m1 = m2, where m1 is the mean in population 1and m2 is the mean in population 2
Assumptions:
alpha = 0.0500 (two-sided)power = 0.8000m1 = 13700.5m2 = 11417.1
sd1 = 15956.8sd2 = 15956.8
n2/n1 = 1.00
Estimated required sample sizes:
n1 = 767n2 = 767
Total Sample: ~1500
8/13/2019 PK PM Jan 2014 Power Handout
12/14
Appendix
8/13/2019 PK PM Jan 2014 Power Handout
13/14
Basic Statistics (1)
What do we mean by detecting an effect?
We are claiming that the mean for the treated isdifferent from the mean of the control.
How do we do this?
A hypothesis test: Test for significance of someclaim
Null Hypothesis: H0: treatment mean =control mean
Alternative Hypothesis: H1: treatment mean !=control mean
8/13/2019 PK PM Jan 2014 Power Handout
14/14
Basic Statistics (2)
Type I Error: Wrongly rejecting the null. That is, the null is correct,but we reject it. Want to keep this as small as possible. P(Type I Error) = Significance level = 5% (traditionally)
Type II Error: Failing to reject a null when it is false
P(Type II Error) = beta, generally unknown Power: 1P(Type II Error) = 1beta
What is the probability that you would be able to reject a false null?I.e. the probability of making the right decision or probability ofnot making a type II error.
Traditionally, we fix this to 80%.
Decision
Reject H0 Do not reject H0
Truth H0 Type I Error Right decision
H1 Right Decision Type II Error