The EEF by numbers

Building Evidence in Education:Workshop for EEF evaluators

2nd June: York6th June: London

www.educationendowmentfoundation.org.uk

The EEF by numbers

83 evaluationsfunded to

date

3,000 schools

participating in projects

34 topics in

the Toolkit

16 independent evaluation

teams

600,000 pupils involved in EEF projects

14 members of EEF team

£220mestimated spend over lifetime of

the EEF

6,000 heads

presented to since launch

10 reports

published

Session 1: Design

RCT design, power calculations and randomisation  Ben Styles (NFER)

Maximising power using the NPDJohn Jerrim (Institute of Education)

RCT designPower calculations and randomisation

Ben Styles

Education Endowment FoundationJune 2014

RCT design

• The ideal trial• Methods of randomisation• Power calculations• Syntax exercise!

A statistician’s ideal trial

• Randomly select eligible pupils from NPD• No consent!• Simple randomisation of pupils to intervention

and control groups• No attrition• No data matching problems• No measurement error

BEFORE YOU START !

1. Trial registration: specification of primary and secondary outcomes in addition to sub-group analyses

2. Recruit participants and explain method to stakeholders

3. Select participants according to fixed eligibility criteria

4. Obtain consent

5. Baseline outcome measurement (or use existing administrative data)

6. Randomise eligible participants into groups (evaluator carries out randomisation)

7. Intervention runs in experimental group; control receives ‘business-as-usual’/an alternative activity

8. Administer follow-up measurement (evaluator)

9. Intention-to-treat analysis followed by reporting as per CONSORT guidelines

10. Control receives intervention (under what circumstances?)

Why we depart from the ideal

• Schools manage pupils!• Nature of the intervention• Contamination – how serious is the risk?

Restricted randomisation?• Use simple randomisation where you can• Timetable considerations in a pupil-randomised

trial → stratify by school• Important predictor variable with small and

important category → stratify by predictor• Fewer than 20 schools → minimise

http://minimpy.sourceforge.net/ • Multiple recruitment tranches → blocked• Pairing → BAD IDEA!

Restricted randomisationSimple randomisation Restricted randomisation

Restricted randomisation more complicated and can go wrong. Take strata into account in analysis: http://www.bmj.com/content/345/bmj.e5840

To remember!

If you have restricted your randomisation using a factor that is associated with the outcome (e.g. school) THEN

INCLUDE THE FACTOR AS A COVARIATE IN YOUR ANALYSIS

Chance imbalance at baseline

• As distinct from bias induced by measurement attrition

• Can be quite large in small trials e.g. on baseline measure

• Include covariate in final analysis

Sample size calculations

• School or pupil-randomised?• Intra-cluster correlation• Correlation between covariate and outcome• Expected effect size• p(type I error)=0.05; power=0.8• Attrition

Rule of thumb

Lehr, 1992

Pupil randomised• ICC = 0• Correlation between baseline and outcome:

http://educationendowmentfoundation.org.uk/uploads/pdf/Pre-testing_paper.pdf and your previous work

• Effect size: previous evidence; cost-effectiveness; EEF security ratings

• Attrition: EEF allow recruitment to be 15% above sample size after attrition

Cluster-randomised

• Same as for pupils aside from ICC• Proportion of total variance that is due to

between cluster variance• EEF pre-testing paper has some useful

guidance• Pre-test also reduces ICC e.g. from 0.2 to

0.15 for KS2 baseline, GCSE outcome

MDES

• Minimum detectable effect size• EEF require this on the basis of real

parameters for the security rating• (avoid retrospective power calculation)• How good were my estimates?

Sample size spreadsheet(fill in the highlighted boxes) Scenario 1Expected number of pupils per school being sampled 180

ROH (Intra-class correlation - percentage of variance in outcome being studied attributable to school attended) 0.15Deff (adjustment for nested design) 27.85Confidence level (of test we will use to assess effect) 95.0%Critical T-value 1.96   Correlation between before and after scores 0.70SD of residuals in scores (if scores have SD of 1) 0.71

Expected effect size (in terms of absolute outcome scores) 0.2Expected effect size (in terms of residual outcome scores) 0.28n(schools) in intervention 31n(schools) in control 31n(pupils) in intervention 5580n(pupils) in control 5580Expected SE of difference between groups (in SDs) 0.10Power 80.0%

0.00 0.05 0.10 0.15 0.20 0.25 0.300%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

n(intervention)=31; n(control)=31

Effect size

Pow

er

Running the randomisation

SYNTAX EXERCISE

• In pairs, explain what each of the steps does• How many schools were randomised in this

block?

Conclusions

• Always think of any RCT (any quantitative impact evaluation) as a departure from the ideal trial

• The design, power calculations, method of randomisation and analysis all interrelate and need to be consistent

Maximising power using the NPD

John Jerrim (Institute of Education)

StructureHow much power do EEF trials currently have?• PISA, power, star ratings and current EEF trials

Exercise• Work in groups to design an EEF trial• Goal = Maximise power at minimal cost

My answers• How might I try to maximise power?

Your answers! / Discussion

Power in context

Effect sizes, PISA rankings and EEF padlock ratings

How powerful are EEF trials thus far?

0 0.1 0.2 0.3 0.4 0.5

Detectable effect

EEF secondary school trialsAs of 01 / 05 / 2014

Detectable effect sizeMean = 0.276Median = 0.25

Between 4* and 5* by EEF guidelines….

Power and the PISA reading rankings

United Kingdom

France

Viet Nam

Switzerland

Netherlands

New Zealand

Liechtenstein

Canada

Chinese Taipei

Korea

Singapore

Shanghai-China

-0.10 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80UK’s current position

Effect size = 0.10

Effect size = 0.20 (EEF 5*)

Effect size = 0.30 (EEF 4*)MEDIAN EEF TRIAL = 0.25

Effect size = 0.40 (EEF 3*)

IMPLICATIONEffect sizes of 0.20 are

damn big

… particularly given pretty small doses we are

giving

Effect size = 0.50 (EEF 2*)

Do we currently have a power problem?

- Quite possibly!- So trying to get more power in future trials very important…..

Exercise

Task: In groups, discuss how you would design the following trial

Intervention = Teaching children how to play chessMaximum number of treatment schools = 20 secondary schoolsYear group = Year 7Level of randomisation = School levelTest = One-to-one non-verbal IQ assessment with trained educationalist (end of year 7)Control condition = ‘Business as usual’Study type = ‘Efficacy’ study (proof of concept)

Objective: Maximise power at minimum cost

How would you design this trial to meet these twin objectives?What could you do to increase power in this trialE.g. Would you use a baseline test? If so, what?

Exercise

My answers

The usual suspects…..…and less obvious options

The usual suspects…..

1. Use a regression model and include baseline covariates…..

- Adding controls explains variance. Boosts power

2. Use Key stage 2 test scores as “pre-test”….- Point of baseline covariates is to explain variance- KS 2 scores in maths likely to be reasonably correlated with outcome (non-

verbal IQ)- CHEAP! From NPD.

3. Stratify the sample prior to randomisation- Potentially reduces error variance. Thus boosts power.- Additional advantages. Balance of baseline characteristics.

4. Really engage with control schools- Make sure we minimise loss of sample through attrition

Less ‘obvious’ options….

Don’t test every child……..There are around 200 children per secondary school…..

…. One-to-one testing is expensive

…Testing more than 50 pupils buys you little additional power

RANDOMLY SAMPLE PUPILS WITHIN SCHOOLS!

Assumptions20 schoolsPre/post corr of 0.7580% powerRho = 0.15

0 20 40 60 80 100 120 140 160 180 2000.35

0.40

0.45

0.50

0.55

0.60

Cluster size

Det

ecta

ble

effe

ct

…..use an unequal sampling fraction

• We all know that ↑ clusters (k) means ↑ power

• This example: limited to only a small number of treatment schools (20)

• ….but control condition was non-intrusive and cheap

• So don’t just recruit 20 control schools as well – recruit more!

• Nothing about RCT’s mean we need equal k for treatment and control

• Power calculation becomes more complex (anybody know it!?)

Use more homogenous selection of schools….

0204060801000.00

0.05

0.10

0.15

0.20

0.25

0.30

Percentage of all UK schools in population

RHO

ALL UK SCHOOLS

LOW PERFORMING SCHOOLS ONLY

PISA 2009 data

All UK schools:

“Worst” 25% of schools only:

Why does rho decline??

0204060801000

20

40

60

80

100Within school variation

Percentage of all UK schools in population

sigma

The within school variation barely changes …..

…. While the between school variation declines substantially

Implications

• As example is an efficacy study why not restrict attention to low performing schools only?

- Boosts power!- Fits with EEF mandate (close performance gap)- Not worried about generalisability

• We implicitly do this anyway (e.g. by doing trials in just one or two LA’s)……

• …..but can we do it in a smarter way???

• Little appreciated trade-off between POWER and GENERALISBILITY

- Long-term implications for EEF- Trial representative of England population very hard to achieve

ConclusionsDo we have a “power problem”?

• Quite possibly• Median detectable effect size = 0.25 in EEF secondary school trials• If were to boost UK reading PISA scores by this amount, we would

move above Canada, Taiwan and Finland in the rankings…..

Ways to potentially increase power• Include baseline covariates (from NPD where possible)• Stratify the sample prior to randomisation• Engage with control schools!• Do you need to test every child? Practical alternatives?• Could you increase number of control schools without adding much

to cost (unequal randomisation fraction)• Could you restrict your focus to a narrower population? (e.g. low

performing schools only)?