Evaluation of Education Maintenance Allowance Pilots Sue Middleton - CRSP Carl Emmerson - IFS
Evaluation of Education Maintenance Allowance Pilots
Sue Middleton - CRSP
Carl Emmerson - IFS
The Policy Problem (1)
• Very large increases in participation in the later 80s and early 90s had levelled off by the mid-90s
020406080
per c
ent
16 year olds
17 year olds
18 year olds
The Policy Problem: Post-16 Participation and Socio-Economic Group
0
10
20
30
40
50
60
70
80
90
Parents' Occupation
per
cent
16 year olds in full-timeeducation 199816 year olds NEET 1998
Source: Youth Cohort Study, Statistical Bulletin 02/2000
The Policy Problem: Post-16 Participation and Gender
0
10
20
30
40
50
60
70
80
1989 1992 1994 1997 2000
per
cent Male 16 years
Female 16 years
Source: DfES Statistics 2001 http://www.dfes.gov.uk/statistics/DB/SFR/index.html
Reasons for Non-participation
• Little evidence about the reasons• Money is one factor among many
“...not surprisingly, money looms large in the accounts given by disaffected young people of their lives. They report one of the key barriers to further and higher education to be (lack of) money”. (Newburn, 1999)
The Policy Response
• Multiple policy responses• Education Maintenance Allowance
- to encourage participation, retention and achievement in post-16 education
- focusing on young people from low income families
Education Maintenance Allowance
• Household income < £13k
- weekly allowance up to £40 per week
• Household income £13k - £30k
- weekly allowance tapers to minimum £5 per week
• Retention and achievement bonuses
- available to ALL awarded EMA
• Receipt subject to compliance with Learning Agreement
• 4 Variants being piloted
EMA Variants Maximum Weekly Amount
Retention and Achievement Bonuses
Paid to:
Variant 1 £30 £50 Young person
Variant 2 £40 £50 Young person
Variant 3 £30 £50 Parent
Variant 4 £30 £80 (retention)
£140 (achievement)
Young Person
EMA Evaluations
• EMA Main
– 10 LEAs
• Leeds and London
– 5 LEAs
• EMA Transport
– 5 LEAs
• EMA Extensions
– 4 LEAs
Evaluation in the LEAs
ROUND TABLE DISCUSSIONS
• EMA Implementation
groups
DATA COLLECTION
• Labour market• Education profile• Take up of EMA• Socio-demographics
INTERVIEWS
• LEA administrators• Careers Service
representative• TEC representative• Employer organisations
AREA VISITS
Qualitative Interviews with Young People/Parents
• Year 1 – Participants/Non-participants
(young people and parents)
• Year 2 – Longitudinal interviews with young people and early leavers
WAVE 1 WAVE 2 WAVE 3 WAVE 4
Face to face
October 1999
Telephone
October 2000
Telephone
October 2001
Telephone
October 2002COHORT 1
COHORT 2
Quantitative Design
Face to face
October 2000
Telephone
October 2001
Telephone
October 2002
Telephone
October 2003
The data• Questionnaires have detailed information
on:- all components of family income
- household composition
- GCSE results
- mother’s and father’s education, occupation and work history
- early childhood circumstances
- current activities of young people
Matching approach
• Involves taking all EMA eligible young people in the pilot areas and matching them with a weighted sum of young people who look like them in control areas
• Difference in full-time education outcomes in pilot and control areas in this matched sample is the estimate of EMA effect
• Crucial assumption is that everything is observed that determines education participation
How is this done?• Don’t match on all X’s, but can instead match on
the propensity score (Rosenbaum and Rubin, 1983)
• Propensity score is simply the predicted probability of being in a pilot area given all the observables in the data
• Use kernel-based matching (Heckman, Ichimura & Todd, 1998)
• The matching is undertaken for each sub-group of interest
• Family background- household composition, housing status, ethnicity, early
childhood characteristics, older siblings’ education and parents’ age, education, work status and occupation
• Family income- current family income, whether on means-tested
benefits• Ability (GCSE results)• School variables• Indicators of ward level deprivation
Variables are matched on:
Analytic Strategy for EMA
Propensity Score Matching:
Measures the Impact of EMA
BUT
Requires Large Sample Sizes
Weighting Issues
Limited Disaggregation
Descriptive Analysis:
Provides Contextual Detail
Allows Disaggregation
Overcomes Weighting Issues
BUT
Cannot Measure Impact
The Impact of EMA on Participation
• EMA has increased participation by 5.9 percentage points
• EMA had a larger effect on young men than young women
0
1
2
3
4
5
6
7
8
Overall Urban Men Urban Women
Base: Eligible young people in Cohort 1 & 2
The Impact of EMA on Participation
• Draw is from both those who would have been in work and those who would have been NEET
Base: Eligible young people in Cohort 1 & 2
-4
-2
0
2
4
6
8
per
cen
t
FTE
Work/Training
NEET
EMA and Retention at Year 13
• Slightly larger impact on participation in year 13
• Suggests that retention has not fallen
Base: Eligible young people in Cohort 1 & 2
0
1
2
3
4
5
6
7
8
per
cen
t
Y12 effect Y13 effect
Participation and Retention by Variant
0
2
4
6
8
10
12
Participation Retention
per
cen
t
Variant 1 Variant 2 Variant 3 Variant 4
Conclusions (1)
• EMA effect around 6 percentage points
• Plays an important role in reducing gender differences in post-16 participation
• Important to control for local area effects- matching on ward level data important
Conclusions (2)
• More effective paying EMA to young person rather than parent
• Bigger retention bonuses have significantly larger effect on retention than other variants
• Increase in participation drawn from both work and other groups
What We Learn, And When: From PSM
Surveys Information on Best Impact Measure
Wave 1 Participation 2002 Report 2
Wave 2 Participation and Retention in Year 13
2003 Report 3
Wave 3 Retention in Year 14
Achievement
Higher Education Entry
2004 Report 4
What We Learn And When: From Descriptives (1)
Surveys Information on
Wave 1 Year 11 decision making
Awareness, Applications and Receipt
Year 12 Courses
Wave 2 Retention bonuses
Year 12 achievement
Destinations
What We Learn And When: From Descriptives (2)
Surveys Information on
Wave 3 (& 4) Achievement Bonuses
Year 13 Achievement
Higher Education:
- Courses and Institutions
- Financial Support
Destinations over time
What We Could Learn
Surveys Information on
All surveys Destinations over time including labour market and higher education entry (for sub-groups)
Part-time work, hours and earnings
Expenditure patterns and responsibility