INCOME EFFECTS AND EDUCATIONAL PROGRESS: METHODOLOGICAL PUZZLES, STATISTICAL PROBLEMS AND SUBSTANTIVE FINDINGS Ian Plewis, CCSR, University of Manchester.

INCOME EFFECTS AND EDUCATIONAL PROGRESS:METHODOLOGICAL PUZZLES,STATISTICAL PROBLEMS AND SUBSTANTIVE FINDINGS

Ian Plewis, CCSR,University of Manchester.

CCSR seminar2 October 2007

BACKGROUND

• The substantive focus of the research reported here can be put rather plainly: do children start to do better at school and behave better at home – and does this improvement last into adulthood - if more money comes into the family, and do they fall back if family income drops?

• Income is the possible cause; educational, behavioural and later economic outcomes are the effects.

BACKGROUNDThe research was funded by the Department for Work and

Pensions (DWP) and the context for this study is provided by:

1. The short, medium and long-term government targets first to reduce and ultimately to eliminate child poverty by 2020.

2. Policies to increase lone parent employment and to reduce the number of children brought up in workless households.

3. Arguments about ‘the marginal pound’.

BACKGROUNDThe original intention had been to analyse data from the National Child Development Study (the 1958 cohort) and BCS70 (the 1970 cohort), both the main cohorts and the children of the cohorts.

However, DWP were persuaded that it was worth replacing the NCDS analyses by analyses of the National Pupil Database and it is those analyses, focusing on educational attainment, that form the main part of this seminar.

Methodological issues1. We would like to establish a causal relation between

income and an outcome but we only have observational data.

2. But we do have longitudinal data and this makes life a bit easier.

3. Our measures of family income are often rather rudimentary.

4. We are faced with a big ‘self-selection’ problem: family income will often rise (or fall) as a result of decisions made by family members that are likely to be related to, for example, a child’s educational progress.

Methodological issues5. Most families’ incomes are relatively stable so our

inferences are likely to be based on a rather small number of families who experience a change in their economic circumstances.

6. We can control for at least some potentially confounding variables at the individual and family level but substantial income changes could change the overall distribution of family incomes and we cannot analyse the effects of these distributional changes.

UNDERLYING MODEL

Fixed Variables

(Unmeasured)

Fixed Variables

(Measured)e.g. mother’s

education

Family IncomeTIME t

Family Income

TIME t+1

Child’s Educational Attainment

TIME t

Child’s Educational AttainmentTIME t + 1

NATIONAL PUPIL DATABASE (NPD)

One way of analysing the relation between changes in economic circumstances and pupils’ progress at school is to use data from the National Pupil Database (NPD) in conjunction with PLASC (Pupil Level Annual Schools Census).

The NPD and PLASC in England are linked datasets, which have been constructed annually by DfES (now DCFS) since 2002 and provide a census of pupils at state schools in England.

NATIONAL PUPIL DATABASE (NPD)

The NPD has the following advantages: 

• It contains both pupil level and some school level data.• It contains rich information on pupils’ Key Stage test

scores.• It is longitudinal, allowing us to control for prior

characteristics of pupils.

There are, however, some drawbacks. In particular, the NPD contains no measures of parents’ social class, educational level or income.

Free School Meals (FSM) receipt is, however, an important variable in the dataset that is directly related to economic disadvantage.

We focus on the period between Key Stage 1 or KS1(when pupils are age seven years) and Key Stage 2 (KS2) when they are eleven, an age that usually marks the end of their primary school career, and on the cohort that reached KS1 in 2002 and KS2 in 2006. The cohort consists of 595,407 pupils.The following groups were omitted:

a)     Two very small LEAs: City of London and the Scilly Isles (n = 71).

b)     Special and independent schools, pupil referral units etc. where the school experiences are very different (n = 22,448).

This left up to:

572,888 pupils (LEVEL 1);

in about 14,750 schools (LEVEL 2);

in 148 LEAs (LEVEL 3).

(The exact numbers of pupils and schools depended on therelatively small amount of missing data at the pupil level inany particular model.)

Our outcome variables are the pupils’ test scores at KS2 inEnglish, maths and science.

We control for attainments in reading, writing and maths atKS1 along with teacher assessments of their pupils’abilities in English, maths and science at that point.

We also include sex and ethnic group in our models.

For each pupil for each of the four years, we use, as an indicator of a family’s economic circumstances, information on whether or not a claim was made (at the time of the annual schools’ census) for free school meals (FSM). Although FSM is, conceptually, a simplistic indicator of what we really want to measure, it is nevertheless a powerful predictor of educational attainments.

Eligibility for free school meals is based on receipt of Income Support, Income Based Jobseekers Allowance or support under part 6 of the 1999 Immigration and Asylum Act. Pupils are identified as receiving free school meals only if they have actually claimed them, and their eligibility has been confirmed.

We generate two measures of economic circumstances from the FSM variables. The first is a score – varying from zero to four – of the number of years that a pupil claimed FSM. The distribution of the FSM score shows that over three quarters of pupils never claim FSM, 24 per cent claim it at least once and 11 per cent claim it for each of the four years. Pupils in this latter group might be assumed to be living in persistent poverty.

Model fitted (using MLwiN):

y: KS2 tests (English, Maths, Science) for pupil i in school j

in LEA k: mean zero, SD = 1.

xp: KS1 tests and teacher assessments.

x*: FSM score, range 0 – 4, mean = 0.70, SD = 1.4

x.*: FSM score, school mean

zq: sex, ethnic group (12 categories), sex*ethnic group,

FSM*ethnic group.

Q

q ijke

jku

kv

qijkzqd

P

p jkxkc

ijkxjb

pijkx

pjaa

ijky

11

*.

*0

RESULTSEstimates of most interest:

FSM: fixed effect for English = -0.047 (s.e. = 0.0015) Maths = -0.044 (s.e. = 0.0016) Science = -0.053 (s.e. = 0.0016)

White British pupils make between 0.18 and 0.21 SD unitsless progress if they experience persistent poverty (i.e.FSM = 4) compared to those pupils never claiming FSM(FSM = 0).

RESULTSHowever, the effect is smaller for all minority ethnic groups for English and Science; about half the size for Indian, Pakistani, Bangladeshi, and Black African pupils, about two thirds the size for the mixed and Black Caribbean groups.

The variation in the FSM effect between ethnic groups for Maths is less marked.

Also, the FSM effect varies from school to school: from zero in some schools to 0.5 SD units in others.

The difficulty with this model is that it does not include a measure of change in ‘income’, either by controlling for a prior measure or by taking a difference, and so alternative explanations related to self-selection cannot be ruled out.

Our second measure is generated from the claims for FSM made each year and explicitly incorporates a representation of change. There are 24 (i.e.16) combinations of FSM claiming behaviour. We group these combinations into seven categories to create a variable we label “FSM_dyn” (for dynamics):

(i)               never claiming (76 per cent)(ii)              possible improvement (3.8 per cent)(iii)             definite improvement (1.7 per cent)(iv)             possible decline (2.7 per cent)(v)              definite decline (1.1 per cent)(vi)             erratic claiming (3.2 per cent)(vii)            always claiming (11 per cent).

Our interest is in the contrast between the ‘definite’ improvers (those who claimed for the first two years but not for the last two) and the ‘definite’ decliners (those who did not claim for the first two years and claimed for the last two). Although pupils in both these groups have the same number of years of exposure to poverty (as represented by FSM), we would expect pupils in the first group to be living in families on an upward economic path and therefore to make more progress than those in the second, whose families are on a downward path.

3 4 5 6

School Year

Fa

mil

y I

nc

om

e

Definite Improvers FSM Threshold Definite Decliners

3 4 5 6School Year

Fa

mil

y I

nc

om

e

"0110" FSM Threshold "0101"

0 - Not Claiming Free School Meals 1 - Claiming Free School Meals

RESULTSENGLISH MATHS SCIENCE

FS_dyn: base never

Possible improvers

-0.094 -0.090 -0.13

Definite improvers

-0.080 -0.082 -0.12

Possible decliners

-0.097 -0.089 -0.12

Definite decliners

-0.11 -0.10 -0.12

Random -0.11 -0.099 -0.13

Always -0.14 -0.12 -0.17

Sample size 507679 509446 514034

We find that, as expected, all pupils with some exposure to poverty in terms of claiming FSM make less progress than the majority who never claim. However, the definite improvers do, as hypothesized, make slightly more progress than the definite decliners in English (0.03 SD units; p < 0.001 (Wald test)) and in maths (0.02 SD units, p < 0.05 (Wald test)) but there is no difference for science.

A difference of 0.03 SD units is equivalent to about two weeks progress.

There is no evidence for differences by ethnic group or by school.

CONCLUSIONS

1. Pupils make a little more educational progress between KS1 and KS2 if they experience apparently improving rather than apparently declining economic circumstances. This is the best evidence for a causal relation.

2. Pupils living in persistent poverty make substantially less progress than pupils living in families with few or no economic problems. This difference is moderated by ethnic group and varies by school – why?

METHODOLOGICAL PUZZLES

1. The status of FSM as a measure of economic circumstances: combining measures from four years is an improvement but issues of stigma might be important.

2. Are school effects endogeneous – do poorer pupils find themselves in less effective schools?

3. What is the causal lag?

STATISTICAL PROBLEMS

1. Establishing the distribution of FSM score in more and less effective schools when slopes for control variables from KS1 are random.

2. Are we misled by the size of NPD?

3. How should we handle changes of school between KS1 and KS2?