Merging Local Research Data with PLASC: An application on the Appropriateness of Free School Meals as a Measure of Deprivation in Educational Research.

Merging Local Research Data with PLASC:

An application on the Appropriateness of Free School Meals as a Measure of Deprivation in Educational Research in Hampshire.

Daphne Kounali and Tony Robinson

Department of Mathematical Sciences

University of Bath

The HARPS studyThe HARPS project is an acronym for ‘Hampshire Research with Primary Schools’

Objective: Assess the impact of school composition upon student academic progress. The main aim of the study is to estimate and better understand compositional effects at the primary school level.

Definitions: Compositional effects are a subset of contextual effects. Compositional effects attempt to operationalise the influences of fellow pupils characteristics on individual pupil’s performance.Compositional variables included in this study will be; social economic status (SES), ethnicity, gender, prior achievement, special educational needs (SEN) and age.

A further remit of the study is to evaluate the probity of FSM as a proxy for the economic element of SES.

Study Design

The research design is both quantitative and qualitative. Like a set of Russian dolls the project design is of 3 nested parts:

1. A large scale analysis of over 300 primary schools 2. A study of a subsample of 46 schools in the

Basingstoke and Dean area.3. More detailed case studies of 12 schools.

The Basingstoke sub-sample

One purpose of the Basingstoke subsample is to test the sensitivity of the quantitative measures and methodologies, including FSM, used to identify compositional effects.

To this end we have collected and are analysing detailed family background information from the year 3 children in the 46 Basingstoke schools. These schools have been selected so as to compare the findings from these schools with those of the overall Hampshire cohort.

This will not only test the probity of measures such as FSM for the Hampshire dataset but will also throw light on their use for school effectiveness studies in general.

The Basingstoke subsample dataset contains family background data on 1653 pupils from a total of 2012 students attending 46 out of all 50 schools in the Basingstoke and Dean area. Relevant to economic status these data include:

• Occupational group, Working status, Home ownership, Whether in receipt of Working Tax Credit, Whether in receipt of FSM.

• Other information was collected on ethnic background, family structure, out-of-school childcare arrangements and cultural behaviour. These data were collected at the beginning of 2004 for year 3 pupils (7 year old) from these schools.

One other measure that is becoming popular in research is the Index of Multiple Deprivation

(IMD)(Noble et al., 2004)

However this relates not directly to individuals but to the small geographical area in which they live, known as a low level Super Output Area (SOA) containing on average about 1500 people. IMD is a composite index based on indices grouped within seven domains:

Income; Employment; Health deprivation and disability; Education, skills and training; Barriers to housing and

services; Living environment; Crime

Our Basingstoke Schools on the Deprivation Map The schools are represented by their DFES codes

The Deprivation Geography of Hampshire based on the Multiple Deprivation Index

As a competitor to FSM, the IMD or its income component could be used as a distant proxy for economic deprivation but technical problems arise when it is desired to assign a deprivation index to an individual who may or may not share the fortunes of an average neighbour.

In general it is not too difficult to assign a deprivation index to a school using IMD information from the SOAs from which the school draws its pupils. This is then an alternative to using the percentage of pupils eligible for FSM as an indicator of economic deprivation at the school level. Note that the education component of the IMD contains achievement data on performance at Key Stages 2, 3 and 4 which may well preclude it as independent information used as input to an analysis of such performance.

FSM and how it is used

FSM is frequently used as a factor in representing economic disadvantage in educational research

(studies of attainment or truancy) LEAs incorporate FSM figures in calculating SEN and

AEN provision Only statutory available information on economic

disadvantage Used as a proxy indicator of economic disadvantage and

more generally low SES

Current FSM eligibilityThe current eligibility criteria are that parents do not have to

pay for school lunches if they receive any of the following:

Income Support Income-based Jobseeker's Allowance Support under Part VI of the Immigration and Asylum Act

1999 Child Tax Credit, provided they are not entitled to

Working Tax Credit and have an annual income (as assessed by HM Revenue & Customs) that does not exceed £14,155

the Guarantee element of State Pension Credit. Children who receive Income Support or income-based

Job Seeker's Allowance in their own right qualify as well.

What is FSM

• FSM eligibility is a measure of extreme socio-economic disadvantage

• At pupil level indicates the pupil’s family economic disadvantage

• At school level its aggregate indicates proportion of families suffering extreme economic disadvantage

• It is not a direct indicator of non-economic SES disadvantage

Criticisms

The popularity of such uses of FSM eligibility in studies in educational research is based on its availability. There is no other measure of socio-economic disadvantage that is universally or even widely available at individual pupil level.

Registration is voluntary – affected by shame, social class differentials?

Schools encourage registration because it affects school income

Do families consistently qualify over time? Often used as a broad indicator of SES but for example

does not directly indicate cultural capital

The use of Large-Scale Data-Bases on Educational Data

• Opportunities for advantages of using large pre-existing data sets

• Conceptual and logistic problems in working with these data sets

• Statistical Issues germane to the analysis of large, complex data bases

The pros• A means for identifying or enumerating cases: they can provide the

sampling frame – can be especially helpful in identifying rare events i.e. our subsample statistics indicate that 7.5% incidence of FSM eligibility in the Basingstoke and Dean area compared with a yearly estimate of 9% between 2002-2005.

• Provide a broad, if imprecise overview: Frequent existing large data bases can be utilized to measure and define the magnitude and distribution of a problem to more definitive study requiring primary data collection. This is particularly important in studies in social sciences and education where the phenomena under study are not themselves directly measurable and must be studied indirectly through the measurements of other observable phenomena.

This is true for both the outcomes we are measuring i.e. we want student progress and we measure test performance

as well as the predictors used i.e. we want socioeconomic disadvantage and we measure a number of

indicators for this including FSM.

• Could provide directions of the size of measurement error in our predictors as well as the direction of bias induced by non-response. Our Basingstoke subsample was found to have higher intakes (baseline tests) in literacy, lower incidence of SEN and FSM compared with the rest of Hampshire.

Provide opportunities to Validate the quality of research data in terms of what these indicators actually measure as well as address missingness. In our sample for example non-responders were found to be twice more likely to be FSM eligible (using the PLASC information)

Conceptual and structural pitfalls

They can be frequently flawed. There are 2 reasons for this and both related to the fact that those who design and compile data bases cannot consider the particular needs of every person who eventually might use them.

First, PLASC is primarily used for administrative functions related to LEA funding and related policies not always explicitly stated and could be changing. It is simply not designed or maintained to maximize data quality or consistency. Codes and records layouts (i.e. SEN) may be changed periodically.

Data elements (i.e. UPN’s from the earlier years might be unreliable pupil identifiers and the same applies for school identifiers after school mergings or closings) are often incomplete and unreliably coded. Information might be inconsistently recorded across LEAs (i.e. literacy tests at reception)

Pitfalls continue …

• Second, even in well-maintained data bases (Hampshire LEA with a long history of research collaborations) the original goals that dictated what data were collected will not coincide with those of subsequent users and coding practices (i.e. SEN assessed externally) might make secondary analysis difficult if not impossible. Inclusion of both persons and services might be linked to program policies that might change over time. Another example is associated with the LEA’s communication with the individual school systems and heterogeneity among schools in the type of the actual tests administered (e.g. QCA). In most cases we do not know:

How were the data collected and processed

The time-frame of data collection

Completeness and coherence among databases of different LEAs

Another example of complex problems in usage

• Our original population cohort consists of 11793 (51% boys) of all Hampshire pupils who took the baseline test during 2001-2002 and their KS1 tests during 2003-2004. We have test results for approximately 84% of this cohort. The Hampshire-wide size of the cohort of Reception pupils from PLASC 2001/02 is 14329 and the size of the Yr2 pupils from PLASC 2003/04 is 14308.

• The pattern of longitudinal losses in terms of test-results seems quite typical for Hampshire for this phase, judging for other historical cohort data that we also had access to. A wave of around 2000 pupils are lost as we move in time because they are leaving Hampshire schools and another wave of 2000 new pupils are coming-into to these schools.

• Because of data-inconsistencies related to correctly identifying and recording school changes (school merging or closing) we end-up looking at 11702 pupils for the phase Baseline to KS1 data. These pupils come from 318 (Infant or Junior and Primary Schools) at baseline and 302 schools at KS1.

Back to FSM in HARPS and the pros

Some subsample statistics Out of the 1653 Basingstoke families 124 were receiving

FSM (7.5%) Of these 13 claimed to receive working tax-credit Single parents and non-working couples account for the

majority of FSM recipients

So FSM is a very coarse indicator, can we do better? What about the economically deprived that do not fall

below the FSM economic threshold? No indication of how much the “better off” actually are?

X1

X2

X3

Y1

Y2

Y3

LatentIndex

A graphical representation of theMIMIC model (Multiple-Indicator-Multiple-Cause)

X1: Working statusX2: SES (assessed by occupation records)X3: education level of the parent

Y1: Tax-CreditY2: FSM eligibilityY3: Home Renting

A scale of economic statusWe estimate a latent score for the unobserved economic status using relevant data

and Use thisInstead of FSM as input to modelling

FSM stability

There is a widely held view that FSM eligibility status is stable over time as the following quote shows:

FSM stability in Hampshire 2002-05

Who of these ever received FSM?

Observations on FSM stability

Assessing the effect of deprivation on school performanceEffect* of Deprivation on KS1-test results:

Comparison between different indices Deprivation Measure Mean Effect SE of effect Concurrent FSM eligiblility 0.013 0.118 Ever FSM eligibility (over a 3-year period)

-0.029 0.101

Economic Deprivation Index 0.035 0.015 Economic Deprivation Index (accounting for measurement error)

0.027 0.017

(*) : Value-added analysis based on an Multilevel Ordered Multinomial model for the KS1 test for maths outcome conditional on baseline tests at Reception, AEN, gender, age and whether English is a second language. (): effects are expressed on the analysis (probit) scale with reference category the highest Level (3) of achievement. Thus, a positive coefficient represents a negative effect on KS1 test performance, since we are predicting the probability of getting less than or equal to particular level.

So, FSM does NOT work

More Consequences of the choice of deprivation index

Ranking school Under-PerformanceSchool Ranks for the Lowest threshold of the KS1 testing scale:Comparison of school rankings based on FSM and our index suggest that schools can be judged performing worse when FSM is used instead of the deprivation index.So, when we use FSM we tend to under-estimate school performance.This is because FSM eligibility under-estimates the prevalence of economic deprivation.Moreover, we also under-estimate the uncertainty associated with these rankings.

The plots depict the school-level residuals (probit scale) against their rank-order.Large negative values in the probit scale characterize the best performing schools.

The differences in rankings affect mostlyAverage (typical) performing schools

The area where FSM under-estimates school performance is below the diagonal line.Higher Residual Ranks indicate Worse Performance

61% - 71% (depending on the residuals considered for each threshold of the KS1 scale) of the schools are judged as performing worse when using FSM rather than using our index.

Moreover, the bias also seem to be differential More deprived schools (in terms of intake and/or FSM eligibility) being affected more.

8.5

2

13.5

2629.5

24

2

43.5

25.5

25.5

4258

4.5

4.515.5

4

5.5

4 3

12.5

75.5

16

2.5

1.5

11

9.5

2

4.5

3

2

1.5

3.5

1.5

5.5

16.5

7

7.5

1

1.5

42.5

5

010

2030

4050

Res

idua

l Ran

k us

ing

Inde

x

0 10 20 30 40 50Residual Rank using FSM

Low Intake Similar RankingsAverage Intake High Intake

Derivation of the index of disadvantageA MIMIC model (Multiple-Indicator-Multiple-Cause) model was used for

the derivation of a scale representing economic disadvantage. Binary data on three indicators of economic disadvantage

Y1: working tax-credit, Y2: FSM eligibility, Y3: Home renting) are used in a two-parameter logistic model (Rasch-model) as follows:

logit[Pr(yijj)]= diji+ dij ij, for indicator i=1,2,3 and subject j

where di’ is a three-dimensional vector for the i-th element equal to 1 and all other elements equal to zeroand is the latent unobserved measure of economic disadvantage.

The parameter estimates for i (location parameters, or intercepts) represent the potency of each indicator.

The parameters i (factor loadings) represent the discriminating ability of each indicator.

We also specified a structural model for the latent economic disadvantage, allowing the mean indicator potencies to differ between groups as follows:

j=0+1X1+2X2+3X3+di1X1,

where X1: working status, X2: SES assessed from occupation records, X3: level of education

The term di1X1 allows us to introduce a direct effect of working tax credit on the receipt of working tax credit, apart from its indirect effect through the latent variable.

This is needed since only employed people would be in receipt of tax-credit.

The intercepts’ estimates suggest that the estimated indicator potencies increase from FSM to renting with FSM being the most difficult benefit to get. FSM eligibility does have the highest discriminating ability (slope) as expected followed by home renting.

As expected the estimated parameter indicates that the odds of receipt of of working tax-credit are significantly reduced for non-working couples compared with the typical couple where one is working full-time and the other part-time.

0.1

.2.3

.4.5

Den

sity

-6 -4 -2 0 2 4Latent economic disadvantage

Higher SES reduces economic disadvantage compared to middle classes

Low or unknown SES increases economic disadvantage.

Higher education level is associated with reduced economic disadvantage.

Non-working couple are significantly more likely to be economically disadvantaged, as expected, compared with the typical working status of a couple in this sample (one working full-time and one part time) with the single part-time parents following closely.

Couples consisting of partners where only one full-time working time are also significantly more disadvantaged compared to the typical working couple which appears similar to couples where both partners work full-time.

It is interesting to note that couples consisting of partners where one is working full-time and the other is self-employed, are significantly less disadvantaged compared with the typical couple

0.2

.4.6

.8P

rob.

of d

isad

vant

age

-4 -2 0 2 4Economic Disadvantage

Middle Class, Further/Vocational Qualific., Full-time+Part time couple

0.2

.4.6

.81

Pro

b. o

f dis

adva

ntag

e

-4 -2 0 2 4Economic Disadvantage

Low SES, Secondary <16 yrs, Full-time+Part-time couple

0.2

.4.6

.8P

rob.

of d

isad

vant

age

-4 -2 0 2 4Economic Disadvantage

FSM

Rent

Tax-credit

High SES, Univ. grad/postgrad., Full-time+Part-time couple

FSM refers to very disadvantaged childreneven among low SES families

Working tax-credit is a muchMore useful indicator especiallyFor middle class occupations

0.2

.4.6

.8

Pro

b. o

f R

eceip

t o

f ta

x-c

red

it

-4 -2 0 2 4Economic Disadvantage

Vocational Qual.

Secondary <16yrs

University grads/postgrads.

Working tax credit indicator characteristic curve for Middle class occupations , Full-time+Part-time couple

0.2

.4.6

.81

Pro

b. o

f R

eceip

t o

f ta

x-c

red

it

-4 -2 0 2 4Economic Disadvantage

Middle SES

Low SES

High

Working tax credit indicator characteristic curve for people with Vovational qualtificat., Full-time+Part-time couple

Receipt of Working tax creditas an indicator of economic disadvantage seem to INDEX occupation based SES, MOSTLY but not always. This might be due to bias associated with reporting occupation and/or assumptions on occupation rankings.

Receipt of Working tax creditas an indicator of economic disadvantage seem to INDEX level of parental education less well.It seems that level of education is consistent with economic disadvantage for the very poor.The economic value of vocational qualifications seem less clear for more typical incomes. This could also be due to the fact that in our sample 90% of the responders are women and only 60% of these women contribute to the determination of SES.

Further developments Improvements in the construction of

the index: Allow for differential measurement

error through joint modelling instead of plugging-in index estimates.

In the structural part of the model include interaction terms for education level and the identity of the major bread-winner.

.7.7

5.8

.85

.9.9

5V

aria

nce

of L

aten

t eco

nom

ic d

isad

vant

age

-6 -4 -2 0 2 4Latent economic disadvantage

Address problems related to endogeneity i.e. economic disadvantage might affect both school intakes as well as KS1 test results, and there might be both direct and indirect effects of economic disadvantage on school performance.

Conclusions

As far as the use of PLASC data is concerned the conclusion is Documentation and Documentation and … The importance for the research community to get involved

with the development and progress of maintaining and improving such invaluable data resources as the PLASC

data.