1 Assessment and Monitoring [email protected] .

1

Assessment and Monitoring

[email protected]

www.cemcentre.org

2

Centre for Evaluation and Monitoring (CEM)

• Part of Durham University

• Monitoring Systems

• 1.1 million assessments delivered each year

• Pupils aged 3 – 18 years

• CEM systems used in 44 countries

3

Why assess?

• Newton (2007) 18 purposes for which educational judgements may be used

– Not exhaustive– Expanding all the time

– Interesting perspective on the historical debate about formative and summative assessment

4

Why assess?

• From birth ~ weight, hearing etc.

• Qualifications ~ driving test, GCSEs, degree

• Profile of strengths and weaknesses for planning appropriate learning experiences

• Indicator of special educational needs

• Monitor progress and attitudes of pupils and cohorts over time

5

• Comparisons– Children within a class– Groups such as boys/girls– Classes within a year-group– Current cohorts with previous ones– Other schools within a consortium and nationally

• Progress over time

• Research– Within school– Nationally and internationally

6

• Layers of information (different levels of detail):– Diagnostic at pupil-level– Group and class trends– School-level information (including trends

over time)– Consortium/Authority-level

7

• Judgements

• Observations– E.g. Behaviour

• Objective tests– E.g. Attainment

• Practical• Pencil and paper• Computer-delivered

Ways to Assess and Cautions

Do you know the psychometric properties of the assessments that you use?Are these methods always reliable and valid?Early Years Foundation Stage Profile?Age 2 – 3 progress checkYear 1 phonics check

8

• Standardised Assessment

– Administration procedure

– Comparison against representative norms

9

• Standards over time• Consistent content• Consistent sample of schools

Merrell, C. and Tymms, P. (2011) Changes in Children’s Cognitive Development at the Start of School in England 2001 – 2008, Oxford Review of Education, Vol. 37 (3), June 2011, p333 – 345.

Example of a high-level use of standardised assessment

10

Initiatives in the Early Years

• Foundation Stage– Age 3 – 5 years– Increased nursery provision– Curriculum– Assessment

• Sure Start

• Education Action Zones

• Etc. Etc.

Have these initiatives changedchildren’s cognitive development at the start of school?

11

PIPS On-entryBaseline Assessment

• Computer delivered assessment of:– Vocabulary– Phonological awareness &

early reading– Early mathematics

• Additional data:– Date of birth– Sex– First Language

12

• Test/Re-test Reliability= 0.98

• Internal Reliability(Cronbach’s alpha)= 0.94

• Correlation withattainment at age 11= 0.68

13

Sample

472 State Schools in England

Academic Year Number of Pupils 2001 15,319 2002 14,928 2003 14,815 2004 14,501 2005 14,099 2006 14,209 2007 14,761 2008 15,262

14

Background Variables

Academic Year

% EAL

% boys Mean age at test

2001 9.9 51.5 4.56 2002 10.3 51.1 4.55 2003 11.5 50.7 4.55 2004 12.0 50.6 4.56 2005 13.5 51.1 4.56 2006 15.2 51.5 4.56 2007 15.0 51.4 4.55 2008 14.2 51.1 4.55

N.B. EAL data were available for 83.7% of children in the study.

15

Mean PIPS T Scores

AcademicYear

Mean total PIPS BLA

T score(SD in brackets)

2001 48.67 (9.66)

2002 48.50 (9.77)

2003 48.31 (9.43)

2004 48.34 (9.49)

2005 48.23 (9.47)

2006 48.25 (9.42)

2007 47.84 (9.41)

2008 47.59 (9.22)

16

Mean PIPS Raw ScoresEarly Reading Early Maths Picture

VocabularyTotal

Academic Year

Mean(max =

170)

SD Mean(Max =

69)

SD Mean(Max =

23)

SD MeanMax =

262)

SD

2001 14.32 13.90 22.71 9.21 13.66 4.48 50.69 23.64

2002 13.96 13.44 22.66 9.17 13.53 4.56 50.14 23.29

2003 13.32 13.02 22.80 9.16 13.53 5.05 49.64 23.15

2004 13.28 13.03 22.84 9.25 13.70 5.04 49.82 23.27

2005 13.20 13.07 22.38 8.79 13.52 5.04 49.10 22.88

2006 13.59 13.93 23.16 9.15 13.66 5.11 50.40 23.82

2007 13.24 13.67 23.62 9.22 13.51 5.11 50.37 23.57

2008 13.14 13.37 23.74 9.18 13.34 5.08 50.25 23.28

Effect Sizes 0.11 -0.07

17

Main Effects & Interactions

• GLMs to analyse changes in BLA scores in relation to:– Year– EAL– Sex

– Age at test was entered as covariate

18

Conclusions

• Statistically significant decrease from 2001 to 2008 for early reading and picture vocabulary. However, the effect sizes of the differences were small.

• Significant increase in early maths scores although again the effect size was small.

19

Why Should the BLA Scores Remain So Stable Over Time?

• Success from early interventions aimed at influencing the development of young children is difficult to achieve

• Reduction in efficacy of small-scale programmes when rolled out

• Limitations with data analysed but nevertheless this is a large dataset that adds to current studies of trends over time

20

Thank you

[email protected]