1 The Appropriate Use of NAPLAN Data National Symposium, 23 July, 2010 Margaret Wu University of Melbourne [email protected] 1.

1

The Appropriate Use of The Appropriate Use of NAPLAN DataNAPLAN Data

National Symposium, 23 July, 2010

Margaret WuUniversity of [email protected]

1

2

NAPLAN TestsNAPLAN Tests

Conducted once a yearAbout 40 test questions per subject

areaTest scores are used to infer

◦the achievement levels of studentsHow reliable can NAPLAN test scores

reflect◦Student achievement level?◦School performance?

2

3

Margin of error in measuring Margin of error in measuring student performancestudent performance

David - a Grade 5 student in 2008. Reading score was 25 out of 40.David’s reading test scores could vary

between 20 and 30, out of 40.◦if similar tests are administered (e.g.,

2009, 2010 tests )One test collects only a small sample

of performance.Variation in scores is called

Measurement Error.3

4

How big an error size is acceptable?How big an error size is acceptable?

The answer is◦It depends.

An example◦Effectiveness of a weight loss program◦Expect a loss of 0.5 kg after one week.◦Measurement scale is accurate to 1kg.◦Not good enough for measuring

individual change◦OK for a group change, if group size is

‘large’.

4

5

On the NAPLAN scale…On the NAPLAN scale…NAPLAN 2008 reading scores

200

300

400

500

600

700

800

grade 3 grade 5 grade 7 grade 9

2.5%tile

mean

97.5%tile

5

6

On the NAPLAN scale…On the NAPLAN scale…NAPLAN 2008 reading scores

200

300

400

500

600

700

800

grade 3 grade 5 grade 7 grade 9

2.5%tile

mean

97.5%tile

6

7

Measuring GrowthMeasuring GrowthNAPLAN 2008 reading scores

200

300

400

500

600

700

800

grade 3 grade 5 grade 7 grade 9

2.5%tile

mean

97.5%tile

Growth measure?

Margin of error of growth measure

± 76 points

Expected growth is 50 points

7

8

Class mean scoresClass mean scoresAverage score for a class

◦Effect of measurement error reducesNew source of error

◦Sampling errorCohort of students changes from year

to yearVariation in class mean score because

of the sample of students in a classClass mean ± 20 points

◦(1 year’s growth)

8

9

Teacher effectTeacher effectA high performing teacher can

raise student standards by one more year of growth as compared to a low performing teacher.

excellent teacher

average teacher

poor teacher

50 points

Margin of error of teacher effect based on two testing occasions: ± 20 points

NAPLAN 2008 reading scores

200

300

400

500

600

700

800

grade 3 grade 5 grade 7 grade 9

2.5%tile

mean

97.5%tile

9

10

MySchool WebsiteMySchool Website

It is a league table◦It compares and ranks schools

It is the worst kind of league table◦Because it is claimed that the red

bars reflect “underperforming schools”

◦Simple league tables do not have this claim.

11

Summary - 1Summary - 1NAPLAN results are NOT suitable

for measuringStudent achievement level

◦beyond a rough “lower”, “average”, “higher” groups

Student progress Teacher effect School performance

11

12

Summary - 2Summary - 2NAPLAN results are for the

systems, e.g.◦Compare girls and boys◦Compare rural and urban◦Trends, if equating design is improved

NAPLAN results should NEVER be published.

Parents/caregivers should not be encouraged to use the results to judge schools.

12

13

Finally…Finally…Conflicting advice from different

experts?An easy way to check out:Ask proponents of MySchool

website to publicly name one underperforming school.

13

14

ReferencesReferencesWu, M.L. (2010). Measurement,

sampling and equating errors in large-scale assessments. Educational Measurement: Issues and Practice, (In press: Volume 29 Number 4).

Nye, B., Konstantopoulos, S., & Hedges, L. (2004). How Large Are Teacher Effects? Educational Evaluation and Policy Analysis, Vol. 26, No. 3 (Autumn, 2004), pp. 237-257 .

15

Leigh, A. (2009). Estimating teacher effectiveness from two-year changes in students’ test scores. Economics of Education Review.

Byrne, Coventry, Olson, Wadsworth, Samuelsson, Petrill, Willcutt and Corley. (2009). Teacher Effects in Early Literacy Development: Evidence From a Study of Twins. Journal of Educational Psychology, 2009.