Interpreting Feedback from Baseline Tests – Predictive Data

Interpreting Feedback from Baseline Tests – Predictive Data

Course: CEM Information Systems for Beginners and New Users

Day 1 Session 3Wednesday 27th February 2013

Rob Smith: CEM Inset Provider

[email protected]

The grades generated by the CEM systems are not predictions, but average performance by similar students in the most recent examination

What are “Predictions”, and how are they generated?

2008 GCSE Maths (representative sample)

0

1

2

3

4

5

6

7

8

9

0 20 40 60 80 100

baseline score

Res

ult

BASELINE SCORE

P

OIN

TS

G

RA

DE

* ** *** *******************

* ** *** ********************************

* ** *** ********************************* *** ** *

* ** *** ************************************ *** ** *

* ** *** ******************************** **** ** *

* ** ***************************** *** ** *

* ** *** ******************* ** * *

* ** *********** ** *

Subject National trend line (regression line)

50% on or above the trend line

50% on or below the trend line

‘PREDICTION’ (expected grade)C

A*

A

B

X

3 key points are:

• The higher the baseline score the higher the final grade

• Any one grade is achievable from a range of baseline scores

• From any baseline score, a range of grades is possible

4 points = D

6.6 points = A/BTrend line

98 128

The word ‘PREDICTION’: Quite probably the most contentious term that is

used!! Concerns include:

• A prediction for GCSE at start of year 7?• What has the baseline test got to do with my

subject?• I know my pupils!• The predictions are too low/too high: not valid!!!• And what about my professional judgement?

• Alternatives include: Expected grade Minimum attainment/achievable grade Baseline suggested grade

• It assumes average progress and you would hope that your students make above average progress

• But you know that all of them don’t!!!!

The word ‘PREDICTION’:

FACTORS THAT WILL INFLUENCE RELIABILITY OF ‘PREDICTIONS’:

• Knowledge of student • Parental support/home life• Peer influences/social life• Student attitude, interest, language• Expectations of staff• Department/institution ethos • Resources • Quality of teaching and learning: pace of lessons• Understanding how children learn………• And the reliability of the predictions......

10

Correlation = 1

Res

ult

Correlation = 0

Correlation = 0.7

subjectMIDYIS( Yr 7)

to GCSEART 0.49BIOL 0.56ENG LANG 0.7FRENCH 0.64HISTORY 0.63MATHS 0.74MUSIC 0.57

CORRELATIONS

SUBJECTGCSE to A2

ART and DESIGN 0.54BIOLOGY 0.69ECONOMICS 0.65ENG LIT 0.71FRENCH 0.6HISTORY 0.69MATHS 0.6MUSIC 0.65PHYS EDUC 0.65PSYCHOLOGY 0.62

stud

ent n

o.

Sex

Form

Mid

YIS

Scor

e

Mid

YIS

Band

Art &

Des

ign

Biol

ogy

Engl

ish

Fren

ch

His

tory

Mat

hem

atic

s

Scie

nce

1 M 7E 131 A 6.8 7.2 7.1 6.8 7.2 7.5 7.0

2 F 7E 120 A 6.3 6.7 6.4 6.1 6.4 6.5 6.3

3 F 7C 110 B 5.8 6.2 5.7 5.4 5.5 5.6 5.5

4 F 7J 101 B 5.4 5.8 5.1 4.8 4.9 4.9 4.9

5 M 7D 82 D 4.5 4.9 3.8 3.5 3.4 3.2 3.5

1 M 7E 131 A A A A A A A*/A A

2 F 7E 120 A A/B A/B A/B B A/B A/B A/B

3 F 7C 110 B B B B/C B/C B/C B/C B/C

4 F 7J 101 B B/C B C C C C C

5 M 7D 82 D C/D C D D/E D/E E D/E

Point and grade ‘predictions’ to GCSE• Compare the

predictions for English and Mathematics.

What pattern do you notice?

• Look at Art and Design, Biology and French. Comments?

50 60 70 80 90 100 110 120 130 140 1501

2

3

4

5

6

7

8

MidYIS Year 7 to GCSE 2011 Value Added Analysis Art & Design

Biology

English

French

History

Mathematics

Science

MidYIS Test Score

GC

SE P

oint

s Sc

ore

The graph below shows the middle 2/3 of some subject trend lines

Some Subjects are More Equal than Others….

E

D

C

B

A

C

Gra

de

B A A*Average GCSE

PhysicsMathsPsychology

Sociology

Latin

Photography

English Lit

>1 grade

Predictions & Chances Graphs to GCSE subjects

based on Year 9 nationally standardised scores

Please refer to the Information Sheet for details of the method used to make these predictions

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Student Surname Forename Sex Form CEM Id Cust Mid

YIS

Sco

re

Mid

YIS

Ban

d

Addi

tiona

l App

lied

Scie

nce

Addi

tiona

l Sci

ence

Art &

Des

ign

Biol

ogy

Busi

ness

Stu

dies

Che

mis

try

Des

ign

& T

echn

olog

y

Dra

ma

Engl

ish

Engl

ish

Lite

ratu

re

Fren

ch

Geo

grap

hy

1 F 9140005J 105 B 4.4 5.3 5.8 5.9 5.1 5.9 5.3 5.2 5.4 5.5 5.0 5.12 F 9140005K 111 A 4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.63 F 91400042 108 B 4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.44 F 91400043 101 B 4.2 5.0 5.6 5.7 4.8 5.7 5.0 5.0 5.1 5.3 4.7 4.7

Teac

her's

Ad

just

men

t (G

rade

s)

Predictions based on 2011 Regression Analysis

MidYIS Year 9 Test Academic Year 2011/2012

GCSE Point PredictionsPoints To Grades Adjust

CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile of Schools

Prior Value added

Generate File for MIS Import

to place school at 75th percentile of all schools

Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores


0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2


YIS

Scor

e

Mid

YIS

Band

Addi

tiona

l App

lied

Scie

nce

Addi

tiona

l Sci

ence

Art &

Des

ign

Biol

ogy

Busi

ness

Stu

dies

Che

mis

try

Des

ign

& Te

chno

logy

Dra

ma

Engl

ish

Engl

ish

Lite

ratu

re

Fren

ch

Geo

grap

hy

1 F 9140005J 105 B 4.6 5.5 6.0 6.1 5.3 6.1 5.5 5.4 5.6 5.7 5.2 5.32 F 9140005K 111 A 4.9 6.0 6.3 6.5 5.8 6.5 5.9 5.8 6.0 6.1 5.6 5.83 F 91400042 108 B 4.8 5.8 6.2 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.64 F 91400043 101 B 4.4 5.2 5.8 5.9 5.0 5.9 5.2 5.2 5.3 5.5 4.9 4.9

Teac

her's

Ad

just

men

t (G

rade

s)

Predictions based on 2011 Regression AnalysisMidYIS Year 9 Test Academic Year 2011/2012


CEM Centre Scale

QCA Scale

Display Legend

Reset Adjustments

Prior Value Prior Value addedadded




0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0


YIS

Sco

re

Mid

YIS

Ban

d

Addi

tiona

l App

lied

Sci

ence

Addi

tiona

l Sci

ence

Art &

Des

ign

Biol

ogy

Busi

ness

Stu

dies

Che

mis

try

Des

ign

& T

echn

olog

y

Dra

ma

Engl

ish

Engl

ish

Lite

ratu

re

Fren

ch

Geo

grap

hy

1 F 9140005J 105 B 4.4 5.3 5.8 5.9 5.1 5.9 5.3 5.2 5.4 5.5 5.0 5.12 F 9140005K 111 A 4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.63 F 91400042 108 B 4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.44 F 91400043 101 B 4.2 5.0 5.6 5.7 4.8 5.7 5.0 5.0 5.1 5.3 4.7 4.7

Teac

her's

Ad

just

men

t (G

rade

s)

Predictions based on 2011 Regression AnalysisMidYIS Year 9 Test Academic Year 2011/2012


CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile of Schools

Prior Value added


Or insert own valuesand click adjust

Predictions & Chances Graphs to GCSE subjects

based on Year 9 nationally standardised scores


0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4


YIS

Sco

re

Mid

YIS

Band

Addi

tiona

l App

lied

Sci

ence

Addi

tiona

l Sci

ence

Art &

Des

ign

Biol

ogy

Busi

ness

Stu

dies

Che

mis

try

Des

ign

& Te

chno

logy

Dra

ma

Engl

ish

Engl

ish

Lite

ratu

re

Fren

ch

Geo

grap

hy

1 F 9140005J 105 B 4.4 5.6 7.2 6.6 5.1 6.6 5.3 6.1 6.7 6.3 5.7 6.52 F 9140005K 111 A 4.7 6.1 7.5 7.0 5.6 7.0 5.7 6.5 7.1 6.7 6.1 7.03 F 91400042 108 B 4.6 5.9 7.4 6.8 5.4 6.8 5.5 6.3 6.9 6.5 5.9 6.84 F 91400043 101 B 4.2 5.3 7.0 6.4 4.8 6.4 5.0 5.9 6.4 6.1 5.4 6.1

Teac

her's

Ad

just

men

t (G

rade

s)




CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile Upper Quartile of Schoolsof Schools

Reset Adjustments




0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4


YIS

Sco

re

Mid

YIS

Ban

d

Add

ition

al A

pplie

d Sc

ienc

e

Addi

tiona

l Sci

ence

Art

& D

esig

n

Biol

ogy

Bus

ines

s S

tudi

es

Che

mis

try

Des

ign

& Te

chno

logy

Dra

ma

Engl

ish

Engl

ish

Lite

ratu

re

Fren

ch

Geo

grap

hy

1 F 9140005J 105 B C/D B/C A A/B C A/B B/C B A/B A/B B/C A/B2 F 9140005K 111 A C/D B A*/A A B/C A B/C A/B A A/B B A3 F 91400042 108 B C/D B A*/A A B/C A B/C A/B A A/B B A4 F 91400043 101 B D B/C A A/B C A/B C B A/B B B/C B

Teac

her's

Ad

just

men

t (G

rade

s)



GCSE Grade PredictionsGrades To Points Adjust

CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile Upper Quartile of Schoolsof Schools

Reset Adjustments


Predictions & Chances Graphs to GCSE subjects based on Year 9 Independent Sector standardised scores



YIS

Scor

e

Mid

YIS

Ban

d

Addi

tiona

l Sci

ence

Art &

Des

ign

Biol

ogy

Busi

ness

Stu

dies

Che

mis

try

Cla

ssic

al C

ivilis

atio

n

Des

ign

& Te

chno

logy

Dra

ma

Engl

ish

Engl

ish

Lite

ratu

re

Fren

ch

Geo

grap

hy

Ger

man

1 F 9140005J 87 D 5.8 6.6 6.4 5.7 6.4 5.8 6.1 6.1 6.1 6.2 5.7 6.1 5.82 F 9140005K 94 C 6.1 6.8 6.7 6.0 6.7 6.1 6.4 6.4 6.5 6.4 6.1 6.5 6.23 F 91400042 90 C 6.0 6.7 6.6 5.9 6.6 6.0 6.3 6.3 6.3 6.3 5.9 6.3 6.04 F 91400043 81 D 5.5 6.5 6.1 5.4 6.1 5.6 5.8 5.9 5.9 5.9 5.4 5.8 5.6

Predictions based on 2011 Independent Sector Regression Analysis


GCSE Point PredictionsPoints To Grades

CEM Centre Scale

QCA Scale

Display Legend


0 0 15

19

39

27

9

10

5

10

15

20

25

30

35

40

45

U G F E D C B A A*

Perc

ent

Grade

Individual Chances Graph for Student 4- GCSE EnglishMidYIS Score 101 MidYIS Band B

Teacher's Adjustment : 0 grades / levels / points

Prediction/expected grade: 5.1 grade C

Most likely grade

Blue bars represent the probability of the student achieving each grade

‘Predictions’ should be used in conjunction with Chances Graphs

Prediction/expected grade: 6.4 grade A/B

Most likely grade

Chances Graphs can serve as

a warning for top scoring students and a motivator for low scoring students

Not a label for life...just another piece of information

• The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score.

• In working with students these graphs are more useful than a single predicted or target grade

• Chances graphs show what can be achieved: – By students of similar ability– By students with lower baseline scores

Student 1

Student 2

Why is the ‘predicted’ grade not always equal to the highest bar ?

Most likely grade

Predicted (‘expected’) grade

Predicted (‘expected’) grade

AT WHICH POINT WILL THE SEE-SAW BE BALANCED?

i.e. the lower grades ‘pull’ the prediction to the left

Student 3

Student 4

Student 4 - IPR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 330

5

10

15

20

25

30

Vocab Item Responses

Question

Resp

onse

Tim

e (s

)

Correct

Wrong

Key points to consider might include:

• Where has the data come from?• What (reliable and relevant) data should we use?• Enabling colleagues to trust the data: Training (staff)• Communication with parents and students• Challenging, NOT Demoralising, students…….• Storage and retrieval of data• Consistency of understanding what the data means

and does not mean

Interpreting Feedback from Baseline Tests – Predictive Data

Documents

grade predictions

baseline score points

final grade

similar students

range of baseline scores

low scoring students

trend line50

average progress