Embedded formative assessment: still more rhetoric than reality National Conference of The Schools Network 2011 Dylan Wiliam 1.

Embedded formative assessment:still more rhetoric than reality

National Conference of The Schools Network 2011

Dylan Wiliam

www.dylanwiliam.net

1

Origins and antecedents

Feedback (Wiener, 1948) Developing range-finders for anti-aircraft guns Effective action requires a closed system within which

Actions taken within the system are evaluated Evaluation of the actions leads to modification of future actions

Two kinds of loops Positive (bad: leads to collapse or explosive growth) Negative (good: leads to stability)

“Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way” (Ramaprasad, 1983 p. 4)

Feedback and instructional correctives (Bloom)

2

What’s wrong with the feedback metaphor?

Feedback is any information given to the student about their current performance

… or at best, information that compares current performance with desired performance.

Much rarer is information that can be used by learners to improve

That’s just data

That’s just a thermostat

That’s a feedback system

In education In engineering

3

Feedback has complex effects

264 low and high ability grade 6 students in 12 classes in 4 schools; analysis of 132 students at top and bottom of each class

Same teaching, same aims, same teachers, same classwork Three kinds of feedback: scores, comments, scores+comments

Butler(1988) Br. J. Educ. Psychol., 58 1-14

Achievement Attitude

Scores no gain High scorers : positiveLow scorers: negative

Comments 30% gain High scorers : positiveLow scorers : positive

4

Responses

What do you think happened for the students given both scores and comments?

A. Gain: 30%; Attitude: all positiveB. Gain: 30%; Attitude: high scorers positive, low scorers negativeC. Gain: 0%; Attitude: all positiveD. Gain: 0%; Attitude: high scorers positive, low scorers negativeE. Something else

Achievement Attitude

Scores no gain High scorers : positiveLow scorers: negative

Comments 30% gain High scorers : positiveLow scorers : positive

5

Students and grades6

Butler (1987) J. Educ. Psychol. 79 474-482

Feedback is not always effective

200 grade 5 and 6 Israeli students Divergent thinking tasks 4 matched groups

experimental group 1 (EG1); comments experimental group 2 (EG2); grades experimental group 3 (EG3); praise control group (CG); no feedback

Achievement EG1>(EG2≈EG3≈CG)

Ego-involvement (EG2≈EG3)>(EG1≈CG)

7

Feedback should feed forward

80 Grade 8 Canadian students learning to write major scales in Music Experimental group 1 (EG1) given

written praise list of weaknesses workplan

Experimental group 2 (EG2) given oral feedback nature of errors chance to correct errors

Control group (CG1) given no feedback

Achievement: EG2>(EG1≈CG)Boulet et al. (1990) J. Educational Research 84 119-125

8

…and should leave learning with the learner

‘Peekability’ (Simmonds & Cope, 1993) Pairs of students, aged 9-11 Angle and rotation problems

class 1 worked on paper class 2 worked on a computer, using Logo

Class 1 outperformed class 2

‘Scaffolding’ (Day & Cordón, 1993) 2 grade 3 classes

class 1 given ‘scaffolded’ response class 2 given solution when stuck

Class 1 outperformed class 2

9

Effects of feedback

Kluger & DeNisi (1996) Review of 3000 research reports Excluding those:

without adequate controls with poor design with fewer than 10 participants where performance was not measured without details of effect sizes

left 131 reports, 607 effect sizes, involving 12652 individuals On average feedback does improve performance, but

Effect sizes very different in different studies 40% of effect sizes were negative

10

Getting feedback right is hard

Feedback indicates performance…

Response typeexceeds goal falls short of goal

Change behavior Exert less effort Increase effort

Change goal Increase aspiration Reduce aspiration

Abandon goal Decide goal is too easy Decide goal is too hard

Reject feedback Feedback is ignored Feedback is ignored

11

Feedback practice audit

How often do students receive ‘feedback’ in the form of scores, levels, sub-levels, or grades?A.Key stages 1 to 3B.Key stage 4C.Key stage 5

1. Every week2. Every two or three weeks3. Every month or half-term4. Termly/twice a year5. Annually

12

Kinds of feedback (Nyquist, 2003)

Weaker feedback only Knowledge or results (KoR)

Feedback only KoR + clear goals or knowledge of correct results (KCR)

Weak formative assessment KCR+ explanation (KCR+e)

Moderate formative assessment (KCR+e) + specific actions for gap reduction

Strong formative assessment (KCR+e) + activity

13

Effects of formative assessment (HE)

Kind of feedback Count Effect/sd

Weaker feedback only 31 0.14

Feedback only 48 0.36

Weaker formative assessment 49 0.26

Moderate formative assessment 41 0.39

Strong formative assessment 16 0.56

14

Feedback practice audit 2

In your school, what proportion of feedback events involve students in responding to the feedback provided immediately, and in class?

1. Less than 10%2. 10% to 30%3. 30% to 70%4. 70% to 90%5. More than 90%

15

Unfortunately, humans are not machines…

Attribution (Dweck, 2000) Personalization (internal v external) Permanence (stable v unstable) Essential that students attribute both failures and

success to internal, unstable causes(it’s down to you, and you can do something about it)

16

Success Failure

Personalization

internal: “I got a good grade because I did a good piece of work”

internal: “I got a low grade because it wasn’t a very good piece of work”

external: “I got a good grade because the teacher likes me”

external: “I got a low grade because the teacher doesn’t like me

Stability

stable: “I got a good grade because I’m good at that subject”

stable: “I got a bad grade because I’m no good at that subject”

unstable: “I got a good grade because I was lucky in the questions that came up”

unstable: “I got a bad grade because I hadn’t reviewed the material before the test”

Specificity

specific: “I’m good at that but that’s the only thing I’m good at”

specific: “I’m no good at that but I’m good at everything else”

global: “I’m good at that means I’ll be good at everything” global: “I’m useless at everything”

Mindset

Views of ‘ability’ fixed (IQ) incremental (untapped potential) Essential that teachers inculcate in their students a

view that ‘ability’ is incremental rather than fixed(by working, you’re getting smarter)

18

Force-field analysis (Lewin, 1954)

What are the forces that will support or drive the adoption of formative assessment practices in your school/authority?

What are the forces that will constrain or prevent the adoption of formative assessment practices in your school/authority?

+ —

19

“Flow”

A dancer describes how it fees when a performance is going well: “Your concentration is very complete. Your mind isn’t wandering, you are not thinking of something else; you are totally involved in what you are doing. … Your energy is flowing very smoothly. You feel relaxed, comfortable and energetic.”

A rock climber describes how it feels when he is scaling a mountain: “You are so involved in what you are doing [that] you aren’t thinking of yourself as separate from the immediate activity. … You don’t see yourself as separate from what you are doing.”

A mother who enjoys the time spent with her small daughter: “Her reading is the one thing she’s really into, and we read together. She reads to me and I read to her, and that’s a time when I sort of lose touch with the rest of the world, I’m totally absorbed in what I’m doing.”

A chess player tells of playing in a tournament: “… the concentration is like breathing—you never think of it. The roof could fall in and, if it missed you, you would be unaware of it.” (Csikszentmihalyi, 1990, pp. 53–54)

20

Motivation: cause or effect?

competence

challenge

Flow

apathyboredom

relaxation

arousal

anxiety

worry control

high

low

low high

Csikszentmihalyi (1990)

21

Providing feedback that moves learning on

Key idea: feedback should: Cause thinking Provide guidance on how to improve

Comment-only marking Focused marking Explicit reference to mark schemes/scoring guide Suggestions on how to improve:

Not giving complete solutions Re-timing assessment:

e.g., three-quarters-of-the-way-through-a-unit test

22

A blossoming of research reviews…

Fuchs & Fuchs (1986) Natriello (1987) Crooks (1988) Bangert-Drowns, et al. (1991) Dempster (1991, 1992) Elshout-Mohr (1994) Kluger & DeNisi (1996) Black & Wiliam (1998)

Nyquist (2003) Brookhart (2004) Allal & Lopez (2005) Köller (2005) Brookhart (2007) Wiliam (2007) Hattie & Timperley (2007) Shute (2008)

23

Effects of formative assessment

Source Effect sizeKluger & DeNisi (1996) 0.41Black &Wiliam (1998) 0.4 to 0.7Wiliam et al., (2004) 0.32Hattie & Timperley (2007) 0.96Shute (2008) 0.4 to 0.8

Standardized effect size: differences in means, measured in population standard deviations

24

Problems with effect sizes

Restriction of range Sensitivity to instruction Ambiguous comparisons

25

Definitions of formative assessment

We use the general term assessment to refer to all those activities undertaken by teachers—and by their students in assessing themselves—that provide information to be used as feedback to modify teaching and learning activities. Such assessment becomes formative assessment when the evidence is actually used to adapt the teaching to meet student needs” (Black & Wiliam, 1998 p. 140)

“the process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning” (Cowie & Bell, 1999 p. 32)

“assessment carried out during the instructional process for the purpose of improving teaching or learning” (Shepard et al., 2005 p. 275)

26

“Formative assessment refers to frequent, interactive assessments of students’ progress and understanding to identify learning needs and adjust teaching appropriately” (Looney, 2005, p. 21)

“A formative assessment is a tool that teachers use to measure student grasp of specific topics and skills they are teaching. It’s a ‘midstream’ tool to identify specific student misconceptions and mistakes while the material is being taught” (Kahl, 2005 p. 11)

27

“Assessment for Learning is the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and how best to get there” (Broadfoot et al., 2002 pp. 2-3)

Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting students’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information that teachers and their students can use as feedback in assessing themselves and one another and in modifying the teaching and learning activities in which they are engaged. Such assessment becomes “formative assessment” when the evidence is actually used to adapt the teaching work to meet learning needs. (Black et al., 2004 p. 10)

28

Which of these is formative?

A. A science adviser uses test results to plan professional development workshops for teachers

B. Teachers doing item-by-item analysis of KS2 math tests to review their curriculum

C. A school tests students every 10 weeks to predict which students are “on course” to pass a big test

D. “Three fourths” of the way through a unit testE. Exit pass question: “What is the difference between mass

and weight?”F. “Sketch the graph of y equals one over one plus x squared

on your mini-dry-erase boards.”

29

What does formative assessment form?

Cycle length

Long Medium Short

Curriculum alignment ✔

Monitoring progress ✔ ✔ ✔

Student involved assessment ✔ ✔

Student engagement ✔ ✔

Teacher cognition about learning ✔ ✔

Responsive classroom practice ✔

30

Formative assessment: a new definition

“An assessment functions formatively to the extent that evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been taken in the absence of that evidence.” (Wiliam, 2009)

Formative assessment involves the creation of, and capitalization upon, moments of contingency in the regulation of learning processes.

31

Unpacking formative assessment

Key processes Establishing where the learners are in their learning Establishing where they are going Working out how to get there

Participants Teachers Peers Learners

32

Unpacking formative assessment

Where the learner is going Where the learner is How to get there

Teacher

Peer

Learner

Clarifying, sharing and

understanding learning

intentions

Engineering effective discussions, tasks, and

activities that elicit evidence of learning

Providing feedback that

moves learners forward

Activating students as learningresources for one another

Activating students as ownersof their own learning

33

Five “key strategies”…

Clarifying, sharing, and understanding learning intentions curriculum philosophy

Engineering effective classroom discussions, tasks and activities that elicit evidence of learning classroom discourse, interactive whole-class teaching

Providing feedback that moves learners forward feedback

Activating students as learning resources for one another collaborative learning, reciprocal teaching, peer-assessment

Activating students as owners of their own learning metacognition, motivation, interest, attribution, self-assessment

Wiliam & Thompson (2007)

34

Unpacking formative assessment

Where the learner is going Where the learner is How to get there

Teacher

Peer

Learner

Clarifying, sharing and

understanding learning

intentions

35

Using evidence of achievement to adapt

what happens in classrooms to meet

learner needs

Clarifying, sharing, and understanding learning intentions

White & Frederiksen, Cognition & Instruction, 16(1), 1998

Sharing learning intentions

3 teachers each teaching 4 Year 8 science classes in two US schools

14 week experiment 7 two-week projects, each scored 2-10 All teaching the same, except: For a part of each week

Two of each teacher’s classes discusses their likes and dislikes about the teaching (control)

The other two classes discusses how their work will be assessed

37

Sharing learning intentions

Comprehensive Test of Basic Skills

Group Low Middle High

Likes and dislikes

Reflective assessment

38

Outcomes

Who will benefit most from the reflective assessment?1. Higher achievers2. Average achievers3. Lower achievers4. All students will benefit equally

39

Sharing learning intentions

Comprehensive Test of Basic Skills

Group Low Middle High

Likes and dislikes 4.6 5.9 6.6

Reflective assessment

40

Sharing learning intentions

Comprehensive Test of Basic Skills

Group Low Middle High

Likes and dislikes 4.6 5.9 6.6

Reflective assessment 6.7 7.2 7.4

41

Sharing learning intentions42

Explain learning intentions at start of lesson/unit: Learning intentions Success criteria

Consider providing learning intentions and success criteria in students’ language.

Use posters of key words to talk about learning: e.g., describe, explain, evaluate

Use planning and writing frames judiciously Use annotated examples of different standards to “flesh out”

assessment rubrics (e.g., lab reports) Provide opportunities for students to design their own tests

Engineering effective discussions, activities, and classroom tasks that elicit evidence of learning

Eliciting evidence

Key idea: questioning should cause thinking provide data that informs teaching

Improving teacher questioning generating questions with colleagues closed v open low-order v high-order appropriate wait-time

44

Medicine Hat Tigers

A major junior (ice) hockey team playing in the Central Division of the Eastern Conference of the Western Hockey League in Canada

Players are aged from 15 to 20 15 year olds are only allowed to play five games until

their own season has ended Each team is allowed only three 20 year olds Total roster 25 players

45

Medicine Hat Tigers46

Eliciting evidence

Getting away from I-R-E basketball rather than serial table-tennis ‘No hands up’ (except to ask a question) ‘Hot Seat’ questioning

All-student response systems ABCD cards, Mini white-boards, Exit passes

47

Nothing new under the sun…48

Eliciting evidence practice audit

In what proportion of lessons in your school would a teacher use an ‘all student response’ system at least every 30 minutes?

1. Less than 10%2. 10% to 30%3. 30% to 70%4. 70% to 90%5. More than 90%

49

Hinge questions

A hinge question is based on the important concept in a lesson that is critical for students to understand before you move on in the lesson.

The question should fall about midway during the lesson.

Every student must respond to the question within two minutes.

You must be able to collect and interpret the responses from all students in 30 seconds

50

Questioning in maths: Diagnosis

In which of these right-angled triangles is a2 + b2 = c2 ?

A a

c

b

C b

c

a

E c

b

a

B a

b

c

D b

a

c

F c

a

b

51

Wilson & Draney, 2004

Questioning in science: Diagnosis52

The ball sitting on the table is not moving. It is not moving because:

A. no forces are pushing or pulling on the ball.

B. gravity is pulling down, but the table is in the way.C. the table pushes up with the same force that gravity pulls downD. gravity is holding it onto the table. E. there is a force inside the ball keeping it from rolling off the table

Questioning in English: Diagnosis (2)

Which of these is correct?A. Its on its way.B. It’s on its way.C. Its on it’s way.D. It’s on it’s way.

53

Questioning in English: Diagnosis (3)

Identify the adverbs in these sentences:

1. The boy ran across the street quickly. (A) (B) (C) (D) (E)

2. Jayne usually crossed the street in a leisurely fashion. (A) (B) (C) (D) (E)

3. Fred ran the race well but unsuccessfully. (A) (B) (C) (D) (E)

54

Questioning in English: Diagnosis (4)

Which of these is the best thesis statement?A. The typical TV show has 9 violent incidentsB. The essay I am going to write is about violence on TVC. There is a lot of violence on TVD. The amount of violence on TV should be reducedE. Some programs are more violent than othersF. Violence is included in programs to boost ratingsG. Violence on TV is interestingH. I don’t like the violence on TV

55

Questioning in history: Diagnosis

Why are historians concerned with bias when analyzing sources?A. People can never be trusted to tell the truthB. People deliberately leave out important detailsC. People are only able to provide meaningful information

if they experienced an event firsthandD. People interpret the same event in different ways,

according to their experienceE. People are unaware of the motivations for their actionsF. People get confused about sequences of events

56

Questioning in MFL: Diagnosis

Which of the following is the correct translation for ”I give the book to him”?

A. Yo lo doy el libro.B. Yo doy le el libro.C. Yo le doy el libro.D. Yo doy lo el libro.E. Yo doy el libro le.F. Yo doy el libro lo.

57

Key requirement: discriminate between incorrect and correct cognitive rules

Version 1

There are two flights per day from Newtown to Oldtown. The first flight leaves Newtown each day at 9:20 and arrives in Oldtown at 10:55. The second flight from Newtown leaves at 2:15. At what time does the second flight arrive in Oldtown? Show your work.

Version 2

There are two flights per day from Newtown to Oldtown. The first flight leaves Newtown each day at 9:05 and arrives in Oldtown at 10:55. The second flight from Newtown leaves at 2:15. At what time does the second flight arrive in Oldtown? Show your work.

58

Activating students as owners of their own learning

Fontana & Fernandes, Br. J. Educ. Psychol. 64: 407-417

Self-assessment: Portugal

45 teachers studying for a Masters degree in Education, matched in age, qualifications and experience using the same curriculum scheme for the same amount of timeControl group (N=20) follow regular MA program

Experimental group (N=25) develop self-assessment with their students

117 students aged 8 years 125 students aged 8 years119 students aged 9 years 121 students aged 9 years77 students aged 10 - 14 years 108 students aged 10 - 14 years

60

Details of the intervention

Weeks Intervention

1 to 2 Individual choice from a range of work provided by the teacher. Student self-assessment using materials provided

3 to 6 Children construct own problems like those in weeks 1 and 2 and select structured math apparatus to aid solutions

7 to 10 Children presented with a new learning objectives, and make up their own problems, without exemplars by the teacher

11 to 14 Children set their own learning objectives, construct appropriate problems, and use appropriate self-assessment

15 to 20 As weeks 1 to 14, but with less monitoring from the teacher and increased freedom of choice and personal responsibility

61

Impact on student achievement

Pre-test Post-test Gain Effect size

Control 65.1 72.9 7.8 0.34

Experimental 58.7 73.7 15.0 0.66

62

Students owning their own learning

Students assessing their own work: With mark schemes or scoring guides With exemplars

Self-assessment of understanding: Traffic lights Red/green discs Coloured cups

63

Activating students as learning resources for one another

Benefits of structured interaction

15-yr-olds studying World History were tested on their understanding of material delivered in lessons

At the end of the lessons, students were given time to review their understanding of the material before they were tested

Half the students had been trained to pose questions as they listened to the lectures

Individual Group

Unstructured Independent review Group discussion

Structured Structured self-questioning

Structured peer-questioning

65

King, A. (1991). Applied Cognitive Psychology, 5(4), 331-346

Impact on achievement

King, A. (1991). Applied Cognitive Psychology, 5(4), 331-346

66

Students as learning resources

Students assessing their peers’ work: “Pre-flight checklist” “Two stars and a wish”

Training students to pose questions/identifying group weaknesses

End-of-lesson students’ review

67

Pulling it all together

Dual-pathway model (Boekaerts, 1993)

“It is assumed that students who are invited to participate in a learning activity use three sources of information to form a mental representation of the task-in-context and to appraise it:1.current perceptions of the task and the physical, social, and instructional context within which it is embedded;2.activated domain-specific knowledge and (meta)cognitive strategies related to the task; and3.motivational beliefs, including domain-specific capacity, interest and effort beliefs.” (Boekaerts, 2006, p. 349)

69

Growth and well-being

Share learning goals with students so that they are able to monitor their own progress toward them.

Promote the belief that ability is incremental rather than fixed; when students think they can’t get smarter, they are likely to devote their energy to avoiding failure.

Make it more difficult for students to compare themselves with others in terms of achievement.

Provide feedback that contains a recipe for future action rather than a review of past failures.

Use every opportunity to transfer executive control of the learning from the teacher to the students to support their development as autonomous learners, and as learning resources for one another

Use random questioning and all-student response systems to provide high-quality evidence to the teacher about theprogress of learning

70

Force-field analysis (Lewin, 1954)

What are the forces that will support or drive the adoption of formative assessment practices in your school/authority?

What are the forces that will constrain or prevent the adoption of formative assessment practices in your school/authority?

+ —

71