Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,

Learning, assessment and technology: in that order

Keynote address to AMEE conferenceSeptember 2009; Malaga, Spain

Dylan Wiliam

Institute of Education, University of London

www.dylanwiliam.net

Overview of presentationSome theoretical precepts About learning About teaching

• Pedagogies of engagement• Pedagogies of contingency• Pedagogies of formation

The role of technology Supporting, rather than replacing, teachers Classroom aggregation technologies

Some thoughts about supporting teachers in changing practice

How do we improve learning?

What gets learnt?

(Denvir & Brown, 1986)

Key insights from C20th psychologyWhat gets learned as a result of a particular sequence of

instructional activities is impossible to predict, but

Student errors are not random

Conclusions Assessment is the bridge between teaching and learning Teaching is interesting because learners are so different, but only

possible because they are so similar

Learning power environmentsKey concept: Teachers do not create learning Learners create learning

Teaching as the engineering of learning environments

Key features: Create student engagement (pedagogies of engagement) Well-regulated (pedagogies of contingency) Develop habits of mind (pedagogies of formation)

Why pedagogies of engagement?Intelligence is partly inherited So what?

Intelligence is partly environmental Environment creates intelligence Intelligence creates environment

Learning environments High cognitive demand Inclusive Obligatory

264 low and high ability grade 6 students in 12 classes in 4 schools; analysis of 132 students at top and bottom of each class

Same teaching, same aims, same teachers, same class work Three kinds of feedback: scores, comments, scores+comments

[Butler(1988) Br. J. Educ. Psychol., 58 1-14]

Engagement and feedback

Achievement Attitude

Scores no gain High scorers: positive

Low scorers: negative

Comments 30% gain High scorers : positive

Low scorers : positive

[Butler(1988) Br. J. Educ. Psychol., 58 1-14]

Responses

What do you think happened for the students given both scores and comments?

A. Gain: 30%; Attitude: all positiveB. Gain: 30%; Attitude: high scorers positive, low scorers negativeC. Gain: 0%; Attitude: all positiveD. Gain: 0%; Attitude: high scorers positive, low scorers negativeE. Something else

Achievement Attitude

Scores no gain High scorers : positive

Low scorers: negative

Comments 30% gain High scorers : positive

Low scorers : positive

Kinds of feedback (Nyquist, 2003)Weaker feedback only Knowledge of results (KoR)

Feedback only KoR + clear goals or knowledge of correct results (KCR)

Weak formative assessment KCR+ explanation (KCR+e)

Moderate formative assessment (KCR+e) + specific actions for gap reduction

Strong formative assessment (KCR+e) + activity

Effect of formative assessment (HE)

N Effect size

Weaker feedback only 31 0.14

Feedback only 48 0.36

Weaker formative assessment 49 0.26

Moderate formative assessment 41 0.39

Strong formative assessment 16 0.56

(Nyquist, 2003; revised values)

Effects of feedbackKluger & DeNisi (1996) review of 3000 research reports Excluding poorly designed studies left 131 reports, 607 effect

sizes, involving 12652 individuals

On average, feedback increases achievement, but Effect sizes highly variable 38% (50 out of 131) of effect sizes were negative

Engagement in learningAttribution (Dweck, 2000) Personalization (internal vs. external) Permanence (stable vs. unstable) Good learners attribute failure and success to internal, unstable

causes. (It’s down to you, and you can do something about it.)

Views of ‘ability’ Fixed (IQ) Incremental (untapped potential) Essential that teachers inculcate in students a view that ‘ability’ is

incremental rather than fixed (by working, you’re getting smarter).

Dual-pathway theory (Boekaerts, 2006)Long-term learning goals are translated into short-term

learning intentions

Dynamic comparisons of task and situational demands with personal resources

Resulting activation of energy along one of two pathways: Well-being Growth

Motivation: cause or effect?

competence

challenge

Flow

apathyboredom

relaxation

arousalanxiety

worry control

high

low

low high

(Csikszentmihalyi, 1990)

Why pedagogies of contingency?

Fuchs & Fuchs (1986)

Natriello (1987)

Crooks (1988)

Banger-Drowns, et al. (1991)

Kluger & DeNisi (1996)

Black & Wiliam (1998)

Nyquist (2003)

Dempster (1991, 1992)

Elshout-Mohr (1994)

Brookhart (2004)

Allal & Lopez (2005)

Köller (2005)

Brookhart (2007)

Wiliam (2007)

Hattie & Timperley (2007)

Shute (2008)

Hinge-point questionAn experimental study of problem-based learning in undergraduate medical education reports that a result was significant (p<0.05). This means that:A. The experimental group out-performed the control group by at least

5%

B. There is only a 5% chance that the experimental group did not out-perform the control group

C. There is a 5% chance that there is no difference between the experimental group and the control group

D. There is only a 5% chance that the observed result would have happened if the experimental and control groups had the same achievement

Other supports for contingencyAll-student response systems ABCD cards “Exit-pass” questions

Exit-pass questionSummarize the key principles of the following schools of psychology on the appropriate coloured card Associationism (blue) Information processing (orange) Constructivism (red) Situated approaches (green)

Pedagogies of formationInstilling disciplinary habits of mind History Philosophy Statistics

Instilling critical perspectives Values

Three generations of pedagogyFirst generation Traditional pedagogy Negligible contingency

Second generation All student response systems Contingency dependent entirely on teacher skill

Third generation Automated aggregation technologies Contingency supported by technology

Evidence-centred designQuality in assessment is essentially a matter of validity Validity is a property of inferences, not of instruments Assessments should be designed “backwards” from the intended

inferences

Four-process architecture for assessment design Task selection Task presentation Evidence identification Evidence accumulation

Almond, Steinberg and Mislevy (2002)

Task selection/Task presentation

Hinge-point questionWhich of the following is the most important difference between the theories of Piaget and Vygotsky?

A. Piaget places greater importance on the role of conservation in cognitive development

B. Vygotsky places greater importance on the role of cultural artifacts in cognitive development.

C. Vygotsky did not believe in distinct stages of cognitive development.

D. Piaget was a social constructivist while Vygotsky placed greater emphasis on cultural-historical activity theory

Evidence identification

Evidence identificationSingle student response systems

All-student response systems Flash-cards/dry erase boards Classroom ‘clickers’ Traditional keyboards (wired/wireless) Anoto pens

Anoto pen Wireless pen

Special coated paper

Pen ‘knows where it is’

Palm with wireless keyboard

Text-based input

Limited task-presentation capability

Portable

Classroom ‘clickers’ (and their progeny)

Discourse®

www.ets.org/discourse

Evidence identificationAutomated essay scoring (e-rater)

Paraphrase analysers (c-rater)

Graphical and equation analysers (m-rater)

Evidence identification technologies

unstructuredstructured

evidence structure

teacher-mediated

automatedclickers

aggr

egat

ion

ABCDcards

dry-eraseboards

c-rater

Discourse®

latent semantic analysis

m-rater

e-rater

Evidence accumulation

Evidence accumulationUnidimensional student models Useful for summative purposes Almost useless for formative purposesMultidimensional student modelsEvidence-centred design Bayesian inference networks

• Proficiency model• Task model• Evidence model• Student model

Four-process architecture for ECD

Mislevy, Almond and Lukas (2003)

Task selection

Task presentation

Evidence accumulation

Evidence identification

Evidence utilizationWhole-class

Sub-groups Homogenous Heterogenous

Individualization

37

Actually, the technology is the easy partWhat’s hard is changing practice

Telling teachers what to do doesn’t work

Context always intrudes…

38

The ancient yogis used logs of wood, stones, and ropes to help them practise asanas effectively. Extending this principle, Yogacharya Iyengar invented props which allow asanas to be held easily, and for a longer duration without strain.

Yogacharya Iyengar in setubandha sarvangasana

This version of the posture requires considerable strength in the neck, shoulders, and back requiring years of practice to achieve it. It should not be attempted without supervision.

39

A model for teacher learningContent, then process

Content (what we want teachers to change) Evidence Ideas (strategies and techniques)Process (how to go about change) Choice Flexibility Small steps Accountability Support

SummaryLearning has to be done by the learner, not for the learner

Teaching as engineering effective learning environments

Features of effective learning environments Pedagogies of engagement Pedagogies of contingency

• All-student response systems• Classroom aggregation technologies

Pedagogies of formation

Learning milieu focused on growth, rather than well-being

Teachers supported to improve practice continually