Assessment as a washback tool: is it beneficial or harmful? Nick Saville Director, Research and Validation University of Cambridge ESOL Examinations October 2008
Dec 16, 2015
Assessment as a washback tool:
is it beneficial or harmful?Nick Saville
Director, Research and ValidationUniversity of Cambridge
ESOL Examinations
October 2008
2Nick Saville, Cambridge ESOL
Outline
1. Introduction
2. Washback and impact: some definitions
3. Impact and the law of unintended
consequences
4. Researching impact: the Cambridge ESOL
approach
5. Conclusion
3Nick Saville, Cambridge ESOL
1. Introduction
• Examinations• access to opportunity
• fairness
• Exert a powerful influence on educational processes• possibly negative?
• concern for social consequences, ethicality, accountability
4Nick Saville, Cambridge ESOL
Questions to be addressed:
• How can educators and assessment providers foster beneficial effects from their tests and examinations?
• How can harmful consequences be avoided or their impact mitigated?
5Nick Saville, Cambridge ESOL
“Impact by design”
• Integral part of a framework for developing and validating examination systems
• A concept akin to social impact assessment
6Nick Saville, Cambridge ESOL
2. Washback and impact: definitions
• Washback (or backwash) has been broadly defined in the assessment literature as the effect of testing on teaching and learning
• One aspect of the broader phenomenon known as impact – as we shall see later
7Nick Saville, Cambridge ESOL
Washback
• Alderson and Wall, 1993:• 15 washback hypotheses • Based on who or what might be affected:
• Teaching• Learning • Content• Rate of learning• Sequence of teaching/learning• Degree/depth of curriculum coverage• Attitudes of teachers/learners• Etc.
8Nick Saville, Cambridge ESOL
Washback
• A continuum - stretching from harmful at one end, through neutral to beneficial at the other end
Negative Neutral Positive
- +
9Nick Saville, Cambridge ESOL
Washback
• Negative?• Restriction of content – narrowing of curriculum• Too much time practising for the test
• Positive?• Transparent objectives and outcomes• Increased motivation of learners• Increased accountability of teachers (?)
• BUT – cause and effect explanations are rarely adequate …..
10Nick Saville, Cambridge ESOL
Washback Models
In the language testing literature:
• Bailey (1996)
• Watanabe (2004)
• Cheng (2004, 2005)
• Green (2007)
11Nick Saville, Cambridge ESOL
Bailey’s Model (1996) - based on Hughes,1993
3 Ps:
Participants• students• teachers
Processes
Products• learning• teaching• materials• curricula
12Nick Saville, Cambridge ESOL
Watanabe’s Model (2004)
• Five dimensions:
• Specificity
• Intensity
• Length
• Intentionality
• Value
• Factors influencing the process of washback are
related to:• the test itself
• status
• stakeholders
13Nick Saville, Cambridge ESOL
FocalConstruct
Test designcharacteristics
item formatcontent
complexityetc.
Overlap
Potential fornegative backwash
Potential forpositive backwash
Perception oftest importance
Perception oftest difficulty
Backwash toparticipant
Important
Unimportant
No backwash
Intense backwash
Easy
Unachievable
Challenging
Washback direction
Washback intensity
Washback variability
Participant characteristics and values Knowledge/ understanding of test demands Resources to meet test demands Acceptance of test demands
Other stakeholdersCourse providersMaterials writers
PublishersTeachersLearners
Green’s Model See: Studies in Language Testing, 25,
2007: IELTS Washback in Context
Washback will be most intense –
have the most powerful effects
on teaching and learning
behaviours – where participants
see the test as challenging and
the results as important (high
stakes)
SEE BLUE ARROW
14Nick Saville, Cambridge ESOL
Impact
Impact concerns the effects and consequences a test can have beyond the classroom and immediate learning context:
• On individual career or life chances
• In educational systems and in society
15Nick Saville, Cambridge ESOL
Impact
• Impact deals with wider influences and includes the “macro contexts” - tests and examinations in society
• Washback is an aspect of impact related to the “micro contexts” of the classroom and the school
16Nick Saville, Cambridge ESOL
• Investigating impact is integral to validation
• an essential component in establishing the usefulness of an
assessment system - fitness for specific purposes and
contexts of use
• Consistent with Messick views of validity (1989, 1996)• Consequential aspects of validity
• fairness and ethics
Impact
17Nick Saville, Cambridge ESOL
• Educational systems – complex phenomena
• Stakeholder constituencies • many different stakeholders in educational processes
• complex network of relationships
Impact
18Nick Saville, Cambridge ESOL
LearnersTeachersTest writers/examiners Receiving institutionsSchool ownersFuture employersGovernment agenciesProfessional bodiesTest centre administratorsMaterials writersPublishersetc
Learners Parents/carersTeachersReceiving institutions EmployersSchool ownersExaminersGovernment agenciesProfessional bodiesAcademic researchersTest writers/Examinersetc
Test constructs
Test format
Test conditions
Test assessment
criteria
Test scores
Stakeholders in the testing constituency
Testing System
Contexts of test use - consequencesInputs to test design
19Nick Saville, Cambridge ESOL
LearnersTeachersTest writers/examiners Receiving institutionsSchool ownersFuture employersGovernment agenciesProfessional bodiesTest centre administratorsMaterials writersPublishersetc
Learners Parents/carersTeachersReceiving institutions EmployersSchool ownersExaminersGovernment agenciesProfessional bodiesAcademic researchersTest writers/Examinersetc
Test constructs
Test format
Test conditions
Test assessment
criteria
Test scores
Testing System
Contexts of test use - consequencesInputs to test design
Stakeholders in the testing constituency
20Nick Saville, Cambridge ESOL
LearnersTeachersTest writers/examiners Receiving institutionsSchool ownersFuture employersGovernment agenciesProfessional bodiesTest centre administratorsMaterials writersPublishersetc
Learners Parents/carersTeachersReceiving institutions EmployersSchool ownersExaminersGovernment agenciesProfessional bodiesAcademic researchersTest writers/Examinersetc
Test constructs
Test format
Test conditions
Test assessment
criteria
Test scores
Testing System
Contexts of test use - consequencesInputs to test design
Stakeholders in the testing constituency
21Nick Saville, Cambridge ESOL
Impact
• Multiple voices
• views, beliefs and attitudes
• Important for examination providers • to collect feedback from the stakeholder• to take stakeholder perspectives into account
• Dynamic relationship between micro and macro contexts
22Nick Saville, Cambridge ESOL
3. The “law” of unintended consequences
• “Any purposeful action will produce some unintended consequences” or side-effects
• “Goodhart’s Law” (or “Campbell’s Law” in the USA)• a variant of the “law” of unintended consequences
23Nick Saville, Cambridge ESOL
“Goodhart’s Law”
• “All performance indicators lose their meaning when adopted as policy targets”
• Examples:• England - school achievement targets - school league tables
• USA – No Child Left Behind (NCLB)
• The clearer you are about what you want, the more likely you are to get it – but the less likely it is to mean what you wanted it to! (Dylan Wiliam, 2008)
24Nick Saville, Cambridge ESOL
Perverse incentives?
• Assessment policy can create a tension between
• educational objectives at the micro level (teaching and learning in schools) and
• a requirement for accountability at the macro level
25Nick Saville, Cambridge ESOL
What role can examination providers play?
How can examination providers ensure that :
• examination systems are “fit for purpose”?
• research is carried out to “find out what is going on” in contexts of use?
• mitigating action is are carried out if/when negative effects and consequences occur?
26Nick Saville, Cambridge ESOL
4. Researching impact
• Cambridge ESOL
• an international examinations board
• Maxims of Test Impact
• An attempt to integrate an action-oriented approach to investigating impact into working practices
• Milanovic and Saville, 1996
27Nick Saville, Cambridge ESOL
Maxims for achieving/monitoring impact
Maxim 1 PLANUse a rational and explicit approach to test development
Maxim 2 SUPPORTSupport stakeholders in the testing process
Maxim 3 COMMUNICATEProvide comprehensive, useful and transparent information
Maxim 4 MONITOR and EVALUATECollect all relevant data and analyse as required.
Milanovic and Saville, 1996 - Considering the impact of the Cambridge examinations
28Nick Saville, Cambridge ESOL
Maxim 1 PlanUse a rational and explicit approach to test development
• a process model – cyclical and iterative
• creates the necessary conditions for appropriate tests to be developed and for on-going validation to take place
• begins with the purpose - including anticipating • how the test should (or might) be used
• how relevant and useful it is likely to be
- social consequences and value implications
• potential (unplanned) side-effects
29Nick Saville, Cambridge ESOL
Maxim 2 Support
Support stakeholders in the testing process
• Involvement of stakeholders - during test design and development• consultation on specifications/syllabus design• participation in pilot tests
• Professional support programmes• for institutions and individual teachers/students etc.
• Training of suitable personnel to work on all aspects of the examination cycle• item writers, examiners, etc.
30Nick Saville, Cambridge ESOL
LearnersTeachersTest writers/examiners Receiving institutionsSchool ownersFuture employersGovernment agenciesProfessional bodiesTest centre administratorsMaterials writersPublishersetc
Learners Parents/carersTeachersReceiving institutions EmployersSchool ownersExaminersGovernment agenciesProfessional bodiesAcademic researchersTest writers/Examinersetc
Test constructs
Test format
Test conditions
Test assessment
criteria
Test scores
Stakeholders in the testing constituency
Testing System
Contexts of test use - consequencesInputs to test design
31Nick Saville, Cambridge ESOL
Maxim 3 Communicate
Provide comprehensive, useful and transparent information
• Explain issues related to assessment adapted for the different stakeholders • the nature of the language constructs being assessed
• the meaning of language test results
• etc.
• A major challenge for all test providers!
32Nick Saville, Cambridge ESOL
Maxim 4 Monitor and EvaluateCollect all relevant data and analyse as required
• For example, feedback from the candidates and their teachers • from local contexts where the test is used (i.e. at the micro level)
• A long-term endeavour as it involves:• the development of suitable instruments for the collection of adequate
data• appropriate research methodologies
e.g. mixed method designs, case studies etc.
• Evaluate the test’s usefulness routinely• determine the need for changes and periodic revisions
33Nick Saville, Cambridge ESOL
Outcomes of impact studies
• The World-wide survey of the impact of IELTS
• The Italian Progetto Lingue 2000 (PL2000 project)
• See Hawkey, (2006) • Studies in Language Testing, 24
The theory and practice of impact studies: Messages from studies of the IELTS test and Progetto Lingue 2000
34Nick Saville, Cambridge ESOL
Progetto Lingue 2000
• The policy was
• “.... to introduce innovation into the teaching and learning of other
languages by putting greater emphasis on the development of
communicative competence in all grades of the school system”
• It included:
• the adoption of the Council of Europe’s Common European Framework
of Reference (CEFR) as the basis for learning objectives and standards
• certification of proficiency – the testing
• (by a certificating body recognised internationally)
35Nick Saville, Cambridge ESOL
Italy - PL2000 Impact Project
Main interdependent language programme stakeholders and dimensions
Learning
goals,
curriculum,
syllabus
Students
Parents
Teachers
Teacher-trainers
Curriculum developers
Testers
Publishers
Receiving institutions
Employers
Students
Parents
Teachers
Teacher-trainers
Curriculum developers
Testers
Publishers
Receiving institutions
Employers
Materials
Methodology Teacher Support
Testing
36Nick Saville, Cambridge ESOL
5. Conclusion• Impact research now reflects the growing importance of
evidence-based approaches to educational planning
and evaluation
• By adopting an “impact by design” approach - through
careful test development and validation strategies, and
by using the findings from impact research to guide
future actions - more effective assessment policies
and practices can be developed to meet the needs of
contemporary education