30 Years of Evidence on the Comparability of Exam Standards: Myths, Fiascos and Unrealistic Expectations Paul E. Newton Centre for Evaluation & Monitoring, University of Durham, 30th Anniversary Conference: 30 Years of Evidence in Education. 23 September 2014. London.
33
Embed
30 Years of Evidence on the Comparability of Exam Standards: Myths, Fiascos and Unrealistic Expectations Paul E. Newton Centre for Evaluation & Monitoring,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
30 Years of Evidence on the Comparability of Exam Standards: Myths, Fiascos and Unrealistic Expectations
Paul E. NewtonCentre for Evaluation & Monitoring, University of Durham, 30th Anniversary Conference: 30 Years of Evidence in Education. 23 September 2014. London.
Statistics vs. Judgement:What Does 30 Years of Research Tell Us About the Best and Worst Way to Maintain Exam Standards?
What does it mean to ‘maintain’ an exam standard?
Grade Awarding
The process of identifying: which marks on this year’s exam correspond to levels of attainment
(i.e. levels of knowledge, skill and understanding) that were associated with
grade boundary marks on last year’s exam.
Why do exam boards need to move grade boundaries?
Because even exams that are designed to measure: exactly the same kind of attainment in exactly the same way may end up being slightly different in terms of the
overall difficulty of their questions
Have we always maintained exam standards like this?
30 years ago – in 1984? 60 years ago – in 1954?
Have we always maintained exam standards like this?
30 years ago – in 1984? 60 years ago – in 1954?
… yes, pretty much!
Attainment-referencing
From one examination to the next, corresponding grade boundaries should be located at marks associated with equivalent levels of attainment.
The myth
HYPOTHETICAL A level pass-rates for UCLES(Summer examinations, Home candidates only)
0
10
20
30
40
50
60
70
80
90
100
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
Latin
French
Physics
Biology
The myth… debunked
50
55
60
65
70
75
80
85
90
95
100
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
A level pass-rates for the 'Cambridge' boardUCLES (1960 to 1984)
Cumulative percentage of A level Sociology students awarded grade E (blue)against total number of results awarded (red)
(for All Boards, Summer Awards, All Modes, by Syllabus Group)
Cum.% E
No. Results
Unrealistic expectations
Three ‘stages’ in understanding comparability
1. statistical auditing problems are routine solutions require ‘back of the envelope’ sums
2. scientific research problems are difficult solutions require rigorous and objective investigations
3. art criticism problems are perhaps insurmountable solutions require value judgements
(Bardell, Forrest and Shoesmith, 1978)
Realistic expectations +Persuasive justifications
Four ‘stages’ in understanding comparability1. statistical auditing
2. scientific research
3. art criticism
4. engineering pragmatism many comparability problems are technically insurmountable…
but some are less insurmountable than others and should be prioritised
all comparability solutions are inevitably imperfect… but some are less imperfect than others and should be prioritised
technically insurmountable problems and inevitably imperfect solutions highlight the fundamental importance of strong arguments in defence of policy and practice