SCROLLA Symposium 6 Feb 2002 1 ICT in Assessment and Learning Developments from the Enigma Project Robert Harding Nick Raikes ITAL Unit Interactive Technologies in Assessment and Learning
Feb 25, 2016
SCROLLA Symposium 6 Feb 2002 1
ICT in Assessment and Learning
Developments from the Enigma Project
Robert HardingNick Raikes
ITAL UnitInteractive Technologies in Assessment and Learning
SCROLLA Symposium 6 Feb 2002 2
Introduction
ICT-led innovation:Exciting image……but assessment is Cinderella!
Holistic nature of ‘The Learning System’ :Assessment is a goal for teachers and learners……but valid assessment must be rooted in learningHow will new methods fit with existing practice?The “Three-legged race”.
SCROLLA Symposium 6 Feb 2002 3
Outline The Enigma Project trials
Scope User interface
Evaluation: ‘Traditional’ style questions (conceptual) ‘New interactive’ style questions (analytical) Administrative issues
Developments triggered as a result Brief resumé of some resulting work Closer look at one aspect:
on-screen marking Conclusion.
SCROLLA Symposium 6 Feb 2002 4
The Enigma Project- Scope Two years, 3+2 schools, about 170+120 pupils Pilot 1: O-Level Physics examination in paper and CBT forms
A few graphics manipulation items Pilot 2: Selection of O-Level Science in CBT form
Conceptual ‘objective’ items Analytical simulation+free text answer items
Marking: Objective/multiple choice: automatic Open answers: printed out, marked by hand
Comparative evaluation of pupils performance.
SCROLLA Symposium 6 Feb 2002 5
User Interface
Navigation
Time count-down
Progress indicator
Candidate bookmark
Audio feature for questions.
SCROLLA Symposium 6 Feb 2002 7
User Interface
Navigation
Time count-down
Progress indicator
Candidate bookmark
Audio feature for questions.
SCROLLA Symposium 6 Feb 2002 8
Evaluation - ‘traditional’
Pilot 1: the evaluators concluded: For MCQ’s, no obvious differences between groups (3%) For open-ended, those on paper theory Q’s better than CBT
(11%)
Pilot 2: poor correlation between pilot and real examinations: Was computer aptitude responsible? Was it mismatch of trial vs real examination?.
SCROLLA Symposium 6 Feb 2002 9
Evaluation - ‘analytical’ Objective - how do students react to ICT oriented
question styles? Use of simulations 6 Q’s: 2 each Physics, Chemistry, Biology Panel of examiners Typical example follows …note:
access to simulation, free text response boxes ability to record student actions.
SCROLLA Symposium 6 Feb 2002 12
Evaluation - ‘analytical’ Student responses:
Only 1 in 6 failed to understand what was required of them… …but 40% said ‘occasions when unsure what to do’.
(but how true of most of us in many situations!)
Poor correlations between this test and ‘real’ examination
BUT … 70% said: “computer based tests are as fair as conventional tests” Computer literacy cited as most common reason for bias.
SCROLLA Symposium 6 Feb 2002 13
Evaluation - overall 40% students said the computer slowed them down:
Interpretation: they are unfamiliar with computers in schoolwork Conclude that learning and assessment must be integrated
40% said computer stopped them answering as wanted Most common complaint - not enough space to write Did not like not being allowed to go back and change answer
70% thought CBT’s to be as fair as paper-based Reasons why not so fair: typing speed and literacy (lack of).
SCROLLA Symposium 6 Feb 2002 14
Administrative issues Schools’ normal IT systems can interact with
assessment software Individual system crashes - incremental backup
essential High level of expert attention was needed Is printing needed? Security - visibility of screens At least one candidate observed using email! Issues of accessibility to other software and data Interference with normal working must be minimised.
SCROLLA Symposium 6 Feb 2002 15
What developed? - Analysis We have not seen a surge of CBT use in schools:
Material circumstances and resources Holistic nature of changes needed
Cast of influences
Can you ‘box off’ summative assessment?
Traditional links between assessment and teaching
Examiner-teachers set questions, not Boards
Feedback loops: Learning process geared to passing examinations Examinations are rooted in ‘educational ambience’.
SCROLLA Symposium 6 Feb 2002 16
What we did - ‘ambience’ Teacher support for using ICT - TEEM
http://www.teem.org.uk/
Syllabus support e-Lists and electronic communities Electronic learning resources:
e.g. Enigma simulations on the www http://www.cie.org.uk/learning/science_practicals/ .
SCROLLA Symposium 6 Feb 2002 17
What we did - assessment
CALM / CUE / EQL collaborationMEI A-Level Maths examinationsIMS standards and ‘QTI’ workgroup
Question test interoperability
Standards for Examination conductOn-screen marking.
SCROLLA Symposium 6 Feb 2002 18
On-screen marking
Scan scripts, or capture electronic scriptsWhat do we want it for?
Faster, more flexible marking, management More efficient quality control Better feedback to Centres Transition to on-line assessment.
SCROLLA Symposium 6 Feb 2002 19
Study - ‘proof of concept’Actual, live scripts::
O-Level: Maths (350) A-Level: Geography (900), Eng Lit (300)
5 examiners per subjectConventional and screen markedWhole script, marking by questionDownload scripts via Internet.
SCROLLA Symposium 6 Feb 2002 20
Features of the software
Ticks and crossesAnchored commentsDisplay control - e.g. zoom, script scrollView marking guideNavigation between question or scriptProgress monitor and referal.
SCROLLA Symposium 6 Feb 2002 21
Examiners’ impressions
Generally managed the downloadingScripts at least as legible as on paperMost felt they gave same marks:
Exceptions in English Lit & Geography ?’s on trial nature, annotation, whole script Maths points re marking by question
All would mark on-screen again.
SCROLLA Symposium 6 Feb 2002 22
Analysis of marksMaths component - consistencyGeography - mostly satisfactory:
one examiner more severe on screen one consistent on paper but not on screen
English: 2 more severe on screen All more consistent on paper than screen Possible link with ‘whole script’ judgement?.
SCROLLA Symposium 6 Feb 2002 23
Conclusions
Holistic nature of system ‘Three-legged race’ - or more?
Central role of teachersChallenge:
Integrate, make the technology invisible
Way forward - ‘open source’?.
SCROLLA Symposium 6 Feb 2002 24
Some URL’s and email ITAL Unit
http://ital.ucles-red.cam.ac.uk/ Teacher support for using ICT - TEEM
http://www.teem.org.uk/ Enigma simulations on the www
http://www.cie.org.uk/learning/science_practicals/
Robert Harding <[email protected]> Nick Raikes <[email protected]> .