Heriot-Watt University The Use of Computerized Peer-Assessment Dr Phil Davies Division of Computing & Mathematical Sciences Department of Computing FAT.

Heriot-Watt UniversityHeriot-Watt University

The Use of Computerized The Use of Computerized Peer-AssessmentPeer-Assessment

Dr Phil DaviesDivision of Computing & Mathematical Sciences

Department of ComputingFAT

University of Glamorgan

General View of Peer AssessmentGeneral View of Peer AssessmentLecturer or Student?Lecturer or Student?

Lectures getting out of doing their jobs

i.e. marking

Good for developing student reflection – So what? Where’s

the marks

How can students be expected to mark

as ‘good’ as ‘experts’

Why should I mark ‘properly’ and waste my time - I get a fixed mark

for doing it

The feedback given by students is not of the same standard

that I give.

I still have to do it again myself to make sure

they marked ‘properly’

Defining Peer-AssessmentDefining Peer-Assessment• In describing the teacher ..

A tall b******, so he was. A tall thin, mean b******, with a baldy head like a light bulb. He’d make us mark each other’s work, then for every wrong mark we got, we’d get a thump. That way – he paused – ‘we were implicated in each other’s pain’

McCarthy’s Bar McCarthy’s Bar (Pete McCarthy, 2000,page (Pete McCarthy, 2000,page 68)68)

AUTOMATICALLY

CREATE A MARK THAT REFLECTS THE QUALITY OF AN ESSAY/PRODUCT

VIA PEER MARKING,

+

A MARK THAT REFLECTS THE QUALITY OF THE

PEER MARKING PROCESS i.e. A FAIR/REFLECTIVE

MARK FOR MARKING AND COMMENTING

What will make Computerized Peer-Assessment ACCEPTABLE to ALL?

THE FIRST CAP MARKING INTERFACE

Typical Assignment ProcessTypical Assignment Process• Students register to use system -

CAP• Create an essay in an area

associated with the module• Provide RTF template of headings• Submit via Bboard Digital Drop-Box• Anonymous code given to essay

automatically by system• Use CAP system to mark

Self/Peer AssessmentSelf/Peer Assessment• Often Self-Assessment stage used

– Set Personal Criteria– Opportunity to identify errors– Get used to system

• Normally peer-mark about 5/6• Raw peer MEDIAN mark produced• Need for student to receive

Comments + Marks

CompensationCompensationHigh and Low MarkersHigh and Low Markers

• Need to take this into account• Each essay has a ‘raw’ peer generated

mark - MEDIAN• Look at each student’s marking and

ascertain if ‘on average’ they are an under or over marker

• Offset mark given by this value• Create a COMPENSATED PEER MARK• It’s GOOD TO TALK – Tim Nice but Dim

EMAILS THE MARKER ..

ANONYMOUS

Below are comments given to students.Below are comments given to students.Select the 3 most Select the 3 most ImportantImportant to to YOUYOU

1. I think you’ve missed out a big area of the research

2. You’ve included a ‘big chunk’ - word for word that you haven’t cited properly

3. There aren’t any examples given to help me understand

4. Grammatically it is not what it should be like5. Your spelling is atroceious6. You haven’t explained your acronyms to me7. You’ve directly copied my notes as your answer

to the question8. 50% of what you’ve said isn’t about the question9. Your answer is not aimed at the correct level of

audience10.All the points you make in the essay lack any

references for support

Order of AnswersOrder of Answers• Were the results all in the ‘CORRECT’ order –

probably not? -> Why not!

• Subject specific?• Level specific – school, FE, HE• Teacher/Lecturer specific?

• Peer-Assessment is no different – Objectivity through Subjectivity

• Remember – Feedback Comments as important as marks!

• Students need to be rewarded for marking and commenting WELL -> QUANTIFY COMMENTS

Each Student is using a different set of weighted comments

Comments databases sent to tutor

First Stage => Self Assess own Work

Second Stage (button on server) => Peer Assess 6 Essays

Comments – Both Positive and Negative

in the various categories. Provides a Subjective Framework

for Commenting & Marking

Feedback IndexFeedback Index• Produce an index that reflects the

quality of commenting• Produce a Weighted Feedback Index

• Compare how a marker has performed against these averages per essay for both Marking + Commenting – Looking for consistency

-5 -4 -3 -2 -1 -0 +0 1 2 3 4 5 6 7 8 9

29 44 41 49 46 53 64 49 53 60 62 69 68 69 82

38 48 47 51 45 54 58 53 62 62 64 65 73

49 51 50 60 57 57 67 66

51 58 53 50 59

57 63

59 65

64

0 4.2 5.0 1.4 3.5 4.0 6.8 4.8 3.6 3.9 4.7 2.5 3.1 2.8 0

29 41 45 48 49 49 56 52 56 58 59 67 64 71 82

The Review ElementThe Review Element• Requires the owner of the file to ‘ask’

questions of the marker• Emphasis ‘should’ be on the marker• Marker does NOT see comments of other

markers who’ve marked the essays that they have marked

• Marker does not really get to reflect on their own marking – get a reflective 2nd chance

• I’ve avoided this in past -> get it right first time

Used on Final Year Degree + Used on Final Year Degree + MScMSc

MSc EL&A• 13 students • 76 markings• 41 replaced markings (54%)• Average time per marking = 42 minutes• Range of time taken to do markings 3-72 minutes• Average number of menu comments/marking = 15.7• Raw average mark = 61%• Out of 41 Markings ‘replaced’ –> 26 changed mark 26/76

(34%)• Number of students who did replacements = 8 (out of 13)• 2 students ‘Replaced’ ALL his/her markings• 26 markings actually changed mark • -1,+9, -2,-2, +1, -8, -3,-5, +2, +8, -2, +6, +18(71-89), -1, -

4, -6, -5, -7, +7, -6, -3, +6, -7, -7, -2, -5 (Avge -0.2)

How to work out Mark (& How to work out Mark (& Comment) ConsistencyComment) Consistency

• Marker on average OVER marks by 10%• Essay worth 60%• Marker gave it 75%• Marker is 15% over• Actual consistency index (Difference) = 5

• This can be done for all marks and comments

• Creates a consistency factor for marking and commenting

Automatically Generate Automatically Generate Mark for MarkingMark for Marking

• Linear scale 0 -100 mapped directly to consistency … the way in HE?

• Map to Essay Grade Scale achieved (better reflecting ability of group)?

• Expectation of Normalised Results within a particular cohort / subject / institution?

Current ‘Simple’ MethodCurrent ‘Simple’ Method• Average Marks

– Essay Mark = 57%– Marking Consistency = 5.37

• Ranges– Essay 79% <-> 31%– Marking Consistency 2.12 <-> 10.77

• Range Above Avge 22% <-> 3.25 (6.76=1)

• Range Below Avge 26% <-> 5.40 (4.81=1)

ALT-J journal entitled ‘ALT-J journal entitled ‘Don’t Write, Just Mark; Don’t Write, Just Mark; The Validity of Assessing Student Ability via The Validity of Assessing Student Ability via

their Computerized Peer-Marking of an their Computerized Peer-Marking of an Essay rather than their Creation of an Essay’ Essay rather than their Creation of an Essay’

ALT-J (CALT) Vol. 12 No. 3 , pp 263-279.ALT-J (CALT) Vol. 12 No. 3 , pp 263-279.

• Took a Risk• No necessity to write an essay• Judged against previous essays from past

– knew mark and feedback index• NO PLAGIARISM opportunity• Worked really well

Some Points Outstanding or Some Points Outstanding or Outstanding PointsOutstanding Points

• What should students do if they identify plagiarism?• What about accessibility?• Is a computerised solution valid for all subject

areas?• At what age / level can we trust the use of peer

assessment?• How do we assess the time required to perform the

marking task?• What split of the marks between creation &

marking

SummarySummary

• Research / Meeting Pedagogical Needs / Improving relationship between assessment & learning– Keep asking yourself WHY & WHAT am I assessing?

• DON’T LET THE TECHNOLOGISTS DRIVE THE ASSESSMENT PROCESS!! i.e. Lunatics taking over the asylum

• Don’t let the ‘artificial’ need to adhere to standards & large scale implementation of on-line assessment be detrimental to the assessment and learning needs of you and your students. ‘Suck it and see’.

• By using composite e-assessment methods are then able to assess full range of learning + assessment

• Favourite Story to finish

Personal Reference SourcesPersonal Reference Sources

• CAA Conference Proceedings http://www.caaconference.com/

• ‘Computerized Peer-Assessment’, Innovations in Education and Training International Journal (IETI), 37,4, pp 346-355, Nov 2000

• ‘Using Student Reflective Self-Assessment for Awarding Degree Classifications’, Innovations in Education and Training International Journal (IETI), 39,4, pp 307-319, Nov 2002.

• ALT-J journal entitled ‘Closing the Communications Loop on the Computerized Peer Assessment of Essays’, 11, 1, pp 41-54, 2003.

• ALT-C 2003 Research stream paper, Peer-Assessment: No marks required, just feedback, Sheffield University, Sept 2003.

• ALT-J journal entitled ‘Don’t Write, Just Mark; The Validity of Assessing Student Ability via their Computerized Peer-Marking of an Essay rather than their Creation of an Essay’ ALT-J (CALT) Vol. 12 No. 3 , pp 263-279.

• Peer-Assessment: Judging the quality of student work by the comments not the marks?, Innovations in Education and Teaching International (IETI), 43, 1, pp 69-82, 2006.

Contact InformationContact [email protected]

Phil Davies J317

01443 482247University of Glamorgan