Review in Computerized Review in Computerized Peer-Assessment Peer-Assessment Dr Phil Davies Dr Phil Davies Department of Computing Department of Computing Division of Computing & Division of Computing & Mathematical Sciences Mathematical Sciences FAT FAT University of Glamorgan University of Glamorgan
48
Embed
Review in Computerized Peer- Assessment Dr Phil Davies Department of Computing Division of Computing & Mathematical Sciences FAT University of Glamorgan.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Review in Computerized Review in Computerized Peer-AssessmentPeer-Assessment
Dr Phil DaviesDr Phil DaviesDepartment of ComputingDepartment of ComputingDivision of Computing & Division of Computing & Mathematical SciencesMathematical Sciences
FATFATUniversity of GlamorganUniversity of Glamorgan
AUTOMATICALLY
CREATE A MARK THAT REFLECTS THE QUALITY OF AN ESSAY/PRODUCT
VIA PEER MARKING,
What do we need to provide to have fully Automated Peer –Assessment System?
AUTOMATICALLY
A MARK THAT REFLECTS THE QUALITY OF THE
PEER MARKING PROCESS i.e. A FAIR/REFLECTIVE
MARK FOR MARKING AND COMMENTING
SLIGHT GAP
THE FIRST CAP MARKING INTERFACE
Typical Assignment ProcessTypical Assignment Process Students register to use system - Students register to use system -
CAPCAP Create an essay in an area Create an essay in an area
associated with the module using an associated with the module using an RTF template of headingsRTF template of headings
Submit via Bboard Digital Drop-BoxSubmit via Bboard Digital Drop-Box Anonymous code given to essay Anonymous code given to essay
automatically by systemautomatically by system Use CAP system to markUse CAP system to mark
Self/Peer AssessmentSelf/Peer Assessment Often Self-Assessment stage usedOften Self-Assessment stage used
Set Personal CriteriaSet Personal Criteria Opportunity to identify errorsOpportunity to identify errors Get used to systemGet used to system
Normally peer-mark about 5/6Normally peer-mark about 5/6 Raw peer MEDIAN mark producedRaw peer MEDIAN mark produced Need for student to receive Need for student to receive
Comments + Comments + MarksMarks
CompensationCompensationHigh and Low MarkersHigh and Low Markers
Need to take this into accountNeed to take this into account Each essay has a ‘raw’ peer generated Each essay has a ‘raw’ peer generated
mark - MEDIANmark - MEDIAN Look at each student’s marking and Look at each student’s marking and
ascertain if ‘on average’ they are an under ascertain if ‘on average’ they are an under or over markeror over marker
Offset mark given by this valueOffset mark given by this value Create a Create a COMPENSATED PEER MARKCOMPENSATED PEER MARK
Below are comments given to students.Below are comments given to students.Select the 3 most Select the 3 most ImportantImportant to to YOUYOU
1.1. I think you’ve missed out a big area of the I think you’ve missed out a big area of the researchresearch
2.2. You’ve included a ‘big chunk’ - word for word You’ve included a ‘big chunk’ - word for word that you haven’t cited properlythat you haven’t cited properly
3.3. There aren’t any examples given to help me There aren’t any examples given to help me understandunderstand
4.4. Grammatically it is not what it should be likeGrammatically it is not what it should be like5.5. Your spelling is atroceiousYour spelling is atroceious6.6. You haven’t explained your acronyms to meYou haven’t explained your acronyms to me7.7. You’ve directly copied my notes as your answer You’ve directly copied my notes as your answer
to the questionto the question8.8. 50% of what you’ve said isn’t about the question50% of what you’ve said isn’t about the question9.9. Your answer is not aimed at the correct level of Your answer is not aimed at the correct level of
audienceaudience10.10. All the points you make in the essay lack any All the points you make in the essay lack any
references for supportreferences for support
Each Student is using a different set of weighted comments
Comments databases sent to tutor
First Stage => Self Assess own Work
Second Stage (button on server) => Peer Assess 6 Essays
Comments – Both Positive and Negative
in the various categories. Provides a Subjective Framework
for Commenting & Marking
Feedback IndexFeedback Index
Produced an index that reflects the quality Produced an index that reflects the quality of commentingof commenting
Produced a Weighted Feedback IndexProduced a Weighted Feedback Index
Compare how a marker has performed Compare how a marker has performed against these averages per essay for both against these averages per essay for both Marking + Commenting – Looking for Marking + Commenting – Looking for consistencyconsistency
Originally in Communications within CAP Originally in Communications within CAP marking process, it requires the owner of marking process, it requires the owner of the file to ‘ask’ questions of the markerthe file to ‘ask’ questions of the marker
Emphasis ‘should’ be on the markerEmphasis ‘should’ be on the marker Marker does NOT see comments of other Marker does NOT see comments of other
markers who’ve marked the essays that markers who’ve marked the essays that they have markedthey have marked
Marker does not really get to reflect on their Marker does not really get to reflect on their own marking – get a reflective 2own marking – get a reflective 2ndnd chance chance
I’ve avoided this in past -> get it right first I’ve avoided this in past -> get it right first timetime
Click on button to get an essay
previously marked +
comments and marks
Click on button to get to view comments of
another marker
Can change any marks
and/or comments they feel
appropriate and submit by clicking button
Trialled with Post-Graduate Trialled with Post-Graduate GroupGroup
13 students 13 students 76 markings76 markings Average time per marking = 42 minutes (range 3-Average time per marking = 42 minutes (range 3-
72)72) Average number of menu comments/marking = Average number of menu comments/marking =
15.715.7 Peer Avge. mark = 59.69% (before review 60.15%)Peer Avge. mark = 59.69% (before review 60.15%) Number of students who did replacements = 10 Number of students who did replacements = 10
(out of 13) (out of 13) 41 ‘Replaced’ markings (54%) 41 ‘Replaced’ markings (54%) Out of 41 Markings ‘Replaced’ –> 26 changed mark Out of 41 Markings ‘Replaced’ –> 26 changed mark
26/76 (34%)26/76 (34%) Only 33 out of 41 REALLY CHANGED ANYTHINGOnly 33 out of 41 REALLY CHANGED ANYTHING 2 students ‘Replaced’ ALL his/her markings2 students ‘Replaced’ ALL his/her markings
How to work out Mark (& How to work out Mark (& Comment) ConsistencyComment) Consistency
Marker on average Marker on average OVEROVER marks by marks by 1010%% Essay worth Essay worth 6060%% Marker gave it Marker gave it 7575%% Marker is Marker is 1515% over% over Would Would expect 10% overexpect 10% over, therefore Actual , therefore Actual
Consistency index (Difference) = Consistency index (Difference) = 55 If the marker on average had If the marker on average had UNDERUNDER marked by marked by
1010% - Difference would have been 2% - Difference would have been 255 Summing and Averaging these differences Summing and Averaging these differences
produces a Marking Consistency Index (low is produces a Marking Consistency Index (low is good – high is poor)good – high is poor)
This can be done for all marks and commentsThis can be done for all marks and comments
Automatically Generate Automatically Generate Mark for MarkingMark for Marking
Linear scale 0 -100 mapped directly to Linear scale 0 -100 mapped directly to consistency … the way in HE?consistency … the way in HE?
Expectation of Normalised Results within Expectation of Normalised Results within a particular cohort / subject / institution?a particular cohort / subject / institution?
Map to Essay Grade Scale achieved Map to Essay Grade Scale achieved (better reflecting ability of group)?(better reflecting ability of group)?
Current ‘Simple’ methodCurrent ‘Simple’ method Average mark for essay e.g. 55%Average mark for essay e.g. 55% Ranges Highest – Lowest marks Ranges Highest – Lowest marks
achieved for essay e.g. 45% <-> 70%achieved for essay e.g. 45% <-> 70% Average Marking Consistency e.g. 5.0Average Marking Consistency e.g. 5.0 Ranges Highest – Lowest consistency Ranges Highest – Lowest consistency
% Mark for % Mark for Marking based Marking based on range diff. on range diff.
(Avge 59.69%)(Avge 59.69%)
Mark they Mark they received for received for
essayessay
11 5.595.59 + 0 .22+ 0 .22 59%59% 72%72%
22 4.554.55 - 0.82- 0.82 65%65% 68%68%
33 2.312.31 - 3.06- 3.06 81%81% 53%53%
44 4.174.17 - 1.20- 1.20 68%68% 51%51%
5
66 7.597.59 + 2 .22+ 2 .22 53%53% 52%52%
77 10.7810.78 + 5.41+ 5.41 43%43% 62%62%
88 4.544.54 - 0.83- 0.83 65%65% 81%81%
9
1010 5.995.99 + 0.62+ 0.62 58%58% 61%61%
1111 3.313.31 - 2.06- 2.06 74%74% 73%73%
1212 6.676.67 + 1.30+ 1.30 56%56% 43%43%
1313 3.733.73 - 1.64- 1.64 71%71% 66%66%
What about the What about the commenting?commenting?
Does not take into account the Quality Does not take into account the Quality of the Commentingof the Commenting
Should look at the Average Feedback Should look at the Average Feedback Differences per marker to get a Differences per marker to get a Commenting Consistency GradeCommenting Consistency Grade
Same as creating Mark ConsistencySame as creating Mark Consistency
Create a Commenting Consistency Create a Commenting Consistency IndexIndex
Student Student NumberNumber
Average Average Feedback Feedback DifferenceDifference
Feedback Feedback ConsistencyConsistency
Avge 2.80Avge 2.80
Consistency Consistency Difference Difference
from from Average Average
% Mark for % Mark for Comments based Comments based
on range diff. on range diff. (Avge 59.69%)(Avge 59.69%)
Mark they Mark they received for received for
essayessay
11 -2.10-2.10 3.053.05 +0.25+0.25 59%59% 72%72%
22 0.750.75 2.532.53 -0.27-0.27 63%63% 68%68%
33 4.434.43 2.942.94 +0.14+0.14 59%59% 53%53%
44 -1.30-1.30 2.212.21 -0.59-0.59 65%65% 51%51%
5
66 4.384.38 1.591.59 -1.21-1.21 75%75% 52%52%
77 -6.00-6.00 7.237.23 +4.43+4.43 43%43% 62%62%
88 -2.29-2.29 2.612.61 -0.19-0.19 62%62% 81%81%
9
1010 1.891.89 3.473.47 +0.67+0.67 57%57% 61%61%
1111 -0.62-0.62 1.071.07 -1.73-1.73 81%81% 73%73%
1212 0.440.44 2.412.41 -0.39-0.39 64%64% 43%43%
1313 -2.05-2.05 1.721.72 -1.08-1.08 73%73% 66%66%
Student Student NumberNumber
MARK FOR MARK FOR MARKINGMARKING
MARK FOR MARK FOR COMMENTINGCOMMENTING
MARK FOR ESSAYMARK FOR ESSAY FINAL MARKFINAL MARK
60 / 20 / 2060 / 20 / 20
11 59%59% 59%59% 72%72% 67%67%
22 65%65% 63%63% 68%68% 66%66%
33 81%81% 59%59% 53%53% 60%60%
44 68%68% 65%65% 51%51% 57%57%
5
66 53%53% 75%75% 52%52% 57%57%
77 43%43% 43%43% 62%62% 54%54%
88 65%65% 62%62% 81%81% 74%74%
9
1010 58%58% 57%57% 61%61% 60%60%
1111 74%74% 81%81% 73%73% 75%75%
1212 56%56% 64%64% 43%43% 50%50%
1313 71%71% 73%73% 66%66% 68%68%
Student Student NumberNumber
MARK FOR MARK FOR MARKINGMARKING
MARK FOR MARK FOR COMMENTINGCOMMENTING
MARK FOR ESSAYMARK FOR ESSAY FINAL MARKFINAL MARK
60 / 20 / 2060 / 20 / 20
11 59%59% 59%59% 72%72% 67%67%
22 65%65% 63%63% 68%68% 66%66%
33 81%81% 59%59% 53%53% 60%60%
44 68%68% 65%65% 51%51% 57%57%
5
66 53%53% 75%75% 52%52% 57%57%
77 43%43% 43%43% 62%62% 54%54%
88 65%65% 62%62% 81%81% 74%74%
9
1010 58%58% 57%57% 61%61% 60%60%
1111 74%74% 81%81% 73%73% 75%75%
1212 56%56% 64%64% 43%43% 50%50%
1313 71%71% 73%73% 66%66% 68%68%
Student Student NumberNumber
MARK FOR MARK FOR MARKINGMARKING
MARK FOR MARK FOR COMMENTINGCOMMENTING
MARK FOR ESSAYMARK FOR ESSAY FINAL MARKFINAL MARK
60 / 20 / 2060 / 20 / 20
11 59%59% 59%59% 72%72% 67%67%
22 65%65% 63%63% 68%68% 66%66%
33 81%81% 59%59% 53%53% 60%60%
44 68%68% 65%65% 51%51% 57%57%
5
66 53%53% 75%75% 52%52% 57%57%
77 43%43% 43%43% 62%62% 54%54%
88 65%65% 62%62% 81%81% 74%74%
9
1010 58%58% 57%57% 61%61% 60%60%
1111 74%74% 81%81% 73%73% 75%75%
1212 56%56% 64%64% 43%43% 50%50%
1313 71%71% 73%73% 66%66% 68%68%
Correlation between
Commenting Consistency and Essay Mark 0.05
Final Grade for
Coursework takes into account Essay
Grade, Mark for Marking and Mark
for CommentingPercentag
es
Split in Marks 60 / 20 / 20Split in Marks 60 / 20 / 20
Is it reasonable?Is it reasonable? Higher Order skills of Marking – worth Higher Order skills of Marking – worth
more?more?
If we’re judging marking process on If we’re judging marking process on consistency – then should be consistency – then should be rewarded for showing consistency rewarded for showing consistency within marking AND commentingwithin marking AND commenting
CONSISTENT CONSISTENT IN MARKING & IN MARKING & COMMENTINGCOMMENTING
MARK FOR MARK FOR ESSAYESSAY
FINAL MARKFINAL MARK
15 / 15 / 10 / 6015 / 15 / 10 / 60
11 59%59% 59%59% 80%80% 72%72% 69%69%
22 65%65% 63%63% 72%72% 68%68% 67%67%
33 81%81% 59%59% 43%43% 53%53% 57%57%
44 68%68% 65%65% 68%68% 51%51% 57%57%
5
66 53%53% 75%75% 43%43% 52%52% 55%55%
77 43%43% 43%43% 81%81% 62%62% 58%58%
88 65%65% 62%62% 68%68% 81%81% 75%75%
9
1010 58%58% 57%57% 79%79% 61%61% 62%62%
1111 74%74% 81%81% 58%58% 73%73% 73%73%
1212 56%56% 64%64% 56%56% 43%43% 49%49%
1313 71%71% 73%73% 74%74% 66%66% 69%69%
Correlation between Final ESSAY grade & Mark for Marking, Commenting &
Consistency is 0.54
Correlation between FINAL MARK & ESSAY
GRADE is 0.85
StudenStudent#t#
MARK MARK FOR FOR
MARKINMARKINGG
MARK FOR MARK FOR COMMENTINCOMMENTIN
GG
MARK FOR MARK FOR BEING BEING
CONSISTENT CONSISTENT IN MARKING & IN MARKING & COMMENTINGCOMMENTING
MARK FOR MARK FOR ESSAYESSAY
FINAL MARKFINAL MARK
15 / 15 / 10 / 6015 / 15 / 10 / 60
11 59%59% 59%59% 80%80% 72%72% 69%69%
22 65%65% 63%63% 72%72% 68%68% 67%67%
33 81%81% 59%59% 43%43% 53%53% 57%57%
44 68%68% 65%65% 68%68% 51%51% 57%57%
5
66 53%53% 75%75% 43%43% 52%52% 55%55%
77 43%43% 43%43% 81%81% 62%62% 58%58%
88 65%65% 62%62% 68%68% 81%81% 75%75%
9
1010 58%58% 57%57% 79%79% 61%61% 62%62%
1111 74%74% 81%81% 58%58% 73%73% 73%73%
1212 56%56% 64%64% 56%56% 43%43% 49%49%
1313 71%71% 73%73% 74%74% 66%66% 69%69%
IS
IT
WORTH
THE
HASSLE??
Student CommentsStudent Comments
Have you used Peer-Assessment in PHave you used Peer-Assessment in Past?ast?
How did you find self-assessment?How did you find self-assessment? Creating the Comments Database?Creating the Comments Database? How did they find using the CAP systHow did they find using the CAP syst
em and peer-assessment?em and peer-assessment? Thoughts on new Review Stage?Thoughts on new Review Stage? Thoughts on Mark for Marking?Thoughts on Mark for Marking?
Two Main Points to ConsiderTwo Main Points to Consider
How do we assess the time required to How do we assess the time required to perform the marking task?perform the marking task?
What split of the marks between What split of the marks between creation & markingcreation & marking
DefinitionDefinition Student or Lecturer CommentsStudent or Lecturer Comments
Department of Computing & Department of Computing & Mathematical SciencesMathematical Sciences
Faculty of Advanced TechnologyFaculty of Advanced Technology
University of GlamorganUniversity of Glamorgan
THE END
General View of Peer AssessmentGeneral View of Peer AssessmentLecturer or Student?Lecturer or Student?
Lectures getting out of doing their jobs
i.e. marking
Good for developing student skills & employability
How can all students be expected to mark
as ‘good’ as ‘experts’
Why should I mark ‘properly’ and waste my time - I get a fixed mark
for doing it
The feedback given by students is not of the same standard
that I give.
The best thing I’ve ever done to make me
reflect
Defining Peer-AssessmentDefining Peer-Assessment
In describing the teacher ..In describing the teacher ..
A tall b******, so he was. A tall thin, mean A tall b******, so he was. A tall thin, mean b******, with a baldy head like a light bulb. b******, with a baldy head like a light bulb. He’d make us mark each other’s work, He’d make us mark each other’s work, then for every wrong mark we got, we’d then for every wrong mark we got, we’d get a thump. That way – he paused – ‘we get a thump. That way – he paused – ‘we were were
implicated in each other’s painimplicated in each other’s pain’’ McCarthy’s Bar McCarthy’s Bar (Pete McCarthy, 2000, page (Pete McCarthy, 2000, page 68)68)
Student CommentsStudent Comments
Used Peer-Assessment in past?Used Peer-Assessment in past?
None to any real degreeNone to any real degree A couple for staff development type A couple for staff development type
activitiesactivities
Student CommentsStudent Comments
How did you feel about performing How did you feel about performing Self-assessment?Self-assessment?
Very Difficult. Very Difficult. Helped to promote critical thinking Helped to promote critical thinking
ready for peer-assessment stage.ready for peer-assessment stage. Made me think about how I was going to Made me think about how I was going to
Very difficult – not knowing what Very difficult – not knowing what comments they’d needcomments they’d need
Weighting really helped me create Weighting really helped me create criteria ready for markingcriteria ready for marking
Could have helped to do dummy Could have helped to do dummy marking marking
Student CommentsStudent Comments
How did they find using the CAP system How did they find using the CAP system and peer-assessment?and peer-assessment?
Very positive & interestingVery positive & interesting Very time consumingVery time consuming Would do it better next timeWould do it better next time Important to maintain anonymityImportant to maintain anonymity Interesting & complex – thought more about Interesting & complex – thought more about
the assessment processthe assessment process Really helped student developmentReally helped student development
Student CommentsStudent Comments
Thoughts on new Review StageThoughts on new Review Stage
Liked 2Liked 2ndnd chance to review own marks chance to review own marks Gained experience going through processGained experience going through process Didn’t really take much note of peers’ Didn’t really take much note of peers’
commentscomments Liked to see that others felt the same Liked to see that others felt the same
about an essayabout an essay
Student CommentsStudent Comments
Thoughts on Mark for MarkingThoughts on Mark for Marking
Good rewarded appropriatelyGood rewarded appropriately Difficult to fully understandDifficult to fully understand Let owner of essay provide mark for Let owner of essay provide mark for