Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis Dr Richard Parsons Centre for Learning and Teaching University of Dundee.
Post on 24-Dec-2015
215 Views
Preview:
Transcript
Self, Peer and Tutor Assessment of Text Online. Design, Delivery and
AnalysisDr Richard Parsons
Centre for Learning and TeachingUniversity of Dundee
r.parsons@dundee.ac.uk
Self, Peer and Tutor Assessment of Text Online. Design, Delivery and
Analysis• Overall position of CAA
at University of Dundee
• Why self and peer assessment?
• Development story• Key features
– Academic author sets exercise online
– Students use web, email and text messaging
– Robust reliable, secure– Checks for copying
• Demonstration – create an exercise, live
• Using the system – key points– Good question design– Involve the students– Teach the students
• Student activities and responses
• Marking moderation and plagiarism
• The future
University of DundeeLearning Technology Systems
• Blackboard VLE (5.5 Unix, upgrading to Enterprise 6.0 Unix)• QuestionMark Perception 3.4, dual server system
– 1. Low stakes/formative server coupled to Blackboard (Oracle Unix database)
– 2. High stakes/summative server runs stand alone (Oracle Unix database)
• Provide full support for CAA including a “CAA Online” staff education course, initial authoring service
• Have Policy & Procedure documents for CAA (based on BS 7988), (also for VLE and eLearning strategies)
• Provide support for a range of additional services including online evaluation forms, plagiarism detection service, online staff eLearning tuition, publicise innovative solutions for supporting and enhancing learning (e.g Self and Peer Assessment Program)
P3 Student Management
System
e-Directory
QMP formative server CAA-01
QMP Exam server CAA-02
VLE Integrated Systems
University of Dundee MLE
User exists 3 weeks before appointmentCompleted daily
Student existsStaff details also stored in SMSModule enrolments of studentsModule assessment mark storageLDAP client?
Create user name and email process LDAP authenticationRemove user process
Library Catalogue accessResource management systemLDAP client?
Module enrolmentBb clientMark return
Special exam enrolmentMark return
Blackboard VLE – ELS 6.0
GroupWise Web Email
Novell Fileservers Student Desktop
LDAP
Staff Assignment Record
Secure web access
Hardware 2 * 480sVersion 6.0.8Users created from SITS SMS systemModule enrolmentStaff assignments from web systemLDAP clientModule assessment mark storageConsultancy visit 12 May – 16 MayTraining teaching staff
Richard Parsons, CLT LEU
Why Self and Peer Assessment?
• Assessment is one of the keys to learning• Being able to assess your own work is a vital skill• Developing peer assessment skills is important
for future careers• By carrying out assessments, students learn
about the assessment process• Shifts tutor work from post to pre-assignment• Web-based and innovative, so can interest the
students
Development Story
• First a system to replace some lectures when I was overseas
• First self-assessment only, but peer assessment seen as important, results emailed and texted to students
• Developed into a system that allowed online authoring• Added the ability to tutor mark as well as self & peer
assessment• Made more robust based on usage
– Remove the back button– Disable right click– Provide more guidance to staff and student users
• Added a check for copying and plagiarism• Tidied up code to W3C compliance
Online Self & Peer Assessment of Text
(Outline of the process)
Question provided, answer researched and
composed
1. Student submits their text answer (may copy and paste)
3. Student completes an assessment of one of their peers
2. Self assessment completed by student using criteria offered
5. Summary of marks presented. Student thanked for completing
exercise, brief feedback completed
4. Student completes an assessment of the second of their peers
Peer 1 assessment emailed
anonymously to student author
Peer 1 assessment emailed
anonymously to student author
Tutor can mark a selection of answers and moderate and
collate results
Final marks are checked and transferred to
register
Instructor designs question, sets word
limits and details criteria and scoring ranges
All files created, URLs provided to instructor
Technical Details
Author completes webform
Perl program creates
HTML pages for input
Perl files to process
input
HTML pages for output
Perl file to process output
User interaction
And sets file protections
Author access to
results
All operates on a Unix web server, with hyperlinks as
required to Blackboard
Sets question & marking criteria
Demonstration of the Program
Create a NEW exercise, live and interactive?
www.dundee.ac.uk/learning/ilt/
Receive the first assignment, question and instructions
Research the first topic, craft your first text answer
Mark your first answer using the criteria provided and mark two peer’s answers
Receive the second assignment and instructions
Research the second topic, craft your second text answer
Mark your second answer using the criteria provided and mark two peer’s answers
Summary marks returned via VLE – Self, peer and tutor marks compared. Statistics provided overall to judge the success of the exercise
Complete a short evaluation form where your opinion of the exercise will be sought.
Week containing Tuesday 11 March
Week containing Tuesday 15 April
Student Instructions – Online Self and Peer Assessment Exercise
Dr Richard Parsons r.parsons@dundee.ac.uk
Tutor marking of exercise 1
Tutor marking of exercise 2
Marks sent to student author
Student Performance
Environmental science exercise - Write a short piece of text that describes the major scientific reasons why global warming is almost certainly occurring at present. The total length of the text must be between 200 to 300 words.
• Students performed well, range of marks awarded and obtained. Reasonable correlation between self, peer and tutor marking
• Students recognised that they were learning about assessment. Positive feedback received
• One of those exercises where you see the students learning
Criterion vs Norm-based Marking
0
20
40
60
80
100
0 20 40 60 80 100
Expert (Norm-based reference)
Self
/ P
eer
/ T
uto
r m
ean
Student comments demonstrate increased understanding
“A good computer program that more classes should use”“A great learning tool for the students. Enjoyable too, better than
the labs”“Useful for understanding what markers look for in a piece of work”“It was interesting to see how the marking process takes place, and
how you can have an essay that in its own right is well written but does not contain the points needed by the marker”
“Assessing others gives me comparison to my own”“Good to see how we can improve our work by our mistakes”“Useful for the ability to receive results via text message”“It was good to be able to compare your own work with others and
offered an insight into the basic processes staff must use”“Makes you realise that you need more depth in your answer and
to read the question” “It is quite funky”
Demonstration of plagiarism/copying/ collusion
detectionA check for copying is now embedded within the program
http://www.personal.dundee.ac.uk/~rparsons/leu/zz02087process.htm
Operates in the following manner:Recursive check for 6 word sets, ignoring most punctuationTutor can insert text from web pages, textbooks and
lecture notes as requiredNo service is used to store the student’s work, so no
complicated legal issuesWill not pick up copying from year to year, or institution to
institution
Online Self and Peer Assessment Exercise
Formative to Summative Assessments: Considerations
Formative Summative
Low stakes High stakes
Random tutor marking, copying detection
Comparative tutor marking, plagiarism detection
No tutor marking, copying detection
Checks
Initial exercise limited training
Students experienced at self and peer assessment
Training in assessment
Experience
Little moderation of marking
Students gain marks for marking correctly
Moderation Anomalous marks adjusted
Further development
Should be able to soon offer• to include images• check submission step • present criteria to students
before exercise• students receive marks for
marking
And perhaps eventually include
• feedback on feedback• computer marking of free text
Self and Peer AssessmentWhat I leave you with
• Online system is available• Relatively easy to use with supporting documentation• It has some features that can engage students (text
messaging) and staff (plagiarism detection)• Academic issues remain important
– Question design – Students trained in assessment techniques– Student understanding of what constitutes plagiarism– Policy for dealing with plagiarism (formative work)
• Further developments are possible, suggestions welcomed
top related