Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis Dr Richard Parsons Centre for Learning and Teaching University of Dundee.

Self, Peer and Tutor Assessment of Text Online. Design, Delivery and

AnalysisDr Richard Parsons

Centre for Learning and TeachingUniversity of Dundee

[email protected]

Self, Peer and Tutor Assessment of Text Online. Design, Delivery and

Analysis• Overall position of CAA

at University of Dundee

• Why self and peer assessment?

• Development story• Key features

– Academic author sets exercise online

– Students use web, email and text messaging

– Robust reliable, secure– Checks for copying

• Demonstration – create an exercise, live

• Using the system – key points– Good question design– Involve the students– Teach the students

• Student activities and responses

• Marking moderation and plagiarism

• The future

University of DundeeLearning Technology Systems

• Blackboard VLE (5.5 Unix, upgrading to Enterprise 6.0 Unix)• QuestionMark Perception 3.4, dual server system

– 1. Low stakes/formative server coupled to Blackboard (Oracle Unix database)

– 2. High stakes/summative server runs stand alone (Oracle Unix database)

• Provide full support for CAA including a “CAA Online” staff education course, initial authoring service

• Have Policy & Procedure documents for CAA (based on BS 7988), (also for VLE and eLearning strategies)

• Provide support for a range of additional services including online evaluation forms, plagiarism detection service, online staff eLearning tuition, publicise innovative solutions for supporting and enhancing learning (e.g Self and Peer Assessment Program)

P3 Student Management

System

e-Directory

QMP formative server CAA-01

QMP Exam server CAA-02

VLE Integrated Systems

University of Dundee MLE

User exists 3 weeks before appointmentCompleted daily

Student existsStaff details also stored in SMSModule enrolments of studentsModule assessment mark storageLDAP client?

Create user name and email process LDAP authenticationRemove user process

Library Catalogue accessResource management systemLDAP client?

Module enrolmentBb clientMark return

Special exam enrolmentMark return

Blackboard VLE – ELS 6.0

GroupWise Web Email

Novell Fileservers Student Desktop

LDAP

Staff Assignment Record

Secure web access

Hardware 2 * 480sVersion 6.0.8Users created from SITS SMS systemModule enrolmentStaff assignments from web systemLDAP clientModule assessment mark storageConsultancy visit 12 May – 16 MayTraining teaching staff

Richard Parsons, CLT LEU

Why Self and Peer Assessment?

• Assessment is one of the keys to learning• Being able to assess your own work is a vital skill• Developing peer assessment skills is important

for future careers• By carrying out assessments, students learn

about the assessment process• Shifts tutor work from post to pre-assignment• Web-based and innovative, so can interest the

students

Development Story

• First a system to replace some lectures when I was overseas

• First self-assessment only, but peer assessment seen as important, results emailed and texted to students

• Developed into a system that allowed online authoring• Added the ability to tutor mark as well as self & peer

assessment• Made more robust based on usage

– Remove the back button– Disable right click– Provide more guidance to staff and student users

• Added a check for copying and plagiarism• Tidied up code to W3C compliance

Online Self & Peer Assessment of Text

(Outline of the process)

Question provided, answer researched and

composed

1. Student submits their text answer (may copy and paste)

3. Student completes an assessment of one of their peers

2. Self assessment completed by student using criteria offered

5. Summary of marks presented. Student thanked for completing

exercise, brief feedback completed

4. Student completes an assessment of the second of their peers

Peer 1 assessment emailed

anonymously to student author

Peer 1 assessment emailed

anonymously to student author

Tutor can mark a selection of answers and moderate and

collate results

Final marks are checked and transferred to

register

Instructor designs question, sets word

limits and details criteria and scoring ranges

All files created, URLs provided to instructor

Technical Details

Author completes webform

Perl program creates

HTML pages for input

Perl files to process

input

HTML pages for output

Perl file to process output

User interaction

And sets file protections

Author access to

results

All operates on a Unix web server, with hyperlinks as

required to Blackboard

Sets question & marking criteria

Demonstration of the Program

Create a NEW exercise, live and interactive?

www.dundee.ac.uk/learning/ilt/

Using the System – Key Points

• Good question design

• Involve the students

• Teach the students

Receive the first assignment, question and instructions

Research the first topic, craft your first text answer

Mark your first answer using the criteria provided and mark two peer’s answers

Receive the second assignment and instructions

Research the second topic, craft your second text answer

Mark your second answer using the criteria provided and mark two peer’s answers

Summary marks returned via VLE – Self, peer and tutor marks compared. Statistics provided overall to judge the success of the exercise

Complete a short evaluation form where your opinion of the exercise will be sought.

Week containing Tuesday 11 March

Week containing Tuesday 15 April

Student Instructions – Online Self and Peer Assessment Exercise

Dr Richard Parsons [email protected]

Tutor marking of exercise 1

Tutor marking of exercise 2

Marks sent to student author

Student Performance

Environmental science exercise - Write a short piece of text that describes the major scientific reasons why global warming is almost certainly occurring at present. The total length of the text must be between 200 to 300 words.

• Students performed well, range of marks awarded and obtained. Reasonable correlation between self, peer and tutor marking

• Students recognised that they were learning about assessment. Positive feedback received

• One of those exercises where you see the students learning

Criterion vs Norm-based Marking

0

20

40

60

80

100

0 20 40 60 80 100

Expert (Norm-based reference)

Self

/ P

eer

/ T

uto

r m

ean

Student comments demonstrate increased understanding

“A good computer program that more classes should use”“A great learning tool for the students. Enjoyable too, better than

the labs”“Useful for understanding what markers look for in a piece of work”“It was interesting to see how the marking process takes place, and

how you can have an essay that in its own right is well written but does not contain the points needed by the marker”

“Assessing others gives me comparison to my own”“Good to see how we can improve our work by our mistakes”“Useful for the ability to receive results via text message”“It was good to be able to compare your own work with others and

offered an insight into the basic processes staff must use”“Makes you realise that you need more depth in your answer and

to read the question” “It is quite funky”

Demonstration of plagiarism/copying/ collusion

detectionA check for copying is now embedded within the program

http://www.personal.dundee.ac.uk/~rparsons/leu/zz02087process.htm

Operates in the following manner:Recursive check for 6 word sets, ignoring most punctuationTutor can insert text from web pages, textbooks and

lecture notes as requiredNo service is used to store the student’s work, so no

complicated legal issuesWill not pick up copying from year to year, or institution to

institution

Online Self and Peer Assessment Exercise

Formative to Summative Assessments: Considerations

Formative Summative

Low stakes High stakes

Random tutor marking, copying detection

Comparative tutor marking, plagiarism detection

No tutor marking, copying detection

Checks

Initial exercise limited training

Students experienced at self and peer assessment

Training in assessment

Experience

Little moderation of marking

Students gain marks for marking correctly

Moderation Anomalous marks adjusted

Further development

Should be able to soon offer• to include images• check submission step • present criteria to students

before exercise• students receive marks for

marking

And perhaps eventually include

• feedback on feedback• computer marking of free text

Self and Peer AssessmentWhat I leave you with

• Online system is available• Relatively easy to use with supporting documentation• It has some features that can engage students (text

messaging) and staff (plagiarism detection)• Academic issues remain important

– Question design – Students trained in assessment techniques– Student understanding of what constitutes plagiarism– Policy for dealing with plagiarism (formative work)

• Further developments are possible, suggestions welcomed