TA webinar 4 oct 2017 slides - Transforming Assessment

4 Oct 2017: 07:00AM UTC/GMT

Student self-assessment: rationale and practice Justtoletyouknow:

Bypar'cipa'nginthewebinaryouacknowledgeandagreethat: Thesessionmayberecorded,includingvoiceandtextchatcommunica'ons(arecordingindicatorisshowninsidethewebinarroomwhenthisisthecase).Wemayreleaserecordingsfreelytothepublicwhichbecomepartofthepublicrecord.Wemayusesessionrecordingsforqualityimprovement,oraspartoffurtherresearchandpublica'ons.

Webinar Series

e-AssessmentSIG

Webinar Hosts Professor Geoff Crisp, PVC Education, University of New South Wales g.crisp[at]unsw.edu.au Dr Mathew Hillier, Office for Learning & Teaching, Monash University mathew.hillier[at]monash.edu

Danny Carroll University of New South Wales

Student self-assessment: rationale and practice Danny Carroll UNSW Business School [email protected]

Theshapeofthistalk

AndradeandDu(2007)

•  Reflec'veprac'ces•  Thestudy(brieflydescribed)•  Self-assessmentaccuracy:quan'ta've•  Self-assessment–qualita'vefeedback•  Discussion

Self-assessment

AndradeandDu(2007)

Developingreflec'veprac''oners

AndradeandDu(2007)

Developingjudgementprocesses(…OR…Howwronghavewegotthis?)

hRp://sta'c.guim.co.uk/sys-images/Guardian/Pix/cartoons/2012/6/20/1340194927119/students-taking-exams-008.jpg

Analterna'vestate…

hRp://emmerapartmentblog.com/wp-content/uploads/2013/04/study.jpg

UNSWUniversityAssessmentPolicy

SelfandPeerassessmentshouldbewidelyembeddedinassessmentac'vity

Self-reflec'oncycle

Self-assessment–whybother?

• Observedinaccuracy&variabilityofstudentself-assessment

• U'lity,validity,reliability

Ididamixed-methodsstudyonself-assessment

…withanaccuracyfocus

Whycriteriabasedself-assessment?

•  Exemplifywhatexpertjudgesvalue•  Providecomprehensibleperformancetargetsforperformance&judgement

•  Interactwithrubrics,scalesandexemplarstoestablish‘quality’

•  Providesaframeworkforfeedback,interpreta'onandcalibra'on

•  StartsaprocessofaRen'on&response

Researchques'ons:Generalpresenta'onofself-assessmentaccuracyCanstudentsreadilyimproveaccuracy?Whoimproves,howdotheyimprove?Otherobserva'ons?

Self-assessmentconceptualmodel(Sargeantet.Al.,2010)

Studentself-assessmentinREVIEW

Twotasks:similarcriteriadeveloped-studentses'mateperformance

Againsteachcriteria,studentsposi'ontheslidertoes'matetheirjudgmentofthequalityoftheirperformanceonthatcriteria

Myresearch:comparedself-assessment(SA)accuracybetweentwotasks

•  SAwasop'onalinallcases–butSApar'cipa'on%washigh

•  2essaytypetaskscomparedineachcourse•  Supportofmarkingstaffandstudentswithaccuracytraining–e.g.class-baseddiscussionofcriteriaandtask

Self-assessmentaccuracybands(differencescorefrommarker)

<+10 >10to<+5 >5to<-5 >-5<-10 >-10

Strongover-es7mators Accurate

assessorsStrongunder-es7mators

Over- assessors Under- assessors

Staff&studentself-assessmentaccuracytraining

Demographics

Course N= %Local %Int %M %F %20-25years

1stYear 131 25 75 59 41 74

2ndYear 178 59 41 38 62 78

Actualself-assessmentac'vityCourse Students %Self-

assessedTask1

%Self-assessedTask2

%DidSurvey

COMM5001(1stYrPG)

131 73 60 73

MGMT2102(2ndYrUG)

178 91 74 85

Summaryfindings…Studentstendedtobeover-assessAccuracygenerallyimprovedInaccurateassessorsgotmoreaccurateAccurateassessors‘wobbled’2ndyearsweremoreaccuratethan1styearsStudentsimprovedbycalibra'ngontheVERYinaccuratecriteriaAccuracy&taskmarkoutcomes…Localsweremoreaccuratethaninterna'onalsMalesreturnedhigherself-marksthanfemales…

Dostudentsover-assess/dotheygetmoreaccurate?

InTask1’s–55–82%ofstudentsover-assessedtheircriteriamarksoutcome

Course-Task

Over Accurate Under

1stYearTask1

64 32 4

Task2 43 54 3

2ndYearTask1

34 51 15

Task2 19 62 17

Aremarksoutcomesassociatedwithself-assessment?(1styear,Task1)

Over-assessorstask1vstask2calibra'on-1stYr

Under-assessorstask1–2calibra'on-2ndYr

1stYear:Under-assessors:criterialevelaccuracychange

Werestudentjudgementsnuanced?

Criteria-levelaccuracy(1stYear,Task1)

1stYr,N=124

%studentsimprovedaccuracyonmostinaccurateTask1criteria(2ndYear)

Individualover-assessor(criteriacalibra7on)

T1_Task_Difference10T2_Task_Difference0.25

Taskmarkoutcomesbyini'alassessortype:whobenefits?

Self-assessor Type Task 1 Task 2 Task mark % change

Under-assessors, n= 19 Mean task mark 74.7 77.1 3.2

Accurate assessors n= 67 Mean task mark 72 75.4 4.7

Over-assessors, n= 38 Mean task mark 67.3 73.5 9.2 Strong over-assessors, n=16 Mean task mark 65.1 72 10.6

Studentviews:u'lityofself-assessment D28

Prac7cingself-assessmentregularly,islikelytoleadtoimprovedprofessionaljudgment

E31Ithinkprac7cingself-assessmentaccuracyisuseful

E33Developingaccurateprofessionaljudgmentisanimportantskill

E34Developingaccurateprofessionaljudgmentisasimportantasbeingjudgedbyteachers/experts

E35Iwanttodevelopbe^erself-assessmentjudgments

1stYearPGSurvey 3.94 3.98 4.26 4.06 4.37

3rdyearUGSurvey 3.57 3.51 4.47 4.03 4.26

2ndYearUGSurvey 3.71 3.49 4.17 3.78 4.09

Score is the mean score of all respondents out of a possible 5 (mid-point is 2.5)

2ndYear:Previousself-assessment%

Qualita'vefeedback–2ndYearUGManagementCourse:FeedbackCategories

•  Generalexperience•  Selfregulatorymechanisms•  Mo'va'on•  Scaffoldingskillsdevelopment–developingprofessionaljudgment

•  Meta-cogni'veskillsdevelopment

Selfregula'on

‘Itmademerealisewhichpartsiexcelledatandwhichpartsneededmorework’.‘…ItalsoletsmetoseewhetherIhaveansweredalltheques'onsornot’.

Recognisingstandards/developingjudgment

‘Thoughtitwasagreatwayforustoreflectonourstrengthsandweaknesses’‘…asanexchangestudentinmyfirstsemester…I'mreallyunfamiliarwithwhatcons'tutesthedifferentgrades…’‘quiteusefulindevelopingjudgementskills’

Metacogni'on

‘Selfassessmentgeneratesthebuildingblockstoimprovement.Iden'fyingwhatstepsandtheelementsofyourworkandwhatneedsimprovementisundoubtedlyimportant’

Summingup…

hRp://www.stonybrook.edu/commcms/advising/undeclared/selfassess.html

BibliographyBoud,David(2000)"Sustainableassessment:rethinkingassessmentforthelearningsociety."Studiesincon,nuingeduca,on22(2):151-167.Bandura,A.(2001)“Socialcogni'vetheory:Anagen'cperspec've”.AnnualReviewofPsychology,52(1),1-26.Cornelius-White,Jeffrey(2007)"Learner-centeredteacher-studentrela'onshipsareeffec've:Ameta-analysis."ReviewofEduca'onalResearch77(1):113-143.Carroll,D.,(2013).Businessstudent’saxtudestocriteriabasedself-assessmentandself-efficacy.Proceedingsofthe30thasciliteconference,Sydney,Australia.hRp://www.ascilite.org/conferences/sydney13/program/papers/Carroll.pdfCarroll,D.(2014)Benefitsforstudentsfromachievingaccuracyincriteria-basedself-assessment,IAEAConferenceProceedings.hRp://www.iaea.info/documents/paper_226e233d61.pdfDunning,D.,Heath,C.,&Suls,J.M.(2004).Flawedself-assessment.PsychologicalScienceinthePublicInterest,5(3),69-106.Haxe,John.(2008)Visiblelearning:Asynthesisofover800meta-analysesrela'ngtoachievement.Routledge.Lew,MagdeleineDN,W.A.M.Alwis,andHenkG.Schmidt.(2010)"Accuracyofstudents'self-assessmentandtheirbeliefsaboutitsu'lity."Assessment&Evalua,oninHigherEduca,on35(2):135-156.

Webinar Series

Webinar Session feedback With thanks from your hosts

Professor Geoff Crisp, PVC Education, University of New South Wales g.crisp[at]unsw.edu.au

Dr Mathew Hillier, Office of Learning & Teaching Monash University mathew.hillier[at]monash.edu Recording available http://transformingassessment.com

e-AssessmentSIG