TA webinar 15 jul 2020 title slides - Transforming Assessmenttransformingassessment.com/sites/default/files/files/TA...AHE Conference 2021 - Preview James Michael Wood (Seoul NaAonal

15 July 2020: 07:00AM UTC Assessment in Higher Education conference panel: looking to 2021 Session chair: Fabio R Aricò (University of East Anglia, UK)

Just to let you know: By participating in the webinar you acknowledge and agree that: The session may be recorded, including voice and text chat communications (a recording indicator is shown inside the webinar room when this is the case). We may release recordings freely to the public which become part of the public record. We may use session recordings for quality improvement, or as part of further research and publications.

Webinar Series

e-Assessment SIG

Webinar Hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com

FabioR.Aricò

[email protected]

AHE:LeadingAssessmentforLearninginHigherEducaAon

FionaMeddings

[email protected]

AHEConferenceCommi.eeHosts

AHEisanindependentnetworkfocusedondevelopingresearch-

informedpracAceinassessmentandfeedbackinHE.

InternaAonalExecuAveCommiKeehostedbytheLEDResearch

CentreattheUniversityofCumbriainEngland.

AHEnetworkandeventsacrossfullrangeofsubjectdisciplines

andprofessionalfields-evaluaAng,researchinganddeveloping

theory,research,policyandpracAceinassessmentand

feedback.


AHENetwork–ahenetwork.org

ForumforcriAcaldebateofresearchandinnovaAvefocused

assessmentandfeedbackpracAceandpolicy.

Keynotes,peer-reviewedpresentaAons,masterclassesfrom

leadingexperts.

30June–2July2021ManchesterMacDonaldHotel

Manchester,England


AHEConference2021


AHEConference2021-Preview

JamesMichaelWood(SeoulNaAonalUniversity,SouthKorea)

‘Feedbackisuselessifno-onecangetwhatitmeans’:Theeffect

oftechnology-mediatedpeerfeedbackonuptakeandliteracy

JoannaTai,MaryDracup,MerrinMccracken,YasminMobayad

(DeakinUniversity,Australia)

Inclusiveassessmentdesign:whatcanwelearnfromassessmentadjustments?

RobLowneandSuzanneStone(DublinCityUniversity,Ireland)

Assessmentanddigitalcompetences:buildingcapacitywithacademic

staff

‘Feedback is useless if no-one can get what it means’: The relationship between technology-mediated peer feedback and

feedback uptake and literacy

DrJamesWood

[email protected]

[email protected]

AHEConferencePanel:TransformingAssessment

Understanding relationship between technology-mediated peer and teacher feedback and feedback uptake and literacy

4 Results and discussion

1 Feedback literacy & paradigm shifts

2 The USM Model

3 Online Feedback practices & methodology

5 Implications & conclusions

Feedback is powerful! But…

(Carless&Boud,2018)

For feedback to be effective…

Itmustbeusedto‘alterthegap’(Sadler,1989)

For feedback to be effective…

Itmustbeusedto‘alterthegap’(Sadler,1989)

Halfofstudentsfailtocollectfeedback,orengagewithitpoorly(Evans,2013;Handley,Price&Millar,2011)

Problemsexacerbatedbyviewoffeedbackas

transmission(Eretal.,2020;Molloyetal.,2019)

However

Feedback literacy

Howcanitbedeveloped?

Through,analysingexemplars,

workingwithassessmentcriteriaPeerfeedbacketc.(CarlessandBoud,2018)

Dialoguesmayenhancebenefitsoftheseprocesses(Carless,2015;Zhu&Carless,2018) (Carless&Winstone,2020)

The changing context of feedback practice COVID19–goingonlinechangesmessageandrelaAonalandpracAcalaspectsoffeedback–mediatetheeffectsofthepracAce(Winstoneetal.,2017)

Gapindialogicstudiesindigitalsenngsthattargetfeedbackuptake&literacy(Ajjawi&Boud,2017;Steen-Utheim&Hopfenbeck,2018)

NeedfornewperspecAvesonfeedbackuptake&literacyfordigitalenvironments

What is the impact of dialogic feedback on feedback uptake in the literature? Helped‘negoAatemeaning’(Zhu&Carless,2018)

anddevelopfeedbackliteracyoverAme(Reddyetal.2020;Han&Xu,2019)

What is the impact of dialogic feedback on feedback uptake in the literature? Helped‘negoAatemeaning’(Zhu&Carless,2018)

anddevelopfeedbackliteracyoverAme(Reddyetal.2020;Han&Xu,2019)

LimitaTonsofdialogicfeedbackin

pracTce

Dialogicpeerfeedbackmet‘contextual

difficulAes’

(Zhu&Carless,2018)

AffecAveandrelaAonalissuesinvolved(Han&Xu,2019)

ThedialogicUSMmodelof

feedbackuptake&literacy

(blendedandonline)

(Wood,2020)

(simplified)

Formative assessment and technology-mediated dialogic peer feedback (Nicol 2010; Carless, 2015)

Feedback viewed as social and constructed dialogic process (Nicol, 2010; Price et al., 2011)

Understanding the assessment

& feedback landscape

Self-assessment, goal setting &

regulation



(blendedandonline)

(Wood,2020)

(simplified)

Formative assessment and technology-mediated dialogic peer feedback (Nicol 2010; Carless, 2015)

Feedback viewed as social and constructed dialogic process (Nicol, 2010; Price et al., 2011)

•  Dialogic and technology-mediated peer (and teacher) feedback (Nicol 2010; Carless, 2015)

•  Feedback as social and constructed dialogic process (Nicol,2010;Priceetal.,2011)•  Emphasis on student role in making feedback processes effective (NashandWinstone,2017)




regulation

Output



(blendedandonline)

(Wood,2020)

(simplified)

Ongoing technology-mediated dialogues support navigation of the USM processes

through co-regulation (Panadero et al. 2018; Er et al. 2020)

•  Dialogic and technology-mediated peer (and teacher) feedback (Nicol 2010; Carless, 2015)

•  Feedback as social and constructed dialogic process (Nicol,2010;Priceetal.,2011)•  Emphasis on student role in making feedback processes effective (NashandWinstone,2017)




regulation

Motivation affect &

receptivity

Motivation affect &

receptivity

Motivation affect &

receptivity

Output

Improved Feedback Uptake & Literacy



(blendedandonline)

(Wood,2020)(simplified)

Ongoing technology-mediated dialogues support navigation of the USM processes

through co-regulation (Panadero et al., 2018; Er et al., 2020)

Online feedback practices (based on USM)

GoogleDriveandDocsforpeerfeedback,(andscreencast/Doc)teacher-

studentfeedback(toenablediscussion)ona1,200-wordessayover16

weekcourse)

Stages:

1.PreparaAon:socialisaAon,exemplars,criteria,modellingandpracAce

ofgoodsindrive.Introduceconceptsofdialogicfeedback,ZPD,

growthmindsetas‘primingforrecepAvity’.

2.Submitfirstdrat:fortechnology-mediatedpeerfeedback&ongoing

dialogue(1week)

Online feedback practices (based on USM)

GoogleDriveandDocsforpeerfeedback,(andscreencast/Doc)teacher-

studentfeedback(toenablediscussion)ona1,200-wordessayover16

weekcourse)

Stages:

1.PreparaAon:socialisaAon,exemplars,criteria,modellingandpracAce

ofgoodsindrive.Introduceconceptsofdialogicfeedback,ZPD,

growthmindsetas‘primingforrecepAvity’.

2.Submitfirstdrat:fortechnology-mediatedpeerfeedback&ongoing

dialogue(1week)

3.Submittoteacher:Teacherscreencastfeedback,anddialogue,(1

week)forsocialpresenceandenhancedfeedbackmessage

4.Finalsubmissiontoteacher:(screencastfeedbackandGoogleDocs

dialogue)teacherfeedbackfollowedbyareflecAontask

Research question

Whatweretheperceivedeffectsonfeedbackuptakeoftechnology-mediateddialogicfeedbackpracAce,basedontheUSMmodel?

Methodology

Insiderresearch,UCLethicsandBERA2018guidelines

ParAcipants:14SouthKoreanstudentsonadvancedacademicwriAngcourse(16weeks)

Socio-construcAvist,process-orientedinterpreAvistperspecAve

Methods

ReflecAvewriAng(N=14)(wriAng-to-learntasks)and

QuesAonnaires(N=14)

Providedprogressivefocusfor:

Methods

ReflecAvewriAng(N=14)(wriAng-to-learntasks)and

QuesAonnaires(N=14)

Providedprogressivefocusfor:

Semi-structuredinterviews(N=13),

InducAvethemaAccoding(Braun&

Clarke,2006)

DeducAveanalysisagainstmodel

Results and discussion

Feedback is useless if no one can get what it means (Jennyinterview)

1. Facilitates ‘negotiation of meaning’ development of peer feedback points

Thisclasswasspecialbecausethefeedbackwasmoretwo-way…Icouldques?onthefeedback,andwhythepeerthoughtIshouldrevisethat

partoftheessay.Thisaspectoftheac?vi?eswasthemosthelpfulforme.

(QuesAonnaire9)

AlsoIcanaskandrefutethefeedback…Throughthisprocess,IcanreflectanddevelopthefeedbackandeventuallyimprovemywriAng.

(QuesAonnaire7)

1. Facilitates ‘negotiation of meaning’ development of peer feedback points

Thisclasswasspecialbecausethefeedbackwasmore

two-way…Icouldques?onthefeedback,andwhythepeer

thoughtIshouldrevisethatpartoftheessay.Thisaspect

oftheac?vi?eswasthemosthelpfulforme.

(QuesAonnaire9)

AlsoIcanaskandrefutethefeedback…Throughthis

process,Icanreflectanddevelopthefeedbackand

eventuallyimprovemywriAng.

(QuesAonnaire7)

IcouldunderstandbeCerwhatotherscommentedabout

mywri?ng,andalsoIcouldmakemypointsclearerand

deliveredbeCerwhentheydidn'tunderstandthem.

(Ques?onnaire3)

Withoutdialoguetherewillbenoresultofthefeedback.

Feedbackisuselessifnoonecangetwhatitmeans,

havingadiscussionandhaving?metoclarifyitcanmake

thatfeedbackuseful.

(JennyInterview)

2. Multiple Learning cycles in online learning communities

Usuallyifit[peerreview]wasinpaperformitwouldbeaone-waythinganditwouldfinish,buthereIcoulddoafollow-upques?onoraskthemforfeedbackonmyanswer“ohisitokaynow?”andtheywouldsay“ohIthinkit’smuchbeCer”.

(Judyinterview)

…Icanexpanduponthatfeedback,andlikenotjustclarifythemeaning…ifsomeonesaidthispartofmysentenceiswrongsoIcould,forexample,say“whataboutthispart,isthispartokay”andtheywouldsay“that’sokay,butinconjunc?onwithmyabovefeedback

ifyoucouldcombinethistomakethis,that’llbeevenbeCer”.

(JunoInterview)

Thecycleshappenedthreeorfour?mes,Ithinkitwasfour

(Hayleyinterview)

Discussion of 1 & 2

1.  NegoAaAngmeaning

AlsonotedbyZhu&Carless,(2018)but‘contextualdifficulAes’werealsonoted.

Inthisstudy:MulApleonlineexchangesoccurredoverAme

NegoAaAonofmeaningofpeerandfeedbackledtopercepAonofenhanceduptakeanduseandimprovedqualityofacAononfeedback(theorisedbyEretal.,

2020b)

2.FacilitatedcollecAvelearningcyclesinlearningcommuniAes

LearnershadopportunitytoelicitandcollaborateoncreaAonofaddiAonalscaffoldingtoimproveacAonabilityoffeedback

StudentsgothroughcompletefeedbackcycleswithpeersonlinetoconfirmachievementsindigitallearningcommuniAes

(seeReddyetal.2020).

3. Dialogue Reduced impact of cultural and emotional barriers to peer feedback engagement

InKorea,having“redmarks”alloveryourpapermeansabadthing.Itusuallyimpliesthatthereissomethingvery‘wrong’withthepaperandmanymayconsiderthisascri/cismratherthanhelpfulsugges?ons.

(JudyreflecAon)

Enablingfeedbacktobelikeatwo-wayconversa?onhelpsinseveralways.First,thiscreatesasenseof

rela?onshipwithpeersandthusstudentsfeelmorecomfortableingivingmanycommentstoeachotherbecausetheyknowthattheyarenotcri?cizingeachotherbutratherhelping.

(QuesAonnaire3)

Ithoughtatfirstfeedbackwasjustlikemarking…nowIknowthatit'shelpingeachother,it'sforothersandforme,alsogivingthefeedbackwouldalsohelpmineimprovelater…thebarrierswentdown

(Kevininterview)

4. Mutually supportive relationships – a pedagogic alliance of peers?

Itmadememoremo?vatedtoactuallyfixit,soIknowthateveryone'sengagedandpeoplearepuXngtheir?meandeffortindoingmyfeedback…itwouldbeasintonotuseit…Ithinkit'sjustlikeamo/vator,it'slikeasynergyeffect…

(JudyInterview)

IfeelI'mbeinganimportantperson,soeveryoneishelpingmetobeabeCerwriter,thatemo/onitselfhelpedmealotinwri?ng.

(Kevininterview)

I'mnotdoingthisworkaloneandthereissomeonewhoaresolookingforwardtomyimproving…Ithinkitwassupportandfeedbackthatmakemereallymo?vatedto

workhardandimprove.

(Jennyinterview)

Discussion of 3 & 4

CulturecannegaAvelyinfluenceunderstandingoftheconceptandfuncAonoffeedback(Evans,2013;Ryan&Henderson,2018)andemoAonalreacAonstoit(Tian&Lowe,2013)

Theneedtomanageaffectiswidelyacknowledgedinfeedbackliteracyresearch(Forsythe&Johnson,2017;PiK&Norton,2017;Priceetal.2011;Carless&Boud,

2018)

Resultsshowtechnology-mediaTon:

•  EnhancedabilitytoposiAvelyhandleaffect(CarlessandBoud,2018)

And

•  therelaAonalaspectsofpeerfeedbackpracAce–a‘pedagogicalliance’(Leighton&BustosGómez,2018)ofpeers

5. Online peer feedback processes nurtured feedback literacy (Carless and Boud 2018)

AppreciaTngfeedbackIlearnedthatwecangivefeedbackwhenwehaveopinion,

adviceandfeelings.Allthiscanhelpwritertoreflectreaders.Methodssuchas…googledrivemadeusgiveorgetfeedbackanywhereany?me,theprac?cesmadeusfreelytalkingwithpeopleregardlessofposi?on(whethersuperiororinferior).

(Ques?onnaire7)

Mybiggesttake-awayisthatfeedbackisopentoeveryone,

everywhereandevery?me.Itmademefeelfeedbackmorecomfortableandeasier.Thisledmetoac?velypar?cipateinfeedbackprocess.

(Ques?onnaire10)

Hmm,Ijustwantmorefeedbacknow,onallmyworks

(Hollyinterview)

Icouldn'tthinkofanyvalueofthefeedbackbefore,butthis

coursemademethinkthefeedbackisreallyvaluableandreallyappreciatesomeonegivingmefeedbackandhelpingmeimprove.

(Nahyuninterview)

5. Online peer feedback processes nurtured feedback literacy

MakingandrefiningjudgmentsQ:Howdidyouunderstandwhatgoodstandardsare?

Q:Howdidyoujudgethequalityofyourwork?

Peerreview!ItwasaliClebithardtotrytobecri?calofthree

otherwri?ngs,butIlearnedhowtobemorecri?calofmyownwri?ng.

(JudyreflecAon)

Givingfeedbackalsohelpsme,becauseIlearnalotaboutgoodwri?ngitselfwhenI'mgivingfeedback…ifIseeotherpeople'swri?ng,thenit'smuchmorecleartomewhatcanbedonebeCer,comparedtolookingatmyownwri?ng

(KylieInterview)

Igotmoreempiricalgroundsforwhyfeedbackisuseful…

moreexamplesoftheevidencebase…seeingthatoverandoverandexperiencingitmyself,hmmyouknowittotallyaffectedme,ittotallymademebelievethatfeedbackissoimportantandthenthischangeofbeliefIthinkitwillcon?nueevenaeertakingthisclassbecauseit'skindofchangeofyouraXtude.

(Kylieinterview)

6. Overcoming barriers/opening new learning spaces

Youknowlikeonnormalprojects…that'sreallyhardfor

peopletosynchronizetheir?me,sotheyusuallyhave

veryfewmee?ngsclosetothedeadlineandthenthat

that'sit.Butdrivedidn’thavea?melimit,soitwasable

tofacilitateconversa?onatall?mes,…thathelpedalot.

(Junointerview)

UsingDriveforpeerfeedbackgivespeerandmeenough/metothinkabouteachother'sworks.Ithinkithelpsustogivehigherqualityfeedback,makingusmore

mo?vated.

(Ques?onnaire4)

Theessaywaslike,eightpages,butthecommentson

thesideswereliketen,twentypages,soIdon’tthinkwe

couldhavedonethatmuchandthatextensivelyifyou

weredoingjustonpaperorduringclassorjusttalking

aboutitface-to-face…

(Junointerview)

6. Overcoming barriers/opening new learning spaces

Itwasalsoefficientintermsoffeedbackbecausewehad

thisunifiedpla{orm/placewherewecanfindallofeach

other'sworks,readthem,commentonthem,get

immediatenoAficaAonswhengenngcommentsonour

work,etc…

UsingDrivegreatlymoAvatedmetoacAvelyparAcipate

infeedback.

(QuesAonnaire10)

Ithinkthetechnologythatweuseduringtheclasswas

reallyimportantforfacilita?ngengagementofthe

feedbackprocess…becauseit'swayeasier,andwayfaster,way[more]approachablethannothavingit

(KylieInterview)

Ithinkthebestthingisthatyoucanaccesstheclass

materialsonmobile(Readingthingsonsubwayis

extremelyconvenient).

(Ques?onnaire6)

Discussion of 6: Overcoming barriers/opening new learning spaces

Undergraduatesreport:morelikelytoengage

withsimpleunified&convenientfeedback

technology (Winstoneetal.

2020)

Theseresultssuggest:

•  convenience(unifiedpla{orm)

•  noAficaAons•  mobilepla{orms

•  Ametoengage

Relateto:

•  Strongengagementindigitalsenngs

•  Openingnewspacesforlearningwithpeers

Lowered

‘behaviouralfricAon’(seeBehaviouralInsights

Team,2014)

Implications

Helpsinformgapregardingimpactofdialogic

peerfeedbackpracAcesinonlinesenngson

assessmentliteracy

DemonstrateshowlogisAcal,spaAaland

resourcelimitaAonsofdialogicfeedbackcanbe

overcome(synchronousdiscussionthrough

zoomalsobeneficial)

Illustratesthattheprocessesinvolvedin

feedbackuptakeandliteracy(Winstoneetal.2017;

Carless&Winstone2020)canbesupportedthrough

onlinepeerdialogue

PosiAveimplicaAonsforcontextswithlarge

cohorts

WorkloadposiAve

USM Offers insight into designing for uptake, & handling relational & practical issues for online & blended environments

(CarlessandWinstone,2020)

Limitations

LimittoclaimsaboutpercepAons,

generalisability.

Deployabilitywillvary.

Purelyonlineenvironmentsneedmore

socialpresence(CarlessandWinstone,2020)and

socialisaAonprocessestoinculcatetrust,for

example,5-stepsocialisaAonmodel(seeSalmon,2013).

Feedbackuptakeandliteracyisalsorelated

touniversityengagement,digitalliteracy(see

GourlayandOliver2018)andecological(Chong,

2020)orsociomaterialfactors(GraveK,2020).

Future research

Generalisability-StudycouldbereplicatedindifferentcontextsanduAlisingcohortstudiestodetermineimpactonaKainmentandsaAsfacAon.

Scalability,workloadneutrality,trainingandtechnologyacceptanceissues

Onlineonlyvsblendedenvironments

Final Conclusions

DemonstratedpotenAalofonlinefeedbackpracAcesbasedontheUSMmodelforfeedbackliteracy

Informsteacherfeedbackliteracyfordigitalsenngs(CarlessandWinstone,2020)

Exemplifieshowbenefitsofonlineenvironmentscanbeleveragedin‘TransformingAssessment’pracAces

ReferencesAjjawi, R., & Boud, D. (2017). Researching feedback dialogue: an interactional analysis approach. Assessment and Evaluation in Higher Education, 42(2), 252–265.

https://doi.org/10.1080/02602938.2015.1102863 Behavioural Insights Team. 2014.East—Four Simple Ways to Apply Behavioural Insights, April 11,

http://www.behaviouralinsights.co.uk/publications/east-four-simple-ways- to-apply-behavioural-insights/(accessed January 2020). Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Carless, D. (2015). Excellence in university assessment: Learning from award-winning practice. Routledge. Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. Assessment and Evaluation in Higher Education, 43(8), 1315–

1325. https://doi.org/10.1080/02602938.2018.1463354 Carless, D., & Winstone, N. (2020). Teacher feedback literacy and its interplay with student feedback literacy. Teaching in Higher Education, 1–14. https://doi.org/

10.1080/13562517.2020.1782372

Chong, S. W. (2020). Reconsidering student feedback literacy from an ecological perspective. Assessment and Evaluation in Higher Education, 0(0), 1–13. https://doi.org/10.1080/02602938.2020.1730765

Er, E., Dimitriadis, Y., & Gašević, D. (2020a). A collaborative learning approach to dialogic peer feedback: a theoretical framework. Assessment & Evaluation in Higher Education, 1–15. https://doi.org/10.1080/02602938.2020.1786497

Er, E., Dimitriadis, Y., & Gašević, D. (2020b). Collaborative peer feedback and learning analytics: theory-oriented design for supporting class-wide interventions. Assessment & Evaluation in Higher Education, 1–22. https://doi.org/10.1080/02602938.2020.1764490

Evans, C. (2013, March). Making Sense of Assessment Feedback in Higher Education. Review of Educational Research. https://doi.org/10.3102/0034654312474350 Gravett, K. (2020). Feedback literacies as sociomaterial practice. Critical Studies in Education, 00(00), 1–14. https://doi.org/10.1080/17508487.2020.1747099 Gravett, K., & Winstone, N. E. (2018, July 11). ‘Feedback interpreters’: the role of learning development professionals in facilitating university students’ engagement with

feedback. Teaching in Higher Education, pp. 1–16. https://doi.org/10.1080/13562517.2018.1498076 Han, Y., & Xu, Y. (2019). The development of student feedback literacy: the influences of teacher feedback on peer feedback. Assessment & Evaluation in Higher

Education, 1–17. https://doi.org/10.1080/02602938.2019.1689545 Handley, K., Price, M., & Millar, J. (2011). Beyond ‘doing time’: investigating the concept of student engagement with feedback. Oxford Review of Education, 37(4), 543–

560. https://doi.org/10.1080/03054985.2011.604951 Hill, J., & West, H. (2020). Improving the student learning experience through dialogic feed-forward assessment. Assessment and Evaluation in Higher Education, 45(1),

82–97. https://doi.org/10.1080/02602938.2019.1608908 Malecka, B., Boud, D., Carless, D., Malecka, B., & Boud, D. (2020). Eliciting , processing and enacting feedback : mechanisms for embedding student feedback literacy

within the curriculum embedding student feedback literacy within the curriculum. Teaching in Higher Education, 0(0),

ReferencesMolloy, E., Boud, D., & Henderson, M. (2019). Developing a learning-centred framework for feedback literacy. Assessment and Evaluation in Higher Education. https://

doi.org/10.1080/02602938.2019.1667955 Nash, R. A., & Winstone, N. E. (2017). Responsibility-Sharing in the Giving and Receiving of Assessment Feedback. Frontiers in Psychology, 8, 1519.

https://doi.org/10.3389/fpsyg.2017.01519 Leighton, J. P., & Bustos Gómez, M. C. (2018). A pedagogical alliance for trust, wellbeing and the identification of errors for learning and formative assessment. Educational

Psychology, 38(3), 381–406. Reddy, K., Harland, T., Wass, R., & Wald, N. (2020). Student peer review as a process of knowledge creation through dialogue. Higher Education Research &

Development, 1–13. https://doi.org/10.1080/07294360.2020.1781797 Ryan, T., & Henderson, M. (2018). Feeling feedback: students’ emotional responses to educator feedback. Assessment & Evaluation in Higher Education, 43(6), 880–892.

https://doi.org/10.1080/02602938.2017.1416456 Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional science, 18(2), 119-144. Steen-Utheim, A., & Hopfenbeck, T. N. (2018). To do or not to do with feedback. A study of undergraduate students’ engagement and use of feedback within a portfolio

assessment design. Assessment & Evaluation in Higher Education, 1–17. https://doi.org/10.1080/02602938.2018.1476669 Steen-Utheim, A., & Wittek, A. L. (2017). Dialogic feedback and potentialities for student learning. Learning, Culture and Social Interaction, 15, 18–30. https://doi.org/

10.1016/j.lcsi.2017.06.002 Tian, M., & Lowe, J. (2013). The role of feedback in cross-cultural learning: a case study of Chinese taught postgraduate students in a UK university. Assessment &

Evaluation in Higher Education, 38(5), 580-598. Winstone,N.E.,Nash,R.A.,Parker,M.,&Rowntree,J.(2017).SupporAngLearners’AgenAcEngagementWithFeedback:ASystemaAcReviewandaTaxonomyofRecipienceProcesses.

Educa?onalPsychologist,52(1),17–37.hKps://doi.org/10.1080/00461520.2016.1207538

Winstone,N.E.,Nash,R.A.,Rowntree,J.,&Parker,M.(2017).‘It’dbeuseful,butIwouldn’tuseit’:barrierstouniversitystudents’feedbackseekingandrecipience.StudiesinHigher

Educa?on,42(11),2026–2041.hKps://doi.org/10.1080/03075079.2015.1130032

Winstone,N.,Bourne,J.,Medland,E.,Niculescu,I.,&Rees,R.(2020b).“Checkthegrade,logout”:students’engagementwithfeedbackinlearningmanagementsystems.Assessment&

EvaluaAoninHigherEducaAon,1–13.hKps://doi.org/10.1080/02602938.2020.1787331

Wood,J.M.(2020).ADialogic,Technology-MediatedApproachtoSupporAngFeedbackEngagementinaHigherEducaAonContext:PerceivedEffectsonLearners’Feedback

Recipience(DoctoraldissertaAon,UCL(UniversityCollegeLondon)).

Zhu,Q.,&Carless,D.(2018).HigherEducaAonResearch&DevelopmentDialoguewithinpeerfeedbackprocesses:clarificaAonandnegoAaAonofmeaningDialoguewithinpeer

feedbackprocesses:clarificaAonandnegoAaAonofmeaning.hKps://doi.org/10.1080/07294360.2018.1446417

Questions?

@DrJoannaT

deakin.edu.au/learning/enhancing-your-practice/cradle

Inclusiveassessmentdesign:whatcanwelearnfromassessmentadjustments?

JoannaTai,MaryDracup,MerrinMcCracken,

YasminMobayad

TransformingAssessmentwebinarseries

AHEconferencepanel:lookingaheadto2021

15July2020

DeakinUniversityCRICOSProviderCode:00113B

@DrJoannaT

InclusiveAssessment:Whyisitimportant?


IncreasingnumbersofstudentswithadisabilityinhighereducaAon(Grimesetal2017)

WearemorallyandlegallyobligedtoensurestudentshavethesameopportuniAestodemonstrateachievement(DisabilityDiscriminaAonAct(Cth)

1992;DisabilityStandardsforEducaAon(Cth)2005)

Adjustmentsdonotaddressstudentneeds(Waterfield&West2006)ANDthereisalackofevidence(Weis&Beauchemin2019)

SomestudentsdonotdisclosetheircondiAon(Grimesetal2019)

@DrJoannaT

InclusiveAssessmentDesign


‘thedesignanduseoffairandeffec?veassessmentmethods

andprac?cesthatenableallstudentstodemonstratetotheir

fullpoten?alwhattheyknow,understandandcando’(Hockings2010,34)

•  LinkedtoUniversalDesignforLearning(CAST2018)•  Manyguidesforinclusiveassessmentdesignexist

(e.g.UniversityofPlymouth–Waterfield&West2006)

@DrJoannaT

CurrentProblemsintheRealWorld


Staffarelessconfidentaboutinclusiveassessmentdesign(Morris,Milton&Goldstone2019)

StudentsaresAllexperiencingconsiderabledisadvantage

inassessment(Grimesetal2019)

Inclusioncouldbealsoconsideredmorebroadly(StenAford&

Koutsouris2020)

Howshouldwedirectourefforts?

@DrJoannaT

ResearchQuesTon&Aims


WhataretheprioriTesforinclusiveassessmentdesign?

Whorequestsassessmentadjustments?

Whatisthenatureoftheadjustmentsrequested?

@DrJoannaT

Methods


2018DisabilityResourceCentreAccessPlanrecords

DescripAvestaAsAcs

QualitaAveanalysisofopen-textentries

Ethicswaiverforaccessingde-idenAfiedUniversity

heldrecords

@DrJoannaT

Results


Total2860AccessPlanscreatedin2018

72%studentshadanongoingcondiAon

17%hadmorethanonecondiAon

54%reportedamentalhealthcondiAon

31%reportedamedicalcondiAon

54%ofstudentsreportamentalhealthcondiTon


n Ofanydisability Ofmentalhealth

Anxiety 826 29% 53%

Depression 531 19% 34%

PTSD 95 3% 6%

Bipolar 94 3% 6%

SchizoaffecAve

disorders&psychosis 41 1% 3%

PanicaKacks 30 1% 2%

@DrJoannaT

Adjustmenttypes


Exams-2016(71%)

Onlinetests–1038(36%)

Assignments

Groupwork–319(11%)

OralpresentaAons–239(8%)

Labreport&pracAcaltasks–177(6%)

InclassparAcipaAon–256(9%)

@DrJoannaT

Examadjustments


Typeofadjustment N %ofallAPs

Restbreaks 1612 56%

AddiAonalexamAme 1414 49%

AddiAonalreadingAme 176 6%

@DrJoannaT

QualitaTvecommentsaboutadjustments


OngoingcommunicaAonandnegoAaAonshouldtake

place

Groupworkisimportantbutnotalwayspossible

Studentsfeltbadaboutlenngothersdown

In-person,in-classtaskscanbedifficult

Temporalflexibilityisvalued

@DrJoannaT

ImplicaTonsforassessmentdesign


HowcanweberesponsivetocondiAonswithfluctuaAngor

variableimpact?

Implementstudentchoiceandflexibilityinassessment

ConsiderassessmentprogrammaAcally

Howcanweensureequivalenceofstandardsacrosstasks?

UseaninvitaAonalmetaphorforcriteria(BearmanandAjjawi2019)

@DrJoannaT

ImplicaTonsforassessmentdesignpt.2


Forexams:

Improveeaseofaccesstoadjustments

Onlineexamsparadoxicallymayprovidebenefits(locaAon;inputopAons)

Forin-classassessment:

CertaintyaboutparAcipaAon&presentaAonschedules

AlternaAveopAons(pre-recorded;videolink)

Forgroupassignments:

SetupcommunicaAon/parAcipaAonagreements

IncludeconAngencyplanstomanageexpectaAons

@DrJoannaT

References


Bearman,M.,&Ajjawi,R.(2019).Canarubricdomorethanbetransparent?InvitaAonasanewmetaphorforassessmentcriteria.Studiesin

HigherEduca?on,1–10.hKps://doi.org/10.1080/03075079.2019.1637842

CAST.2018.“UniversalDesignforLearningGuidelinesVersion2.2.”2018

DisabilityDiscrimina?onAct(1992)CommonwealthofAustralia

DisabilityStandardsforEduca?on(2005)CommonwealthofAustralia

Grimes,S.,Scevak,J.,Southgate,E.,&Buchanan,R.(2017).Non-disclosingstudentswithdisabiliAesorlearningchallenges:characterisAcs

andsizeofahiddenpopulaAon.AustralianEduca?onalResearcher,44(4–5),425–441.hKps://doi.org/10.1007/s13384-017-0242-y

Grimes,S.,Southgate,E.,Scevak,J.,&Buchanan,R.(2019).Learningimpactsreportedbystudentslivingwithlearningchallenges/disability.

StudiesinHigherEduca?on,0(0),1–13.hKps://doi.org/10.1080/03075079.2019.1661986

Hockings,C.(2010).InclusivelearningandteachinginhighereducaAon:asynthesisofresearch.RetreivedAugust,(April),1–67.

Morris,C.,Milton,E.,&Goldstone,R.(2019).Casestudy:suggesAngchoice:inclusiveassessmentprocesses.HigherEduca?onPedagogies,

4(1),435–447.hKps://doi.org/10.1080/23752696.2019.1669479

StenAford,L.,&Koutsouris,G.(2020).WhatareinclusivepedagogiesinhighereducaAon?AsystemaAcscopingreview.StudiesinHigher

Educa?on,0(0),1–17.hKps://doi.org/10.1080/03075079.2020.1716322

Waterfield,J.,&West,B.(2006).InclusiveAssessmentinHigherEduca?on:AResourceforChange.hKps://doi.org/10.4324/9781315045009

Weis,R.,&Beauchemin,E.L.(2019).AreseparateroomtestaccommodaAonseffecAveforcollegestudentswithdisabiliAes?Assessmentand

Evalua?oninHigherEduca?on,0(0),1–16.hKps://doi.org/10.1080/02602938.2019.1702922

Assessment & digital competences: building capacity with academic staff

The DCU-IUA Enhancing Digital Teaching and Learning Project

Suzanne Stone & Rob Lowney Teaching Enhancement Unit, Dublin City University

[email protected] ; [email protected] @suzielearning ; @lowneyrob

Slides at: bit.ly/EDTLAHE20

Overview Background to the EDTL project

Structure of project

#IUADigEd Community

DCU pilot development

Technology-enhanced assessment professional learning programme

Post-pilot implementation

Adapting to the Covid-19 crisis

Background to project: Europe

New Skills Agenda for Europe 2016 - Digital technologies are driving change in the global economy and we need to ensure our students are suitably prepared

EU Digital Education Action Plan 2018 - 11 actions to support technology use and the development of digital competences in education: 1. Making better use of digital technology for teaching and learning 2. Developing digital competences and skills

Courtesy of Dr Sharon Flynn, IUA

Background to project: National

National Skills Strategy 2025 - technology is one of the key drivers of change and improved digital skills will be vital for Ireland’s future

National Strategy for Higher Education to 2030 - teaching methods should increasingly be accompanied by e-learning and blended learning opportunities


Background to project: National

Digital Strategy for Schools 2015-2020 - provide rationale and action plan for embedding digital technologies in teaching, learning and assessment


Background to project: Sectoral / Local

IUA Universities Charter 2018 - need to build on the quality of the student experience in a digital age, commit to developing a coherent national programme in digital learning

DCU Strategic Plan 2017-2022 - University-wide curriculum review and an increase in digital learning initiatives and blended learning modules


Opportunity

IUA devised the project

Applied for funding under HEA Innovation and Transformation Call 2018

Digital learning programme


European context

Irish national context

University sector context

Institutional

context

Enhancing Digital Teaching and Learning (EDTL) Three-year nationally-funded project. Project aim:

enhance the digital attributes and educational experiences of Irish university students . develop, pilot, review and roll out an ambitious staff development programme to enhance the digital confidence, skills and competences of those who teach in Irish universities.


Approach

This project aims to mainstream digital in teaching and learning activities in Irish Universities, by addressing the professional development of all who teach or support teaching and learning.


Approach Courtesy of Dr Sharon Flynn, IUA

Pedagogy first - DigCompEdu Courtesy of Dr Sharon Flynn, IUA

European Framework for the Digital Competence of Educators (DigCompEdu)

Project structure Courtesy of Dr Sharon Flynn, IUA

Steering group ○  Senior management

IUA project manager ○  Dr Sharon Flynn

Project team ○  Project leads in 7

universities ○  Student intern working

at national project level ○  Local level interns at

each university

See edtl.blog for more information

One pilot, seven flavours Courtesy of Dr Sharon Flynn, IUA

Dublin City University

Focus on assessment through structured workshops and development activities, involving 3 pilot groups.

Trinity College Dublin

Redesigning an accredited module in Technology Enhanced Learning to focus on the Flipped Classroom.

University College Dublin

Leveraging digital for student feedback in the College of Science with the move to the new VLE ; initial needs analysis in College of Engineering, focus on creation of interactive animations for engagement.

Maynooth University

Conducted a needs assessment with 3 separate pilot groups, focus on information literacy and student skills, and aligning the learning outcomes of a new accredited module.

University College Cork

Leveraging the move to the new VLE to focus on staff digital skills.

University of Limerick

Conducting a needs assessment with 2 pilot groups, enhancing staff and student digital skills to support peer and self-assessment/

NUI Galway Focus on curriculum (re)design and development of digital resources through a structured approach. Alignment with existing accredited CPD.

#IUADigEd Community Courtesy of Dr Sharon Flynn, IUA

See edtl.blog for more information

#IUADigEd webinars

EDTL Approach: Consider Content & Activities, Project Team, Monday 20 July.

International Perspectives on Teaching Online, panel session with Maha Bali, Jesse Stommel and

Sukaina Waljil, hosted by Kate Molloy, Monday 27 July at 4pm.

EDTL Approach: Consider Communication & Engagement, Project Team, Monday 10 August.

EDTL Approach: Consider Assessment & Feedback, Project Team, Monday 17 August.

DCU Pilot phase (Sep-Dec 2019) DCU’s focus: technology-enhanced assessment (TEA)

Why assessment?

4 work packages: Staff Development Programme

Development and Maintenance of Online Resources

Ongoing Technical Enhancements for the VLE (Moodle)

Communications and Dissemination

Staff development programme based around suite of 10 workshops drawing on established professional learning opportunities and staff expertise

DCU Pilot phase (Sep-Dec 2019)

Focus on ‘Why’ of TEA, ‘How to’

later

Starting point: workshops already

offered by TEU

Refine workshops for discipline

specific needs Input to come later

Staff development Programme: Underlying principles

Iterateandsustain Reviewand

refine ImplementTEAassessment Ongoingsupport

PlanningaTEAassessment Individual/groupconsultancies

Workshopx3

engagement Pre-workshoplearning

Customworkshop

development Developmentofsuiteof

Digitalassessmentworkshops

Iconscourtesyof

Fontawesome.com

Disciplinespecificneedsanalysis:

EducaAon&Psychology

Project structure

Staff Development Programme: Lessons/Adaptations Timing is crucial! Early engagement with schools required Flexibility needed for delivery Even before Covid-19! Discussion more important than practical work - removed Pedagogy focused workshop needs to be followed by technology ‘how to’ workshop - now built into programme Pre-workshop tasks too time consuming - removed DigCompEdu: Participants need scaffolding to engage. Gamification Digital Pursuit adapted from Jisc Digital Pursuit* Development of additional online resources (non-tech focused) required - work ongoing, collaborations with participants explored

*Hibberson, S. (2020). Digital Pursuits - playing the digital capability game! - Inspiring learning. [online] Inspiring learning. Available at: https://inspiringlearning.jiscinvolve.org/wp/2019/03/digital-pursuits-playing-the-digital-capability-game/ [Accessed 13 Feb. 2020].


Evaluation underway Student voice = key - focus groups Bamber’s framework - beyond quantitative data towards evidence of impact on practice National Forum badge on offer to participants Reference Bamber, V. (Ed.). (2013). Evidencing the value of educational development. SEDA Special No 34. London: SEDA. ISBN 978-1-902435-56-5.

DCU Pilot phase Evaluation

At national level - project refocused to support the blended/online delivery for 2020/2021

EDTL approach - curation of resources/framework

Crisis = Opportunity: Alternative assessment

Impact of pilot: EDTL participants felt prepared for crisis particularly in respect of designing alternative assessments (anecdotal feedback)

Adapting to Covid-19

Driven by identified needs at various levels

Discipline-specific

Tailored to participants’ needs

Scaffolded and ongoing support

Student voice central to align skills development for staff with student needs

Mixture of type of engagement

Adaptation to needs resulting from Covid-19 crisis - a real opportunity

In summary

Participants: Testimonials “I was delighted to have the opportunity to partake in the EDTL Project. The team were fantastic and provided a great platform for learning. It was very practical. There was a great system of support provided when we were trying the new techniques which gave me confidence in using these. I enjoyed embedding these new strategies in my teaching as I could feel the level of engagement [from students] rising. The students were definitely engaged and enjoyed asking questions (e.g. mentimeter) and receiving feedback on their learning (E.g. in Kahoot)! Feedback from the students on these techniques was very positive. these were definitely informing teaching and learning”


Participants: Testimonials “Participating in the EDTL workshops benefitted me because while there was an overall framework for the CPD, the detail was built around the identified needs of the group in question and this was really useful and one of the key reasons (I think) that staff engaged well with the initiative.”


Assessment and digital competences: building capacity with academic staff The DCU-IUA Enhancing Digital Teaching and Learning Project Suzanne Stone & Rob Lowney Teaching Enhancement Unit [email protected] ; [email protected] @suzielearning ; @lowneyrob Slides at: https://bit.ly/EDTLAHE20

Webinar Series

Webinar Session feedback With thanks from your hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com Recording available http://transformingassessment.com

e-Assessment SIG

Next session 5 Aug 2020 Adapting to COVID19 by

ignoring proctoring: catalysing alignment of

online teaching, learning and assessment

Register http://taw.fi /5aug2020

TA webinar 15 jul 2020 title slides - Transforming Assessmenttransformingassessment.com/sites/default/files/files/TA...AHE Conference 2021 - Preview James Michael Wood (Seoul NaAonal

Documents