Anonymous Assignments: Core product or Customisation?

Anonymous assignments: Core product or customisation?

[email protected]

University of York

2

Session overview

1. EMA in the UK

2. EMA workflow requirements

3. Anonymous assignment submission @ York

4. Blackboard’s anonymous assignment submission

5. Ways forward: core product or bespoke development?

3

Session outcomes

• You– what we do at York– what BB does themselves– where do you need to be?

• Me– benchmark York within the sector– inform use of BB anon assignment – sanity check

• BlackBoard– anon assignment user feedback

4

Electronic Management of Assessment (EMA) in the UK

• Growing adoption and interest across the sector (UCISA 2014)

• Strategic priority for JISC / UCISA / HELF identified benefits for– Students (convenience)– Admin (efficiencies)– Academics (pedagogic)

• Turnitin Grademark leading e-submission solution, 63%

• End to end solution remains elusive– JISC, Southampton, Northumbria, BB/SITS integration?

5

Assessment and feedback lifecycle

1 - Specifying

2 - Setting

3 - Supporting

4 - Submitting

5 - Harvesting

the work

6 - Marking and

feedback

7 - Returning marks and feedback

8 Reflecting / evaluation

Lifecycle model adapted from Manchester University TRAFFIC Project

6

EMA workflow requirements

• http://padlet.com/simon_davis/4y0m09uh4i5i

7

EMA @ York: Contextual factors

• Anonymity mandatory “except where unfeasible”– Assessment principles: Equity, Openness, Clarity, Consistency

• No current institutional EMA policy /mandate, The “York Way”

• ~14,000 students

• Change is coming…– New L&T strategy and faculties– Growing DL provision

8

Anonymous assignment submission at York

• Bespoke building block deployed since 2007 for file based workflow

• Iterative development to support submission and feedback return (2014/15)

• Complete interdependence on SITS data; student exam number and assessment records

• Student work anonymised on submission and tracked throughout the rest of the lifecycle by exam number

• Flexible “barebones” toolset maps onto familiar admin processes

• 20,558 files submitted, 11,267 feedback files returned online this academic year (Oct 14 - March 2015)

9

York Anonymous Assignment: Workflow & features

• Deadlines and late submission alerts• Restrict file types (access / annotation)Setting

• Submitted files (30MB max) labelled with exam no & timestamped• E-mail receipt and alertsSubmitting

• Associate submissions with SITS assessment record • Download options; submissions, feedback (blank / template / annotated

work) marksheet, non-submission and tracking info

Harvesting the work

• Create feedback in standalone form / annotate work• Marks recorded in marksheet

Marking / feedback

• Completed feedback and marks batch uploaded and made available to students through SITS e:vision

Returning marks / feedback

VLE

Dept defined (shared drive

/ Google…)

SITS e:vision

10

York Anonymous Assignment: Evaluation Highlights

• Institution: Need to fill gaps in policy and guidance

• Admin (n=18): Easy to use and saves time

• Students (n=261): Generally +ive, FB quality, confidence issues

• Markers (n=48): Mixed reaction to all aspects of workflow– Most negative response to reading work on screen– Broadly positive response to feedback production – Areas for development:

• Improved workflow and speed / More managed system• Annotations – awareness, workflow, reusable, workload implications• Resistance to greater use of technology

11

Polarised response

“I can see no merit in this system. I have other experience of on-line marking from my external examining and have found it very difficult, time consuming and problematic. There are some parts of the pedagogic process that are not amenable to on-line systems and marking essays is one of these. The student sget a worse experience, worse feedback and and I have found it tiresome, unnecessarily complicated and much much slower. I conclude that it is inefficient use of time, makes turning the essays round longer, and pedagogically worse outcome.”

“This was my first year using this system and it was revolutionary. I could mark anywhere (just using a USB stick and a backup) and there was no copying of disks. Also there was no printing out of forms at the Dept Office. The whole thing was slick and easy.”

12

York Anonymous Assignment: Pros / Cons

• Pros– Familiar and evolutionary

• In use since 2007• File based – closely aligned with paper based workflows• Significant “organic” opt-in from Departments

– Batch upload of feedback created offline– Low entry point – markers do not need to engage with VLE– Technology imposed where it counts– No possibility of anonymity being circumvented

• Cons– Feedback production cumbersome for some– Anonymity is permanent; sharing work with supervisors etc…– End to end workflow– Missed opportunity for greater use of TEL?

13

BlackBoard Anonymous assignment – step by step

1 – Activate / deactivate when editing an assignment

14

BlackBoard Anonymous assignment – step by step

2 – Students submit and advised of anonymity

15

BlackBoard Anonymous assignment – step by step

3 – Student submissions obfuscated in the GradeCentre

16

BlackBoard Anonymous assignment – step by step

4 – Read and provide feedback anonymously

17

BB Anon Assignment @ York?

“I think that having the materials submitted online is good, and I prefer marking digitally. I found the VLE comments section less useful. I downloaded the essays and used the track changes function of Word, which worked great for me.”

“I've also been trialling non-anonymous marking through the VLE for one module - where we annotated on screen. I really liked providing feedback this way.”

• Marker preference for working offline– Batch upload of feedback– Downloading doesn’t respect Smart Views / Delegation– No delegation without reconciliation

• Interface– Learning a new system– Limited display area – is it as good as reading / annotating in Word?

• Integration with SITS e:vision – untested @ York

• Anonymity via randomly generated number– Tracking non submissions etc– All or nothing

18

Comparing York and BB Anonymous submission

Feature York BB

Specifying Limited to 30MB No limit

Setting File restrictionsRequires SITS

Formative / summative

Supporting Staff for more advanced use Greater staff to get started

Submitting Anonymous tracking Tracking difficulty

Harvesting the work

Admin download filesFile management issues

Direct access by markersDownload issues

Marking and feedback

Offline onlyFamiliar feedback approachesSimple paper readingDisability stickers

Paper logistically trickyFlexible Feedback toolsManual entry into VLE – no batch upload

Returning marks / FB

Pass to AdminBatch upload to SITS

Direct through VLEUntested SITS integration

Reflecting / evaluation

Anonymous identifier cannot be shared

Possibility within the VLELongitudinal view problems

19

Does BB Anon Assignment meet our workflow requirements?

• http://padlet.com/simon_davis/4y0m09uh4i5i

20

Detail on Evaluation outcomes follows

21

Evaluation: Administrators (n=18 from 11 depts)

• Easy to use, adoption simplified by fit with current practices

• Saves time – Student submission – 4.67 / 5– Sharing files with markers – 4.25 / 5– Adding marks to e:vision – 4.43 / 5– Returning feedback to students 5 / 5– Complete workflow 4.69 / 5

• Minor enhancement requests

• Highlighted policy and guidance blindspots

22

Evaluation: Students (n=261 from 15 depts)

• Clear preference for “easy” online submission– Positive impact on student experience

• Preference for online feedback (delivery mode)– Feedback “utility” not linked to delivery mode– Correlation between feedback utility and feedback type (annotations

preferred)

• General lack of confidence in IT systems– Potential to address with guidance / training

23

Markers: overall reaction

• Mixed reactions across all aspects of the workflow; – access to files– reading and marking– feedback production & quality– recording marks

• Broad acceptance from most markers, vitriolic resistance from some opponents

Neutral38%

Postive34%

Negative28%

24

Markers: reading and marking work

• 83% read all work on screen despite options to print– 41% - “about the same as reading on paper”– 40% - “generally worse than paper”– 19% - “generally better than on paper”

• Polarised / negative response to marking on tablets– 33% - “Not at all – I hate the idea”– 29% - “Not bothered either way”– 23% - “Yes I really want to be able to do this”– 15% - “Other” (mainly qualified enthusiasm)

25

Markers: Top suggested areas for development

• Improved workflow and speed– “The whole system is very cumbersome and clunky.”

• Annotations– Awareness– Workflow– Reusable– Workload implications / strategic design of feedback

• More managed system; VLE tools / Grademark– “Many other institutions use more stable systems where scripts and marks are

linked and can be accessed and viewed more easily.”

• Resistance to greater use of technology– “I would much prefer to leave things as they are and continue with the paper-based

system. I work on the computer all day and do not want to mark on the screen as well.”

26

27