Assessment Futures: The Role for e-Assessment?

Assessment Assessment Futures: role for Futures: role for e-Assessment?e-Assessment?

Peter Hartley, University of [email protected]

http://www.brad.ac.uk/educational-development/aboutus/team/Full_details_27414_en.php

This sessionThis session

o Why bother? Why worry?o Assessment trends and futureso What can e-assessment contribute?

And the argumentAnd the argument

o Assessment is a problem which will ‘get worse’.

o Assessment futures: trends re strategy and environment.

o Significant potential for ‘e’: some examples.

o And so the role for e-assessment must be part of a broader strategic initiative.

Assessment is Assessment is a problem, #1 a problem, #1

• Ask a student of X:what makes a ‘First Class X?

Assessment is Assessment is a problem, #1 a problem, #1

• Ask a student of X:what makes a ‘First Class X?’o How would they respond?o Can they give you a coherent summary of the

main programme outcomes (or just the algorithm that determines the marks)?

Assessment is Assessment is a problem, #1 a problem, #1

• See useful summaries of major issues:o the the PASS Project Issues Paper

Please comment/feedback and use.• http://www.pebblepad.co.uk/bradford/viewasset.aspx?oid=260486&type

=file

o Recent article:• Price, M., Carroll, J., O'Donovan, B. and Rust, C. (2011)

''If I was going there I wouldn't start from here: a critical commentary on current assessment practices', Assessment & Evaluation in Higher Education. Vol. 36, No. 4, 479-492.

Assessment is Assessment is a problem, #2 a problem, #2

• From PASS, would highlight:o Assessment ‘drives and channels’o What/why are we measuring: the‘slowly

learnt’problem.o Limitations of grading (e.g. marks are not

numbers).o Implications for course structures/regulations.

Problem #3: multi-Problem #3: multi-purpose purpose

& multi-audience& multi-audience

Problem #4: Problem #4: The meaning of The meaning of

feedbackfeedback• Cannot we ‘recapture’ the ‘original’ meaning of

feedback: enabling self-correcting behaviour towards a known goal.

• This means rediscovering the ‘feedback loop’ whereby information must be ‘fed back’ so that it:• relates to the goal.• is received.• is correctly interpreted.• enables corrective action

cf. the work of Royce Sadler in Higher Education, e.g. http://www.northumbria.ac.uk/sd/central/ar/academy/cetl_afl/earli2010/themes/rsadler/

Problem #5 Problem #5 • More

demanding ‘consumers’:

• E.g. the NUS Charter

Assessment futures:Assessment futures:strategy and strategy and environmentenvironment

Environment:Environment:TESTA project

• NTFS group project with 4 partners: ‘aims to improve the quality of student learning through addressing programme-level assessment.’

• starting from audit of current practice on nine programmes: osurveyed students using focus groups and AEQ – Assessment Experience Questionnaire – Graham Gibbs et aloalso using tool to identify programme level ‘assessment environments’ (Gibbs)

Consistent practice?Consistent practice?

Characterising programme-level assessment environments that support learning by Graham Gibbs and Harriet Dunbar-Goddet Published in: Assessment & Evaluation in Higher Education, Volume 34, Issue 4 August 2009 , pages 481 - 489

Data from TESTAData from TESTA

Assessment Assessment environment and environment and

impactimpact• Interim findings from TESTA

o variety of assessments can cause problemso Issues over understanding assessment criteria,

marker variation, and feedbacko variation across programmeso QA ‘myths and traditions’ can get in the way

The need for strategyThe need for strategy• An example finding from Gibbs

o ‘greater explicitness of goals and standards was not associated with students experiencing the goals and standards to be clearer’

• And what did make a difference?

The need for strategyThe need for strategy• An example finding from Gibbs

o ‘greater explicitness of goals and standards was not associated with students experiencing the goals and standards to be clearer’

• And what did make a difference?o Formative-only assessmento More oral feedbacko Students ‘came to understand standards

through many cycles of practice and feedback’

Programme-based Programme-based assessment: assessment: PASS

• NTFS group project over 3 yearso development and investigation leading to

pilots and implementation

• Consortiumo Led by Bradfordo 2 CETLs – ASKE and AfLo Plus Exeter, Plymouth and Leeds Met.

What are we What are we investigating?investigating?

How to design an effective, efficient, inclusive and sustainable assessment strategy that delivers the key course/programme outcomes.

Also look out for:Also look out for:

And also …And also …• New JISC Programme

on Assessment and Feedbacko See the Strand A projects.

Typical student concerns Typical student concerns (based on PASS)(based on PASS)

• perceptions of ‘the course’ variable.• assessment experienced as ‘fragmented’.• anxieties re move to more integrated

assessment – perceived risk in terms of performance.

• Concerns about feedback and timing.

Searching for typesSearching for types

Searching for typesSearching for types

An example from PASS: An example from PASS: Peninsula Medical Peninsula Medical

SchoolSchoolIncludes:•four assessment modules that run through the 5 year undergraduate medical programme and are not linked directly to specific areas of teaching•focus on high-quality learning (Mattick and Knight, 2007)

Further case studies Further case studies being exploredbeing explored

• Brunelo New regulations which separate study and

assessment blocks.o Visit last week.

• Liverpool Hopeo New regulations which ‘abandon modules’ in

all undergraduate programmes.o ‘Key Honours Assessment’.o Visit now arranged

Brunel: the regsBrunel: the regs• 120 credits per year of study.• Course/programme can include mix of

study, assessment and modular blocks.• Option blocks must be modular.• Blocks must be in multiples of 5 credits• Maximum assessment block is 40

points

Examples from BrunelExamples from Brunel• Biomedical Sciences

o Study and assessment blocks in all years.o Cut assessment load by 2/3rds; generated

more time for class contact.o Synoptic exam in all three years.

• Mathematicso Conventional modules in final year only.o Improved understanding and ‘carry-over’ of

‘the basics’ into year 2.

Do you PASS?Do you PASS?

What can What can e-assessment e-assessment

achieve?achieve?

1. Processes and 1. Processes and systemssystems

Examples from the Examples from the Curriculum Design Curriculum Design

Prog.Prog.• eBioLabs (Bristol)

o By combining interactive media with formative self-evaluation assessments students learn the methods and techniques they will use in the lab, without risking valuable time, equipment or materials. Because students first experiment on-line there is a reduced chance of cognitive overload during the practical and they are more able to concentrate on the wider aims of the experiment, rather than blindly following the lab instructions.

o Because eBiolabs includes tools that automatically mark student assignments, and tools that allow academics to easily track student attendance and achievement, the marking and administrative burden associated with running practicals is very significantly reduced.

Curriculum Design Curriculum Design exampleexample

• CASCADE project (Oxford)o Online assessment submission

• Students can now submit assignments much more easily at any time from anywhere in the world. It is also possible to predict significant efficiencies in assignment handling time for the Registry staff who deal with student submissions for approximately 260 course assignments across 48 course cohorts a year: a saving of 30 minutes per assignment or more soon cumulates savings in the order of half a day per week. Other advantages of the new online system are the reduction in paper handling and photocopying, as well as better auditing and control. Reduction in paper storage is a further advantage both in terms of less physical space being required and also in terms of less staff time being required to retrieve data from the archive.’ 

Curriculum Design Curriculum Design exampleexample

• ESCAPE project (Hertfordshire)o Effectiveness vs efficiency. (watch the video)

o Next two slides from Mark Russell, Deputy Director of the Blended Learning Unit, University of Hertfordshire

ESCAPE Themes …ESCAPE Themes …

Good assessment for learning:…Engages students with assessment

criteria…Supports personalised learning…Focuses on student development…Ensures feedback leads to improvement…Stimulates dialogue…Considers staff and student effort

Numerous legacy resources

2. Feedback 2. Feedback and self-evaluationand self-evaluation

Example 2.1: audioExample 2.1: audio• The ASEL project

o led by Bradford with Kingston as partner.o various uses of audio, including feedback, in

different disciplines.

o See the intro by Will Stewart as part of the recent Leeds University Building Capacity project:

• Jorum resource • Direct weblink

ASEL noted:ASEL noted:• Technology now easy and accessible.• Positive student reactions.• Different tutor styles and

approaches.• Serendipity – e.g. feedback

stimulated podcasts.And• A different form of communication?

ASEL main ASEL main conclusionsconclusions

o … audio is a powerful tool, providing opportunities for personalising learning, promoting greater student engagement, and encouraging creativity. In introducing audio into their practice, lecturers were required to rethink their pedagogical approaches and learning design, adopting new and innovative ways to enable students to be more actively involved in the learning process.

ASEL main ASEL main conclusionsconclusions

o … (audio) allowed lecturers to provide more personal and richer feedback to students, and increased the level of interaction and dialogue amongst students and between students and lecturers.

(Will Stewart and Chris Dearnley)

Example 2.2:Example 2.2:audio and videoaudio and video

• Growing number of examples.o ALT/Epigeum Awards 2010: see the

ALT Open Access Repository

• See the winning entry by Read and Brown from Southampton:o Organic Chemistry.o Use of tablets to show solutions and working.o Focus on self-assessment.

Example 2.3:Example 2.3:clickers are comingclickers are coming

• Student Response Systems at the moment?o They work … they can change staff and

student behaviour and performance.

• Buto can be cumbersome and fiddly.o setup time.o need strong commitment and support (e.g. see

experience at Exeter Business School).

Example 2.3:Example 2.3:clickers are comingclickers are coming

• Student Response Systems in the future?o They will radically change staff and student

behaviour.o They will be flexible and easy to use.o They will be on the student’s own device!

• e.g. experimentation at University of Bradford arising from our JISC Building Capacity project – work with TxtTools by John Fairhall et al)

Example 2.4:Example 2.4:adaptive systemsadaptive systems

• PBL with consequences – you get immediate feedback on the consequences of your decisions.o e.g. The G4 project at St George’s

• http://www.generation4.co.uk/ • iEthics case online

• Adaptive assessmento e.g. the work of Trevor Barker

3. Integrating systems3. Integrating systems

Example 3.1:Example 3.1:integrating systems: integrating systems:

CAACAA• ITS4SEA project at Bradford

o 100-seater facility, now plus break-out 30 seats.

o Thin client technology.o QMP as University standard for summative

assessment.o Procedures agreed with Exam Office.o Design of room (available as cluster outside

assessment times)o Teaching potential.

The main The main CAA room at BradfordCAA room at Bradford

And the growth …And the growth …

And recent changes …And recent changes …• growth in ‘hybrid exams’:

o mix of automatic marking (QMP) and open ended response items (e.g. short answer questions).

o short answers to spreadsheet & marked by human.o likely use of word-processing in future.o Impact on teaching - new flexibility.

• Example of impact: o ‘reduced my marking load for this module from 5

days to one day, whilst still enabling assessment of higher order cognitive skills.’

Example 3.2:Example 3.2:integrating integrating applicationsapplications

• Use of mobile technologyo e.g. CampusM at Bradford:

• http://www.campusm.com/• http://www.techrepublic.com/software/university-of-bradford-about-

uob-10-mobile/2194295?tag=content;selector-1

And finally … And finally … back to the roleback to the role

• E-assessment can play an important role re:o Assessment authenticity and diversityo Feedback quantity and qualityo Profiling and ‘mindset’.

BUT ONLY IF• Embedded in meaningful

course/programme strategy/environment

And what is an ‘effective And what is an ‘effective assessment strategy’?assessment strategy’?

• Will it explain to staff, students and external agencies:o How the course/programme assesses the main

learning outcomes?o How assessment and teaching are linked?o How assessment both supports ‘high-quality

learning’ and develops it over the course?

And finally …And finally …assessment/identity assessment/identity

interfaceinterfaceStudents as ‘conscientious consumers’

(Higgins et al, 2002).But:• personal identity as ‘mediator’.

o e.g. apprentice (‘feedback is useful tool’) cf. victim (‘feedback is another burden’).

So need to change the mindsets of some students.

And finally finally …And finally finally …some other contactssome other contacts

PASSo Project Manager: Ruth Whitfield

[email protected]

ASELo Project Manager: Will Stewart

[email protected]

• CAA (building on ITS4SEA)o Project Manager: John Dermo

[email protected]

And maybe students And maybe students will never again see …will never again see …

59% Excellent.

This was the only tutor comment on a student assignment. How do you think the student reacted and felt?

And some other And some other interesting stuffinteresting stuff

• Challenging students to search Wikipedia creatively: C-Link

• http://www.conceptlinkage/clink /

• Helping students review and evaluate their interview performance

• http://www.gowerpublishing.com/isbn/9781409411369

• Helping research students to prepare for their viva: Interviewer VivaContact me for further info/demo.

Thank you for listeningThank you for listening

Peter [email protected]