Transforming assessment through the tenets Sally Brown and Brenda Smith 23 April 2013.

Transforming assessment through the tenets

Sally Brown and Brenda Smith 23 April 2013

2

Some thoughts on the nature of assessment as a transformative process to enhance student learning;

An overview of the six tenets;A review of how tenets one and two integrate holistically with

the other tenets;Some practical suggestions to support your work based on

tenets 1 and 2;Opportunities to develop your work and share ideas and issues

so you can start to make some meaningful changes at your institutions.

This session aims to offer:

Stems from firm evidence that assessment is not‘fit for purpose’ (Race 2010, Brown, 2010);

Aims to take a radical approach, recognising that it is time for significant reappraisal of assessment policy and practice (ASKe, Weston Manor Manifesto);

Builds on expertise, perspectives, and previous work; Takes an evidence-informed approach; Encourages for assessment to be seen as an integral part

of the learning experience.

Background to ‘A marked improvement’

3

QAA subject reviews Burgess Report - “system no longer fit for purpose” (p5) QAA - “…it cannot be assumed students graduating …. will

have achieved similar standards” (2007) Media accusations of dumbing down and grade inflation National Student Satisfaction Survey “There is considerable scope for professional development in

the area of assessment” (Yorke et al, 2000, p7) Students become more interested in the mark and less

interested in the subject over the course of their studies (Newstead 2002, p2).

Why change is needed

4

“The debate on standards needs to focus on how high standards of learning can be

achieved through assessment. This requires a greater emphasis on

assessment for learning rather than assessment of learning”

Tenet 1

5

Our current systems focused on marks and grades are not working: Belief that it is possible to distinguish the quality of work to a precision

of one percentage point (Elander & Hardman, 2002); Belief that double-marking will ensure fairness and reliability (Polani,

1998); Belief that consistency can be achieved through conformity, and simple

numerical rules (e.g. level 1 essay 3,000 words, level 3 essay 5,000; or no more than two pieces of assessment per module);

The combination of scores, which obscures the different types of learning outcome represented by the separate scores;

The distortion of resulting degree classifications by the application of idiosyncratic institutional rules (e.g. Armstrong et al, 1998).

Why change is needed

6

Students developing as learners

Informal feedback

Formal feedback

Practice, rehearsal

Formative and

summative

Authentic assessment

Assessment for learning

7

Adapted from Sambell, McDowell and Montgomery (2012, p5)

“Even when lecturers say that they want their students to be creative and thoughtful, students often recognise that what is really necessary, or at least what is sufficient, is to memorise” (Gibbs, 1992, p10)

We need to emphasise for students the importance of spending rich and purposeful time on task, not just going through the motions

We need to focus on high standards of achievement and to clarify what we seek

8

Are you confident that assessment tasks demand high standards of learning?

Is assessment for learning given emphasis in relation to assessment of learning?

Do you ensure an appropriate balance between formative and summative assessment?

Is assessment and feedback planned within and across a programme to ensure appropriate student preparation and practice before summative assessment takes place?

(From ‘A marked improvement’)

Key questions: To what extent and how do you evidence this?

9

“When it comes to the assessment of learning, we need to move beyond systems focused on

marks and grades towards the valid assessment of the achievement of intended programme

outcomes”

Tenet 2

10

Meaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan et al. 2004)

“We can know more than we can tell” (Polanyi, reprinted 1998, p.136)

Verbal level descriptors are inevitably ‘fuzzy’ (Sadler 1987)

There is a cost (in terms of time and resources) to codifying

knowledge which increases the more diverse an audience’s

experience and language (Snowdon, 2002) Tacit knowledge is experience-based and can only be revealed

through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991).

Why change is needed

11

Is their an emphasis on assessment for learning over systems focused on marks, grades and reliability?

Does the assessment design process ensure valid assessment of the intended learning outcomes?

Is there a trade off between reliability and validity of assessment?

Are assessment decisions in relation to design, development and variety made within a programme context and focused on learning outcomes?

(From ‘A marked improvement’)

Key questions: To what extent and how do you evidence this?

12

“ Limits to the extent that standards can be articulated explicitly must be recognised since ever more detailed specificity and striving for reliability, all too frequently,

diminish the learning experience and threaten its validity. There are important benefits of higher

education which are not amenable either to the precise specification of standards or to objective assessment”

(for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)

Tenet 3

13

The most significant factor in student academic success is student involvement fostered by student/staff interactions and student/student interactions (Astin, 1997)

The only common factor in a study of departments deemed excellent in both research and learning and teaching is high levels of student involvement (Gibbs, 2007)

“Participation, as a way of learning, enables the student to both absorb, and be absorbed in the culture of practice” (Elwood & Klenowski, 2002, p. 246)

Why change is needed

14

“Assessment standards are socially constructed so there

must be a greater emphasis on assessment and feedback

processes that actively engage both staff and students in

dialogue about standards. It is when learners share an

understanding of academic and professional standards in

an atmosphere of mutual trust that learning works best”

(for further discussion at later targeted forums, but here as part of a holistic

discussion of the tenets as a whole)

Tenet 4

15

Important aspects of complex, high-level learning outcomes can only be achieved when students are allowed time to ‘come to know’ the standards in use by the community:

Slowly learnt academic literacies require rehearsal and practice throughout a programme (Knight and Yorke 2004)

The achievement of high-level learning requires integrated and coherent progression based on programme outcomes

Where there is a greater sense of the holistic programme students are likely to achieve higher standards than on more fragmented programmes (Havnes, p. 2007)

Students need to engage as interactive partners in a learning community, relinquishing the passive role of ‘the instructed’ within processes controlled by academic experts (Gibbs et al, 2004)

Why change is needed

16

“Active engagement with assessment standards needs to be an integral and seamless part of course design and the learning process in order to allow students to develop their own, internalised, conceptions of

standards and monitor and supervise their own learning”

(for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)

Tenet 5

17

Changes in higher education (e.g. massification, reduced unit of resource, expectations of increased productivity in staff) threaten the ‘health’ of disciplinary communities and their ability to share and exemplify professional judgement.

There has been slow progress in the professionalisation of university teachers.

There has been limited attention paid to professional assessment practice.Reliance on the external examiner system to mediate standards within the

system is misplaced (Newstead and Dennis,1994).

If some aspects of high-level learning can only be assessed using professional judgement then we need to ensure that judgement is indeed –

professional!!

Why change is needed

18

“Assessment is largely dependent upon professional judgement and confidence in such judgement

requires the establishment of appropriate forums for the development and sharing of standards within

and between disciplinary and professional communities”

(for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)

Tenet 6

19

The current approach to assessment standards is inadequate;

The quest for reliability is getting in the way of learning;We should be enabling students to achieve high level,

complex learning;Active involvement by staff and students in the learning

community is essential to reach common understandings of assessment standards;

To achieve change there may need to be a review and evaluation of the allocation of time and resources within HE.

Why change is needed

20

Using the framework on the next slide, focus on tenet one to interrogate your own practices to date and where you can make improvements;

Identify between three and five ways in which you would like to make changes in assessment practice to make your assessment focus more on assessment for learning.

Prepare some summary bullet points on a flip chart sheet for sharing with other HEI teams.

Task: working in institutional teams

21

22

Tenet 1 – Key PointsAssessment for learning

Learning and assessment should be integrated, fully aligned, challenging & on-going Assessment should build on previous knowledge There should be a balance between formative and summative assessment Students need time for preparation & practice A range of approaches to feedback are needed Students need to give and receive feedback and feed-forward and engage in a dialogue re their learningTasks should involve the active engagement of students developing the capacity to find things out for themselves & learn independently

Are your students enabled to broaden their skills, knowledge and experience? Are a range of assessment methods used and constructively aligned? Is your assessment linked, cumulative and challenging? Are your students given opportunities to prepare and practice? Do your students engage in self and peer assessment & feedback? Do you help students and staff to understand the role of assessment and feedback in learning?Do your assessment tasks encourage independent thinking?Is your assessment spread throughout the academic year?

Using the framework in the next slide, discuss with colleagues what you consider your top priorities to work on from the questions and suggestions provided there;

Identify ways that you would measure and evaluate changes as a result of the actions you plan to take;

Ask a spokesperson to be prepared to report back on this in plenary.

Task: working in institutional teams

23

24

Tenet 2- Key PointsEnsuring assessment is fit for purpose

Systems that focus on marks and grades should be reviewed Emphasis should be on assessment validity Students should be offered a choice of assessments suited to their needs Assessments should demonstrate students have achieved the programme learning outcomes Course/programme teams needs to collaborate Assessment methods should be chosen for more than reliability

Do your students summative marks reflect the achievement of the programme learning outcomes? Is formative assessment built into your assessment cycle and does it feed forward? Do your colleagues work collaboratively across a programme to ensure coherence and validity? Do your staff ensure assessments are linked to the learning outcomes?Are you confident that students being marked by different people or the same people at different times (inter & intra-tutor reliability) will achieve equivalent marks? Does your University have a strategy for staff engagement and development in assessment and feedback to ensure up-to-date knowledge & experience?

25

Programme learning outcomes should reflect what students should achieve

Making it clear to students what is expected of them; Making it clear to teachers what students are expected to learn in

their own and other modules; Helping teachers to select the most appropriate teaching strategy for

the intended learning outcomes e.g. lecture, seminar, tutorial, group work, discussion, student presentation, laboratory work.

Helping teachers to select the most appropriate assessment style to assess the achievement of the learning outcomes, e.g.

project, essay, performance assessment, multiple‐choice questions, exam;

Having a focus on programme learning outcomes – staff therefore need time to collaborate;

Are you confident that students being marked by different people or the same people at different times (inter and intra-tutor reliability) will achieve equivalent marks?

Methods of AssessmentAssignments and the learning opportunity they

offer

Communicate information and ideas in other than the written

word

Become more adept at writing in different forms, formats and

‘genres’

To understand the benefits and challenges of working

collaboratively

26

Methods of AssessmentAssignments and the learning opportunity they

offer

Communicate information and ideas in other than the written

word

Become more adept at writing in different forms, formats and

‘genres’

To understand the benefits and challenges of working

collaboratively

Seminar, poster, video, video conferencing or other

multimedia

Design or proposals; book reviews; case report/study; web

pages; journal articles

Group problem solving; joint book reviews; team presentations; role

plays; group exhibition

27

Is the schedule of assignments and the feedback on these assignments made clear to students across the year?

Do you provide multiple opportunities for small low stake assessments with feedback?

Are your students given opportunities to prepare and practice?

Is there an appropriate balance between summative and formative assessment?

Is assessment and feedback planned within and across programmes and are staff given time to ensure this happens?

Balancing and timing formative and summative assessments

2013 University of Poppleton

28

29

Total study effort = 68 hours

Figure 1: A student’s estimate of his weekly study effort (in hours) on a programme with no

assignments and an exam in week 12

Gibbs (2010)

30

Total study effort = 78 hours

Figure 2: A student’s estimate of her weekly study effort (in hours) on a program with three assignments due in weeks 4, 7 and 10 and an exam in week 12

(Gibbs 2010)

Assessment for Learning

Do your students always know what the criteria are for all work they are given?

How do you help students to understand the criteria?

Could the students re-write the criteria in their own words?

31

Students can’t read our writing There is too much emphasis on grades and marks at the expense of learning The feedback given is often not very useful and comes too late Students are not actively encouraged to self reflect Little or no use is made of peer/self assessment and feedback Little dialogue takes place around feedback Students have little opportunity to collate feedback over time and act upon

it Most feedback does not feed forward, it only tell students what they have

done that is incorrect Very little use is made of feedback as a normal part of the learning and

teaching process Staff vary in their approach to feedback

Key issues in feedback – develop the capacity to use feedback effectively

32

33

Some thoughts on promoting the engagement of students by helping them to get confident

with assessing Give students in groups a selection of essay or other relevant piece of work

Excellent, Good, Pass, Near pass, Fail Let the students decide individually which is which and then compare notes In groups get them to tease out the differences between them and express

them as criteria Produce suggestions for writing an effective essay or relevant piece of work Get them to write appropriate feedback comments Exchange the comments with a different group and ask them to comment on

the feedback. Is it helpful, would they know what to do differently, does it feed forward etc?

What suggestions would they give to the group to enhance the feedback comments?

34

How you might help colleagues develop the capacity to use feedback effectively

In small groups exchange your assignments and compare the comments received

Do you understand the comments, if not discuss what you think they might mean?

How would you respond to them?

What would you do differently next time if you redid the same assignment and applied the learning to a new and different assignment?

Return marked work back to the individual but with no grade or feedback

Get them to mark their own work Collect in their grades (Pass a

sheet round) 9 out of 10 students will be

within 5% Arrange to talk to those where

the difference is more than 5% or one grade

Giving generic feedback within 24 hours on the VLEOffering face- to-face feedback with whole group/small groupsUsing peer feedback in groupsUsing technologies including:

A sound file – AudacityDragon software

Statement Banks“Sounds good project”

Encouraging students to use self feedbackHelping students to recognise that feedback can come from many

different sources (including themselves) and given in different ways

Saving time – a range of approaches to feedback need to be in place

35

Involve the students- Feedback as a dialogue

For Student Completion For Staff Completion

These are areas of my work that I think are good for the following reasonsPlease comment on the following areas of work

What I want to improve or do differently next time

The mark I think this piece of work deserves is

“When I get to sit down with my tutor and discuss my work – that’s probably the best form of feedback I can get” 36

Promoting the engagement of students across modules and programmes

LearningMost significant feedback commentsWhat did you understand these to mean?

AnalysingThings that I did that attracted positive commentsThings that I did that attracted critical commentsAny recurring trends or comments?

Planning and feeding forwardThings I can do to build on the positive feedbackThings I can do to address the critical feedback

(after Race, 2007). 37

All the tenets together provide a coherent agenda to bring about a marked improvement in assessment in your HEI;

The focus of the rest of the morning is to present your initial plans of action for enhancement with another pilot institution so that ideas and practices can be shared, developed and critiqued.

Bringing this thinking together

38

39

1. Using some ideas from the presentation and the publication “A Marked Improvement”, review how you might work with your colleagues back in your own institution to embed issues arising from Tenet 1 (Assessment for Learning) and Tenet 2 (Ensuring assessment is fit for purpose).

2. Working with another institutional task group, please present your plans to date and share your ideas and any issues that have arisen.

3. Now consider what are the opportunities and challenges that these two tenets offer and how might your plans change as a result of your conversations with colleagues here?

Paired activity in your institutional task group

40

References and further reading 1 Armstrong, M., Clarkson, P. and Noble, M. (1998), Modularity and credit frameworks: the

NUCCAT survey and 1998 conference report, Newcastle-upon-Tyne, Northern Universities Consortiuum for Credit Accumulation and Transfer.

ASKe Weston Manor manifesto http://www.brookes.ac.uk/aske/Manifesto/ (accessed April 2012)

Assessment Reform Group (1999) Assessment for Learning : Beyond the black box, Cambridge UK, University of Cambridge School of Education.

Astin, A. (1997), What matters in college? Four critical years revisited. San Francisco: Jossey Bass.

Bloxham, Sue. "Marking and moderation in the UK: false assumptions and wasted resources." Assessment & Evaluation in Higher Education 34.2 (2009): 209-220.

Brown, G. with Bull, J. and Pendlebury, M. (1997) Assessing Student Learning in Higher Education, London: Routledge.

Brown, S. and Knight, P. (1994) Assessing Learners in Higher Education, London: Kogan Page.Brown, S. Rust, C. & Gibbs, G. (1994) Strategies for Diversifying Assessment, Oxford: Oxford

Centre for Staff Development. Brown, S. and Glasner, A. (eds.) (1999) Assessment Matters in Higher Education, Choosing and

Using Diverse Approaches, Maidenhead: Open University Press.Brown, S. and Race, P. (2012) Using effective assessment to promote learning in Hunt, L. and

Chambers, D. (2012) University Teaching in Focus, Melbourne: Australian Council for Educational Research( Acer Press). P74-91

41

References and further reading 2Brown, S. (2011) First class: how assessment can enhance student learning in Blue Skies: new thinking

about the future of higher education, London: Pearson.Boud, D. (1995) Enhancing learning through self-assessment, London: Routledge.Burgess, Robert. Beyond the honours degree classification: Burgess Group final report. Universities UK,

2007. Elander, J. and Hardman, D. (2002), ‘An application of judgement analysis to examination marking in psychology’, British Journal of Psychology 93, pp. 303-328.

Elwood, J. and Klenowski, V. (2002), ‘Creating communities of shared practice: the challenges of assessment use in learning and teaching’, Assessment and Evaluation in Higher Education, 27, pp. 243-256.

Gibbs, G. (1992), Improving the quality of student learning. Bristol: TESGibbs, G (2007) ‘Departmental leadership of teaching’ Presented at The Oxford Learning Institute research

seminar series, University of Oxford. Hilary termGibbs, G. 2010, Using assessment to support student learning, Leeds Met Press, Leeds.Gibbs, P., Angelides, P. and Michaelides, P. (2004), ‘Preliminary thoughts on a praxis of higher education

teaching’, Teaching in Higher Education, 9, pp. 183-194.Havnes, A. (2007), ‘What can feedback practices tell us about variation in grading across fields?’ Presented

at the ASKe Seminar Series, Oxford Brookes University, 19th September.Laming, D. (1990) The reliability of a certain university examination compared with the precision of

absolute judgements, Quarterly Journal of Experimental Psychology Section A—Human ExperimentalPsychology, 42 (2), pp. 239–254.

Knight, P.and Yorke, M (2004),Learning, Curriculum and Employability in Higher Education. London: Routledge.

McDowell, L. and Brown, S. (1998) Assessing students: cheating and plagiarism, Newcastle: Red Guide 10/11 University of Northumbria.

42

References and further reading 3Nicol, D. J. and Macfarlane-Dick, D. (2006) Formative assessment and self-regulated learning: A

model and seven principles of good feedback practice. Studies in Higher Education Vol 31(2), 199-218.

Newstead, S. E. and Dennis, I. (1994), ‘Examiners examined: the reality of exam marking in psychology’, The Psychologist, 7, pp. 216-19.

Nonaka, I. (1991), ‘The knowledge-creating company’, The Harvard Business Review, November-December, pp. 96-104.

O’Donovan, B., Price, M. and Rust, C. (2004), ‘Know what I mean? Enhancing student understanding of assessment standards and criteria’, Teaching in Higher Education, 9, pp. 325-335.

Pickford, R. and Brown, S. (2006) Assessing skills and practice, London: Routledge. Race,P., Brown, S. and Smith, B. (2005) 500 Tips on Assessment: 2nd edition, London: Routledge

Polanyi, M. (1998) The tacit dimension, in: L. PRUSAK (Ed.) Knowledge in Organization (Boston, MA,Butterworth Heineman).

Race, P. (2010) Making learning happen (chapter 4) Sage, LondonRace, P. (2001) A Briefing on Self, Peer & Group Assessment, in LTSN Generic Centre

Assessment Series No 9 LTSN York.Race P. (2006) The lecturer’s toolkit (3rd edition), London: Routledge.Race (2007) How to get a good degree , London, Routledge.Rust, C., Price, M. and O’Donovan, B. (2003) Improving students’ learning by developing their

understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education. 28 (2), 147-164.

43

References and further reading 4Sambell, K McDowell, L and Brown, S 1997, ‘“But is it fair?” An exploratory study of

student perceptions of the consequent validity o assessment’, Studies in Educational Evaluation, vol. 23 (4), pp. 349-371. Knight, P. and Yorke, M. (2003) Assessment, learning and employability Maidenhead, UK: SRHE/Open University Press.

Sadler, D. R. (1987), ‘Specifying and Promulgating Achievement Standards’, Oxford Review of Education, 13, pp. 191–209.

Sadler, DR 1989, ‘Formative assessment and the design of instructional systems’, Instructional Science, vol. 18, pp. 119-144.

Sadler, R. (2008) Assessment of Higher Education, in International Encyclopaedia of Education

Yorke, M. (1999) Leaving Early: Undergraduate Non-completion in Higher Education, London: Routledge.

Yorke, Mantz, and Bernard Longden. (2007)"The first-year experience in higher education in the UK: Report on Phase 1 of a project funded by the Higher Education Academy."

Yorke, M., Bridges, P and Woolf, H. (2000), ‘Mark distributions and marking practices in UK higher education; some challenging issues’, Active Learning in Higher Education, vol. 1, no. 1, pp. 7-27.