Experiences of audio assessment feedback: Staff and Student experiences Carol Ekinsmyth Department of Geography University of Portsmouth ([email protected])

Experiences of audio assessment feedback: Staff and Student experiences

Carol EkinsmythDepartment of GeographyUniversity of Portsmouth

([email protected])

Why audio feedback might be important

Teaching and learning in higher education is often confounded by dominant cultures and quality audit frameworks that prevent the achievement of these desirable goals:

The sharing between ‘teachers’ and ‘learners’ of integrated knowledge structures (Kinchin, Lygo-Baker and Hay (2008)

The breaking down of “pedagogic resonance” (Trigwell and Shale (2004), Kinchin et al (2008)

Helping students arrive at and move beyond “liminal space” and develop understanding of “threshold concepts” (Meyer and Land, 2005)

Assessment and feedback

Researchers agree that the majority of student effort (learning?) is geared towards assessment (Carless et al, 2006; Brown, 2005; Gibbs, 2006)

Carless et al (2006) encourage us to engage in “learning orientated assessment”

Smith (2007, 11) argues: “If assessment is for learning, then feedback needs to focus on what the student could do differently and how to apply that learning to future work.”

Price (2007) and Baumard (1999) argue that the trend towards over-reliance on written feedback in HE crushes the ability to effectively encourage tacit knowledge formation and concentrates instead only on explicit knowledge

Assessment and feedback

“Good feedback comprises not just commentary about what has been done, but suggestions for what can be done next” (Brown, 2007)

And she argues;

“I believe that concentrating on giving students detailed and developmental formative feedback is the single most useful thing we can do for our students” (Brown, 2007)

Race (1999) argues “...feedback quality and quantity are probably the most important factors in enhancing student learning”

And yet, my experience suggests that Staff are reluctant to spend a lot of time on feedback ......

Why the reluctance over feedback?

Time pressures Uncertainty about what feedback is for:

Kinchin et al (2008) talk about more fundamental barriers – complicity between teachers and students in “non-learning outcomes”

“Correcting errors, providing guidance for future work, justifying the mark, satisfying quality

assurance processes, diagnosing difficulties, the list goes on.” – Price (2007)

How can audio feedback help?

It can’t - if teachers are unwilling to use it We perhaps need to concentrate on overcoming the

barriers to teacher-engagement Audio feedback seems especially suited to feed-

forward as a supplement to assessment proformas that often concentrate on feedback

Used in carefully identified places in the curriculum, it may be a powerful tool in breaking down pedagogic resonance

Why?

Because it enables the students to get closer to the thought processes (and thus community of practice) of the academic than written feedback does

Face-to-face contact might not be better as it can be intimidating and the details often cannot be recalled by students (my research findings)

The Project

GEES/LTSN (JISC ‘Sounds Good’ project)

Aims:

To evaluate the utility, efficiency, popularity and effectiveness (amongst students and staff) of audio feedback on assignments (range of types) for undergraduate students within the discipline of Geography.

Other reasons to bother

UK National Student Survey (2005-2008) shows that the area that students are least satisfied with is ‘assessment and feedback’ (Race and Pickford 2007)

The centrality of assessment to the student experience is widely acknowledged (Ribchester et al (2008), Brown 2005)

Research has shown that on average, 90% of all student study activity is directly related to assessment (Ribchester et al (2008))

Feedback that academics provide may not be effectively used by all students (Higgins et al (2002))

Academics and students have been shown to differ on their perception of what makes good feedback (Carless (2006))

“…the literature on student experiences of feedback tells a sorry tale” (Handley et al (2007, p1) quoted by Ribchester et al (2008))

Shriver (1992) has shown that hearing feedback helps writers appreciate the effects of their writing on the reader (Merry and Orsmond (2007))

The technologies exist so why not experiment!

An example

My feedback – A , B

Probably easier to leave feedback when

the work is good rather than poor.

Reactions: Student experiences

Student audio files - Response D Response A

Response B

Response D

Student Reactions: Positive thoughts

Key aspects of audio feedback that students liked were; Its more detailed nature Tone of voice (enriches experience) Clarity – “You can take it in easier” Not reliant on handwriting Its more personal nature – “I listened

carefully because it is more personal” Its potential for feed-forward

Students taking more notice?

“I think it is more articulate than written text. I think it allows you as a marker to articulate your points a lot better than a written line of text does - and it allows you as a student to take the feedback on

board more. I mean, I normally dismiss the written feedback form if I’ve done really badly”

“And you can’t read ahead. You have to listen to it all. You don’t know what’s coming (general

laughs). When you’ve got a sheet you can skim-read it ….”

Student Reactions: Positive thoughts

It is definitely

more memorable because

it is a different

way”

“Written feedback is often too brief and difficult to read”

“I take information in better when it is in a spoken form”

“I find it interesting to hear how you are criticising my work (tone)”

“Overall I think it is extremely useful”

“I really liked it”

“Very easy to use”

“It was the most feedback I have ever had on a single piece of work”

Student Reactions: Negative thoughts

It doesn’t replace the written feedback

Nothing visual to look at Can be too personal for comfort You can’t answer back/ engage in

dialogue

Student Reactions: Negative thoughts

Strange at first – “felt

like my lecturer

was in the room with

me”

“..perhaps made you more nervous.. Builds a bit of tension”

“I found that some of the feedback was very critical and it upset me more as an audio feedback

than a written feedback”

“Personally I do prefer hard copy –

I’m a bit old fashioned like that … so I do

think you shouldn’t loose

hard copy of feedback – you

can keep that for life then”

“My main dislike was that I had nothing visual to look at“

“It was a bit more personal, therefore I feel harder to take the criticism”

“It would wear off. It

can’t be used all the time – or it won’t go in it won’t be registered at

all well”

Staff Reactions: Focus group discussion

There was a reluctance to like this method – why?

Reactions focused on issues of: Exposure and feedback quality Feasibility/practicality Appropriateness Best applications Context limitations and litigation Language Skill acquisition

Staff Reactions: Exposure and feedback quality

Do we ‘hide’ behind written feedback conventions?

Staff Reactions: Exposure and feedback quality

Do we ‘hide’ behind written feedback conventions?

“When I was doing audio, I couldn’t hide anything… it made me think more about

what good feedback constitutes”“You can hide

more in written feedback – you are much more explicit when

speaking something so you need to think much

more carefully”

This isn’t to suggest that teachers are intellectually lazy, but under inevitable

time pressures during summative marking

processes, it was felt that feedback short-hand

comments require less time and effort than

comprehensive spoken feedback.

Do feedback proformas ease or exacerbate this?

Staff Reactions: Exposure and feedback quality

Written feedback, especially that coded onto generic feedback sheets perhaps leads us to write feedback in a form of short-hand that needs to be deciphered by the students.

Do we teach students to decipher this short-hand well enough?

Staff can ‘hide behind’ this short-hand which, for reasons of lack of time or the repetition of the process, reduces the need for great qualification of opinions or development of comments.

In audio feedback, you are able to develop this feedback with “for example.............”

“You do not develop your arguments thoroughly enough”

“You need to contextualise

these findings by referring to

debates in the academic literature”

“Your discussion is too anecdotal”

Staff reactions: A consensus

Feed-forward rather than feed-back is where the greatest strength of this method lies

Not everyone however is convinced of the need for thorough feed-forward – why? Unitised curricula structures Goal-oriented teaching and learning structures “Economies of practice” for teachers and

students (Kinchin and Hay 2007, 98) Teachers and students complicit in a cycle of

“non-learning” (Kinchin and Hay 2007, 98)

Staff reactions: Feasibility and practicality

It takes longer if you are going to do it properly (i.e. Use the method to its greatest potential)

“I think that if had undertaken better preparation for my 5

minutes of audio feedback, by preparing the comments in

advance, then the work necessary (and time taken)

would have been much greater than what I do when I’m giving written feedback,

and we do need to keep coming back to that“

““My gut feeling will still be … phew…. it’s a lot of work”“I don’t think it will ever be

widely applicable”“At the moment based on

my own experiences, I think the same, its beyond us but it may be because it is new, its learning a new language

and new skills””

Staff reactions: Practicality

“You do need a quiet room to sit down … we

work in a fragmented way and slot in marking … to make it efficient. Sitting

down to record a narrative was

problematic. To have a continuous narrative for 5

minutes was difficult.”

Staff reactions: Practicality and disembodiment

“The practicality of it, to me, is my main barrier – I think ideologically and pedagogically there are some benefits but the shear practicality, the fact that we almost disembody the feedback from the written text and then to reconnect – I’m not sure there is anything lost but I expect there is… but simply the recording of it, thinking of what you are going to record, it is a two-stage process

whereas when writing, it is all in one”

Staff reactions: Language and Litigation

“It is much more difficult if you are being natural to control the language that you are using. Now in our feedback we have to work to language criteria – it’s a very controlled environment – excellent is a first, very good is a 2.1 – so audio feedback, it’s almost impossible to work within those limitations - and if you do work to them, it is pointless.”

“The audio feedback has to be different from the written in that it can display some emotion or passion”

Do we need to accompany audio feedback with written guidance as a disclaimer?

Staff reactions: Applications

“I’d be happier giving feedback in a formative sense released from the

constraints of grade point

criteria and the associated language”

“We have sanitised our written feedback so it is probably less memorable. There’s clearly a

role for it, you can motivate and stimulate much more easily it is

much more memorable.”

Formative feedbackFeed-forward

But what of the novelty value?

“I’d be a little wary of replacing written comments with audio feedback – not least from a

quality perspective, records for External Examiners and so on”

Staff reactions: Re-accessibility

“The re-accessibility aspect is important. They can go back to parts of the written feedback but would

they listen to a 5 minute file again if they only needed a bit? Also, they have to listen to the

negative criticism again if there is some, and they might not want to do this and thus may not be able

to bear to listen again”

Will students keep it and listen to it again?

How do they keep it with the work?

Do they need instructions?

Conclusions/ Recommendations I

Excellent for formative feedback/ feed-forward – perhaps not so useful (or necessary) for summative feedback.

Staff were more cautious than students, remaining unconvinced overall of the cost-benefits but recognising great strengths and potentials if used judiciously.

For summative feedback, neither staff nor students recognise audio feedback as a replacement for the rule-governed written feedback.

Students in particular felt that audio feedback provided a much more detailed and richer account of the strengths and weaknesses of the work.

Conclusions/ Recommendations II

Students felt that hearing feedback was more effective and memorable than reading it but both staff and students felt that the sporadic use of audio feedback was important in this respect.

Thus the judicious use of use of the audio medium at the most appropriate moments would be the best approach.

Staff had concerns about the need to be very careful about language and tone in audio feedback, and felt that a separation of intent between written feedback and audio feedback would be useful.

In a nutshell

Excellent for feed-forward Saves time in delivering detailed feed-forward Not time-efficient for vast piles of scripts Best times to use method in the curriculum/degree

programme need identifying A key problem is breaking through established

economies of practice of teachers with regard to feedback

Central to this is a re-evaluation amongst practitioners of the goals, possibilities and importance of feedback

Thank you

Carol EkinsmythPrincipal Lecturer

Department of GeographyUniversity of Portsmouth

References

Baumard, P (1999) Tacit Knowledge in Organizations, London, Sage Brown, S (2007) Feedback and feed-forward, Centre for Biosciences Bulletin, 22, Autumn 2007 Brown, S. (2005) Assessment for learning. Learning and Teaching in Higher Education 1, 81-89. Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education,

31, 219-233. Carless D, Joughin G and Mok M (2006) Learning-orientated assessment: principles and

practice. Assessment and Evaluation in Higher Education 31, 4, 396 Gibbs G (2006) How assessment frames student learning. In C Bryan and K Clegg (Eds.),

Innovative Assessment in Higher Education, Routledge, London. Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more:

Student’s experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc

Higgins, R., Hartley, P., Skelton, A. (2002).The conscientious consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27(1), 53-64

Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more: Student’s experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc

References

Kinchin I, lygo-Baker S and Hay D (2008) Universities as centres of non-learning, Studies in Higher Education 33, 1, 89-103

Merry, S., Orsmond, P. (2007) Feedback via MP3 audio files. Bioscience HEA Academy Centre for Excellence Bulletin, Autumn 2007

Meyer J and Land R (2005) Threshold concepts and troublesome knowledge (2) – epistemological considerations and a conceptual framework for teaching and learning, Higher Education, May

Price M (2007) Should we be giving less written feedback?, Centre for Biosciences Bulletin, 22, Autumn, 9

Race, P., Pickford, L. (2007). Making Teaching Work: Teaching Smarter in Post-compulsory Education. Sage, London.

Race, P (1999) Enhancing Student Learning, Birmingham: SEDA Ribchester, C., France, D., Wakefield, K. (2008) ‘It was just like a personal tutorial’: Using

podcasts to provide assessment feedback, paper presented at the Higher Education Academy Conference July 2008

Shriver, K. (1992) Teaching writers to anticipate reader’s needs. Written Communication 9(2) 179-208

Smith B (2007) Is assessment really for learning?, Centre for Biosciences Bulletin, 22, Autumn, 11

Trigwell K and Shale S (2004) Student learning and the scholarship of University teaching, , Studies in Higher Education 29, 4, 523-36

Appendix

Practicalities

Phases and progress

Phase one – Experimentation

1. Using audio feedback in a range of ways (individual feedback, group feedback, generic feedback, ‘bite-sized’ feedback

2. Trialling WIMBA as the enabling platform

Phases and progress

Phase 2

1. Staff experiences of using audio feedback recorded in a videoed focus group. Video to be podcasted for use in staff-training workshops.

2. Student experiences and evaluations of the method elicited through Wimba voice email, focus groups and questionnaires.

Phases and progress: Future plans

Phase 3Investigation into where in the curriculum audio feedback can be most effective

Phase 4Evaluation and dissemination (including production of staff-training materials)

Constraints

With limited time and large student numbers, any method will need to be measured in terms of a cost-benefit analysis

Giving more detailed feedback takes time – so what methods are available that might help in this respect?

Phil Race (2008) – on feedback(www.phil-race.co.uk retrieved 12.12.08)

Two of his key recommendations are:

Making sure they get feedback quickly enough so that they still care about it

Ensuring that they get plenty of feed-forward, so they can make their next piece of work better

Audio feedback – my method

Read work, make notes against the assessment criteria.

Also make notes on each section of the work (perhaps paragraph by paragraph for a traditional essay). This might be in the margins of the work.

Take a note of overall feedback. Make a voice recording of this feedback. Send to student. Provide summary notes on a conventional

marking sheet for return to the student.

Staff Reactions: My experiences

Can be daunting at first leaving the recording but you quickly gain confidence and do not feel the need to listen to the message

Despite worries about tone of voice or in authenticity of sound, if you are natural and speak as you normally would, the recordings are well-received

You get quicker at the whole process – at first you think it will never work. Having said this, it is more time-consuming than methods I am used to (but the quality of feedback is much greater)

In spoken form, I could go into depth about issues and make suggestions that I simply wouldn’t have time (or inclination) to do in written-form.

Minute for minute therefore, it is much quicker to speak than to write.

You need to record the feedback after you have read an

Audio feedback- Colleague A’s method

Read work and record feedback instead of writing it on marking sheet – trying to take no longer than he would normally take in conventional marking/ feedback production

Audio feedback – Colleague B’s method

Provide feedback in a PowerPoint presentation – scanning in sections of the student work and using ‘narration record’ to provide explanation of points.

Hardware & Software The recordings could be made using any

MP3 or digital recording devise but sending files then takes time.

VLE/Wimba Voicemail can be used and for this, only a headset is needed.

PowerPoint can be used for audio-visual feedback

Wimba audio archive