Analysis of feedback webinar

JISC Assessment and Feedback Programme

Webinar

Analysis of Feedback

24 June 2013 12.00 - 13.00

Holly Smith IoE Assessment Careers [email protected]

Anne Jones QUB e-AFFECT [email protected]

Maria Fernandez-

Toro

Open University eFEP [email protected]

Peter Chatterton Daedalus e-

World

Critical Friend [email protected]

Institute of Education

www.ioe.ac.uk/assessmentcareers

Queen’s University Belfast

http://www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/e-AFFECTproject/

Open University http://www.open.ac.uk/blogs/efep/

Speakers

http://www.ioe.ac.uk/study/departments/lce/59520.html

http://www.ioe.ac.uk/study/departments/lce/59520.html





http://www.open.ac.uk/blogs/efep/


http://www.ioe.ac.uk/assessmentcareers






To explore the following:

• Why analyse feedback?

• Approaches and tools for analysing feedback

• Institutional experiences of using the tools - resulting

feedback profiles and audits

• Benefits, impact and challenges from using the tools

Webinar objective

Why analyse feedback?

Why analyse feedback?

• Widely inconsistent practice in feedback (quality & quantity)

• Lack of learner understanding of, engagement with, and dialogue/action & on feedback

• High teacher effort - low efficiency

• Not utilising self/peer feedback

• Transmitted feedback creates dependency on teacher

• Reduced staff satisfaction as evidence of feed-forward not seen

• NSS scores

Dependent learners

feedback “done to them”

Independent learners

capable of self-review

Approaches and tools for analysing feedback

IOE Coding framework

• The score is the number of times a classification appears in the feedback

• The default unit for analysis was the sentence

• Where a sentence contains clauses that make distinct points, it was split into separate clauses, each of which was classified separately.

• Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified

IOE Feedback tool

P1 Giving praise

P2 Recognising progress or ipsative feedback

Criticisms

C1 Correction of errors

C2 Factual criticisms

C3 Criticism of approach

Giving advice

A1 Specific to content current assignment

A2 General skills in current assignment

A3 For future assignments

Q Clarifications and questions

O Other unclassified statements

Adapted from Orsmond & Merry, 2011 including Hughes, 2011.

e-AFFECT - What the students say

Last year their feedback

pointed out spelling

mistakes or referencing

mistakes, but we were

not told how to do things

right

What do they mean

by clear and concise?

Sometimes they say

something to

encourage that is not

really true – am I

excelling at it or are you being nice?

I liked feedback which

helped me improve my

work the next time. Despite

this, I felt that my marks

never really changed much

and they tended to stay at

the same level

What do ‘?’ and

‘What?’ in the

margin mean?

e-AFFECT - Analysis of coursework

• Content analysis of depth of feedback using adapted categories developed by Glover and Brown (2006)

• Indication that there is an strength/error/weakness or omission (Level 1)

• Provides correction or appropriate response/indication why a strength (Level 2)

• Provides explanation as to why the student’s response was incorrect or inappropriate or why suggestion was preferable or how a strength can be built upon (Level 3)

Essays Lab reports Total

Year 1 40 26 66

Year 2 61 24 85

Total 101 50 151

• Analysis criteria based around two dimensions:

o Whether feedback focuses on strengths or weaknesses

o How much information the feedback provides, cf ‘depth’ of

feedback (Brown & Glover 2006) layers of scaffolding

• Analysis tool: Feedback Analysis Chart for Tutors (FACT)

Provides a visual ‘profile’ of a tutor’s feedback

Analysing assignment feedback

Screencast description: http://www.open.ac.uk/blogs/efep/?page_id=114

http://www.open.ac.uk/blogs/efep/?page_id=114

Layers Comments focusing on weaknesses Comments focusing on strengths

1 Error identified only

Strength identified only

2

Error categorised, but not corrected

Strength categorised or described as per

marking criteria

3

Error corrected

Illustrated with specific example from

student’s

performance

4

Explanation given

Explanation given

5

Advice given on how to prevent errors in

future performance

Advice given on how to develop

existing strengths in future

Manos blancos Manos blancos Good work Good work ?? ??

Layers of scaffolding in assignment feedback

Manos blancos Agreement Manos blancos Agreement

Although it ends in O, ‘mano’ is

a feminine noun.

Although it ends in O, ‘mano’ is

a feminine noun.

Manos blancos blancas Manos blancos blancas

Revise section 6.1 of

your grammar book

Revise section 6.1 of

your grammar book

You use a wide range of

language structures

You use a wide range of

language structures

No digo que quieran... Good use of the subjunctive

No digo que quieran... Good use of the subjunctive

This connector makes it very clear

that a new section is starting here.

This connector makes it very clear

that a new section is starting here.

Good, you could

also look up...

Good, you could

also look up...

Feedback Analysis Chart for Tutors (FACT)

Possible uses:

Enables us to compare...

feedback relating to different criteria

feedback given by different tutors

feedback given to more/less proficient students

feedback related to different types of assignment

(e.g. spoken presentations vs. written essays)

feedback delivered through different media

(e.g. written vs. audio-recorded feedback)

Language

Weaknesses Strengths

Identified only Identified only

Categorised Categorised / Described

Corrected / Modelled

Exemplified

Explained Explained

Future-oriented Future-oriented

Example of FACT analysis grid

(tutor A: beginner assignment)

Weaknesses Strengths Identified

only 6

Identified

only

Categorised 2 Categorised / Described


8 Exemplified

Explained 3 Explained

Future-

oriented

Future-

oriented





1 Exemplified

Explained Explained



only

Identified

only



Exemplified

Explained Explained

Future-

oriented 1

Future-

oriented

Content Feedback form

Notes on script

Language





6 Exemplified



Example of FACT analysis grid

(tutor B: advanced assignment)


only 4

Identified

only

Categorised 1 5 Categorised / Described


2 6 Exemplified

Explained 5

Explained

Future-

oriented

Future-

oriented





4 2 Exemplified

Explained 1 1 Explained



only

Identified

only

Categorised 1 6 Categorised / Described


2 Exemplified


Future-

oriented

Future-

oriented

Content Feedback form

Notes on script

Questions?

Institutional experiences of using the tools

IOE Data

• Analysed formative and summative assessment

feedback for modules on 5 postgraduate programmes

at the IOE (total 228 pieces)

• Recorded the total number of comments in each

category and the average per script

• Ranked the categories to obtain a feedback profile at

programme level as well as an aggregate profile of the

5 programmes.

IOE Profile for Summative Assessment

Total comments for 5 programmes

Summative assessment (N= 165)

Category of feedback Average per script Rank

P1 Praise 4.4 1

P2 Ipsative (progress) 0 (negligible) 5

C1-C3 Critique 2.7 2

A1-A3 Advice for current or future

assignments

1.9 (mostly for current

assignment)

3

Q Questions and clarification

requests

0.1 4

e-AFFECT - Comments on essays

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

Student response Student skills Achievement

%

Distribution of comments at Year 1

Strengths

Weaknesses

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Student response Student skills Achievement

%

Distribution of comments at Year 2

Strengths

Weaknesses

e-AFFECT - Further learning comments

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

Sourcematerials

Dialogueand

reflection

Future work

%

% Year 1 work

% Year 2 work

e-AFFECT - Depth of feedback

Depth of feedback

Indication that there is an

strength/error/weakness or omission (Level 1)

Provides correction or appropriate

response/indication why a strength (Level 2)

Provides explanation as to why the student’s

response was incorrect or inappropriate or why

suggestion was preferable or how a strength can

be built upon (Level 3)

-30 -20 -10 0 10 20

1

2

3

%

Depth of feedback

Year 2

Weakness Strength

-15 -10 -5 0 5 10 15

1

2

3

%

Depth of feedback

Year 1

Weakness Strength

-40 -20 0 20 40

1

2

3

%

Depth of feedback

Year 1 Module A

Weakness Strength

-30 -20 -10 0 10 20 30

1

2

3

%

Depth of feedback

Year 1 Module B

Weakness Strength

-60 -40 -20 0 20 40

1

2

3

%

Depth of feedback

Year 2 Module C

Weakness Strength

-40 -20 0 20 40

1

2

3

%

Depth of feedback

Year 2 Module D

Weakness Strength

e-AFFECT – Module depth of feedback

e-AFFECT - Feedback workshop

Analysing what?

Feedback on language assignments at the OU

Our sample:

100 writing assignments

100 speaking assignments

4 levels (9 tutors per level)

108 students (3 per tutor)

Feedback consists of:

200 e-feedback forms

100 annotated scripts

100 audio files

Tutor Student

Well done!

Summary of FACT analysis results: Use of the four media

E-feedback forms (writing/speaking)

Contain the lowest proportion of…

Comments on strengths

Comments on content

Categorised strengths/weaknesses

Advice for future (proportion on script = 0%).

Quite low everywhere else (both in the audio

feedback and on the e-feedback forms)

Script annotations/Audio feedback

Contain the highest proportion of…

Comments on weaknesses (often

adressed at more than one depth)

Comments on language

Corrected errors

Explanations (especially high occurrence

in the audio feedback)

Contain the lowest proportion of…

Comments on weaknesses

Comments on language

Corrected errors/Exemplified strengths

Explanations

All of these occur a bit more frequently for

the speaking assignment

Contain the highest proportion of…

Comments on strengths

Comments on content

Categorised strengths/weaknesses

All of these occur even more frequently

for the written assignment

Questions?

Benefits, impact and challenges from using the tools

IOE Benefits, Impact and Challenges

• Enables initiation of discussion of feedback at the programme team level

• Facilitates reflection on the purpose of feedback within programme teams

• Feedback practices are very entrenched and resistant to change

e-AFFECT

Benefits, impacts and challenges from using the tools

Benefits

• Staff seeing the real issue rather than an imagined issue

• Engendering dialogue

• Raising awareness of feedback messages

Impacts

• Work in progress

Challenges

• Reaching consensus of the level and quantity of

feedback

• Useful research tool: Overall patterns of use of different media for giving

feedback on language assignments at the OU

• Results need to be interpreted with caution (e.g. ‘deeper’ feedback is not

necessarily the most appropriate in all contexts)

• Not suitable for quantitative evaluation by practitioners: Coding requires

complex guidelines in order to be reliable

• Suitable for awareness-raising purposes in staff training events. Materials

include: sample of marked assignments + coding grids + student webcasts

giving their ‘feedback on feedback’

(Online training event: 67% ‘very useful’ – 33% ‘possibly useful’)

• FACT criteria now also presented as a simplified checklist for reflection

• Informs new research strand focusing on feedback alignment

Benefits, impact and challenges

from using the FACT analysis tool

Questions?

Further info on projects

More info on projects

Assessment Careers-Institute of Education

www.ioe.ac.uk/assessmentcareers

http://youtu.be/VSaGbPoXPh0

References:

1. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg (Eds) Innovative assessment in higher education. London, Routledge), 81-91.

2. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational Research, 77, 81-112.

3. Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 367.

4. Orsmond, P. & Merry, S. (2011) Feedback alignment: effective and ineffective links between tutors’ and students’ understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2): 125-126.

Holly Smith [email protected]

http://www.ioe.ac.uk/assessmentcareers









Queen’s University Belfast

e-AFFECT


http://blogs.qub.ac.uk/e-affect/

Anne Jones [email protected]















Open University

The eFeedback Evaluation Project (eFEP)


Screencast description:


Maria Fernandez-Toro [email protected]





Analysis of feedback webinar

Education

feedback analysis chart

different tutors feedback

different criteria feedback

ioe feedback toolp1

written essays feedback

audiorecorded feedback

analysis criteria

good use