Evaluating your Teaching 2014-15 Edition

Centre for Learning and Teaching

Study Pack

Evaluating your Teaching 2014-15 Edition

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

2

Contents Evaluating your Teaching ........................................................................................................... 3

Introduction ............................................................................................................................ 3

WHAT do you want to evaluate? ............................................................................................ 3

Performance anxiety ........................................................................................................... 3

Activity 1 ............................................................................................................................. 3

Dealing with feelings ........................................................................................................... 4

WHY do you want to evaluate your teaching? ........................................................................ 4

WHEN is the best time to conduct evaluations? ..................................................................... 4

Activity 2 ............................................................................................................................. 4

WHO will you get evidence from? ........................................................................................... 5

WHO is evaluation for? - the wider picture .............................................................................. 5

Activity 3 ............................................................................................................................. 5

Activity 4 ............................................................................................................................. 5

HOW will you carry out and act on your evaluation? ............................................................... 6

Self -generated feedback .................................................................................................... 6

Feedback from colleagues .................................................................................................. 6

Activity 5 ............................................................................................................................. 6

Students ............................................................................................................................. 7

Feedback within a session .................................................................................................. 7

Activity 6 ............................................................................................................................. 7

Focus groups and interviews............................................................................................... 8

Designing a questionnaire ................................................................................................... 8

Activity 7 ............................................................................................................................. 8

Activity 8 ............................................................................................................................. 8

Activity 9 ............................................................................................................................. 9

Activity 10 ........................................................................................................................... 9

Activity 11 ........................................................................................................................... 9

Activity 12 ........................................................................................................................... 9

Designing your evaluation .....................................................................................................10

References and further reading .............................................................................................10

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

3

Evaluating your Teaching

Introduction

Evaluating the effectiveness of your teaching is an essential tool for improving it, and there are a number of key issues to consider: • WHAT do you want to evaluate? • WHY do you want to evaluate it? • WHEN is the best time to conduct the evaluation? • WHO will you get evidence from, and who is it for? • HOW will you do it and how will you act on the outcome? This pack is designed to clarify these issues and to help you:

develop your understanding of the main principles and purposes of evaluation

inform yourself about current evaluation practices in your own department and critically appraise them in the light of your readings and other research

explore a range of practical techniques for gathering feedback on your own teaching

plan and carry out an evaluation of at least one aspect of your teaching and act on the results

How to approach using this pack We suggest you go through whole pack, decide what you are going to evaluate, and then select the activities that are most well suited to help you do this.

WHAT do you want to evaluate?

There is little point choosing evaluation techniques until you are clear in your own mind what it is you really want to test - for example:

Your performance as a lecturer?

The overall quality of student learning?

The design of the curriculum?

Student enjoyment?

Performance anxiety When we first start teaching, our main worry is often “How am I doing?” rather than “How are the students learning?” In the same way, student evaluations frequently focus on how much they've enjoyed a course rather than on what they have actually learned. This isn't entirely

Activity 1

Which of the above list is most important to you at the moment?

What other elements would you add?

What kinds of evidence might be most useful to evaluate each aspect?

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

4

unreasonable – a good lecturer can motivate and inspire, and nobody ever learnt much from someone who sent them to sleep - so you do need to identify current strengths and weaknesses. We will be looking at a range of ways to get relevant feedback to help with this. However, bear in mind that your activities as a teacher are only a means to an end - improving student learning - and that there are many different ways in which you, your colleagues and the learning environment as a whole can contribute to that goal. You may need different kinds of feedback to help you evaluate each element. At the same time, it is common sense to focus on evaluating those aspects which you can do something about. If you are a new lecturer, you may prefer to concentrate initially on your teaching 'technique' and the design of individual teaching sessions and modules, rather than spending a lot of time evaluating the overall curriculum design, which you are unlikely to be able to change at this stage.

Dealing with feelings Evaluating our own teaching - and in particular dealing with feedback from students and colleagues - can be painful. It is important to recognise and deal with the anxiety this can engender, or we may ask only those questions to which we are likely to get favourable answers, or react defensively when faced with some negative feedback. Keep reminding yourself that this is about improving your students' learning - not about whether you are a 'good' teacher. At the same time, try to learn from any negative emotions that do surface. They should help you understand and deal more helpfully with students feelings whenever you assess their work.

WHY do you want to evaluate your teaching?

"Evaluation is not at heart about collecting evidence to justify oneself, nor about measuring the relative worth of courses and teachers. It is about coming to understand teaching in order to improve student learning." (Ramsden, 2003: 225)

WHEN is the best time to conduct evaluations?

The short answer is: all the time. However, different kinds of evaluation may be more effective at particular times and for specific purposes. You also need to be aware of your School practices. In the middle of a class you can respond to the cues alerting you that students may be confused or bored, by adjusting the pace or giving them something more active to do for a while. After each session, as well as at the end of the module, take time to reflect and make a note of what went well or badly, and how you could improve on it next time. Students often say,

Activity 2 Read Ramsden, 2003, Chapter 11 (reference at end of pack), which locates evaluation within some broader contexts and debates.

In order of priority, what are your main reasons at the moment for evaluating your teaching? elements would you add?

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

5

at the very end of their degrees or even a year or two afterwards, that they have only just realised the value of a particular element of the course - so bear in mind long-term evaluation as well as the instant sort.

The timing of more formal kinds of evaluation exercise can be vital, not just for the reliability of what you find out, but for its positive or negative impact on the people involved. Students are likely to resent repeated or apparently pointless surveys, so don't rush to carry out any exercise in student feedback until you have made sure what else is happening in your department. Timing is also important to letting people know what actions you have taken in response. (See 'Closing the Feedback Loop' below).

WHO will you get evidence from?

There may be a number of sources of data available to you for your evaluation. The most obvious is, of course, your own students, but you may wish to complement this with evidence from other sources, including, for example:

Peer observations from colleagues (or those outside of your subject area) – which you are required to do if using this pack as part of undertaking the PGCert in Learning & Teaching in HE).

External examiners.

WHO is evaluation for? - the wider picture

Although teaching evaluation should be conducted primarily for the benefit of your students and yourself, it is also carried out for a range of other reasons, both academic and managerial. Ramsden (Chapter 11) and Fry & Ketteridge (Chapters 13 & 14) include some discussion of these questions, which raise much wider issues than can be covered in this pack. However you should familiarise yourself with the quality management and enhancement systems used in your own department, and with national debates and practices, especially in your own discipline. Academic health and monitoring procedures within the University and partner colleges require us to collect evaluation data in a number of ways, and report on this through via a number of reporting structures and committees. You may need module evaluation to feed into a course academic health report, for example, which in turn will feed into School and Faculty health reports. More information on the academic health reporting processes can be found at: http://staffcentral.brighton.ac.uk/xpedio/groups/Public/documents/staffcentral/doc003272.pdf.

Activity 3 Take another look at your list of WHAT you want to evaluate (Activity 1). When would be the best times to carry out evaluations for each?

Activity 4

Read Hounsell, D. (2003) The evaluation of teaching, particularly pp 203 - 205 List all the different sources of feedback on your own teaching that are currently available to you, including 'incidental feedback’.

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

6

Information to help you to evaluate the effectiveness of your teaching can come from many different sources; for clarity, these can be divided into three main categories:

yourself – self-generated feedback

your colleagues - peer

your students

HOW will you carry out and act on your evaluation?

Self -generated feedback Making time to reflect on and analyse what works well - or not - in your own teaching is a key step to becoming a more effective lecturer. As well as basing these reflections on your memories of particular sessions, or notes made shortly afterwards, you could arrange to record a teaching a session (audio or video) to allow you to make a more sustained analysis. For instance, recording a seminar discussion can be done without too much technical intervention and may help you to analyse patterns of participation. You can use it to spot those moments where you jumped in too soon and answered your own questions; hindsight makes it much easier to think of ways to encourage and build on student contributions or to draw in the quieter members of the group. Recording one of your lectures calls for more planning but can be extraordinarily useful. You may get embarrassed by your own appearance or any mannerisms you didn't know about, but try not to focus solely on these aspects. Checklists can help you to take a more systematic approach.

Not all self-evaluation has to be retrospective - use the advice available in the other packs and your readings to help you assess your course design and other plans as you go along.

Feedback from colleagues Get into the habit of actively seeking informal feedback from all your colleagues, especially in the early stages of your teaching career. Some departments will allocate a mentor to offer more extended support, but most colleagues should be happy to share their experience to help you develop your skills. Rather than repeated general questions to everyone around you ("Is this OK? Should I have done this differently?") be selective and choose your informants and your questions carefully. Formal peer observation practices are becoming more widely used at Brighton and elsewhere. They can be an extremely powerful tool for improving teaching, as valuable for the observer as the observed, and a very good starting point for encouraging departmental discussions about

Activity 5

Use the self-evaluation questionnaire in Gibbs and Haigh (1985) or adapt one of the forms developed for peer observation of teaching (see below) to think about one of your teaching sessions, either immediately afterwards or via a recording. Identify one relatively minor aspect that you think could be improved quite simply. Plan and implement the change for the next session. Don't forget to evaluate the result!

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

7

student learning. You may find the University of Wisconsin-Madison pages on peer review useful: https://tle.wisc.edu/teaching-academy/definition-peer-review.

Students Since improving student learning is at the heart of the evaluation of teaching, it follows that students should be a key source of information. However, don't assume that surveys of student opinion will give you all the answers. A frequently neglected source of student feedback is what they do rather than what they say. Analysing their actual performance in all areas of your course for which you have any responsibility is a vital part of your continuous self-evaluation. If a substantial proportion of student assignments or examinations display similar weaknesses or errors, you need to ask yourself how you could improve their understanding of the topic. The more closely we align curriculum design, teaching and learning activities and assessment tasks, the clearer becomes the relation between student performance and staff evaluation. How students choose to prioritise their time can also be very revealing - could a change in your teaching help to modify their behaviour? If they don't sign up for your options, you should also consider the reasons, although beware of jumping to the conclusion that they simply don't like you or your subject - the causes are likely to be more complex than this.

Feedback within a session Some of the most effective ways of getting student feedback are also the simplest. For instance, get students to write down or discuss at the end of a class:

the main point (or three) they learned today

the main thing they still don't understand about the topic The advantage of this kind of approach is that it encourages students to think more actively about their own learning, which can have a very positive impact on their academic development. More elaborate ways to encourage students to reflect on their own learning, such as journals and logbooks, can be similarly beneficial for them while also providing a rich source of data to help you improve your teaching. Even informal surveys need a response, and 'instant' questions of the kind described above require that you make time to clarify any problem areas immediately or at the start of the next class. Similarly, sampling student lecture notes (Gibbs & Habeshaw, 1988:121) should be followed up as soon as possible - along the lines of "several of you seemed a bit confused about x, so I obviously didn't explain that very clearly. Let's see if we can sort it out together before we move on to the next topic…". This kind of open engagement helps to encourage questions and create a culture of active dialogue and collaboration.

Activity 6

Consider the various methods described in Gibbs & Habeshaw (1998) 53 Interesting Ways to Appraise your Teaching pp 117-123.

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

8

Focus groups and interviews As an alternative to the sometimes endless questionnaires, it is also useful to consider collecting qualitative data from your students, via interviews or focus groups. For an introduction to these methods, consult Glynis Cousin’s book (reference below) or use a standard social science research methods texts, such as Denscombe or Robson (again, references below).

Designing a questionnaire

Before you embark on asking students direct questions about your teaching, please bear the following key points in mind:

Students, like the rest of us, can suffer from questionnaire fatigue. Don't embark on any survey, however informal or well designed, until you know what questions they are being asked by other people about their experience of the course.

There is nothing more alienating than being asked for your opinion and then feeling it has been ignored. Students can become very cynical if they complete regular course questionnaires and see no evidence of any changes following. "Closing the feedback loop" is vital.

It is equally important not to act too hastily on the information derived from student feedback exercises. Remember that it may be unrepresentative, or based on only a partial knowledge of the course as a whole, or the questions asked may have been poorly designed, measuring popularity rather than the actual quality of student learning. Make sure you place it in the context of all your other sources of evaluation evidence.

Activity 7

Read Cousins, G. (2009) Researching learning in higher education: an introduction to contemporary methods and approaches, Chapter 4 on using focus groups. When might it be appropriate to use this method, and to answer what type of questions?

Activity 8 Designing the questionnaire

Decide whether this should be an individual questionnaire [see Appendix A of the Student Feedback pack from the University of Kent (2004) which is available in the studentcentral folder for this study for this pack].

If the student questionnaire normally used in your department is appropriate in content and timing, then you could use that. Otherwise, choose one of the examples in your specified readings, and adapt it to your needs.

Would 'open' or 'closed' questions be most useful at this stage?

Ask a colleague for comments on the draft version and incorporate any useful suggestions.

If you are planning an online survey then you could contact your Learning Technology Advisor for advice.

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

9

Activity 9

Find out about the following (consult colleagues if you are new to the department):

How does your own department collect student feedback: o On individual modules? o On the course as a whole?

What were the NSS scores?

What is done with this information?

How do students know what action has been taken in response to their comments?

On the basis of your work so far on this topic, are there any improvements you would make?

Activity 10

Read one or two recent evaluative documents in your department, preferably at different levels (e.g. module tutor's report, annual course monitoring document, external examiner's report, faculty academic health report...). These should be available in your school or faculty office

Is it clear how evaluative evidence has been collected?

Is it clear how the process has contributed to improving the quality of student learning?

Ask your colleagues to explain how the system works and whether they feel it could be improved

Activity 11

Having chosen your time (most appropriate point in the module, beginning or end of a scheduled class), explain to the students why you are conducting the evaluation. It's usually better to say something like "I'm looking for ways to improve what you get out of the course" rather than "I want to know what I'm doing wrong", and tell them how and when you expect to feed back the results

Once you've collected the data, analyse the responses. Do any patterns emerge? Were the answers what you expected? What short-term and long-term adjustments might they suggest you could make to your teaching?

Before you take any decisions, discuss the results with one or more of your colleagues, who may be able to put them in a broader perspective. Decide what actions are appropriate - now or in the future.

Activity 12 Closing the feedback loop In general, your response to students should be made as soon as you have had time to consider the issues properly, and should include the following elements:

summarise the positive points made as well as the main suggestions for improvement

let them know what modifications you plan to make during the remainder of the module

outline any changes that would only be possible the next time the course is run

explain what aspects you don't feel it would be appropriate to change - and why

give them an opportunity to comment on your response, but don't argue or become defensive

ask them to comment on the process and suggest their own improvements to the questions and timing

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

10

Take time to note your own responses to the whole process, and to monitor the effects of any changes in your teaching over the next few weeks, and in the longer term. If you were carrying out a similar evaluation exercise again, how would you do it differently?

Designing your evaluation

From working through this pack decide on your approach …

Identify one or more aspects of your teaching that you particularly want feedback on from students

Consider carefully the timing of the exercise (you may need to wait a few weeks if you have just started teaching a module).

References and further reading

Biggs, J. & Tang, C. (2011) Teaching for Quality Learning at University: what the student does. 4th ed. [electronic resource]. Maidenhead: Open University Press. Available at: http://capitadiscovery.co.uk/brighton-ac/items/1228819 (this links via the library catalogue where you need to follow the “text online” link to view the e-book). [The 3rd edition of this text is also available as an ebook] Brown, S. et al (1993) Observing Teaching. Birmingham: SCED Paper 79 [available in the studentcentral folder for this module] Cousins, G. (2009) Researching learning in higher education: an introduction to contemporary methods and approaches. New York: Routledge. Denscombe, M. (2010) The Good Research Guide for small scale social research projects 4th ed. Maidenhead: McGraw-Hill: Open University Press. Available at: http://capitadiscovery.co.uk/brighton-ac/items/1236226 (this links via the library catalogue where you need to follow the “text online” link to view the e-book). Denscombe, M., 2002. Ground Rules for Good Research: a 10 point guide for social researchers. Maidenhead: Open University Press.

Eastcott, D. (1992) Evaluating your teaching, SCED paper 68. Birmingham: Standing conference on educational development. [available in the CLT library]

Fry, H., Ketteridge, S. & Marshall, S. (2003), A Handbook for Teaching and Learning in Higher Education 2nd ed. London: Kogan Page Also available as an ebook in myilibrary: Fry, H., Ketteridge, S. & Marshall, S. (2003) Handbook for Teaching and Learning in Higher Education, A: Enhancing Academic Practice. [online]. Taylor & Francis. Available from: http://ezproxy.brighton.ac.uk/login?url=http://www.myilibrary.com?id=7419 Gibbs, G. & Habeshaw, S. (1988) 53 Interesting ways to appraise your teaching Bristol: T&ES

Produced by the University of Brighton, Centre for Learning and Teaching 2014-15

11

Gibbs, G. & Haigh, M. (1985) A compendium of course evaluation questionnaires Birmingham: SCED Paper 17. [available in the studentcentral folder for this study pack]. Ramsden, P. (2003) Learning to teach in higher education. 2nd ed. London: Routledge. Also available as an ebook via myilibrary: Ramsden, Paul. 2003., Learning to Teach in Higher Education. [online]. Taylor & Francis. Available from: http://ezproxy.brighton.ac.uk/login?url=http://www.myilibrary.com?id=2596 Robson, C. (2002) Real world research: a resource for social scientists and practitioner-researchers 2nd ed. Oxford: Blackwell.

University of Wisconsin, Peer review of teaching [online]. Available at: https://tle.wisc.edu/teaching-academy/peer-review-teaching.

Useful Websites

http://shop.brookes.ac.uk/browse/extra_info.asp?compid=1&modid=1&deptid=47&catid=227&prodid=948 Part of the excellent “First Words” on-line guide for new teachers

http://www.uio.no/for-ansatte/arbeidsstotte/kvalitetssystem/kvalitetssystem-

utdanning/verktoykasse/feedback.pdf Contains documents from the Sharing Excellence Together project (SET), including a Resource

Pack for Academic Staff on the university’s Peer Review System, and a Student feedback starter pack, with excellent guidance on developmental peer observation of teaching and a useful range of proformas. http://www.prodait.iehosting.co.uk/approaches/observation/ ProDAIT’s (Professional

Development for Academics Involved in Teaching, at the University of Birmingham) pages on peer observation; see also their pages on assessment tools for Evaluating teaching: http://www.prodait.iehosting.co.uk/approaches/observation/ All the web links cited in this pack were checked 8th September 2014.