Student evaluations of teaching: Perceptions determining teacher behaviours
Sarah Stein, Jo Kennedy, Trudy Harris, Stuart Terry, Lynley Deaker, Dorothy Spiller
Presentation at the Higher Education Research and Development Society of Australasia (HERDSA) conference, Hobart, 2012.
2
The project
Title: “Unlocking the Impact of Tertiary Teachers’ Perceptions of Student Evaluations of Teaching”
Ako Aotearoa (National Centre for Tertiary Teaching Excellence) National Project Fund grant ($150k)
2010 - 2011
3
Project partners
from left:top: Stuart, Sarah, Lynleyfront: Jo, Dorothy, Trudy
Otago Polytechnic
• Stuart Terry
University of Waikato
• Dorothy Spiller• Trudy Harris
University of Otago
• Sarah Stein• Jo Kennedy• Lynley Deaker
4
Issues
academics’ hostility towards student evaluations
academics are resigned to the notion of student evaluations (Beran & Rokosh, 2009)
=/= improvements in teaching (Kember, Leung & Kwan, 2002) or serious engagement for development (Beran & Rokosh, 2009; Burden, 2008)
two (competing?) purposes for student evaluations:1. audit (monitoring, gauging teaching effectiveness)2. development of teaching and courses
claims that the two purposes are complementary (e.g., Ramsden, 1992)
BUT perceptions are very important
5
Issues
interrelated factors (contextual, philosophical, practical and personal) influence academics’ perceptions of evaluations and their use of them:
validity and reliability are still cited, despite the evidence (e.g., Benton & Cashin, 2012; Marsh, 1987; Theall & Franklin, 2001)
institutional expectations and community norms (e.g., Nasser & Fresko, 2002)
limitations of student judgement (Aleamoni, 1981)
quality of the institution’s evaluation/appraisal instruments (e.g., Ballantyne, Borthwick & Packer, 2000; Penny & Coe, 2004)
institutional ownership and use of evaluations (e.g., Edström, 2008; McKeachie, 1997; Nasser & Fresko, 2002)
individual teacher’s teaching beliefs (Hendry, Lyon & Henderson-Smart,
2007) and emotions (e.g., Moore & Kuol, 2005)
6
Issues
Perceptions do matter
“it doesn’t matter much what the institution’s intended purpose is [or what research evidence suggests]. What is important is what the individual teachers perceive to be
the purpose”
(Edström, 2008, p. 100, emphasis in original)
7
Research questions
How do current formal student evaluation/appraisal processes and practices influence teachers’ thinking and behaviours in relation to student learning at all stages of the teaching and learning cycle?
What are the perceptions that tertiary teachers hold about student evaluation/appraisal?
What factors (causes, influences) affect these views?
How do tertiary teachers engage with student evaluations/appraisals?
8
Design & Methods
overarching interpretivist research approach (Erikson, 1998)
combination of quantitative and qualitative data: online questionnaire semi-structured interviews
key ideas and issues identified through the literature review contributed to the development of the data gathering tools and to the analysis of the resulting data
9
Online questionnaire
Likert-scale and open response questions
4 parts: Section A - explored current practices (Q1-8) Section B - explored perceptions of student
evaluation data and influence on practice (Q9-22)
Section C - demographic information (Q23-32) Section D - interview availability (Q33)
2,426 teaching staff invited 1,065 responses received (44%)
10
Semi-structured interviews
20 from each institution, purposively selected (Patton,
1990) from volunteers (Q33)
range of academic discipline, career stage, seniority
core interview questions based on the key themes identified in the questionnaire responses
teaching and learning beliefs students’ capacity to make judgements personal, emotional factors other factors e.g., timing, promotion,
engagement with evaluation
11
Analysis
Questionnaire comments & interviews thematic analysis - involved searching
for themes, using a constant comparative technique (Dye, Schatz, Rosenberg & Coleman, 2000; Silverman, 2001)
Likert scale questionnaire questions ANOVA, Kruskal–Wallis test
12
Findings: Questionnaire data
widespread recognition in the questionnaire that collecting evaluation/appraisal data was worthwhile
(Q17 Do you personally think it is worthwhile to gather student evaluation/appraisal data about teaching and courses/papers?)
OU WU OP0
102030405060708090
100
freq
uen
cy o
f re
spo
nse
s (%
)
very worthwhile (1&2)
middle ground (3)
not at all worthwhile (4&5)
combined institutionsby institution
13
Findings: Questionnaire data over half the respondents find their centralised system
effective at gathering useful/meaningful data for them (1 or 2 rating). Sixteen percent, on average, found the centralised system not effective (4 or 5 rating)
(Q19 How effective is your institution’s centralised evaluation/appraisal system in gathering useful/meaningful student data for you?)
very ef-fective (1&2)
middle ground
(3)
not ef-fective (4&5)
OU WU OP0
102030405060708090
100
frequency
of
responses (
%)
combined institutions by institution
14
Findings: Questionnaire comment dataQ18 Please explain your answer to Q17
enhancing role of evaluations
[Evaluation] is great way to learn about what is good and what is bad - hard to be totally objective about your methods etc and students are great at honesty in this forum!
(Q18 sub theme 1b, OP, lecturer teaching position, 0-5 years’ tertiary teaching experience, permanent, Sciences)
limiting role of evaluations
What is not good in our system is the standard format which is unsuited to so many diverse courses.
(Q18 sub-theme 2c, WU, lecturer teaching position, 11-15 years’ tertiary teaching experience, continuing, Humanities)
I do not however, approve of the institution's tendency to use them [evaluations] as weapons against staff. A heavy-handed hierarchical approach from academics with little knowledge of the course or the students and even less interest in teaching or its context, is counter-productive.
(Q18 sub theme 2a, OU, lecturer teaching position, 16-20 years’ tertiary teaching experience, permanent, Humanities)
15
Findings: Questionnaire comment & interview data
mistrust about students’ reliability
You should treat them [student evaluations] with a pinch of salt. Students don’t have a long term perspective. Personality is a big factor. (OU, interview)
I used to believe that [students can make judgements about teaching], but now I no longer believe that. I think in terms of how…students are believing they are buying a qualification..it’s more like purchasing their degree. (WU, interview)
Students have bullied staff and they use evaluations as an opportunity to dump on staff. (OP, interview)
preferences for other forms of student feedback
The rating questions are rather useless but perhaps useful for a promotion committee to make broad judgements. That is their sole value, nothing else. The reason for that is that they do not specifically tell you what is wrong or what is right. The comments do that best. Also, as mentioned earlier, the statistical rigour in many of these appraisals would make a real statistician seriously question their meaning.
(Q18 sub-theme 2a, WU, lecturer teaching position, 11-15 years’ tertiary teaching experience, continuing,
Sciences)
16
Questionnaire comment & interview data
suspicion of institutional use of student evaluations It is valuable, but only if taken in context! One negative comment and 100
positive ones is a very good result. However, management have a tendency to focus on that one comment. Often there are reasons other than the quality of the teaching for negative evaluations.
(Q18 sub-theme 2a, OP, senior teaching position, 6-10 years’ tertiary teaching experience, permanent, Health Sciences)
lack of faith in the process and instrument The institution tries to do too much with this limited data. (OU, interview)
The student evaluations instrument is unreliable, contrived and manipulated. (OU, interview)
suspicion of manipulation by colleagues People are more careful to choose questions that are more likely to yield
a positive response. (OU, interview)
17
Findings: Summary
Teachers are generally positively disposed towards the student evaluations, although not well informed about student evaluation purposes and effective use;
Perceptions of student evaluations seem to be connected to:
their expressed teaching beliefs and emotions;
their concerns with the quality of student evaluation instruments;
misgivings about students' competency to judge;
disenchantment with a student evaluation system that can be manipulated easily by academics;
lack of institutional support for, and recognition of, teaching; and
their preoccupation with research.
(Context and personal experience determined the extent of these views.)
Evaluation tends to be seen as an individual and private activity.
Many teachers (predominantly university) have grave reservations about institutional reliance on a single evaluation instrument to measure the quality and effectiveness of teaching/courses.
18
Implications for practice & change: Making connections or ‘closing the loop’For institutions:
ensure that there is a clear alignment between institutional vision/policy statements and processes of implementation;
recognise and acknowledge that student evaluation is first and foremost about development, and therefore the developmental and auditing purposes of student evaluation should be clarified within that frame;
be aware that expectations about roles and responsibilities in evaluation/appraisal can be ambiguous, and so connections among performance, evaluation and reward need to be clearly understood by all (teaching and administration/personnel staff and students).
19
Implications for practice & change: Making connections or ‘closing the loop’For evaluation systems:
recognise and acknowledge the that staff perceptions about evaluation vary and provide appropriate support and resources, to address teacher expectations and needs, without compromising institutional intents and purposes
include processes and practices (including an ongoing professional development strategy) that target both developmental and auditing purposes, while recognising the complementarity of the purposes
recognise that multiple forms of evaluation are more likely to represent a ‘well-rounded’ description of teaching and courses.
20
Questions? More information?Sarah SteinHigher Education Development Centre (HEDC)University of OtagoDunedin, New [email protected]
Look for the full and summary reports, soon to be on the Ako Aotearoa website