Engagement: Tensions and Solutions Dr Colin Mason Dean, Teaching and Learning.

Engagement: Tensions and Solutions

Dr Colin Mason

Dean, Teaching and Learning

Tensions in Tertiary Education

TEACHING

RESEARCH

ACCOUNTABILITY EVALUATION

LEARNING QUALITY

FORMATIVE ASSESSMENT

SUMMATIVE ASSESSMENT

ELITE HOMOGENOUS TRADITIONAL LECTURES INDUCTION

MASS DIVERSE APPLIED INNOVATION ENGAGEMENT

Outline

• Some of the tensions in tertiary education (worldwide)

• Case Studies: – Scottish Approach to Quality assurance (ELIR and

Enhancement themes)– Research and Teaching linkages– Assessment practice– A case for individualised student support?

• A model for solutions to tensions in continua• Staff engagement too

Teaching Quality – Enhancement-led Institutional Review (ELIR)

• Universities monitor quality themselves at subject level

• Periodic review by QAA

• Different systems in England and in Wales

• Scotland better?

• Emphasis on Enhancement

Scottish Peer Review 5 Yr Visits

• Visit Team 5 people including– 3 Academic staff, – 1 student, – review secretary- administrator

• Process– Reading documentation– Interviewing students/staff who come to the

reviewers– No visits to individual Schools or Units

SFC/QAA Approach to Enhancement Themes (2003-4)

• Institutional focus groups and surveys– good practice– development needs

• Identifying international best practice to meet development needs identified through the surveys

• National and International experts invited to share their expertise widely in Scotland

• One-two day National conference for students/staff

QAA (Scotland) Enhancement Themes

• 2003– Responding to Student Needs– Assessment

• 2004– Employability– Flexible Delivery

• 2005– Integrative Assessment– The First Year

• 2006– The First Year– Research-Teaching Linkages

Assessmenthttp://www.qaa.ac.uk/scottishenhancement/events/default.htm

Review of degree classification:– Assessment seminar: Considering the Honours

degree classification system - 11 May 2004

8 Workshops on Good practice– (Jan – June 2004)– Key note (international) presentations– Case Studies (Scottish)– Discussion groups

Book

Booklet and 2 Volumes

published: 17 June 2005

STOP

PRESS

Assessment Workshops

1 - Streamlining assessment - How to make assessment more effective and more efficient - 13 January 2004

2 - Using assessment to motivate learning - 5 February 2004

3 - Constructive alignment of learning outcomes to assessment methods - 27 February 2004

4 - Developing a variety of assessment methods, including self and peer assessment - 19 March 2004

5 - Assessing online - 16 April 2004

6 - Issues of validity, reliability and fairness - 7 May 2004

7 - Improving feedback to students (link between formative and summative assessment) - 4 June 2004

8 - Assessment of Personal Transferable Skills - 29 June 2004

QAA (Scotland) Enhancement Themes

• 2003– Responding to Student Needs– Assessment

• 2004– Employability– Flexible Delivery

• 2005– Integrative Assessment– The First Year

• 2006– The First Year– Research-Teaching Linkages

Projects undertaken by the sector

‘Expertise’ delivered to the sectorE

VOLUTION

How this affected student learning quality at St Andrews

• Learning outcomes, curricula and assessment, including programme specifications

• The student learning experience

• Mentoring for staff

• Peer observation of teaching

• Funded Innovation

• Raising the status of teaching?

Tension between Research and Teaching

“If it wasn’t for all these (UG) students I could do more

research”

“Without all these (UG) students how could I do any

research”

Research-Teaching Linkages

• Boyer Commission– Reinventing undergraduate education: A

blueprint for America’s Research Universities (1995)

– 10 ways to change undergraduate curriculum

Boyer’s Ten ways to …..

1. Make research-based learning (“mini-projects”) the standard (involve UGs in research process; mentor for every student; internships)

2. Inquiry-based first year (“curiosity, problem-based learning; including group projects”)

3. Building in the second year (including integration of direct entry)

4. Remove barriers to inter-disciplinarity (flexible course ‘swaps’)

Boyer’s Ten ways to …..

5. Link communication skills and course work (“Presentations and writing – ‘down’ as well as ‘up’”)

6. Use Information Technology creatively eg electronic classrooms (“exploit digital technology and media”)

7. Culminate with a capstone experience – (“compulsory project or dissertation”)

Boyer’s Ten ways to …..

8. Graduate students as apprentice teachers (“Postgraduate tutors”)

9. Change faculty reward systems – highest standards in teaching and research (“and reward systems to match”; evaluate teaching – look for integration of research with teaching)

10. Cultivate a Sense of Community (“strong social as well as academic opportunities”; see diversity as an asset)

A model of Research-Teaching linkage in curriculum design

Curriculum is research-led; – current content-based teaching or lecturing

Curriculum emphasises teaching focused on disciplinary values, ethics, approaches

Curriculum is designed around students writing & presenting on academic articles

Curriculum engages students in inquiry or problem-based learning

Res

earc

h C

on

ten

tR

esearch P

rocess

Teacher Focus

Student FocusGriffiths, Brew, Healy: 2004-2005

Some practical ways forward

• Jenkins and Healey (2005)– Develop research policies and strategies to

strengthen the link • “Schools (Departments) are expected to identify research

policies that support the curriculum and underpin teaching (Gibbs, 2003)”

• Mason 2006 (ELIR visit)– Fund for Innovation in Learning, Teaching and

Assessment (FILTA); criterion for project bids that explicitly link how author’s research will be integrated with the teaching innovation

An interlude (with a purpose)

Q. Write down the names of 10 colours

A. 1 Black2 White3 Yellow4 Blew5 Green6 Red7 Orrange8 Indigo9 Broon10 Violet

10

8.5

7

5

0

Almost any number

(0-10)

Score this answer out of 10

9

Assessment Marking Criteria COLOUR “Seen” as

a “colour” Misspelt -

Lose 1 mark

Misspelt - Lose ½ mark

Accepted as a colour

& spelt correctly

Black 1 1 1 0 White 1 1 1 0 Yellow 1 1 1 1 Blew 1 0 ½ 0 Green 1 1 1 1 Red 1 1 1 1 Orrange 1 0 ½ 0 Indigo 1 1 1 1 Broon 1 0 ½ 0 Violet 1 1 1 1

Total 10 7 8½ 5

Tensions in Assessment

RELIABILITY

VALIDITY

ATOMISTIC

HOLISTIC

TUTOR ASSESSOR PEER ASSESSOR

MASTERY RECALL

CREATIVITY ORIGINALITY

EXAMINATIONS

COURSEWORK

INDIVIDUAL GROUP

FORMATIVE

SUMMATIVE

Forms (Modes) of AssessmentDIAGNOSTIC

FORMATIVE

SUMMATIVE

FORMATIVE

SUMMATIVE

DIAGNOSTIC

‘Theory’ ‘Reality’

Complex outcomes of learning

• ‘Complex’ learning outcomes including:– higher order academic abilities (analysis,

synthesis, evaluation) and – ‘soft skills’ (eg teamwork, communication, time

management) – are rarely, and often poorly and inconsistently

defined.

• Such learning is:– Advanced and difficult to ‘measure’– Slow to develop (100s or 1000s of hours?)

– Fuzzy (precision & reliability often only possible at the expense of validity).

Myths about assessment

• Assessment ≠ measurement

• Problems with summative assessment (Knight 2002)

• Limits of number• High-precision, high reliability• Threats to validity• Representing all aspects of student achievement

• So ……

Shift to Formative Assessment

• Knight, PT (2002). “Summative Assessment in Higher Education: practices in disarray”. Studies in Higher Education, 27, No.3, 275-286.

Assessment Feedout

(Summative)

Student deception

Feedback

(Formative)

Student disclosure

A shift in the balance of modes of assessment?

• Low-stakes or formative purposes– Create informed feedback on achievement in order to

improve future achievement: •SENLEF Higher Education Academy and Scottish Higher Education Developers

– http://www.heacademy.ac.uk/ourwork/learning/assessment/senlef

•REAP, Scottish Funding Council 2005-07 (D Nicol, Strathclyde)

– http://www.reap.ac.uk/

• Specifically think about ‘Feed-forward’ as well as feed-back

How to increase ‘feed-forward’?

• Encourage students to submit drafts – provide feedback and then they re-submit

• First years are particularly vulnerable– What about having no summative

assessments in Semester 1?

• For coursework:– Apply progressive weighting to assignments

to minimise impact of adjusting to a new environment

A model for assignment weightings to improve ‘feed-forward’

Coursework Examination

Ass

ignm

ent

1

Ass

ignm

ent

2

Ass

ignm

ent

3

10% 20% 30% 40%

Summary

• High-stakes, summative assessment has limitations for ‘complex’ learning

• Reliability & precision is expensive• Low-stakes, formative assessment is good for

learning• Low stakes can reach achievements that elude

high-stakes approaches (eg teamwork)• Blended assessment: preferably planned at a

programme level.

So, what about the first year?

• Induction is a problem for anyone new to an institution

• Cultural differences may exacerbate the problem

• What is happening in the UK (Scotland)?

When you are in deep shit, say nothing, and try to look like you know what you are doing

Entering the university: the differentiated experience of two Chinese international

students ….

• Longitudinal study on a single course (module?) and their personal experiences to establish an identity

• B. Business Studies, 12 students enrolled, 2004• 3x semi-structured interviews – Semester 1 (&

follow up)• 2 groups emerged:

– Negotiated a pathway for survival– No obvious headway in new setting

Capturing the journey

Before I come I take IELTS and I think, ‘Oh, they speak English and study in that is very good’

‘But when I come to here many things is different’ ..

Different learning and no people to help you in the English .. And you don’t know what it means

‘Mike’

New approaches (when expected to learn for themselves -DIY)

.. When I come I feel I can taste a little bit sweet of the successful, because I achieve a goal and I never give up, and keep trying, keep trying, keep trying

‘Saul’

.. Before the lecture I read the books and see the WebCT and then the tutor say anything, I know. Because I read the books, textbooks. And I think it is not difficult.

‘Mike’

A reality check – the first test

.. Actually, I have no point about what is the lecturer want us to catch … I don’t know which part I should focus on, should concentrate.

‘Saul’

.. The first few weeks of study I have no idea. I haven’t read the book and I just attend the lecture and then listen and no [no comprehension]. At home I have read no books

‘Mike’

Mike finds a strategy (how to learn)

.. After the test I thought, oh, the lecturer just ask us to focus on this .. That is teacher’s way to test us which part is important and which is unimportant.’

‘Saul’

.. I borrow my friend’s last year’s test, I saw that and, oh, this question relates to the textbook and my notes and I should read my notes and text carefully with the lecture notes!

‘Mike’

Both fail the exam but … Mike reflects on why; Saul is bewildered

.. ‘You know, I go to every workshop and I go to every lecture and I take marvellous notes, and before the tests I used one week to review.’

‘I think whole semester is quite bad’

‘Saul’

.. ‘I realise when I read the text or books I think [if] I understand, I can remember, so I just general read the books and then actually I can’t remember.’

‘Mike’

Author’s interpretations

• Both students tried to meet course demands

• Both used methods based on previous experience

• Both were confused

• Both were able to reflect but …

• Mike was able to develop new insights and developed a sense of self-efficacy

Author’s interpretations

• Large first year classes were not conducive to approaches in previous language learning classes

• So …• Mike recognised the value of reading before

lectures (and he recognised he needed to work harder)

• Saul was pre-occupied with discovering what was important or necessary and less with how necessary it was to understand (deeply)

Author’s conclusions

• The institution and staff did not have good systems in place to support these students.

• Even for Mike, smaller groups would be necessary, with early opportunities to engage with teaching staff

• Every Cloud ….– Saul repeated the course ….

Saul finds the silver lining

.. ‘Because this is my second time to study this course so I already know all things and all the key point and all the stuff which the lecturer want to ask to understand it, you know, so our study group, it’s like I’m the leader of that.’

.. ‘Oh, Saul said this’. ‘Oh, that’s very important, better write that down.’ Saul thinks …’ You know, like little teacher, you know.

‘Saul’

Entering the university: the differentiated experience of two Chinese international students in a New Zealand University

[Massey]

Gillian Skyrme

Studies in Higher Education, 32, 3, pp357-372, 2007.

QAA Enhancement Theme:The First Year - Projects

• Curriculum design for the first year• Formative and diagnostic assessment and

feedback• Peer support in the first year• Personal development planning• Personalisation of the first year• Introducing scholarship skills• Transition

£10,000; Collaborative, multi-institutional

Conclusions

• Engagement is (and always will be) about curriculum design

• What appear to be polar opposites aren’t necessarily – consider ‘horse-shoe’ models?

• Staff engagement is as vital as student engagement

So, how is Unitec doing?

Main themes in Conference Abstracts

02468

Induction

Authentic or real learning orassessment

Problem-based learning

Portfolios and reflectivelearning

Enthusiasm and motivation

Evaluation of teaching

Finally, some ways to ‘pull’ more staff along with you

• Funded projects: to promote research-linked, teaching innovation and professional development

• Prizes for teams, as well as individuals, to develop innovative approaches

• Disseminate good practice; and develop ‘others’ (mentoring and coaching colleagues) as well as self

• Accountability– Performance review based on evidence (with follow-up)– Report on professional development opportunities– Academic (e)-portfolios and profiles for promotion (and prizes)

• CTLI Support & Guidance: – School/individual focus – Teaching symposia; Mini-symposia

‘Away-days’; Communities of Practice; informal ‘conversations