New challenges in assessment of clinical competence

www.monash.edu.au

Neil SpikeDirector of Training & AssessmentSchool of Primary Health CareMonash University, Australia

New challenges in assessment of clinical competence

www.monash.edu.au

2

Answers?

We’re not actually going to tell you how to play the game; that’s something you’ll have to figure out for yourself. After all the object of the game is to discover The Meaning of Life, and life does not come with an instruction book. If it did, you’d be wasting all your days in the troubleshooting section, hoping to figure out where you went wrong. That would be no fun.

Manual for Monty Python’s Meaning of Life CD-Rom game.

www.monash.edu.au

3

Answers?

If you have the same ideas as everybody else but have them one week earlier than everyone else then you will be hailed as a visionary. But if you have them five years earlier, you will be called a lunatic.

Barry Jones

www.monash.edu.au

4

Questions?

Confidence never comes from having all the answers; it comes from being open to all the questions.

– Earl Gary Stevens

www.monash.edu.au

5

Overview

• Other parameters of utility• Changes in students• Changes in academic roles• Performance assessment• General professional competencies

www.monash.edu.au

6

Overview

• Other parameters of utility• Changes in students• Changes in academic roles• Performance assessment• General professional competencies

www.monash.edu.au

7

Utility function

• U = Utility

• R = Reliability

• V = Validity

• E = Educational impact

• A = Acceptability

• C = Cost

• W = Weight

U = wrR x wv

V x weE x wa

A x wcC

Adapted from Cees van der Vleuten

www.monash.edu.au

8

Utility function

• U = Utility

• R = Reliability

• V = Validity

• E = Educational impact

• A = Acceptability• C = Cost• W = Weight

U = wrR x wv

V x weE x wa

A x wcC

www.monash.edu.au

9

Utility function

• Introduce a new factor:– Feasibility

www.monash.edu.au

10

www.monash.edu.au

11

www.monash.edu.au

12

www.monash.edu.au

13

Overview

• Other parameters of utility• Changes in students• Changes in academic roles• Performance assessment• General professional competencies

www.monash.edu.au

14

Universiteit Maastricht

Student-centred assessment

- self-assessment

- peer-assessment

- co-assessment

self-evaluation skills

assessing the unmeasurable

assessing the unmeasurable

peer pressure?

peer-evaluation skills

self, peer and co-evaluation skills

peer pressure?

joint responsibility

www.monash.edu.au

15

Student changes

• Increasing diversity– Ethnicity

– Socio-economic status

– Previous school/educational experience

– Equity categories

– Basis of admission

Adapted from Frank Tesoriero

www.monash.edu.au

16

Student changes

• Student diversity means that previous assumptions on student characteristics may no longer hold true– Common values

– Preferred learning styles

– Levels of language proficiency

www.monash.edu.au

17

Student centredness

• Early needs assessment• Individual learning plans & portfolios• Changes to staffing ratios• Professional development for students• Increased time for increased assessments• Collating numerous assessments• Translating the standard into a cut score

www.monash.edu.au

18

www.monash.edu.au

19

Overview

• Other parameters of utility• Changes in students• Changes in academic roles• Performance assessment• General professional competencies

www.monash.edu.au

20

Universiteit Maastricht

SuggestionsAssessment is not mainly a measurement problem but more an educational design problem.

- built-in quality control procedures and bodies

- sampling in stead of (over) structuring

- variation of methods, compensation, regulations

- saturation of information approach

- training of students, teachers, examiners, board members

Masterplan of the assessment system

www.monash.edu.au

21

Academic changes

• In addition to traditional roles:– Research– Clinical service– Teaching– Consultancy

• Management of casualised teaching & assessment teams

www.monash.edu.au

22

Academic changes

• Need to maintain program integrity when teachers are casual and teach outside of a full program context.

• Need to adequately select, induct, support and manage of casual staff both individually and collectively.

• Need to engage casual staff who are often at margins of academic life, in the teaching and learning debates which have significant impact on them, their teaching and their students.

Adapted from Frank Tesoriero

www.monash.edu.au

23

Overview

• Other parameters of utility• Changes in students• Changes in academic roles• Performance assessment• General professional competencies

www.monash.edu.au

24

Universiteit MaastrichtBroaden the measurement time frame

Specific test occasions

Longitudinal testing

- specific preparation of students

- inflexibility

- unwanted influence on learning

- certain aspects unmeasurable

- no specific preparation - congruent with flexible curricula

- wanted influence on learning

- more performance assessment

www.monash.edu.au

25

Performance Assessment

• Indirect methods– Chart audit– Practice audits– Prescription data………

• Direct methods– Clinical (supervisor) ratings– Masked standardized patients– Video assessment– Patient reports– Peer reports– Clinical work samples………

www.monash.edu.au

26

Overview

• Other parameters of utility• Changes in students• Changes in academic roles• Performance assessment• General professional competencies

www.monash.edu.au

27

Universiteit Maastricht

General professional competencies

dealing with tasks

dealing with others

dealing with oneself

planning and time management

information gathering

co-operation

communicating

giving feedback and criticising

self-responsibility

self-evaluation

self-reflection

www.monash.edu.au

28

The aim of education is the knowledge not of facts but of values.

William Ralph Inge

www.monash.edu.au

29

Professional competencies

Quality assurance focus

• Traditionally includes:– The curriculum– The teaching and learning process– The assessment of competence

• Should also include:– Monitoring– Evaluation– Staff development– “Fitness for purpose”

www.monash.edu.au

30

Where do we start?

‘Would you tell me please, which way I ought to go from here?’

‘That depends a good deal on where you want to get to,’ said the Cat.

‘I don’t much care where’ said Alice.‘Then it doesn’t matter which way

you go,’ said the Cat.

Alice’s Adventures in Wonderland - Lewis Carroll

www.monash.edu.au

31