Workplace assessment Dr. Kieran Walsh, Editor, BMJ Learning.

Workplace assessment

Dr. Kieran Walsh, Editor, BMJ Learning.

Workplace assessment – pluses

• Lots of clinical material available• Skilled clinicians/teachers available• Strong educational impact

Workplace assessment – potential difficulties

• Broad range of problems/patients/procedures required

• Integrated skills • Potential to practice • Specific assessment experience/expertise is

often rare • Small number of trainees in each discipline

in an average sized hospital

Workplace assessment – theory

• Knows• Knows how

• Shows how• Does

Workplace assessment – judgement based on….

Single encounter• Actually observed performance • Case specific Routine performance• Observation over a period on different

occasions on different problems • Reduces problems with case specificity

Workplace assessment – nature of judgement

Did it happen? • Checklist approach • Objective• Structured• Consistent• But not very discriminatory

Workplace assessment – nature of judgement

Was the trainee any good? • Global rating• More valid

Workplace assessment – nature of judgement

Was the trainee good enough for a particular purpose?

Workplace assessment – influence on learning

• Formative assessment and feedback

• Direct observation (in the past many trainees never observed in a patient encounter)

Workplace assessment – faculty development

Workshops needed

• Training in behavioural observation • Performance dimension training• Frame of reference training

Workplace assessment – reliability

• Number of encounters observed

• Number of assessors

• Aspects of performance being assessed

Workplace assessment – is it a fair test?

• Different patients• Different assessors

• Faculty development• Common problem list• Number of assessors for each trainee

Workplace assessment – stakes

• Assessments influenced by what is at stake• Colleagues reluctant to make negative

assessment when stakes are high

BUT• External examiners• Faculty development • Anonymity in peer assessment• Should trainees choose their assessor?

Workplace assessment – costs

• Planning• Time of trainees and assessors • Real patients• Training of both groups• Administration • Material• Collating and processing results• Training • Cost effective?? If used for correct purpose

Workplace assessment – CbD

• Case based discussion • Trainee selects case studies, assessor

chooses one of these • Assessor should ask why this selection• Assesses quality of … diagnosis,

assessment, management, record-keeping• 15 minutes (5 minutes for feedback)

Workplace assessment – MiniPAT

• Peer assessment tool• 8 assessors• Including senior colleagues, nurses, AHPs• Including self assessment • Routine performance• Feedback reviewed with trainee and

supervisor • Agreed action plan

Workplace assessment – MiniCEX

• Faculty member observes a trainee interacting with a patient in a clinical setting

• 15 minutes (+5 minutes for feedback) • Interviewing skills, physical exam,

professionalism, communication skills• Formative feedback

Workplace assessment – MiniCEX

• Not for high stakes exams• Not to rank trainees• Not to compare training programmes• 4-6 times in the year• Different assessor each time• Encounter chosen by the trainee, confirmed

by the assessor

Workplace assessment – DOPS

• Direct observation of procedural skills• Real patients• 15 minutes (+ 5 minutes feedback) • Trainees select from an approved list of

procedures• Indications, communication, technique,

analgesia, asepsis

Workplace assessment – DOPS

• 6 times in the year • Valid • Global rating scales

Workplace assessment – portfolios

• Different types of information• Summary of educational experiences• Reviewed by an assessor • Reflection• Action plan• Purpose must be clear (e.g. for summative use)• Need be specific about what portfolio should

contain

Workplace assessment – screening

• If you do really well on your first 2 MiniCEX’s, how will you get on in your next 4?

Workplace assessment – checklists vs global ratings?

• Scores are broadly correlated

• Checklists – objective, reliable, lower level of expertise

• Global ratings – subjective, more valid

• Only experts can use global ratings• Anyone can be trained to use a checklist

Workplace assessment – references and further reading

Norcini JJ. Workplace assessment. In Understanding Medical Education: Evidence, Theory and Practice. ASME 2010

Miller A, Archer J. Impact of workplace based assessment on doctors' education and performance: a systematic review. BMJ. 2010 Sep 24;341

Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach. 2009 Sep;31(9):790-801.

Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007 Nov;29(9):855-71.

Wilkinson TJ. Assessment of clinical performance: gathering evidence. Intern Med J. 2007 Sep;37(9):631-6.