Portfolio Assessment in Clerkship

Portfolio Assessment in Clerkship

Michelle Gibson - Geriatrics(thanks to Chris Frank and Melissa Andrew too)

Objective Participants will discuss the use of portfolios in assessing medical trainees (after a brief review of some basic principles…)

Background Change to the Internal Medicine clerkship in January 2007 4 weeks to 6 weeks

Opportune time to review the assessment process in geriatrics Previously, 8 assessment cards filled out by MD supervisors

Challenges with old system Cards were often filled out the last week of the rotation

Depended solely on supervisor impression

No direct observation was incorporated

Students were assessed using only one modality

Reasons for change Desire for more formative assessment opportunities

Desire for more robust assessment

(Need to meet course requirements!)

Formative assessment Assessment that occurs during instruction to provide feedback to teachers and students

Describes needs for future learning

Assessment for learning

Summative Assessment Assessment that occurs at the end of an instructional unit to document student learning

Certifies learning Assessment of learning

Is this portfolio formative or summative?

High Quality Assessment Clear & appropriate learning targets Appropriateness of assessment methods*

Validity Reliability Fairness Positive consequences* Alignment* Practicality & efficiency*

Reliability & Validity Reliability: concerned with the consistency, stability, and dependability of the scores

Validity: the appropriateness of the inferences, uses, and consequences of the assessment Is it assessing what you want it to assess?

Portfolio A systematic collection of student products to assess progress

Structured portfolio The assessor determines the mandatory content of the portfolio

Literature review Assessment of housestaff in clinical rotations Portfolio literature reviewed Specific instruments examined

Portfolio components were selected to provide multiple methods of assessment for each clerkship objective

Components by MethodologyMethodology Portfolio components

Foundational evaluations

Assessment cards - 6

Direct observations Mini-CEX - 2

Practice- & Data-based learning

Medication ReviewSelf-assessment of case write-up

Multi-source evaluations

TAB - 2

Miscellaneous (optional)

Student’s choice

Mini-CEX

TAB

Card

Medication Review - written Based on an actual patient, preferably one for whom the student has responsibility brief summary of patient, meds he/she is taking including whether or not they are appropriate

suggestion of other medications he/she should be taking

Self-assessment of write-up H & P from early in rotation Students asked to read and identify 2 things done well 2 things that could be improved

Reviewed with undergrad coordinator on the last day

Components & CPE CategoriesCPE Category EC MC TAB MR Case

Knowledge Base X X X X

Motivation to Learn X X X X X

Team Functioning X X

Communication X X X X X

Relationship with patients/families

X X X

Resource Management X X X X

Process Orientation with students on Day 1 Team, faculty, & residents are oriented to the portfolio

Students meet with undergrad coordinator at mid-term, review progress, identify any gaps, etc.

Last day - review portfolio with students, completion of CPE

Evaluation of Pilot Focus on acceptability

Students and assessors Student evaluations ongoing - not seen by the undergrad coordinator until collated in April 2008

Will redo faculty portion in December 2007

Challenges noted - 1st time First rotation ever for clerks

Time spent learning how to be a clerk Couldn’t assemble half the portfolio in 3 weeks

Exam stress “detracted from our learning”

Supervisors were absent more than usual

*all except exam stress have resolved

Strengths noted Multidisciplinary team members provided high-quality feedback

Direct observation occurred Forms were felt to be user-friendly, acceptable

Students appreciated discussing their medication review and being observed

Assessor ratings of forms TAB

5/5 for ease of use; 5/5 for usefulness Mini-CEX

5/5 for ease of use; 4/5 for usefulness Time spent by assessor: 13.5 minutes

Encounter card 4.5/5 for ease of use; 4.5/5 for usefulness

Time spent by assessor: 15 minutes

My time? 5-10 minutes on day 1 for orientation Plan to develop “talking power point”

15 min per student at the end of week 3 Mid-term review Could be any faculty

15 min per student on the last day Final evaluation Could be any faculty

My *impressions* Students receive more feedback in a timely fashion

Direct observation doesn’t take long, and is very valuable

TAB form is very useful, and team members are pleased to be filling them out

Mid-term meetings are very useful

Future directions Continue with student and assessor feedback

Incorporate other tools? Peer assessment? Patient assessment?

Study it more Implement it for residents

Factors to note Small unit, discrete group of nursing and other staff

Small division Very workable for us … but for others?

Questions to consider Is this high quality assessment? How could it be improved?

Can it be extended into other areas?

Is there a role for a “whole clerkship” portfolio? “Pre-clerkship” portfolio?

Residents & non-MDs as assessors? Electronic?

Comments about our students “ .. Is very attentive… he listens well to pt. & staff, & then goes off & evaluates what he’s been told”

“excellent verbal communication observed with all team members”

“better H & P than most residents”