Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen

Accreditation Council for Graduate Medical Education

© 2014 Accreditation Council for Graduate Medical Education

Assessment in Residency Training:

Global Insights

Eric Holmboe

Disclosures

• Eric Holmboe works for the ACGME and

receives royalties from Mosby-Elsevier for a

textbook.


Outline

Major trends - summary

CBME and system outcomes

Overview of competencies, milestones and

EPAs

Importance of informed judgment

Group process


Major Trends in Assessment

Developmental and competency-based

Greater focus on connecting educational and

clinical outcomes

Moving away on over-reliance of

psychometrically based assessment (e.g. tests)

Concept of entrustment for supervision

Operationalizing CBME through Milestones and

entrustable professional activities (EPAs)

Programs of assessment

Group process in judgments of competence and

professional development



Why These Changes in

Assessment?


CBME: Start with System Needs

7Frenk J, et al. Health professionals for a new century: transforming

education to strengthen health systems in an interdependent world.

Lancet. 2010


What Are The Outcomes?

• A competent (at a minimum) practitioner

aligned with:

CMS Triple Aim


How is Sweden doing as a nation on the

triple aim?

What is the responsibility and role of the

medical education system in achieving

the triple aim?

Pair and Share


Diagnostic Errors

IOM Report

Released September 2015

• At least 5 percent of U.S.

adults who seek outpatient

care each year experience a

diagnostic error.

• Postmortem examination

research shows diagnostic

errors consistently contribute

to ~ 10 percent of patient

deaths.

• Diagnostic errors account for

6 to 17 percent of hospital

adverse events.


CBME and Assessment

11Frenk J, et al. Health professionals for a new century: transforming

education to strengthen health systems in an interdependent world.

Lancet. 2010

Assessment and

Curriculum must be

integrated:

Assessment drives

learning; learning

drives assessment

Early Principles: CBME

• World Health Organization (1978):

• “The intended output of a competency-

based programme is a health

professional who can practise medicine

at a defined level of proficiency, in accord

with local conditions, to meet local

needs.”

McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based

Curriculum Development in Medical Education. World Health

Organization, Switzerland, 1978.


CBME Today

An outcomes-based approach to the

design, implementation, assessment

and evaluation of a medical education

program using an organizing framework

of competencies1

1Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical

education: theory to practice. Med Teach. 2010; 32: 638–645



Competencies

Competency: Definition

Competency: An observable ability of a

health professional, integrating multiple

components such as knowledge, skills,

values and attitudes.

The International CBME Collaborators, 2009

Competency-frameworks*

CanMeds

Medical expert

Communicator

Collaborator

Manager

Health advocate

Scholar

Professional

ACGME Medical knowledge

Patient care

Practice-based learning& improvement

Interpersonal and communication skills

Professionalism

Systems-based practice

GMC Good clinical care

Relationships with patients and families

Working with colleagues

Managing the workplace

Social responsibility and accountability

Professionalism

*From CPM Van Der Vleuten


Linking Clinical and Educational Outcomes

National Health Service – UK.

http://www.wipp.nhs.uk/tools_gpn/unit6_education.php

Competencies

Triple Aim

http://www.wipp.nhs.uk/tools_gpn/unit6_education.php



Milestones


PC1. History (Appropriate for age and impairment)

Level 1 Level 2 Level 3 Level 4 Level 5Acquires a

general medical

history

Acquires a basic

clinical history

including

medical,

functional, and

psychosocial

elements

Acquires a

comprehensive

clinical history

integrating medical,

functional, and

psychosocial

elements

Seeks and obtains

data from secondary

sources when needed

Efficiently acquires

and presents a

relevant history in a

prioritized and

hypothesis driven

fashion across a

wide spectrum of

ages and

impairments

Elicits subtleties and

information that may

not be readily

volunteered by the

patient

Gathers and

synthesizes

information in a

highly efficient

manner

Rapidly focuses on

presenting problem,

and elicits key

information in a

prioritized fashion

Models the

gathering of subtle

and difficult

information from the

patient

Competency

Developmental

Progression or Set of

Milestones Sub-competency

Specific

Milestone

Purposes and Implications

ACGME

• Accreditation – continuous monitoring of programs; lengthening of site visit cycles

• Public Accountability – report at a national level on competency outcomes

• Community of practice for evaluation and research, with focus on continuous improvement

Training Programs

• Framework for CCC• Guide curriculum development• More explicit expectations of trainees• Support better assessment• Enhanced opportunities for early

identification of under-performers

Certification Boards

• Research for CBME

Residents and Fellows

• Increased transparency of performance requirements

• Encourage informed self-assessment and self-directed learning

• Better feedback

Milestones

20

Milestones are a Formative Assessment Framework


Milestones as Roadmap

Observations:

1) Journey not a

straight line

2) More than one

path (but not

infinite paths)

3) “If you don’t know

where you are

going, any road

will get you there”


Dreyfus & Dreyfus Development Model

Dreyfus SE and Dreyfus HL. 1980

Carraccio CL et al. Acad Med 2008;83:761-7

Time, Practice, Experience

Novice

Advanced Beginner

Competent

Proficient

Expert/

Master

MILESTONESCurriculum

Assessment

Curriculum

Assessment

Curriculum

Assessment

Curriculum

Assessment

Curriculum

Assessment

Development is a

non-linear phenomenon

In general, Milestones are …

Describe the educational outcomes of the

individual

Developmental in nature

Grounded in Dreyfus model of expertise

development

Behaviorally-based

Describe a learner using narratives

Independent of level of training

Overarching trajectory of training



Entrustable Professional Activities


Video Exercise

What was this intern entrusted to do

without direct supervision?


EPAs represent the routine professional-life

activities of physicians based on their specialty

and subspecialty

The concept of “entrustable” means:

‘‘a practitioner has demonstrated the necessary

knowledge, skills and attitudes to be trusted to

perform this activity [unsupervised].’’1

1Ten Cate O, Scheele F. Competency-based postgraduate

training: can we bridge the gap between theory and

clinical practice? Acad Med. 2007; 82(6):542–547.

An Entrustable Professional Activity

ten Cate et al.

Acad Med 2007; 82: 542-47

Part of essential work for a qualified professional

Requires specific knowledge, skill, attitude

Acquired through training

Leads to recognized output

Observable and measureable, leading to a

conclusion

Reflects the competencies expected

EPA’s together constitute the core of the

profession

“Patients’ and instructors’ … entrustment

of responsibility to a trainee is an essential

concept in this approach…”

Question: What do you currently

entrust your learners to do with only

indirect, reactive supervision?


ten Cate et al.

Acad Med 2007; 82: 542-47

Attending physicians assess a multi-dimensional construct of “trustworthiness” when deciding a level of supervision

Entrustment implies a level of competence

Entrustment in Residency Training

Kennedy, et. al.

Acad Med 2008; 83(10 Suppl): S89-92



How Does All of This Fit

Together?!?

EPAs, Competencies, and Milestones

ten Cate, O. (February 2015) Entrustable Professional Activities as a Framework for

Assessment, Presentation given at the 2015 ACGME Annual Educational Conference.

Date and graphic from ten Cate, et al. 2015 (under review).

Competency Milestones EPA in Training EPA in

Practice

Medical

Knowledge

MK1

MK2 “Lead” a

care team

Lead & work

within IP health

care teams.Patient Care PC1

PC2

Professionalism Prof1

Care for patients

with chronic

illness with

indirect

supervision

Manage care of

patients with

chronic

diseases

Prof2

Interpersonal

Skills

ISC1

ISC2

Systems-based

Practice

SBP1

SBP2 Participate in QI

and pt. safety

initiatives

Enhance patient

safety.

Improve quality

of health care

Practice-based

learning

PBLI1

PBLI2

Analyze to

Understand

Synthesize to

Educate and

Evaluate

Shared Mental Models and Frameworks

Physicians competent to

meet the health care

needs of the population


U.S. Model of Integration

COMPETENCY

Entrustable Professional Activity1

MILESTONES

COMPETENCY

MILESTONES

COMPETENCY

MILESTONES MILESTONES

COMPETENCY

COMPETENCY COMPETENCY COMPETENCY COMPETENCY

MILESTONESMILESTONESMILESTONESMILESTONES

Entrustable Professional Activity2

“White space” “White space”

EPA’s as an assessment strategy

Faculty “get it” (or do they???)

EPA provides meaningful context

Synthetic work-based assessment

Vehicle for faculty to provide honest and specific feedback for growth

Reflect desired outcomes of profession, public and policy makers



Programmatic Assessment


The Professional Self-regulatory “System”

Assessments within

Program:

• Direct observations

• Audit and

performance data

• Multi-source FB

• Simulation

• ITExam

Qual/Quant

“Data”

Synthesis:

Committee

Residents

Faculty, PDs

and others

Milestones and EPAs

as Guiding Framework and Blueprint

Accreditation

Unit of Analysis:

Program

Certification and

Credentialing

Unit of Analysis:

Individual

J

U

D

G

M

E

N

T

D

FB

FB

DD FB

P

U

B

L

I

C


Structured Portfolio

Medical record audit andQI project

MSF: Directed per protocol

Twice/year

Practice-based learning and improvement

Systems-based prac

Mini-CEX:10/year

Interpersonal skills and Communication

ITE:1/year

Patient care

Faculty Evaluations

EBM/Question Log

Medical knowledge

Professionalism

“Sources” of Assessment

■ Learner-directed ■ Direct observation

Time

AssessmentActivities

TrainingActivities

SupportingActivities

v v v v v v

Inte

rme

dia

te E

va

l

Inte

rme

dia

te E

va

l

Fin

al E

va

lua

tio

n

= learning task

= learning artifact

= single assessment data-point

= single certification data point for mastery tasks

= learner reflection and planning

= social interaction around reflection (supervision)

= learning task being an assessment task also

Model For Programmatic Assessment(With permission from CPM van der Vleuten)

Committee


Group Decision Making

Key Issues

What is the environment in which the committee

performs its work?

What is the local culture?

Groups within groups

What is the medical culture of your institution?

What are the effects of hierarchy on group

decision making?

Berg: Medicine one of the most hierarchical of all

professions

Single variable of effectiveness: extent to which

people are willing to say “positive” and “negative”

comments and observations in a group


Group Process in U.S.

Institutional Culture

Info Sources:

• Faculty Evals

• Direct Obs

• Multisource FB

• Patient surveys

• ITExams

• +/- Simulation

• Critical events

• Informal (e.g.

“hallway talks”)

Pre

-me

etin

g D

ata

Pre

pa

ratio

n

Group ProcessKnown Variables:

• Group composition

• Info presentation

• Evidence vs.

verdict

• Hierarchy

• Info context

• Time pressures

• Additional info

Judgment

Program

Culture

Feedback

“Filter”

Institutional Culture

Learner

“Wisdom of the Crowd”

• Hemmer (2001) – Group conversations more

likely to uncover deficiencies in professionalism

among students

• Schwind, Acad. Med. (2004) –

• 18% of resident deficiencies requiring active remediation became apparent only via group discussion.

• Average discussion 5 minutes/resident (range 1 – 30 minutes)

Narratives and Judgments

Pangaro (1999) – matching students to a

“synthetic” descriptive framework (RIME) reliable

and valid across multiple clerkships

Regehr (2007) – matching students to a

standardized set of holistic, realistic vignettes

improved discrimination of student performance

Regehr (2012) – faculty created narrative

“profiles” (16 in all) found to produce consistent

ranking of excellent, competent and problematic

performance

Rules for future assessments of trainee:

Be crystal clear on the purpose of a curricular

activity/experience

Measure the expected outcome(s) for a defined

activity or rotation (based on purpose)

Provide meaning to faculty

Provide meaning to trainee

Provide meaning to CCC (in time)

Reflect needs of our health delivery system

Will necessitate educational redesign

Direct observation of trainees

Purposeful assessment at times of entrustment

Capture what you already do!

Coordination and cooperation between training programs and healthcare professions

Detailed mapping between assessments and milestones in electronic evaluation system

Faculty Development

It is very unlikely that you can create a

successful assessment system without faculty

training

Evaluation tools are only as good as the person

using them

At the present time – still need faculty judgment

Need shared mental models and an

understanding of the expected outcomes

Questions and Discussion

[email protected]

Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen

Documents