Fellowship Grassroots Session · • Stakeholder panel for Q&A • SPIN update – Rich Mink ... Session Goals and Objectives To develop a workplace-based assessment tool that fits

Fellowship Grassroots SessionAPPD Fall Meeting

Thursday, September 26, 2019

Agenda

• Community building• Stakeholder panel for Q&A• SPIN update – Rich Mink• Brief discussion re: mentoring• Skill building: assessment

Stand up if…

• You are a:• FPD• F-APD• Coordinator• Vice Chair

• Your specialty is…

Name one word….

• To describe what is valuable or enriching about being a FPD/F-APD/coordinator/Vice-chair

• To describe what is most challenging about your role

Stakeholder Q&A

• ABP – Suzanne Woods, MD – Executive VP• ACGME – Caroline Fischer – Executive Director for RCs and Susie

Buchter, MD – Peds RC chair• COPS – Debra Boyer, MD – Executive Committee Chair

SPIN Update

Mentoring Discussion

MSF

Clinical Practice ITE

ScholarlyProduct

Checklist

Milestone

OSCE

Individual Fellow Assessment:From Programmatic Design

to Individual OutcomesAmanda R. Emke, MD, MHPE

APPD, Fall 2019Disclosures: Grant Support [COMSEP]

Session Goals and Objectives

To develop a workplace-based

assessment tool that fits into a competency-

based program of assessment and meets

the needs of all stakeholders.

Objectives• Define a competency-based

program of assessment• Compare and contrast current

frameworks in medical education

• Discuss the advantages and limitations of direct observation

• Review tools for capturing assessment data based on direct observations

An Important Distinction

MSF


ScholarlyProduct

Checklist

MilestoneOSCE

= Program Evaluation

An Important Distinction

MSF


ScholarlyProduct

Checklist

MilestoneOSCE

= Assessment

Perfo

rman

ce

Time Spent Learning

Goal of Fellowship Training

Assessment

What is a Competency Based Program of Assessment?

Individual Assessment

Data

Expert Judgement

Decision that Competency is

Attained allowing for

Promotion or Advancement

Reflections

360

Checklist

Sim

Clinical Practice Review

ITEWorkplace


Data

Expert Judgement

Decision that Competency is

Attained allowing for

Promotion or Advancement

Reflections

360

Checklist

Sim


ITEWorkplace



Data

Expert Judgement

High-Stakes

Decision

Reflections

360

Checklist

Sim


ITEWorkplace


Creating a Competency Based Program of Assessment for Your Fellowship

Year 1

Program Objectives instructed upon and assessed in this Year of fellowship training


Year 1

Program Objectives instructed upon and assessed in this Year of fellowship trainingLearning activities used to instruct on the Program Objectives


Year 1

More to Come!

Program Objectives instructed upon and assessed in this Year of fellowship trainingLearning activities used to instruct on the Program ObjectivesAssessments



Year 1

Learning activities used to instruct on the Program ObjectivesAssessmentsStructured, documented learner reflection & projectionRequired meeting between learner and coach

Rotation



Year 1

Learning activities used to instruct on the Program ObjectivesAssessmentsStructured, documented learner reflection & projectionRequired meeting between learner and coach

Rotation


Year 1


Data

Rotation


Year 1


Data

DecisionExpert Judgement

+

What are your Program Objectives?

Do you have specific learning activities?

What assessments do you currently employ?

Is everything arranged in a programmatic way?

First Breakout

Assessment Strategies

General Challenges to Workplace Based Assessment

• Lack of a unifying framework

• Shift to competency assessment

• Complexity of assessment task

• Assessors as sources of error

3 Frameworks for Workplace-Based Assessments

Entrustable Professional Activity

Competency

Competency

Competency

Describes progress along a trajectory.

Divides competence into domains.

Combines domains into tasks.

Physician

Competency

Competency

Competency

Competency

Competency

Competency

Developmental Analytic Synthetic

AMEE Guide No 78. Medical Teacher. 2013

Comparing the ThreeASSUMPTIONS DISADVANTAGES

• Each stage supersedes prior

• Domains progress differently• Understanding true difference

between levels

ADVANTAGES

• Demonstrates individual progress

EXAMPLE

Dreyfus





between levels

ADVANTAGES


• Combining discrete elements equals competence

• Elements can be measured discretely

• Covers all aspects of being a “good” physician

• Allows assessment & feedback on specifics

• Extensive descriptions not easily comprehended by clinicians

• Weak connection with clinical activities

EXAMPLE

ACGME





between levels

ADVANTAGES


• Combining discrete elements equals competence

• Elements can be measured discretely

• Covers all aspects of being a “good” physician

• Allows assessment & feedback on specifics

• Extensive descriptions not easily comprehended by clinicians

• Weak connection with clinical activities

• Clinical tasks are complex, social, & require multiple domains applied simultaneously

• Strong connection with true clinical work

• Highly authentic

• Specific reasons for or against progression may not be identified

EXAMPLE

EPAs


The Chocolate Experiment

Second Breakout

Really a combination…

OR

Addressing the Complexity and Limitations of Competency Assessments in the Workplace Based

Addressing the Complexity and Limitations of Competency Assessments in the Workplace Based

InferenceObservation

Objectivity Subjectivity

Inference Observation

PC1 – Provide transfers of care that ensure seamless transitions

PC2 – Make informed diagnostic and therapeutic decisions that result in optimal clinical judgement

MK1 – Locate, appraise, and assimilate evidence from scientific studies related to patient care

PBLI3 – Use information technology to optimize learning and care delivery

PROF4 – The capacity to accept that ambiguity is part of clinical medicine


PC1 – Provide transfers of care that ensure seamless transitions

PC2 – Make informed diagnostic and therapeutic decisions that result in optimal clinical judgement

MK1 – Locate, appraise, and assimilate evidence from scientific studies related to patient care

PBLI3 – Use information technology to optimize learning and care delivery

PROF4 – The capacity to accept that ambiguity is part of clinical medicine


Bias

• Leniency or Generosity• Recency• Spillover• Central tendency• Halo or Confirmatory• Unconscious

Bias

• Broad, systematic sampling of clinical situations• Short rating instruments• Immediate feedback for teaching and learning • Maintenance of written performance notes support

delayed ratings• Feedback to raters about their ratings• Use formal and informal observations• Promotion decisions made via group review• Establish the meaning of scale items or scores

Mitigating Bias

Williams et al. Teach Learn Med. 2003.

Examples of Workplace Based Assessment Tools

• Mini-Clinical Evaluation Exercise (mini-CEX)• Clinical Encounter Cards• Clinical Work Sampling• Blinded Patient Encounters• Direct Observation of Procedural Skills• Case-Based Discussion/Chart Stimulated Recall• Multi-Source Feedback

Examples of Workplace Based Assessment Tools:Clinical Encounter Cards & Clinical Work Sampling

Purpose Rate performance based on direct observation of patient encounterRating Scale Based on expected level of performanceCapture Mechanism 4x6 card (traditionally)

Electronic platformSupporting Evidence Feasible, valid, reliable

(minimum 8 encounters)Unique Components to the Assessment

Development & commitment to an action plan

• Analytic Framework would rate each skill separately

• Synthetic Framework would rate global encounter

Examples of Workplace Based Assessment Tools:Clinical Encounter Cards & Clinical Work Sampling

• Trainee Name• Observer Name• Date• Clinical Location or Context• General Problem

• Airway/Breathing• CVS/Circulation• GI• Neuro• Psych/Behavior

• Patient Complexity or Acuity• Written Comments

• Strengths• Suggestions for Improvement• Agreed Actions or Learning Plan

• Performance • unsatisfactory • below expectations • at expectations • above expectations • ready for advancement/independence

• Entrustment • observation only • direct observation • indirect observation • unobserved • supervises others

Examples of Workplace Based Assessment Tools:Direct Observation of Procedural Skills

Purpose Assessment of procedural skillsRating Scale Based on expected level of performanceCapture Mechanism Paper or electronicSupporting Evidence Differs based on toolUnique Components to the Assessment

• Documents number of times assessor has observed any trainee• Documents number of times trainee has performed this procedure• Assesses individual steps/components of procedure PLUS overall

ability to perform procedure

3 with good validity evidence1. Operative Performance Rating Scale2. Mayo Clinical Skills Assessment Test3. Ottawa Surgical Competency Operating Room Evaluation

Expected time commitment is 20 minutes – 15 minutes of observation time & 5 minutes for

feedback at completion of procedure

Examples of Workplace Based Assessment Tools:Case-Based Discussion & Chart Stimulated Recall

Purpose Varies but should focus on only oneRating Scale Based on expected level of performanceCapture Mechanism Paper or electronicSupporting Evidence Good triangulation with other assessments of clinical reasoningUnique Components to the Assessment

• Multiple experiences with different faculty• Very useful ”diagnosing” struggling learners• Discussion focuses on only one area of competency but form can

collect data on all of them



• Clinical Reasoning• Referral or Consultation Requests• Follow Up & Future Planning• Professionalism• Documentation

Examples of Workplace Based Assessment Tools:Case-Based Discussion & Chart Stimulated Recall

Purpose Varies but should focus on only oneRating Scale Based on expected level of performanceCapture Mechanism Paper or electronicSupporting Evidence Good triangulation with other assessments of clinical reasoningUnique Components to the Assessment

• Multiple experiences with different faculty• Very useful ”diagnosing” struggling learners• Discussion focuses on only one area of competency but form can

collect data on all of them



• Clinical Reasoning• Referral or Consultation Requests• Follow Up & Future Planning• Professionalism• Documentation

Considerations When Developing a Workplace Based Assessment

Framework

Context

Rater

Tool

Bias

• Should the same assessments be conducted everywhere?• Should the same competencies, skills, knowledge, attitudes be captured in all locations?• What are the facilitators and barriers to capturing the assessments?

• What facets of competence can be observed or should be taken into account?• How do my faculty currently talk about superior, adequate, and unacceptable performance?

• Who is the right person for the job?• How does that person receive appropriate training?

• How can I promote co-production?

• How can I minimize bias?

References

• Louis Pangaro & Olle ten Cate (2013) Frameworks for learner assessment in medicine: AMEE Guide No. 78, Medical Teacher, 35:6, e1197-e1210, DOI: 10.3109/0142159X.2013.788789

• Reed G. Williams , Debra A. Klamen & William C. McGaghie (2003) SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings, Teaching and Learning in Medicine, 15:4, 270-292, DOI: 10.1207/S15328015TLM1504_11

• Holmboe, E. S., & Hawkins, R. E. (2018). Practical guide to the evaluation of clinical competence. 2nd Edition. Philadelphia, PA: Mosby/Elsevier.