Portfolio assessment of behavioral competencies at the Feinberg School of Medicine Celia Laird O’Brien, PhD
Portfolio assessment of behavioral competencies at the Feinberg School of Medicine Celia Laird O’Brien, PhD
Competency Based Medical Education
4
Does
Shows How
Knows How
Knows MCQ exam
Clinical vignette, essays
OSCE, simulation
Workplace Based Assessment
Assessment of behaviors • Many data points are necessary, collected from
multiple observers, in different contexts, across time.
6
Assessment of behaviors
• How do we organize and review this data to determine competence?
7
Result: Lots of data!
The portfolio approach
8
• The Feinberg electronic portfolio is used to organize this data: a “learning chart”
• Allows students to review and reflect on their performance across time
• Allows us to measure and enhance behavioral competencies in our students
The portfolio approach
Three goals:
1. Permit assessment of competencies not easily measured by traditional grading methods. 2. Develop students’ capacity to self-assess their skills and abilities using external feedback. 3. Develop students’ ability to engage in self-directed learning and improvement.
9
Assessment at Feinberg
10
Fail/Pass/High Pass/Honors Pass/Fail Fail/Pass/Honors
Traditional grading system
Portfolio review Portfolio review
The portfolio approach
11
Competencies subject to portfolio review:
• Effective Communication and Interpersonal Skills (ECIS)
• Patient Centered Medical Care (PCMC)
• Professional Behavior and Moral Reasoning (PBMR)
• Systems Awareness and Team Based Care (SATBC)
• Continuous Learning and Quality Improvement (CLQI)
The portfolio approach Examples of assessment data:
• Faculty and peer evaluations of small group work
• Clinical performance evaluations
• Feedback from standardized patients
• Feedback from nurses
•Narrative data is most informative. 12
Portfolio review process
• Portfolio review committees comprised of experienced clinician educators
• Each portfolio is reviewed by at least two clinicians, who judge competence by the following:
a) Contents of the portfolio (institutional assessment data, external uploads)
b) Quality of reflections and learning plans
17
Portfolio review process •Decisions:
• Progressing towards competence
• Progressing towards competence with some concern
• Progressing towards competence pending additional development
18
Portfolio review process •Assessment of competence isn’t about the numbers.
•Process depends on judgment from experienced educators.
• Involves analysis of qualitative data; uses qualitative methodology to collect evidence of validity
19
Portfolio review process •Reviewers work in pairs
• Independently score portfolios, then come together to reach consensus through discussion and debate
•Third reviewer used if consensus cannot be reached
•Final decisions reviewed by committee as a whole
20
Portfolio review process •Multiple training sessions are held, including group standard setting exercise
• Important to work from a shared frame of reference
•Reviewers look for patterns and themes in performance across time
21
Results – Phase 1
22
25
39
30 38
16
1
4
4
4
1
0
5
10
15
20
25
30
35
40
45
50
CLQI ECIS PCMC PBMR SATBC
PC PAD
Evaluation
• Statistical analyses show a significant association between results of the portfolio review in Phase 1 and clerkship performance in Phase 2, even after controlling for medical knowledge.
23
Evaluation
24
3
2
1 1
2
3
0
1
2
3
4
5
6
7
Honors High Pass Pass
Median number of clerkship grades awarded by group
No concerning behavior (n=111) Concerning behavior (n=24)
p<.01 p<.01
Evaluation
25
82.3
79.4
70
72
74
76
78
80
82
84
86
88
90
No concerning behavior Concerning behavior
Mean clerkship performance score
p<.001
27
“The last two decades have been characterized by happy
portfolio developers and grumpy portfolio users.”
- Erik Driessen, 2016
Lessons learned
• Many students don’t see value of mandated reflection or self-improvement efforts
28
29
4.4
3.2
1
2
3
4
5
My mentor is able to helpme understand my
evaluation data as it relatesto the competency
standards.
Writing reflections about myprogress in the
competencies has proved tobe a useful exercise.
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
Lessons learned
• Numbered ratings proved to be misleading and often contradicted narrative comments.
• Vast majority of students are rated “above average”… even when they’re not.
30
Lessons learned
• There is inherent overlap between these competency domains.
31
Communication Professionalism Teamwork
Lessons learned
• We need to re-examine our remediation efforts and determine how to best support students with deficiencies.
33
Lessons learned
• We had to pull back plans to transmit results of the portfolio review to residency directors
34
Conclusion
• Portfolio assessment is a feasible way to assess behavioral competencies but requires investment of dedicated time and resources.
35
References 1. O’Brien C.L., Sanguino S.M., Thomas J.X., Green M.M. Feasibility and outcomes of implementing a portfolio
assessment system alongside a traditional grading system [published online ahead of print March 29, 2016]. Academic Medicine.
2. Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Academic Medicine. 2007;82(5):493-502.
3. Driessen E, van der Vleuten C, Schuwirth L, van Tartwijk J, Vermunt J. The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study. Medical Education. 2005;39(2):214-220.
4. Johnston B. Summative Assessment of Portfolios: An Examination of Different Approaches to Agreement over Outcomes. Studies in Higher Education. 2004;29(3):395-412.
5. Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Academic Medicine. 2010;85(7):1212-1220.
6. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Medical Teacher. 2012;34(3):205-214.
36