1 Creative Commons License Attribution-NonCommercial-ShareAlike 2.0 You are free: to copy, distribute, and display this presentation, and/or to make derivative.

1

Creative Commons LicenseAttribution-NonCommercial-ShareAlike 2.0

You are free:• to copy, distribute, and display this presentation, and/or• to make derivative works Under the following conditions:• Attribution. You must give the original authors credit. • Noncommercial. You may not use this work for commercial

purposes. • Share Alike. If you alter, transform, or build upon this work, you

may distribute the resulting work only under a license identical to this one.

See http://creativecommons.org/licenses/by-nc-sa/2.0/ for full license.

Portfolios in Medical Education:Bridging the Gap from Data to

Discovery

CREOG 2007 Education Retreat

Rio Mar, Puerto Rico

Lee A. Learman, MD, PhD

Director of Curricular Affairs

UCSF Office of Graduate Medical Education

Learning ObjectivesLearning Objectives

Participants will emerge able to:Participants will emerge able to: define and list essential components of define and list essential components of

a learning portfolioa learning portfolio distinguish an evidence database, distinguish an evidence database,

formative portfolio and summative formative portfolio and summative portfolioportfolio

describe potential uses in GME, describe potential uses in GME, including the proposed ACGME including the proposed ACGME learning portfoliolearning portfolio

Portfolios for AssessmentPortfolios for Assessment

Purposeful collections of evidence Purposeful collections of evidence used by students to document and used by students to document and reflect on learning outcomes over reflect on learning outcomes over time.time.

Potential uses:Potential uses: Evidence database: by program vs. Evidence database: by program vs.

learnerlearner Formative: coaching, feedbackFormative: coaching, feedback Summative: decisions regarding progressSummative: decisions regarding progress

Dannefer EF, Henson LC. Acad Med 2007;82:403.

Why Portfolios?Why Portfolios?

Exciting and innovative tool for resident Exciting and innovative tool for resident learning and developmentlearning and development Already used in K-12, higher education, and Already used in K-12, higher education, and

multiple professionsmultiple professions Relieving burden while increasing Relieving burden while increasing

accuracy for program directorsaccuracy for program directors ““added” or “lifted” - “eye of the beholder” added” or “lifted” - “eye of the beholder”

Building a community of practice for GMEBuilding a community of practice for GME Within and across specialties to raise the barWithin and across specialties to raise the bar

http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

Why Portfolios?Why Portfolios?

Needs identified in the literature:Needs identified in the literature: Focus on complex tasks, integrated Focus on complex tasks, integrated

competencies competencies More feedback needed as learning objectives More feedback needed as learning objectives

become defined and measured more accuratelybecome defined and measured more accurately Context-dependence: sample multiple sources Context-dependence: sample multiple sources New ways to analyze, summarize all the dataNew ways to analyze, summarize all the data

Portfolios can meet these needsPortfolios can meet these needs

Dannefer EF, Henson LC. Acad Med 2007;82:403.

Self-Reflection & PortfoliosSelf-Reflection & Portfolios

Evidence Warehouse

Learners

Mentors

A method for assessing reflection

Instructions on reflection

Bridge from Data to Bridge from Data to DiscoveryDiscovery

Learner receives, Learner receives, instructions and instructions and

selects experienceselects experience

Evidence Warehouse

Reflects and Reflects and summarizessummarizes

Discusses Discusses reflection with reflection with

mentor (formative)mentor (formative)

Reflection is Reflection is scored as evidence scored as evidence

of a specific of a specific competency or competency or

PBLI (summative)PBLI (summative)

Reflection at UCSF: Reflection at UCSF: InstructionsInstructions

Describe the situation that “taught you the Describe the situation that “taught you the most” about [specific competency]most” about [specific competency]

Describe challenge(s) faced, strategies usedDescribe challenge(s) faced, strategies used Describe sources of feedback (people, data)Describe sources of feedback (people, data) Relate the situation to previous similar onesRelate the situation to previous similar ones Include details to illustrate challenges you Include details to illustrate challenges you

faced and lessons you learnedfaced and lessons you learned List conclusions re: strengths, opportunities List conclusions re: strengths, opportunities

for improvement, and use examples to justify for improvement, and use examples to justify conclusionsconclusions

Learman LA, Autry AM, Pliska L, O’Sullivan PS. WGEA Meeting 2006.

Sent: Wed 1/10/2007 10:04 PM Subject: Proposal for HROB at SFGHHello all!

As many of you know, Thursday mornings at SFGH pose a challenge to even the most efficient of residents.  Juan and I were discussing the following minor changes to resident roles and Thursday mornings HROB clinic that could potentially make a major difference…

Please look over these proposed changes. I would love your feedback – most of this was born from the question of how to make M&M a more consistently educational experience for all (i.e., how can the OB chief and R2 get to OB M&M on time?) -

1)      All residents (including OB team) arrive at 5M on Thursdays at 8:00 am to start prepping charts; attending and fellow arrive at 8:30 to hear presentations of prepped charts. Conference should be finished by 9 am. If chart prep not complete and patients present at 9 am, at least 2 residents should leave conference to start seeing patients.

2)      Night float chief helps to prep some charts on Wed night for Thursday morning so that conference can be finished by 9 am

3)      OB Chief must leave clinic at noon to start M&M - supported by clinic attending

4)      OB R2 next to leave, ideally at noon as well

5)      Gyn R2 and Clinic Chief last to leave clinic, to cover those 12:30 must-see patients (Gyn R1 covering pager and consults if Gyn R2 needed after 12)

6)      If >8 patients remain to be seen at 12:00, then OB R2 stays with the rest of the residents while OB Chief leads M&M alone. If needed, other attendings (mobilized by Dr.V) help to see patients

Without compromising patient care for education, I'm hoping this may help both clinic flow and M&M utility. Thanks for your time!

The EvidenceThe Evidence

Sent: Thu 1/11/2007 9:38 PMTo: OBGYN Resident Class of 2007 Subject: RE: Proposal for HROB at SFGH

If all the rest of the chiefs agree with this plan, I'll forward a summary to the rest of the residents and we'll make a go of it next week.

As for the main concern of folks arriving at an earlier time,

1) Gyn team is almost always finished rounding by 7:30 because the OR chief must be at OB board rounds. In the remaining half hour, juniors can complete a bit of work. 2) OB Chief will make sure that am rounds finish by 8 am (just as we make our best effort to do on Wed am for conference) and bring charts down from L&D, where at least some have been prepped by the NF Chief.

3) GYN OR Chief (covering for OB Chief) will remind juniors on GYN team to be at conference by 8am and help to get the OB team out of rounds by 8 am 4) GYN Clinic Chief will arrive at 5M at 8:00 instead of 8:30

5) Jeopardy and other clinic residents (OB intern) will arrive at 5M at 8:00 instead of 8:30.

Dr. L's suggestion of ensuring that our ancillary services are aware that patients must be seen/in line to see an MD by 11:30-11:45 should also be addressed with M and G (Would you mind doing this, T?).

G is aware and enthusiastic about the proposed plan - she has her chart preparations completed by Wednesday evenings, so it shouldn't be a problem to get the charts to the NF chief.

Thanks to everyone for being so adaptable!

The EvidenceThe Evidence

Reflective Self-Assessment of Systems-based PracticeInstructions: Select a clinical situation you remember during the past year that taught you the most about either practicing cost-effective health care that does not compromise quality of care OR assisting patients in dealing with healthcare system complexities (e.g., surmounting logistical barriers to optimal care – appointments, diagnostic tests).

(1) Describe the setting and context including who was present.Setting: San Francisco General Hospital, Thursday morning moderate risk obstetrics clinic. Those involved in the system change are residents, faculty staffing the clinic, clinic staff and clinic flow nurse

(2) What challenges did you face in practicing cost-effective healthcare or surmounting systemic barriers to optimal care?

The challenge I hoped to address was optimizing the care of patients with complex obstetric problems in the outpatient setting. The major obstacles have been in existence for years: many patients requiring both clinical attention and ancillary services, within a limited time.

One additional challenge is finishing the clinic at a time that would allow some or all of the residents to attend M&M conference, a major learning opportunity.

(3) Describe what efforts you made to surmount the challenges. What past experiences did you bring to this situation?

In the past, there has been a common feeling that “the clinic is just that way – it’s hopeless to try and change it.” I tried to separate myself from that because I wanted people to want to change!

In conjunction with one of our faculty members, I came up with ideas on how residents could contribute time that did not risk violating duty hour regulations in order to improve clinic flow. For example, the night float team helping to prepare charts, residents arriving a half hour earlier to finish preparing charts, and then presenting to attendings two at once so that we could start seeing patients as soon as they were placed in rooms.

I trusted that that every resident in our program would be interested in bettering the system for all. The proposal involved and that a small sacrifice on one rotation would mean educational benefit for them in another rotation (i.e., what goes around, comes around). I used this reasoning when I sent out the email proposal to my fellow chief residents and the program/clinic directors.

The ReflectionThe Reflection

(4) How did you obtain feedback and from whom? Include details providing evidence that you made cost-effective decisions without compromising the quality of care or that you succeeded in assisting the patient in dealing with system complexities?

For the last two weeks, our goal of getting at least two residents to the M&M conference has been successful, but the rest of the residents are not having the opportunity to benefit from this educational conference as I had hoped.

I have been asking the faculty who staff the clinic, residents, and the clinic flow nurse how the system has been working. The reviews have been mixed, and since it has been only been implemented for a short time, it’s difficult to determine whether or not this system change is actually beneficial.

(5) List what conclusions you drew from situation regarding your strengths and opportunities for improvement, and use examples from the situation to justify your conclusions.

I found that motivating people towards change was simple and that those involved were willing to attempt to motivate change themselves, once the activation energy was there. I felt that communicating the importance of this change was one of my strengths. However, putting emotional investment in it means that I have been disappointed that in the first two weeks it hasn’t seemed to work.

(6) What changes, if any, do you plan to make if you face a similar situation in the future?

I plan to encourage future residents to continue with this system and possibly find even more adjustments to improve its efficiency

The ReflectionThe Reflection

Reflection Scenarios: ICS & P

• Addressing challenging patients• Angry or frustrated• Worried, scared or guarded (including DV)• Difficult or controlling (care-seeking or

refusing)• Language barrier• Transgender

• Confronting own limitations• Stereotypes (PSA, IVDU, dwarfism)• Disclosing difficult diagnosis (HIV, cancer)

Learman LA, Autry AM, Pliska L, O’Sullivan PS. WGEA Meeting 2006.

Reflection Scenarios: Surgical

• Surgical Skills• 9 Routine Cesarean with complication• 3 Emergency Cesarean delivery• 6 Surgical decision-making (including

conflict with attending)• 6 Gyn surgery – complications• 4 Gyn surgery – technical challenges• 2 Other

Learman LA, Autry AM, Pliska L, O’Sullivan PS. WGEA Meeting 2006.

ACGME ACGME Portfolio Management Portfolio Management

ToolTool““An interactive web-based An interactive web-based

development tool that residents can development tool that residents can use throughout their residencies to use throughout their residencies to record and organize their learning record and organize their learning and to reflect and receive feedback and to reflect and receive feedback on their skills as physicians, on their skills as physicians, building evidence that allows them building evidence that allows them to chart their own progress over to chart their own progress over time.” time.”

http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

ACGME ACGME Portfolio Management Portfolio Management

ToolTool ““First and foremost . . .a learning tool First and foremost . . .a learning tool

for residents that enables them to”:for residents that enables them to”: track their experiencestrack their experiences self-reflect on those experiences share their insights with mentors, share their insights with mentors, receive real-time formal feedback receive real-time formal feedback

A repository for resident work products A repository for resident work products and professional documents meeting and professional documents meeting the needs of many groups including the needs of many groups including licensing and certification boardslicensing and certification boards

http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

Portfolio Functions - Portfolio Functions - ACGMEACGME

Growth Model (formative) – tracks Growth Model (formative) – tracks learner development over timelearner development over time

Showcase Model (summative) – Showcase Model (summative) – snapshot demonstrating snapshot demonstrating achievement of identified achievement of identified outcomes as for a grade, outcomes as for a grade, promotion, or graduationpromotion, or graduation

Hybrid – supports both purposesHybrid – supports both purposeshttp://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

Portfolio Approach at Portfolio Approach at CCLCMCCLCM

Dannefer EF, Henson LC. Acad Med 2007;82:403.

Evidence-based EssentialsEvidence-based Essentials

Time for reflection and mentorshipTime for reflection and mentorship Separation of formative and Separation of formative and

summativesummative Students select evidence of learningStudents select evidence of learning Essays required to aid reflection on Essays required to aid reflection on

integration of competenciesintegration of competencies Rigorous measurement standards for Rigorous measurement standards for

summative assessments (fair, valid, summative assessments (fair, valid, reliable)reliable)

Dannefer EF, Henson LC. Acad Med 2007;82:403.

UCSF Scoring RubricUCSF Scoring Rubric

1. 1. DescribesDescribes encounter only. encounter only.

2. 2. Unsupported opinionsUnsupported opinions about lessons learned. about lessons learned.

3. 3. Superficial justificationSuperficial justification of lessons learned. of lessons learned.

4. Discussion 4. Discussion well-supported with exampleswell-supported with examples of of challenges, techniques and lessons learned.challenges, techniques and lessons learned.

5. 5. Analyzes factors from experienceAnalyzes factors from experience that contribute that contribute to progress. to progress.

6. 6. Justifies strategies usedJustifies strategies used and evidence for and evidence for effectiveness.effectiveness.

moremoresuperficisuperfici

alal

deeperdeeper

Learman LA, O’Sullivan PS. AAMC-RIME 2006.

Reflection Growth by PGY

Surgical skill^ 3.30 (0.73) 3.13 (1.07) -0.19

Professionalism 2.70 (1.18) 2.68 (1.66) -0.01

Communication 2.63 (1.19) 2.63 (1.20) 0.00

Evidence-based medicine

2.35 (1.07) 3.07 (1.08) 0.65

Systems-based practice

1.97 (1.31) 2.37 (1.60) 0.28

Practice improvement

1.53 (1.20) 1.78 (1.26) 0.20

OVERALL# 2.28 (0.47) 2.64 (0.62) 0.36

Skill for Reflection Years 1-2Mean (SD)

Years 3-4Mean (SD)

Effect Size*

* Effect size = (difference in means)/pooled standard deviation^Skill scores range from1-6 and include all exercises completed by 32 residents.#Overall scores include only the 25 residents who completed all six reflections.

Learman LA, O’Sullivan PA. AERA Annual Meeting 2007.

Resident FeedbackResident Feedback

Focus groups after 1Focus groups after 1stst year of reflection 05/06: year of reflection 05/06: Valued reflection, but had incomplete and variable Valued reflection, but had incomplete and variable

understanding understanding Preferred reflection-in-action (in real time) over Preferred reflection-in-action (in real time) over

reflection-on-action (delayed)reflection-on-action (delayed) Discounted value of written (versus oral) reflectionsDiscounted value of written (versus oral) reflections Felt that specific assignments were constraining Felt that specific assignments were constraining

and artificialand artificial Wanted feedback on and discussion of reflectionsWanted feedback on and discussion of reflections

Foster-Barber, Chittenden, Learman, O’Sullivan. UCSF Education Day 2007

Improvements in 06/07Improvements in 06/07 Better explanation of role of reflection in Better explanation of role of reflection in

medical educationmedical education Choice among 6 options, 3 for each semi-Choice among 6 options, 3 for each semi-

annual meetingannual meeting Clearer instructionsClearer instructions Exercises divided into discrete tasks Exercises divided into discrete tasks More lead time for sharing with peersMore lead time for sharing with peers More lead time for review by program directors More lead time for review by program directors

prior to feedback sessionprior to feedback session

Foster-Barber, Chittenden, Learman, O’Sullivan. UCSF Education Day 2007

Lessons Learned on Lessons Learned on ReflectionReflection

Optimal design of exercises unclearOptimal design of exercises unclear Need to:Need to:

Hone residents’ understanding of reflectionHone residents’ understanding of reflection Give more freedom in content/timing of Give more freedom in content/timing of

reflectionsreflections Provide mechanisms to ensure timely Provide mechanisms to ensure timely

feedbackfeedback Introduce reflection exercises earlier in Introduce reflection exercises earlier in

medical education to improve their medical education to improve their acceptability to residentsacceptability to residents

Foster-Barber, Chittenden, Learman, O’Sullivan. UCSF Education Day 2007

Reflecting About PortfoliosReflecting About Portfolios

What They Are

PurposefulPurposeful collections of evidence collections of evidence used by students to document and used by students to document and reflect on learning outcomes over reflect on learning outcomes over time.time.

A learning tool enabling residents to:A learning tool enabling residents to: track their experiencestrack their experiences self-reflect on those experiencesself-reflect on those experiences share their insights with mentors, share their insights with mentors, receive real-time formal feedback receive real-time formal feedback

Consider

What do you think?What do you think? Fad or Formidable InnovationFad or Formidable Innovation

Which aspects seem more/less Which aspects seem more/less valuable?valuable? Tracking progressTracking progress Refection and self-assessmentRefection and self-assessment Feedback and mentorshipFeedback and mentorship

Their Purposes or Functions

To store evidence of learning To store evidence of learning

≠ ≠ a true portfolioa true portfolio To promote feedback, reflection, To promote feedback, reflection,

growth growth

= formative= formative To determine advancement, To determine advancement,

graduation graduation

= summative= summative

Consider

Which functions will be easiest to Which functions will be easiest to implement in your residency implement in your residency program?program? Data warehouseData warehouse Formative review of growth, Formative review of growth,

developmentdevelopment Summative decisionsSummative decisions

Why? Why?

The Essentials for Success

Time for reflection and mentorshipTime for reflection and mentorship Separation of formative and summativeSeparation of formative and summative Learners own access, grant permissionLearners own access, grant permission Essays assigned to aid reflection Essays assigned to aid reflection Summative decisions are based on fair, Summative decisions are based on fair,

valid, and reliable assessmentsvalid, and reliable assessments

» » Not even the best technology can make-up Not even the best technology can make-up for an absence of any of these essentials!for an absence of any of these essentials!

Consider One

Mentorship and FeedbackMentorship and Feedback How is time currently set aside for mentorship? Are How is time currently set aside for mentorship? Are

mentors different than faculty responsible for advancement mentors different than faculty responsible for advancement and graduation? Do they review evidence of learning and and graduation? Do they review evidence of learning and provide formative feedback? How might this be improved?provide formative feedback? How might this be improved?

Owning the EvidenceOwning the Evidence Currently, are any assessments or evidence of learning Currently, are any assessments or evidence of learning

“owned” by your residents and shared with you at their “owned” by your residents and shared with you at their discretion? What kinds of evidence could be? discretion? What kinds of evidence could be?

Reflecting on LearningReflecting on Learning What opportunities (or time) do your residents have to do What opportunities (or time) do your residents have to do

structured reflections about their progress? How could this structured reflections about their progress? How could this be created or expanded? be created or expanded?

ACGME Learning Portfolio Timeline

Now: Alpha test, cultivate early adopters, Now: Alpha test, cultivate early adopters, specialty user-groupsspecialty user-groups

Early 2008: Create beta phase prototypeEarly 2008: Create beta phase prototype Mid 2008 – Late 2009: Beta testing phaseMid 2008 – Late 2009: Beta testing phase Early 2010: Finalize initial roll-out prototypeEarly 2010: Finalize initial roll-out prototype 2010: Initial roll-out available, voluntary2010: Initial roll-out available, voluntary 2016: Full implementation, available for all2016: Full implementation, available for all Ongoing: Consider linkages to UME and MOCOngoing: Consider linkages to UME and MOC

http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

Coming Sooner Than 2016

PIF Transition DocumentPIF Transition Document PBLI: “Describe one learning activity in PBLI: “Describe one learning activity in

which residents engage to identify strengths, which residents engage to identify strengths, deficiencies, and limits in their knowledge deficiencies, and limits in their knowledge and expertise (self-reflection and self-and expertise (self-reflection and self-assessment); set learning and improvement assessment); set learning and improvement goals; identify and perform appropriate goals; identify and perform appropriate learning activities to achieve self-identified learning activities to achieve self-identified goals (life-long learning)”goals (life-long learning)”

New PIF likely to elaborate . . .New PIF likely to elaborate . . .

Opportunities

• Studies across GME programs to establish fairness, validity and reliability of measures

• Faculty development for giving high-quality, behaviorally-anchored feedback

• Faculty development for advisors and summative assessors

• Optimize how self-reflection happens• Participate in beta-testing of ALP

To Learn More . . .

Dannefer EF, Henson LC. The portfolio approach to Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic competency-based assessment at the Cleveland Clinic Lerner College of Medicine [of Case Western Reserve Lerner College of Medicine [of Case Western Reserve University]. University]. Academic MedicineAcademic Medicine 2007;82:493-502. 2007;82:493-502.

Challis M. AMEE Medical Education Guide No. 11 Challis M. AMEE Medical Education Guide No. 11 (revised): Portfolio-based learning and assessment in (revised): Portfolio-based learning and assessment in medical education. medical education. Medical TeacherMedical Teacher 1999;21(4):370- 1999;21(4):370-86. 86.

O’Sullivan PS, Cogbill KK, McLain T, Reckase MD, O’Sullivan PS, Cogbill KK, McLain T, Reckase MD, Clardy JA. Portfolios as a novel approach for Clardy JA. Portfolios as a novel approach for residency evaluation. residency evaluation. Academic PsychiatryAcademic Psychiatry 2002;26(3):173-79.2002;26(3):173-79.

Acknowledgments

• Meg Autry, Laura Pliska, and Patricia O’Sullivan for development and implementation of the reflection exercises

• Patricia O’Sullivan for feedback regarding earlier versions of this presentation