AGS Annual Meeting May 2005 Converting the Geriatric Functional Assessment Standardized Patient Instructor …into an OSCE Karen E. Hall, M.D., Ph.D. Clinical Assistant Professor Division of Geriatric Medicine University of Michigan, Ann Arbor VAMC
Dec 20, 2015
AGS Annual MeetingMay 2005
Converting the Geriatric Functional Assessment
Standardized Patient Instructor
…into an OSCE
Karen E. Hall, M.D., Ph.D.Clinical Assistant Professor Division of Geriatric Medicine
University of Michigan, Ann Arbor VAMC
AGS Annual MeetingMay 2005
GFA SPI: CharacteristicsGFA SPI: Characteristics
• ““Essential Skill Set” for all physiciansEssential Skill Set” for all physicians
• Emphasis on function, not diseaseEmphasis on function, not disease
• Validated, rapid screening tools Validated, rapid screening tools
• FeedbackFeedback
• EducationEducation
• Multiple types and level of learnerMultiple types and level of learner
AGS Annual MeetingMay 2005
Steps to convert the SPI into an Steps to convert the SPI into an OSCEOSCE
1. Review scenario for reliability and validity1. Review scenario for reliability and validity (high stakes exam requires rigor)(high stakes exam requires rigor)
Is task is appropriate for level of learner and time allotted?Is task is appropriate for level of learner and time allotted? Err on side of “too little content” rather than “too much”Err on side of “too little content” rather than “too much”
Faculty review to ensure content is clinically relevant (validity)Faculty review to ensure content is clinically relevant (validity)
Review checklist items with SPs to ensure performance and Review checklist items with SPs to ensure performance and scoring is accurate. scoring is accurate.
Faculty should observe a percentage of OSCEs (at least 5-Faculty should observe a percentage of OSCEs (at least 5-10%) and score using same checklist as SPs (reliability)10%) and score using same checklist as SPs (reliability)
AGS Annual MeetingMay 2005
Case ScenarioCase Scenario
•85-year-old male or female85-year-old male or female
•Inpatient or Emergency RoomInpatient or Emergency Room
•““Ready for Discharge” BUT...Ready for Discharge” BUT...
•Functional impairmentFunctional impairment
•Social issue: caregiver stressSocial issue: caregiver stress
•Focus on functional assessment Focus on functional assessment
not diagnosisnot diagnosis
AGS Annual MeetingMay 2005
Administration of SPI in Learning Administration of SPI in Learning Resource Center (LRC)Resource Center (LRC)
Total time: 1 hour Minutes
Introduction 5SPI 20Learner Survey 5SPI feedback 10Faculty debrief 15
Learner evaluation of SPI 5
AGS Annual MeetingMay 2005
Administration of GFA OSCE in Administration of GFA OSCE in Learning Resource Center (LRC)Learning Resource Center (LRC)
Total time: Minutes
Review case on door 3SP OSCE 10
Learner self assessment 2
Entire CCA is 4 hours with 14 stations
Acute abdomen, Health beliefs, Chest pain, Chest pain note, Geriatric, Back pain, Pediatric, XRays/Imaging, EKG, Psychiatry, Evidence Based Medicine + 2 breaks
AGS Annual MeetingMay 2005
Type of Data available from OSCE
By Student:
• raw scores (not done=0,needs improvement=1, done=2) by item
• mean +/- SD of raw scores (“1.7 +/- 0.4”)
• percentage score (tasks; communication) – identifies pass/fail
AGS Annual MeetingMay 2005
Type of Data available from OSCE
Whole class:
• Histogram of scores by item (e.g. “done vs. “not done”) - Identifies deficiencies in class performance ? Curricular deficiency?
• Percentage for all stations “overall score” – distribution of the class: “high”; “average”; “low”
AGS Annual MeetingMay 2005
Educational GFA SPI:M1
0
25
50
75
100
Eatin
g
Transf
erin
g
Toiletin
g
Bathin
g
Dress
ing
Groom
ing
Per
cen
tag
e o
f M
1 st
ud
ents
Not Asked (1) Asked (2) Asked in Detail (3)
Evaluative GFA OSCE:M3
0
25
50
75
100
Per
cen
tag
e o
f M
3 S
tud
ents
Not Asked (0) Needs Improvement (1) Done (2)
AGS Annual MeetingMay 2005
OSCE: IADL Assessment
Students remembered to ask about meals and grocery shopping more frequently than medications. Students often neglected to ask in enough detail to score “done”.
0
25
50
75
100
Medications Prepare Meals Transportation Other
% o
f M
3 S
tud
ents
Not Asked (0) Needs Improvement (1) Done (2)
AGS Annual MeetingMay 2005
OSCE: Falls, Observe Gait, Timed Up and Go
Most students remember to ask about falls and observe gait, but few do objective test (TUG).
0
25
50
75
100
Asked about falls Observed Gait Timed Up and Go
% o
f M
3 S
tud
ents
Not Done (0) Needs Improvement (1) Done (2)
AGS Annual MeetingMay 2005
Special Communication
0
25
50
75
100
Facilitated Visual Faciliated Auditory
% o
f M
1 st
ud
ents
Poor (1) Adequate (2) Skillful (3)
0
25
50
75
100
Glasses Hearing
% o
f M
3 S
tud
ents
Not done (0) Needs Improvement (1) Done (2)
EducationalSPI
EvaluativeOSCE
AGS Annual MeetingMay 2005
Communication (9 items)
0
25
50
75
100
All Stations Geriatric
M1
Rat
ing
(1-
5 sc
ale)
Best performance (100%): “addressed by name”; “didn’t interrupt”Worst performance (60-62%): “asked open-ended questions”; ”summarized discussion”
AGS Annual MeetingMay 2005
GFA OSCE Data from PilotGFA OSCE Data from Pilot
1. Numerical percentage score on GFA
17 task items (max score =32)
Converted to percentage of 100%
Mean +/- SD for tasks 43 +/- 15%
Communication score 84 +/- 5 %
AGS Annual MeetingMay 2005
GFA OSCE Data from PilotGFA OSCE Data from Pilot
2. Learner self-evaluation score (% of 100) on GFA: “Estimate your average score for this station”
Mean +/- SD 74 +/- 12%
Clearly there is a discrepancy between the student’s assessment and their performance score of 43%!
AGS Annual MeetingMay 2005
Overall Performance Overall Performance (excluding GFA)(excluding GFA)
Overall score on CCA 76 +/- 6%
Passing average set at… 64%
Out of 162 students…..
62 students passed all stations
97 failed 1-3 stations – retake failed stations
3 students failed entire CCA
Significant correlation between failed 2 or more stations and very low GFA scores (less than 30) – overall lack of knowledge?
AGS Annual MeetingMay 2005
Hofstee Method to define Hofstee Method to define “pass/fail” for 2005 CCA“pass/fail” for 2005 CCA
“What is the lowest % students you would accept to fail station” - usually 0%
“What is the highest % students you would accept to fail station” - high for vital material, low for unimportant
“What is the lowest % cut-off you would accept to fail station” – low if hard material, high if easy
“What is the highest % cut-off you would accept to fail station” – high if vital
AGS Annual MeetingMay 2005
Hofstee Method to define cut Hofstee Method to define cut points for 2005 GFA OSCEpoints for 2005 GFA OSCE
As of March 2005 (AGS handouts due)….
Don’t know (still being analyzed – will know before May 2005) – Still don’t know!
Given poor performance and curriculum revision during 2004-2006: may need generous cutoff criteria
0%-40% failure rate, 45%-60% pass cut off
May be more like 10% failure, 50% cut off
AGS Annual MeetingMay 2005
Take home points for converting Take home points for converting an SPI to an OSCEan SPI to an OSCE
Keep it simple
Be prepared to justify your choice of material – to faculty and students (particularly those that fail!)
Make sure tasks are taught in curriculum!
Be prepared to review a lot of performances
Keep checking the reliability of the SPs
Fight off all attempts to change the OSCE checklist in mid-assessment (data will not be analyzable!)
AGS Annual MeetingMay 2005
Reference sitesReference sitesTalk will be on my website:
http://sitemaker.umich.edu/khallinfo
GFA Student manual and pocket card on POGOe (registration required):
http://www.pogoe.org/px/login.cfm
I can email other material if needed: