From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

From Learning Objectives to Outcomes

Assessment During SimulationPaul E. Phrampus

DirectorWinter Institute for Simulation, Education and

Research (WISER)University of Pittsburgh

• Why• What• When• Where• How

Assessment

The Future of Simulation………

What is a simulation for?

1. Education2. Assessment3. Fun4. 1 and 2

TESTING TEACHING

The Simulation Advantage

Psychomotor SkillsPsychomotor Skills

Decision MakingDecision Making

Skills and DecisionsSkills and Decisions

Individuals Individuals Teams Teams

What is “Simulation”….. Really…..

The Job of Healthcare Educators

Psychomotor Skills

Communications Skills

Professionalism Skills

Decision MakingBase Knowledge

Teamwork Skills

Why is this so hard?

Learning Objective

• You have to have a clear reason of why you are running any simulation.

Lets Watch

• What is most important aspect to debrief?

Where does a simulation fit?

• It depends on what your doing!

EACH Sim is part of a bigger plan!

What is this?

Does it belong here?

Course (Global) objectives v sim objectives

Global Objectives

• Course Objectives– 1. Demonstrate a proper patient assessment– 2. Demonstrate following the emergency protocol– 3. Demonstrate Airway Management Skills

Course Schedule / Flow

• Sim 1• Sim 2• Lecture• Skills Lab• Sim 3• Sim 4

Course Objectives

Course Objectives

Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol

Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol

How do you determine “success” of a simulation?

Students

liked...

Learn

ing objec...

Students

passe...

Not S

ure

25% 25%25%25%1. Students liked it2. Learning objectives

achieved3. Students passed the

scenario4. Not Sure

How do you determine “success” of a simulation?

Instructor Number 1

Instructor Number 2

Instructor Number 3

How do you determine “success” of a simulation?

Instructor Number 1 Instructor Number 2 Instructor Number 3

Measure Here

You can only keep track of so many things at one time

How many of you feel like this ?

Tools for Success

• Make it easy for your faculty to stay on track

• “In my opinion…..”

Assessment Methodology– Identify methods for measuring core aspects in

simulation education / assessment– Discuss development and deployment of

measurement tools

Student Learning

OutcomesCurriculum

Teaching & Learning Assessment Evaluation

Assessment and Evaluation

• Assessment and Evaluation often used interchangeably

• Assessment – learner outcomes• Evaluation – course or program outcomes

Performance Assessment

Rate the Performance

• Pass or Fail

• Score Performance

• Procedural Check

Assessment

• Why• What• When• Where• How

Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim Healthcare 3:2008;33–41.

ACTION

PERFORMANCE

COMPETENCE

KNOWLEDGE

Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

Assessment of clinical skills/competence/performance

Performance Assessment

• Basic to performance – Do they know it and know how?

• Competence – Can they do it?

• Performance – Do they do it?

DOES

SHOWS HOW

KNOWS HOW

KNOWS

Knows When

Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

Assessment of clinical skills/competence/performance

DOES

SHOWS HOW

KNOWS HOW

KNOWS

Knows When

Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

Assessment of clinical skills/competence/performance

Knowledge Tests

MCQs, Essay, Oral exam

Clinical Based Tests

Pt Mgt Qs, Essay, Oral

Performance Tests

OSCE, TaskDemo, SIM/SP

Practice Review

Audits, Video, UC-SP, 360O

Assessment• Formative Assessment

– Lower stakes assessment– One of several assessments over time of course or program– May be evaluative, diagnostic, or prescriptive– Often results in remediation or progression to next level

• Summative Assessment– Higher stakes assessment– Generally final course or program assessment– Primary purpose is performance evaluation– Often results in a Go-No Go outcome

Assessment

• Assessors– Instructors– Observers– SIM/Patients– Peers– Participants

• Instruments– AV annotation– Pencil/Paper– Scantron form– TabletPC or PDA

• Assessment data sources– Live observation– Video recording– SIM device logs

STUDENT

EVALUATOR

ASSESSMENTPATIENT

Variables in Assessment

Assessment Validity

• Validity – – Degree to which an assessment tool measures what it is

intended to measure– Construct validity

– Accuracy of tool to measure intended construct

• Concurrent validity• Content validity• Predicative validity

Assessment Reliability• Test Reliability –

– consistency of a test to yield the same measure over time (does not assure validity)

– Test-Retest Reliabilityconsistency among different administrations

– Parallel Forms Reliabilityto assure that memory effects do not occur use a different pre- and posttest

– Inter-Rater Reliability two or more observers rate the same subjects with strong positive correlation of their ratings

STUDENT

EVALUATOR

ASSESSMENTPATIENT

Variables in Assessment

Assessment Improvement

• Improving Reliability and Validity– Base assessment on outcome/objectives >

event triggers > observable behavior > behavioral rating > assess against competencies

– Define <low – medium – high> performance– Use of Rubric or rating metric– Use (video) training examples of performance– Employ a quality assurance/improvement system

Assessment Based on Objectives and Events

Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.

Event – Behavior – Scale - Report

Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare. Sim Healthcare 3:2008;33-41.

Assessment Metrics

• Procedural or Check List assessment• Global Rating assessment

Assessment Metrics• Procedural or Check List assessment

BCLS Y N

Open Airway

Check Breathing

BCLS Y N

Open Airway(< 5 sec of LOC)

Check Breathing(< 5 sec of Airway)

BCLS Y N

Open Airway

Check Breathing

A

Rating Score +1 -1 0 *Assist

Assessment Metrics• Global Rating assessment

Code Blue P F

CPR and

ACLS

Code Blue

CPR<1(low) - 5(Hi)> points

ACLS<1(low)- 5(Hi)> points

Code Blue H M

CPR

ACLS

L

Rating Score +1 -1 0

Pts.

Assessment Metrics• Procedural or Check List assessment• Global Rating assessment

< EXAMPLES>

• Pros and Cons to each• Multiple modes & points vs. Single assessment• QA/QI Metric – collective vs single assessment

Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics 121(3):2008;597-603.

Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1)July 1998:8-18.

Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.

Interrater Reliability

Evaluator9

Evaluator8

Evaluator7

Evaluator6

Evaluator5

Evaluator4

Evaluator3

Evaluator2

Evaluator 1

5 ScenarioVideos

Novices

Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations

Andrus, Phrampus, Wang

Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations

0

0.2

0.4

0.6

0.8

1

1.2

9 Raters

6 Raters

3 rookies

Andrus, Phrampus, Wang

Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations

• CONCLUSIONS: – Simulation instructors exhibited good overall

inter-rater agreement but showed poor agreement for some key airway tasks.

– Variability in instructor ratings may influence real-time evaluation of simulation performance.

– Likely could be improved with careful instructor orientation / training

Andrus, Phrampus, Wang

Evaluation

• Individual Trainee Evaluation– Per episode performance– Course of the Day

Evaluation

• Group Trainee Evaluation– Benchmark

Evaluation

• Instructor Evaluation– Per episode performance– Measure Against Benchmarks

Testing Points

T I M E

Behind the Scenes

Behind the Scenes

Behind the Scenes

Behind the Scenes

Behind the Scenes

Summary

• Know why you are doing the sim• Each sim is part of a bigger picture• Design tools for consistency• Make it easy for your faculty!

Questions and discussion