IPSSW Pre-Conference Workshop 7 (2016, Glasgow)

Simulation Research:Improving Rigor, Impact, and

Methodology

May 8, 2016

Glasgow, UK

IPSSW Pre-Conference Workshop

International Network for Simulation-based Pediatric Innovation, Research and Education

Introduction / Faculty

• Ralph MacKinnon

• Todd Chang

• Vinay Nadkarni

• David Kessler

Disclosures

• No relevant financial conflicts of interest

• INSPIRE receives funding/support from



Mission

We aim to improve the delivery of medical care to acutely ill children

by answering important research questions pertaining to resuscitation,

technical skills, behavioral skills, debriefing and simulation-based

education



Who are you?

• Where do you work with Simulation?

• What is your simulation experience?

• What is your research experience?


Learning Objectives

• Define the 2 types of simulation-based research

• Evaluate characteristics of a simulation-based research question or protocol that are conducive to quality research and output

• Identify 6 specific challenges in simulation-based research that threaten rigor and quality of research output

Objectives

1. Develop a study design using simulation as the subject of research

2. Develop a study design using simulation as an investigative methodology

3. Use Reporting Guidelines to improve study methodology


Schedule


Didactic Discussion Small group work Break

Closing Discussion & Networking

Objectives






2 Types of Simulation Research

Research subject

(e.g. Team training)

Research method

(e.g. Test new defibrillator)


1: Simulation as the research subject

• Is simulation an effective training methodology?

• How do we maximize effects on patient outcomes?

2: Simulation as the research method

• Leverages standardization of simulation

• Questions that might not be safe, feasible, ethical, timely in traditional research

Simulation as the subject of research



INSPIRE Research Themes

Simulation as the subject

Debriefing Develop/assess/implement effective techniques for debriefing real/sim events

IPE, Teamwork, Communication

Develop/assess/implement effective techniques for team training

Procedural, Psychomotor Skills

Develop/assess/implement effective techniques for skills development retention


• Research examining whether or not specific features of simulation experiences are effective




#1 start with strong conceptual model

What we know…

• 609 studies

• 23% RCT’s

• 12% multicenter studies

• 5% reported patient and/or healthcare outcomes

• Sim is an effective educational intervention compared to no intervention


Pediatric Research

• 57 studies

• 1/3 RCT

• compare sim to nointervention

• Large pooled effective size

• Knowledge

• Skills in simulation

• Time to completion

• < 10% on patient outcome


2 Lessons


1. Avoid the null comparison • (Don’t compare to no intervention)


Varying Instructional Design

1. Feedback ***

2. Repetitive Practice ***

3. Distributed Practice

4. Curriculum Integration

5. Clinical variation

6. Range of difficulty

7. Individualized learning

8. Multiple learning strategies

9. Defined Outcomes

10. Valid simulator


What examples of Instructional Design Features have you

investigated?Discussion 1

2 Lessons


2. Measure patient outcomes • (Or at least provider behavior)

Knowledge/skills

Attitudes/perceptions

Learners’ satisfaction/reaction

Yadley 2011, BEME 2005, Miller 1990, Kirkpatrick 1967

Δ practice

Δ behavior

Knowledge/skills




Popul-ation

Patient

Δ practice

Δ behavior

Knowledge/skills




Framing your Research Outcomes

• Goal of medical education• To transmit knowledge, impart skills, and inculcate

the values of medicine

that will improve health outcomes

Cooke, Irby, O’Brien. Carnegie Foundation Report: Educating Physicians: A call for reform of medical school and residency. 2010


Selecting outcomes

• Important to patient• Measurable• Dependent on provider skills/behaviors

• Quality• Satisfaction, comfort

• Procedure related **• Success/failure• Complications (Erb’s Palsy, Line Infections)

• Survival • Cardiac resuscitation, obstetrics, NICU


INSPIRE Examples


Infant Lumbar Punctures

• Essential pediatric procedure

• Decreasing experience

• Novice success rate 35%

• Bench top simulator• Low cost

• Easy to use

Auerbach et al Pediatric Emergency Care 2013, Gaies Pediatrics 2007


Infant Lumbar Punctures

• Instructional Design principles: • Distributed Practice (JITT)

• Repeated Practice

• Defined Outcomes

• Outcomes selected:• Clinical Lumbar Puncture success


Research Question

• Population

• Intervention

• Comparison

• Outcome


Research Question

• P – 1st year residents (‘interns’)

• I – Mastery Learning LP Training + Just-in-Time Training on LP Simulator just prior to clinical LP

• C – Mastery Learning LP Training only

• O – Clinical LP success rate


Auerbach et al, Pediatric Emergency Care 2013

Overall Performance Score


Let’s design a Research Project: Simulation as the subject

Small Group 1

Objectives





Simulation as an investigative methodology



INSPIRE Research Themes

Simulation as the method

Technology Develop/assess/implement novel technologies designed to improve processes of care and pediatric patient outcomes

Acute Care and Resuscitation

Develop/assess/implement novel techniques for improving care of pediatric patients

Human Factors Assess the role of human factors when providing care to pediatric patients

Patient Safety Explore the key variables that influence patient safety and assess strategies to mitigate

Simulation as the method

• The simulated environment is used as an experimental model to study factors affecting human and systems performance in healthcare.



Why use simulated instead of real scenarios

• Standardization of “noise”

• Patient- age, clinical status, parent actions

• Individuals- experience, stress, fatigue

• Team- structure, numbers

• Environment- structure, distractions, noise

• Technology- equipment availability, size, safety

• Ask questions that might not be safe, feasible, ethical, timely in traditional research

Threats to internal validity


Threats to validity

• Educator training

• Simulator Selection

• Scenario Design

• Confederates

• Realism (physical, conceptual and emotional)

• Debriefing

• Video Capture and Review

• Study OutcomesCheng et al, Pediatrics, 2014

• ADD SLIDES ON EDUCATIONAL DESIGNS- e.g. solomon 4, cluster randomized etc…(martin’s paper, selin workshop)

• Ethics- rct not always possible. (perfume example)

What examples of Threats to Validity have you encountered?

Discussion 2

INSPIRE Examples


Technology

• A QCPR simulator was

used to show that

Feedback Devices

improved CPR

performance


Systems/Human Factors

• Individuals

• Teams

• Environments

• Technological Factors

• Systems Factors

• Patient Factors


SEIPS 2.0. Ergonomics . 2013 November ; 56(11) Holden, Carayon et al


SEIPS 2.0. Ergonomics . 2013 November ; 56(11) Holden, Carayon et al


New Environments


http://www.google.com/imgres?imgurl=http://www.childlife.org/conference09/data/Sponsors.cfm_files/UMass_Memorial.jpg&imgrefurl=http://www.childlife.org/conference09/data/sponsors.htm&usg=__Wzeyyd9PXcVncn3N5Hfa859qPh8=&h=440&w=982&sz=89&hl=en&start=1&zoom=1&tbnid=6AshcxsNsklhPM:&tbnh=67&tbnw=149&ei=moPkUKi_EObg0QHX8YG4BA&prev=/search?q=umass+children's+medical+center&um=1&hl=en&gbv=2&tbm=isch&um=1&itbs=1

http://www.google.com/imgres?imgurl=http://www.childlife.org/conference09/data/Sponsors.cfm_files/UMass_Memorial.jpg&imgrefurl=http://www.childlife.org/conference09/data/sponsors.htm&usg=__Wzeyyd9PXcVncn3N5Hfa859qPh8=&h=440&w=982&sz=89&hl=en&start=1&zoom=1&tbnid=6AshcxsNsklhPM:&tbnh=67&tbnw=149&ei=moPkUKi_EObg0QHX8YG4BA&prev=/search?q=umass+children's+medical+center&um=1&hl=en&gbv=2&tbm=isch&um=1&itbs=1


Safety / Quality

• In-situ simulation• Mobile unit brought into work environment

• Use real life equipment in “context”

• Work with real life team

• Identify latent safety threats

• Provider level - insufficient staffing/training

• Systems level - missing/broken equipment


Direct comparison: same patient at different hospital

Crash Testing, not Normal Testing

• Identify and remediate threats to safety • Active

• Latent: unrecognized errors in design or organization


Let’s design a Research Project: Simulation as the method

Small Group 2

Objectives





Reporting Guidelines

• Standardized reporting will enhance quality of studies and assist journal reviewer/editors

• Explanation and elaboration document will support implementation of reporting guidelines

• Reporting guidelines currently under review• Eventual adoption for author guidelines


Reporting Guidelines

• Extensions to prospective trial design:

• CONSORT (randomized-control)

• STROBE (observation cohort)


How do we use Reporting Guidelines?


Let’s review our Research Project: Using Reporting

GuidelinesSmall Group 3

Objectives





What changes will you be making?


What changes will you be making?



Questions?

www.inspireSim.com


IPSSW Pre-Conference Workshop 7 (2016, Glasgow)

Science