Nadim Lalani MD. Vanilla Sky Tom Cruise 2001 Existential Mind warp Deals with cryogenics and the possibility of living a virtual life after death Blending.

Nadim Lalani MD

Vanilla Sky

Tom Cruise 2001 Existential “Mind warp” Deals with cryogenics and the

possibility of living a virtual life after death

Blending of the technologic and biologic worlds “plugged in’

?Virtual [technologic] world to supplement [real world] EM medical education

Objectives Definition Background Literature Review

Med Ed Resident Ed Professional Development

What it might look like Future Directions

What is “Online learning”? Online Learning [e-learning] = is digital Evolved from CD/computer labs Everyone does it! Performance Support [ for software e.g] Web page [e.g. Uptodate] Self-paced Web-based [CME] Leader-led [ Distance Learning] Blended [or hybrid] learning

combines conventional with digital learning

Advantages of e-learning Rich environment:

Media-filled [esp in EM] transfer of difficult concepts Links to sources

Convenient, efficient & flexible Asynchronous Can be accessed from a distance

Adult learning principles: Self-paced and self-directed Flexible/ home access efficiency

Background: Why bother?

U of C Medical School current enrollment = 130 students goal 150 Mandatory EM rotation / increasing

competencies Resident numbers also increasing Result more learners in the ED Relative shortage of preceptors, relevant

clinical encounters and curricular time Will be worse when our program expands

usurp learning opportunities

Why bother

Deficiency in learning encounters = a performance gap future physicians do not have the

adequate exposure to emergent problems.

imperative we equip students, clerks & residents with the skills and training.

why bother Increased digitalisation is a key

strategic goal of the U of C Learners are unique with

mulitdimensional learning styles. Adult learning principles Attract the best candidates Provide a method of training

students and clerks at two different campuses

Provide consistency in teaching

Why bother? Provide efficient means of knowledge

transfer to residents Increasing number of competencies

[CANMEDs] Better use of academic half-day.

Provide more effective professional development: Asynchronous don’t have to be there Interactive discussion board Consistent, evidence-based standard of

practice Increased self-efficacy

E-learning not a panacea there is more to training and

education than e-learning Certain skills do not lend themselves

to e-learning The key will be selecting the best

delivery method. Cannot simply upload old material. Learner – focused no one solution [blended may work for

residents].

Process: Can it be done?

“Fail to prepare … prepare to fail”

Need to address several key questions:1. Purpose? Added value?2. What support and expertise exist?3. Ongoing upkeep?4. Stakeholders?5. Team?6. Instructional design/Pedagogy

Literature Review

Same Search terms in PUBMED Bibliographies of relevant articles scanned Missing 1 Med Ed & 1 CME [both foreign

language]

Literature general Comments More Literature exists for Med Ed

Pre 1990 Limited by lack of internal validity

Few Randomised Controlled Studies Emerging Lit wrt Resident

experience Despite lots of experience with

online CME Little Literature … mostly Descriptive

Literature General Comments Terminology inconsistent Interventions vary. ? “prototypes” of today’s

technology? Don’t address some of the uniqueness

[internet] Comparing apples to oranges

Can E-learning be used to replace/augment Traditional Methods?

E Learning & Med Ed

Study Dartmouth Med School 328 Students randomised to:

Interactive Case-based study guide on Computer*

Case-Based Printed study guide Anemia and Cardiology Courses

Outcomes: Performance on higher order MCQ tests,

exams Self-reported Efficiency

* media-rich images, blood smears and EKG’s

Results

No Difference in Test Scores No difference on board exams The vehicle is an acceptable means of

delivery

Time Spent

Cardiology Computer

4.4 ± 0.6h

Cardiology Workbook

9.4 ± 0.5h P = 0.0001

Anemia Computer

5.5 ± 0.5h

Anemia Workbook

8.0 ± 0.5h P = 0.001

Limitations Self reporting of efficiency! Confounders [other text books/practice exams/time-spent

cramming] Doesn’t really tell us about dynamic problem-solving/

clinical judgment

Board Exams

Anemia CV

Class + CAL

81.5 ± 6.5 80.0 ± 6.7 NS

CAL 82.7 ± 6.0 82.4 ± 5.6 NS

179 Paeds Clerks in [2 sites Chicago] Randomised to Lecture via:

Multimedia Text Book* Lecture Printed Text No intervention

Paeds airway diseases Outomes:

MCQ Test Score [at end of rotation & at 1 y later]

* Only different in audio/video

Results

Score / 26

MMTB Printed Book

Lecture Control

N = 89 21 19 23 26

Initial Test 16.6 [2.5] 16.5 [3.6] 15.9 [2.5] 14.2 [2.9]

Final Test 15.3 [2.8] 15.3 [2.9] 14.6 [2.3] 14.5 [3.4]

Limitations

51% Attrition rate! Clerks at one site had mail-in repeat

exam Confounders

One hour instruction embedded in a 6 week clerkship

75 Med students [Brisbane Australia] Randomised after pretest to:

Computer Tutorial Focus on knowledge Computer tutorial create ideal patient

for dx + feedback [every 10 cases] Computer Tutorial both knowledge &

decision + three different types of feedback [after every 10 cases]

Looking at diagnosing abdo pain [ 30 cases]

Outcomes and Results Outcomes:

Attained knowledge Diagnostic accuracy Decision-making confidence [self reported]

Results: Students focusing on facts did not improve

decision-making All feedback groups improved diagnostic

accuracy Type of Feedback not important.

Self reported confidence improved

Limitations

Small study Very convoluted method ?

reliability

Can E-learning be used to Teach Procedural Skills?

E- Learning & Med Ed

82 Medical Students [Toronto & Augusta] Randomised to:

Computer Tutorial + knot board Lecture with Feedback + knot board Two-Handed Knot tying

Tested right after [filmed] Outcomes:

Proportion Square/ Time to tie Knot Performance score [blinded surgeons] Student Questionnaire

Results

NO difference in “Cognitive” portion Lower performance score in CAL group 89% Students would have preferred

Lecture Session Lack of feedback cited as negative

Limitations

Apples and Oranges! ? Not controlled for hands-on feedback Maybe CAL better if it described pitfalls /

showed video of good and bad knots? Reliability of performance score [not

included]

42 Clerks U of T Randomised to:

Computer Tutorial [rich text, animations, interactive –Q&A, no audio/video]

small group seminar [also interactive] Epistaxis Management

Outcomes: Short Answer written Test Practical Test [16 point performance

scale]

Results:

Poor Prior knowledge No difference in written scores Slightly better practical skills with

CBL

Limitations:

Small numbers Examiners NOT blinded ?reliability of performance score [not

included] Practical was on dummy

? transferability

69 Medical Students [Wisconsin] After pre-test Randomised to:

Didactic Session/Q&A¥

Video-Tape Session* Computer Tutorial*

Post Intervention: MCQ test, Practical Skills test [2 blinded

obs] Repeat testing at one month¥ no feedback . * Instructor present

Outcomes: MCQ Test Scores Timed observation of skills Critical Skills evaluated via checklist Performance Quotient calculated

Results

Higher initial mean % correct / % complete in CBT group [p<0.01]

Significantly better PQ in CBT group at 1 month [p < 0.01]

Results

Didactic group better on immediate MCQ [63% vs 49% for video/CBT p < 0.01]

Difference in MCQ still there at 1 month

Results

Bigger change in PQ with CBT at 1 month [ P< 0.01]

Limitations

Small study Video vs CBT essentially the same

intervention ? Why CBT would do better than Video

? Reliability of checklist and PQ?

Can E-Learning be used for Emergency Medicine Rotations?

What About the ED Experience?

100 Clerks [Mt Sinai] Randomised by blocks

EM rotation with access to EM Website Modules [ACLS, Tox], Xrays, Pix, Paeds

Cases EM Rotation without access Outomes:

Exam Scores Student Satisfaction

Results

ONLY 28% intervention group used it. 72% Cited lack of time

NON sig difference in exam score [72.8 vs 68.2 p = 0.058]

Non sig difference in satisfaction [ 77.5% vs 66% p = 0.23]

Baseline only 26% > 1h /wk online [cf 96% next class]

Baseline 65% wanted online component

Limitations

< 30% in intervention group didn’t reach power. WAS ITT so results would have been

+ve with more participants Problems with randomising by block

rotations given away on lottery [ CARMs]

Unmotivated learners? ?generalisable to clerks in 2008

23 Clerks [U of T Sick Kids] Volunteered for study, Randomised

to Access to Web-based Modules No Access to Web-based modules

ED Procedures [lac, LP, splint] Outcome:

Performance on MCQ Test

Results

Statistically higher competence [ p = 0.0001]

Cohen’s d Effect size r = 0.79

Limitations

Small sample size Volunteers [EM /techno gung-ho] Methodology:

Unclear when test was administered in relation to rotation

?randomised to learning vs no learning? Validity of MCQ vs Observed skills

Transfer of knowledge? MCQ vs Observed skills

350 Urology Clerks [4 med schools US] Randomised [two-group crossover] to:

Web-based Tutorials [BPH,ED,PC,PSA] No Access to Web Tutorials Served as the controls for the modules they

didn’t have access to online Outcomes:

Performance on test [pre/post] [Cr = .79] Durability of learning/ Learning efficiency in

SubG

Results

Results:

Results

Learning Efficiency 0.10 vs 0.03 [p<0.001] Test scores still improved without WBT [12%

BPH, 6% ED, 24% PC, 20% PSA] Web-based alone had Cohen’s r = 24.9!

Limitations:

Volunteers with unequal participation b/w sites [93% HMS vs 52% BUSM]

High Drop out rate 210 /350 completed

? Generalisability of repeated measures

? Generalisability to EM

Summary: E-Learning & Med Ed Content can be delivered Appears to be transfer of learning of

Cognitive skills … Perhaps also Psychomotor skills

Still need for experienced clinician feedback

Increased student satisfaction Attempt to make instructional design

identical [validity] undermines uniqueness of EL Always will be apples and oranges

Summary E-Learning & EM: Controlled Interventional Literature

conflicts Learners:

Identify time constraints as a barrier in ER Want more visual aids [Ekg, XR, photo & WEB]

Effective strategies for ER teaching include: Using Resources Going beyond patient care Improving the learning environment

Adult principles/ learners may be driving force despite dearth of evidence

How Can we Augment Med Ed?

Currently Clerks Use LMS “Osler”Download PDFs of Content/CasesCore Content Reviewed with Preceptor

We can do more …

Procedures/ Anatomy Pictures Xray / Ekg interactive ppt

E Learning for Residents

109 IM Residents [US] during “clinics” year

Randomised [Crossover] to: Access to Web-based Practice Guidelines

On WebCT [rich format with links to sources] Printed Practice Guidelines

Outcomes: Format Preference Performance on Final test

Results

Strong preference for Web Based material Men > Women

NO difference on test between groups Non-Significant reduction in time spent ++ problems with WEbsite

Limitations

Volunteer Significant dropout 145 eligible

109 enrolled 51 completed all aspects

? generalisability

162 FM & IM Residents Randomised to:

Web-based tutorial [hyperlinks, graphics] Printed material ACC/AHA Guidelines Management post AMI

Outomes: Test Score Efficiency Satisfaction

Results:

Limitations

Only enrolled 30% [550 eligible] Participants not blinded to hypothesis ? Generalisable

Voluntary + monetary honorarium

22 EM residents & Staff [U of T] Noninferiority for U/S vascular access

course Randomised [after pretest] to:

Web Tutorial + practice* 1h Lecture + practice*

Outcomes: [2 weeks later ..] MCQ test 4 OSCE stations [blinded obs used checklist]*Non-precepted

Results

Same pass rate All web users logged on from home

Limitations

Small study [did have power though] No controlling of practice session

Stronger coaching the weaker? No interrater reliability for OSCE

Summary: E-Learning + Residents Learning can be delivered this way

Weak evidence of non-inferiority for learning guidelines in other specialties

Modest evidence for use prepping for a hands-on session relating to EM.

Weak evidence for Psychomotor skill acquisition

Consistent satisfaction with the method

How to Augment Resident Ed

Website / Subscriptions / RemergS HPS / AHD

Better Use of AHD …

Longitudinal course on LMS? Already Licensed by U of C

Used by Anesthesia

Interaction is key

Learner-Sources, Learner-learner, Learner-Clinician

E Learning and CME

52 Physicians [US] randomised to: Web-Based CME Normal Instruction Office Dermatologic procedures

Outcomes: Performance on test Satisfaction Self-reported performance change

Results:

More improvement in test score with WBL [13.2 vs 9.6]

General satisfaction with WBL Increased self-reported competency Fair amount of interaction in

asynchronous forums

Limitations

Small study Self-reporting ? Reliability of MCQ test

? Direct observation of procedures

99 Physicians [US] Randomised to:

Web based CME for domestic violence Regular Instruction [only did two surveys]

Outomes: Self-Efficacy

Externally validated Survey instrument [CR 0.7]

Change in screening

Results:

Increase in self efficacy + 18% intervention vs - 0.6% control [p =0.01]

Positive other endpoints in survey NO difference in DV screening

Limitations

Honoraria to WEB participants Methods

?No learning to controls? generalisability

103 physicians [US] randomised to: Web-based CME F-2-F small group CME session Cholesterol management

Outcomes: Performance on tests Chart audit [20 docs from each group 25

charts] Satisfaction survey

Results:

More improvement with Online NO difference in Satifaction between

groups Online spent 3.8 h / 3 sessions Chart review statistically signif [small

difference] in guideline adherence [but no difference in cholesterol screening]

Limitations

170 eligible only 103 included ? Different groups? [online better at

baseline] Live CME event happened before Online

[?cross contamination] Hard to reconcile change in

behaviour when no change in screening

? generalisability

87 Physicians self-selected into: Online CME [3 sessions 8h] F-2-F CME [one 8 h session] Treatment of opioid dependence

Outcomes: Post intervention test Satisfaction

Results

NO difference in improvement No difference in satisfaction scores

Limitations

30% attrition [87 entered data for 62]

NON-randomised ?reliability of MCQ test ?Generalisability

Summary: E-Learning and CME Certainly can be used to disseminate

info In keeping with adult learning principles

Transfer of cognitive component ? Psychomotor NO change in practice patterns Subjective increased self-efficacy

Enhancing CME

Combine LMS with narrated slides Provide facilitation for your topic

Narrated Options for Quizzes

Future Directions

International Emergency Medicine Disseminate info before experts hit the

ground Collaboration with other programs

Online Electives? EBM Toxicology Radiology

Thoughts?

Appendix: How you get there

Discerning the Context

Learning Context Power Dynamics

PGME Regionalisation

People dynamics Resistance change Relatively few champions of Med Ed

Supporting Cast

Stakeholders: University UME /PGME RMES

Planning Committee: Content experts Educational specialists/Instructional

design IT [graphic design/ media] End-users

Identifying Program Ideas Core content

Literature Current material

Needs analysis Narrow focus Not redundant

Sorting and Prioritising

Prioritise content Important? Feasible?

Developing Objectives

Needs to be competency based Reflect goals of UME/ Colleges

Instructional Plan

How are you going to deliver this? Need interaction:

Content, Instructors & peers Dedicated ED computer WEBSITE LMS

Instructional Plan

Instructors: few innovators Clinicians Well suited “many hats” Need to “train the trainers”

Learners: Self-directed & Computer savvy Clinical/ procedural Skills Blended approach

Instructional Plan Variety & content:

Anatomy and radiology PowerPoint, Flash files, video streaming. e-case modules LMS online discussion

Logisitics: LMS powerful tool

Free ones available STARS uses Moodle U of C has licenses & tech Support Regionalisation