Triola Medibiquitous 2015 - MedBiq · Using&Analytics&to&Drive&Change&in&Health& Professions&Education Marc%M.%Triola,%MD NYU%School%of%Medicine @marctriola

Using&Analytics&to&Drive&Change&in&Health&Professions&Education

Marc%M.%Triola,%MD

NYU%School%of%Medicine

@marctriola

Why%Now?

• Analytics%Driving%Changes%in%Care:

– Changes%in%health%care%delivery%

– Explosion%of%scientific%knowledge%

– Demand%for%a%data%literate%physician%workforce

• Analytics%Driving%Changes%in%Education:

– Personalized%education

– CompetencyKgated%pathways

– Increasing%amounts/capacity%for%data%collection

– New%learning%analytics%methods%to%derive%actionable%

knowledge

"I%know%more%about%my%11KyearKold%son's%sixthK

grade%basketball%team%than%the%average%college%

faculty%member%knows%about%their%incoming%

class,%in%terms%of%key%variables%that%are%going%to%

make%them%successful%or%not%successful”

»Mark%Milliron,%Bill%&%Melinda%Gates%Foundation

The&Education&Data&Continuum

Curriculum

USMLE%

Exams%and%Evaluations

MCAT%/%

Admissions

Competency%Portfolios

Clinical% patient%outcomes

CME%Performance

Medical%School Training Clinical%Practice

NRMP ACGME

Boards%and%licensure

Warehouse%of%longitudinally% integrated%learner%data

Institution%A Institution%B Institution%C

The&Education&Data&Continuum

Med%

SchoolClinical% Practice

Residency

College

Density%of%Education%Data

Density%of%Patient%Data%and%Outcomes

Inspired%by%Dr.%Tom%Aretz,%Harvard%Medical%School

How%to%Drive%Change?

• Our%ability%to%collect%data%has%vastly%outpaced%

our%ability%do%gain%insight%and%action%from%it.

• Most%data%and%systems%are%not%integrated%nor%

interoperable.

• Local%resources%are%needed,%as%is%expertise%

not%normally%a%part%of%the%education%mission.

• An%ongoing%strong%partnership%between%

Informatics/IT,%faculty%and%administration%is%

key%– it’s%a%culture%change.

Education&Data&Warehouse

Education%Data%

Warehouse

Education%Data%

Warehouse

LMS

Curricular%

Content

AdmissionsExams

ePortfolioPatient%

Log

Learning%

ModulesEvaluations

SIS

Reporting&and&AnalyticsData&Marts&for&

Education&Research

Simulation

Clinical%Data%

Warehouse

Clinical%Data%

Warehouse

NYU%

Practice%

Network

Epic%EMR

Division%of%Educational%

Informatics

Institute%Structure

Curriculum%Mapping

Multimedia%and%Medical%

Visualization

Educational%Technology

Division%of%Education%Quality%

and%Analytics

Instructional%Design

Education%Analytics%for%

UMEKGME%Continuum

Benchmarking%and%

Outcomes

UME/GME%Assessment%

Program

Digital%Press

Faculty%iBook% Publishing

Commercialization% and%

Licensing

UMEKGME%

Continuum

Education%Data%

Warehouse

Learning%analytics%refers%to%the%interpretation%of%a%

wide%range%of%data%produced%by%and%gathered%on%

behalf%of%students%in%order%to%assess%academic%

progress,%predict%future%performance,%and%spot%

potential%issues

KU.S.%Department%of%Education%2012

Analytics

• Descriptive:&&How%did%I%do%on%the%assessment%items?%

• Diagnostic:&&Did%I%do%better%on%application%or%recall%questions?

• Predictive:&Based%on%my%prior%data,%what%is%my%estimated%USMLE%score?%

• Prescriptive:&My%learning%data%indicates%I%should%practice%genomics%and%pharmacology%using%

application%questions.

Analytics

• Early&warning&system

• Early&reward&system

• Anecdote&to&epidemiology

Individual%Dashboards%for%Learners

Individual%Dashboards%for%Faculty

Dashboards%for%Edu%Leadership%and%Mentors

Dashboards%for%Learning%Modules

Clinical%Data%and%the%Education%

Continuum

• Use%of%EMR%data%in%the%medical%school%curriculum%has%

tremendous%potential

• Can%our%clinical%data%drive%our%curricular%content?

• Which%learner%and%patient%outcomes%that%can%be%used%

to%monitor%the%impact%of%curriculum%on%clinical%

effectiveness?

Predictive%Analytics

• 7,859%students%in%36%classes%entering%Jefferson%Medical%College%between%1970%and%2005

• Could%MCAT%predict:%performance%in%medical%school,%USMLE%scores,%and%ratings%of%clinical%competence%in%residency

MCAT%as%Predictor

• MCAT%scores%moderately%correlated%with%

medical%school%performance%and%with%scores%

on%USMLE%(up%to%4%years%later).

• Predictive%validity%coefficients%for%women%were%

consistently%higher%than%those%for%men.

• MCAT%performance%did%not%predict%clinical%

competence%or%professionalism.

Predicting%USMLE

• Base%Model:%Gender,%MCAT,%College%GPA

• Exams:%Preclinical%exam%scores

• Knowledge%domains:%%Performance%on%topics,%

weighted%by%correlations%to%prior%student%

USMLE

Base%Model Exams Knowledge%Domains

Domains%+%Base

Within&10&points 55% 58% 52% 65%

Within&15&points 76% 86% 82% 86%

Within&20&points 85% 95% 95% 97%

Pusic et.%al.,%unpublished% data

Driving%Change

• Educational%technologies%and%analytics%should:

– drive%the%move%towards%individualized% learning%and%%

personalized%progression

– accelerate%the%transformation% towards%health%

education% that%is%competencyKdriven,%affordable,%and%

accessible%to%each%learner

• Faculty%need%new%skills%and%expertise%in%the%

effective%use%of%educational%technologies%and%

learning%analytics%to%enhance%the%teachingK

learning%process.

Driving%Change

• Shared%goals,%shared%pressures,%robust%data,%

and%analytics%can%bridge%the%gap%between%our%

educational%and%clinical%missions.

• Insights%from%the%overlap%of%education%and%

clinical%data%can%enhance%the%value%of%

teaching%and%learning%to%the%healthcare%

delivery%system

Driving%Change

• Educational%technologies%should%be%

interoperable%with%respect%to%both%content%and%

outcomes%data.

• Learners%and%faculty%should%own%their%data.

• Analytics%and%data%integration%should%be%easy.

• Impact%of%educational%innovations%will%be%

maximized%by%sharing%across%platforms,%

teams,%schools,%institutions,%and%the%public.

• Organizations%like%MedBiquitous are%key!