Supercharge Your Improvement Efforts with Predictive Analytics · Supercharge Your Improvement Efforts with Predictive Analytics Chris DeRienzo, MD, MPP, FAAP Chief Quality Officer

Supercharge Your Improvement Efforts with Predictive Analytics

Chris DeRienzo, MD, MPP, FAAPChief Quality Officer &

Neonatologist, Mission Health

Andrew O. Johnson, PhD

Manager, Data Science Clinical & Business

Analytics, Mission Health

Session #29

Learning Objectives• Explain the value proposition for building a robust internal data science team into an

integrated continuous improvement analytics strategy.

• Identify the required elements of a “Data Science Starter Kit” that can upgrade your organization’s analytics capabilities from reporting to predicting.

• Illustrate the role data scientists can play within a health system through the development of an all-cause, 30-day inpatient readmission model.

We are dead in the water without a culture of continuous improvement, grounded in analytics, that permeates everything we do, and all that we are.

The U.S. health system

ranks last among 11

industrialized countries on

measures of access, equity, quality, efficiency,

and healthy lives.

Wasteful spending in the health

system has been calculated at up to $1.2 trillion.

70% of hospital strategic initiatives

fail.

Only 32% of healthcare IT projects

meet their objectives,

while 24% totally fail, and 44% have difficulties

in meeting their goals.

About Mission Health

• Mission Health is westernNorth Carolina’s onlynot-for-profit, independent community healthcare system.

• Mission’s BIG(GER) Aim is to get every person to their desired outcome, first without harm, also without waste, and always with an exceptional experience for each person, family, and team member.

• Employing over 13,000 dedicated professionals, the system is comprised of seven hospitals including tertiary, critical access, and inpatient rehabilitation, 750 employed/aligned providers, and one of the largest Medicare Shared Savings ACOs in the nation.

Western North Carolina 18-County AreaPopulation (2016): 882,581Percent over 65: 22%

The Need at Mission HealthMajor process improvement was facilitated through the centralized performance improvement team—a highly skilled team of process engineering professionals.• Created internal dependency on the centralized team for complex projects.• Access to data only through IT and Informatics.• Difficult to provide real-time operational data or management roll-up data.

The Need:1. Provide health systemwide data in a way that allows broad access for analysis as well

as operational uses ranging from daily unit management to board-level discussion.

2. Create a culture of continuous improvement by outfitting key clinical and operational roles with skills, tools, and a sense of ownership for improving their own processes.

3. Create the capability to leverage both operational and clinical predictive analytics.

Where does your organization spend most of its time right now?

1) A2) B3) C4) D5) E6) F7) G8) Unsure or

not applicable

Poll Question #1

Standard Reports

Ad-hoc Reports

Query Drill-down

Alerts & Triggers

Statistical Analysis

Forecasting

Predictive What-if Analysis

Optimization

What happened?

How many? How often? Where?

What action is needed?

What exactly is the problem?

What is the best course of action?

How can I influence the future?

What are likely future outcomes?

Why did this occur?

A

B

C

D

E

F

G

Analytics

The Turning Point

Process transformation after“low hanging fruit.”• Early wins always lead

to more difficult and complex follow-up projects.

• Requires higher level of leadership, particularly from physicians.

Clinical program development.• Simply not possible

without strong physician leadership.

• Requires sensitive project facilitation.

Better together.• Clinical and

operational leaders and data scientists can accomplish more together than they can separately.

Results Successfully kick-started the journey “up the curve” in clinical analytics.

• Significant reductions in sepsis and stroke mortality, length of stay for bowel surgery and renal patients, and population screening for breast and colorectal cancer through workflows built into 50+ care process models.

Moved further up the curve toward predictive operational analytics.

• Large scale unit and flow simulations for building projects, regional transport optimization, and census prediction.

Launched integrated team to drive to meaningful clinical predictive analytics (e.g., readmissions predictor).

How We Did It

Began with the BIG(GER) Aim and asked “what do we need to best deliver on this promise in a population health world?”

Created a vision and support for data-driven continuous improvement grounded in analytics across Mission Health.

Built our analytics team with in-house data science expertise.

Cultivated a strong organizational motivation, capability, and process to move to predictive analytics operationally and clinically (initially focused on readmissions).

Tempering Your Organizational Enthusiasm

Moving From Reporting to Prediction

Things you must have to be successful: • Senior leaders with vision, budgetary impact,

and desire for change.• A trusted data infrastructure.• Well-defined problems with predictive

solutions.• Skilled and empowered data scientists.• The rest of the analytics team.• Processes to develop and sustain data

science projects.

Value Proposition for an In-house Data Science Team

Decide which analytic avenues to investigate.

1

2

3

4

5

6

Multiply returns on prior investments in data assets unique to your organization.

Rely on trusted in-house statistical advisors aligned with your goals.

Enjoy having allies when evaluating analytics

vendors’ products.

Tune prediction models to the “home-field advantage.”

Benefit from intense scrutiny over the validity

of data in your enterprise data warehouse (EDW).

Forecast ahead 18 months…Where do you want your organization spending most of its time?

1) A2) B3) C4) D5) E6) F7) G8) Unsure or

not applicable

Poll Question #2

Standard Reports

Ad-hoc Reports

Query Drill-down

Alerts & Triggers

Statistical Analysis

Forecasting

Predictive What-if Analysis

Optimization

What happened?

How many? How often? Where?

What action is needed?

What exactly is the problem?

What is the best course of action?

How can I influence the future?

What are likely future outcomes?

Why did this occur?

A

B

C

D

E

F

G

Analytics

Required: A Trusted Data Infrastructure

“76% of data scientists view data preparation as the least enjoyable part of their work.”

If you don’t have a reliable data infrastructure or EDW,you are wasting your time trying to operationalize data science.

Source: Press, G. (2016). Cleaning big data: Most time-consuming, least enjoyable data science task, survey says. Forbes Magazine. Retrieved from https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says

Required: Well-Defined Problems with Predictive Solutions

There are still so many beautiful things to be said in C-Major.

~ Sergey Prokofiev

There is still much good music that can be written in C-Major.

~ Arnold Schoenberg

““

Start with the essential quantities that the organization cares about.

Use existing data mart or reporting tables when available.

Trust your data scientists to know when to abandon

(or refuse to intake) a project.

Proceed incrementally.

Try to beat the performance of an existing model, or show

up sooner than it can.

Scopes creep when people get excited!

Twelve Key Questions to Intake a Predictive Project

12

3

4

5

67

8

9

10

11

12

1. What is the business problem?

2. What needs to be predicted?

3. How much improvement is needed?

4. Who is the targeted population?

5. How early in the process do you need it?

6. How often do you need updated predictions?

7. What time of day you need the predictions?

8. Have there been other projects that focused on this?

9. What will you do with the predictions?

10. Are there predictions being made now?

11. Are there any models to benchmark against?

12. Are there any minimum required levels of performance?

Required: Skilled Data Scientists…Where Is Everyone?

“Of the approximately 6,000 data scientists in the U.S., only 180 are estimated to work in the hospital and health care field. Given that there are nearly 6,000 hospitals and just 400 academic medical centers in the U.S.,

that’s stretching the available labor force a bit thin.”

Source: Huesch, M. D., & Mosher, T. J. (2017). Using it or losing it? The case for data scientists inside health care. NEJM Catalyst. Retrieved from http://catalyst.nejm.org/case-data-scientists-inside-health-care/

No Unicorns Need Apply!

“Specialization is for insects.”– Robert A. Heinlein

“In terms of species diversity, total biomass, range of habitat, adaptations to adverse conditions, and intra-species pro-social behavior, insects are the most successful animals on Earth.”

– Every biology textbook ever

The Ability to Learn and Adapt Is a Key Requirement

On the Proper Care of Data Scientists

Top 5 rules for my data science team:

1Always ask questions. 2Try to be

comfortable with failure and uncertainty. 3Do not use

methods you do not understand. 4Before going

down a technical rabbit hole, ask

“What’s the value to Mission?” 5Always be nice

to the data architects.

Require them to continue learning and to set aside work time to do this.

A Tale of Two Job Offers

… The other issue is that I was uneasy about working with the physicians I’d met, as I felt their project vision and expectations exceeded their appreciation of the organizational, technical, and personnel requirements involved. When I asked them what they wanted from the person in the data scientist position, I heard: a statistical analyst, a project manager, a health economist, a strategy manager, and small amount of EMR build consulting. I can’t do all of that myself, …

Andy,…to be honest, we do expect all of those things from our data scientists. …

A Tale of Two Job Offers

… Can you give me any info on how I would be evaluated, and what sort of expectations you have for this position?

Hi Andy,Good to hear from you. Here are answers to your questions:

… The expectation would simply be to add value to our highly collaborative analytics program.

Required: “The Rest of the Team”

• Identify data sources.• Collect new data if

needed.• Merge, join, and

augment as appropriate.

• Understand structure of the data set - what does the data mean?

• Identify errors, outliers, case-deficient groups, etc.

• Understand statistical properties of the data set.

• Methodology assessment.

• Train model.• Test model.

• Literature review.• Identify competing

models.

• Automate.• Integrate into

application.• Validate.• Train.

DescriptiveAnalytics

PredictiveAnalytics

DataDiscoveryResearch ImplementationProblem

Definition

• Who is the customer?• What problem are we

trying to solve?• Is this the right

problem?• What is the value

added?

Knowledge Engineers

DataArchitects

BIDevelopers

QA andTraining

Integration with Analytics TeamReadmission Predictor v1Data Scientist:• Prototype input data frames.• Create prediction model.• Validate model performance.• Create model task/timing/output structure.

Knowledge Engineer:• Evaluate initial project proposal.• Set scope, deliverables, and feasibility.• Steer stakeholder validation of existing data

elements (LACE/Readmission Explorer).• Monitor progress to milestones.

Data Architect:• Productionize subject area mart (SAM) models

for input data.• Acquire additional fields from other sources.• Validate DS-created input data.• Build or modify application using model output.

BI Developer:• Develop best practices for in-app visualization.• Build or modify application using model output.

Training and QA Analyst: • QA check involved apps.• Maintain release schedule.• Communicate new releases.• Train end users.

Required: Processes to Develop Data Science Projects

Readmission Prediction v1 at MissionExisting tool: LACE

Problem: LACE is good, but not great, and we have unique patient populations and unusual data assets for building our own model.

Goal: To construct and automate the calculation of a risk model for 30-day, all-cause inpatient readmission.

Requirements:• Performance must beat LACE in our patient population.• Be available before 8 a.m. day after discharge.• First version must use fields currently existing in EDW.

A Relatively Simple ImplementationEDW

source mart

External source data

Internalsource data

EDW SAM

Predictive model input SAM

Predictive model output

SAM

R on DS server

R model

Staging server

R output to forecast model

database

Mission Analytics Portal

Visualizationapplication

Model Performance Comparison

Appreciate the importance of a skilled data science team.

Key Takeaways

Create the capability to

leverage both operational and

clinical predictive analytics

Create a culture of continuous improvement by outfitting key clinical and operational roles

with skills, tools, and a sense of ownership for

improving their own processes.

Provide systemwide data in a way that allows

broad access for analysis as well as

operational uses ranging from daily unit

management to board-level discussion.

Thank You