Session 25 Interactive Forum: Make Risk Your Friend – Next ... · Make Risk Your Friend – Next Generation Claim Prediction Nick Ortner, FSA, MAAA. Consulting Actuary Milliman

Session 25IF, Make Risk Your Friend – Next Generation Claim Prediction

Moderator/Presenter:

Nickolas J. Ortner, FSA, MAAA

Presenters: Elena V. Black, FSA, EA, MAAA, FCA

Yi-Ling Lin, FSA, MAAA, FCA

SOA Antitrust Disclaimer SOA Presentation Disclaimer

Make Risk Your Friend –

Next GenerationClaim Prediction Nick Ortner, FSA, MAAA

Consulting Actuary Milliman – Brookfield, WI

[email protected] (262) 796-3403

Overview / Today’s Agenda

History / simple worldEvolution / revolutionCurrent modelsComplex / emerging variablesSamplesPrediction and executionRegulation / challengesTransition

2

Our Simple (Past) World

Fee for service

Funding limits?

Past = future

Metrics Participation Attendance Clinical

3

“Business Case” Revolution

Insurers paying differently

Slowing funding spigot

Employers / missions

Impact on systemsNetworksProviders (facilities, physicians)

4

Transformative Models

Current models

Concept mainstreaming

What don’t we know

5

Harnessing Complex Variables

Traditional variablesDemographic (age, gender, area)Plan design“Clinical” (diagnosis, Rx)

Emerging variablesVariable interactions/combinationsSocial determinantsCommunity/connections

Evolving techniques

6

Sample Projects

Insurers Wearables Periodic check-ins and changes

Employers Long-term sustainability Proactive, with requirements

7

Sample Projects (continued)

Emergent care risksMedicare: community/interaction

Opioid addiction riskCommonalities = heightened risk

Value of changing measures

8

Prediction Execution

Gamification = participation

Tailored messaging “Meet targets where they are”

9

Regulation and Challenges

Transparency

Privacy

Other challenges

10

Nick Ortner, FSA, MAAA

Consulting Actuary – Milliman – Brookfield, WI

[email protected] (262) 796-3403

2018 SOA Health MeetingELENA BLACK, PHD, CFA, FSA, EA, MAAA, FCA

THE TERRY GROUPSession 25 – Make Risk Your Friend-Next Generation Claim PredictionJune 25, 2018

SOCIETY OF ACTUARIESAntitrust Compliance Guidelines

Active participation in the Society of Actuaries is an important aspect of membership. While the positive contributions of professional societies and associations are well-recognized and encouraged, association activities are vulnerable to close antitrust scrutiny. By their very nature, associations bring together industry competitors and other market participants.

The United States antitrust laws aim to protect consumers by preserving the free economy and prohibiting anti-competitive business practices; they promote competition. There are both state and federal antitrust laws, although state antitrust laws closely follow federal law. The Sherman Act, is the primary U.S. antitrust law pertaining to association activities. The Sherman Act prohibits every contract, combination or conspiracy that places an unreasonable restraint on trade. There are, however, some activities that are illegal under all circumstances, such as price fixing, market allocation and collusive bidding.

There is no safe harbor under the antitrust law for professional association activities. Therefore, association meeting participants should refrain from discussing any activity that could potentially be construed as having an anti-competitive effect. Discussions relating to product or service pricing, market allocations, membership restrictions, product standardization or other conditions on trade could arguably be perceived as a restraint on trade and may expose the SOA and its members to antitrust enforcement procedures.

While participating in all SOA in person meetings, webinars, teleconferences or side discussions, you should avoid discussing competitively sensitive information with competitors and follow these guidelines:

• Do not discuss prices for services or products or anything else that might affect prices• Do not discuss what you or other entities plan to do in a particular geographic or product markets or with particular customers.• Do not speak on behalf of the SOA or any of its committees unless specifically authorized to do so.

• Do leave a meeting where any anticompetitive pricing or market allocation discussion occurs.• Do alert SOA staff and/or legal counsel to any concerning discussions• Do consult with legal counsel before raising any matter or making a statement that may involve competitively sensitive information.

Adherence to these guidelines involves not only avoidance of antitrust violations, but avoidance of behavior which might be so construed. These guidelines only provide an overview of prohibited activities. SOA legal counsel reviews meeting agenda and materials as deemed appropriate and any discussion that departs from the formal agenda should be scrutinized carefully. Antitrust compliance is everyone’s responsibility; however, please seek legal counsel if you have any questions or concerns.

2

Presentation Disclaimer

Presentations are intended for educational purposes only and do not replace independent professional judgment. Statements of fact and opinions expressed are those of the participants individually and, unless expressly stated to the contrary, are not the opinion or position of the Society of Actuaries, its cosponsors or its committees. The Society of Actuaries does not endorse or approve, and assumes no responsibility for, the content, accuracy or completeness of the information presented. Attendees should note that the sessions are audio-recorded and may be published in various media, including print, audio and video formats without further notice.

3

Risk Scoring in Health, SOA Studies, and Professionalism Issues

Risk Scoring Modeling in Healthcare

5

Application of data analytics in healthcare for risk scoring

Traditional to emerging– Methodologies: from linear regression models to machine learning

algorithms

– Data: traditional claims and enrollment data (Rx, Dx, demographic, prior year costs, etc.) to new and emerging data, e.g. socio-economic factors

– Types of risk scoring models: concurrent vs. prospective fitting well into data analytics spectrum

Health risk scores are used for variety

of purposes

Many sources of uncertainty

Wealth of Information on Risk Scoring Models

6

A Comparative Analysis of Claims-Based Methods of Health Risk Assessment for Commercial Populations (2002)

A Comparative Analysis of Claims-Based Tools for Health Risk Assessment (2007)

Uncertainty in Risk Adjustment (2012)

Nontraditional Variables in Healthcare Risk Adjustment (2013)

Accuracy of Claims-Based Risk Scoring Models (2016)

Risk Scoring in Health Insurance: A Primer (2016)

SOA studies related to risk scoring models in healthcare

Potential Issues and Professionalism

7

Professional guidance (list not exhaustive)

The Code of Professional Conduct and Actuarial Standards of Practice (ASOPs)

ASOP 12: Risk Classification

ASOP 23: Data Quality

ASOP 25: Credibility Procedures

ASOP 38: Using Models Outside the Actuary’s Area of Expertise

ASOP 41: Actuarial Communications

ASOP 45: The Use of Health Status Based Risk Adjustment Methodologies

Assumptions Setting ASOPs (27, 35)

Risk ASOP (51) and Modeling ASOP

Exciting things often come with challenges and potential pitfalls

Challenges/issues• Messy, often high-dimensional with missing

values, data and data quality issues

• Potential bias in data

• Use of proxies

• Non-discrimination, security and confidentiality

• Transparency vs. “black box”

• Spurious correlations: correlation vs. causality

• Interpretability and replicability

• Overfitting and overreliance

• Business purpose appropriateness and applicability

… and… many more

Data Analytics Spectrum and Risk Scoring Modeling in Health

Spectrum of Data Analytics

9

Descriptive analytics

Diagnostic analytics

Predictive analytics

Prescriptive analytics

What happened?

Why did it happen?

What will happen?

What should I do?

Analytical sophistication

Valu

e to

war

ds b

usin

ess

solu

tions

Adapted from Gartner’s Data Analytics Maturity Model

Risk Scoring in Healthcare in Data Analytics Spectrum

10

What happened?

Why did it happen?

What will happen?

What should I do?

Healthcare costs dashboardsDescriptive

statisticsData clustering

Healthcare cost trends

Cost driving features

Concurrent risk scoring modeling

Prospective risk scoring modelingRecalibration off-

the-shelf risk scores

Custom risk scoring models

Risk stratification and care

managementChoice modeling,

simulation and optimization

Adapted from Gartner’s Data Analytics Maturity Model

Calibration of Risk Scoring Models

11

Calibration to adjust existing models to specific population

Methodologies– Full calibration (transparent models, e.g. HHS-HCC)

– Residual calibration to same/similar features (e.g. linear regression on demographic and diagnosis variables)

– Ridge regression residual calibration

– Custom risk scoring models or risk stratification models

– Residual custom off-the-shelf model recalibration o Additional variables/features

o Different modeling techniques

Custom risk scoring methodologies

Full Calibration Example

12

Calibration of HHS-HCC model to specific population

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

7.00%

0.000

5.000

10.000

15.000

20.000

25.000

30.000

35.000

40.000

45.000

001

004

009

012

019

023

029

035

038

045

048

056

062

067

070

074

082

089

096

103

108

111

114

118

121

126

129

132

145

150

154

159

162

184

203

207

217

251

incidence Current Weights 2018

Diabetes ~6.5%

Asthma COPD ~4.5%

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

Demographic Profile: off-the-shelf HCC versus custom calibration

Custom weights-M Custom weights-F

Off-the-shelf-M Off-the-shelf-F

Major depressive and bipolar

disorders ~3%

Linear regression model based on age/gender and condition bins

Case study for illustration purposes only

Case study for illustration purposes only

Custom weights HCC weights

Custom Off-the-shelf Model Recalibration

13

Putting model calibration and ensemble concepts together

SOA 2016 paper briefly explored ensemble idea as analytics question

Ensemble learning

• Improves predictive analytics results by combining several models (“weak learners”)

o Bagging (variance decrease)

o Boosting (bias decrease)

o Stacking (improves predictions)

Custom Recalibration

• Adjusts to specifics of a given population

• Can use off-the-shelf risk scores as inputs (stacking)

• Potentially reflects additional variables

• Can use different methodologies from original off-the-shelf model

• New spin on residual calibration

Emerging Programming Paradigm and Model Evaluation

New Programming Paradigm: Machine Learning• Humans Input data & answers• And how to “learn”… and what does it mean to be

wrong…• Example: clustering algorithm or neural networks or

decision tree/Random Forest

Traditional Modeling versus Machine Learning

15

Could computer automatically learn the rules by looking at data?

Traditional

Classical programming model• Humans input data and set of rules/function

on how to arrive at answers • Also how close they want data to fit to the

“model”…• Example: linear regression or generalized

linear regression

Data

Rules

AnswersHumans input:

Machine Learning

Data

Rules

Answers

Humans input:

New DataPotential feedback loop:

• Examples of ensemble models

• Based on decision trees

o Random forest: multitude of trees trained (random subsets of data) and results averaged

o Gradient boosting: trees are trained in succession on residuals of target versus sum of previously trained trees

Decision Trees and Ensemble Methods

16

Ensemble approaches often result in robust models

Rules based decision tree • Perfect for classification problems, but can be used for

regression• Transparent and easy to interpret• Training is done by optimizing given “loss” function• …. But a “weak” learner

Combine many trees

Case study for illustration purposes only

Risk Scoring Model Evaluation

17

Model evaluation is an important part of any modeling project

• Relevance and importance of criteria

• Appropriate and consistent with purpose

• On “unseen” or “test” sample of data

• Examples of criteria/metrics

Standard statistical measures (R squared, RMSE, MAE, etc.)

Predictive Ratios: grouped A/E type measures (demographic groups, diagnostic groups, cost groups, random groups, etc.)

Tolerance curves

ROC curves for Cost Groups

Correlation and comparison with naïve and standard models

Cautionary tale!Famous Anscombe’s quartet: all four datasets have the same statistical properties, including R squared=0.67, means and variance of x and y, correlation and linear regression model: y=3+0.5x

Case Study: Custom Risk Scoring Modeling

52% Male

42% Male

4%

18%

12%

6% 5% 7% 7% 8% 9% 10% 9%5%

0%

10%

20%

30%

40%

50%

60%

baby child 18-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65+

Demographic profile of the population

Case Study: Descriptive Analytics

19

Dashboards, distributions, descriptive statistics

In this case study babies under age of 2 were excluded, and population shown were enrolled at least for one month in both years

Shaded area illustrates male percentile

This is traditional analysis to inform what actually happened and the first step in

any modeling project

160237

374

501605 593

710 657

704780 805

964

395 348 331

405456

348422

482

744884

1,221

1,541

0

200

400

600

800

1,000

1,200

1,400

1,600

1,800

baby child 18-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65+

Average PMPM (Medical and Rx) by year and gender

F-2016 PMPM F-2017 PMPM M-2016 PMPM M-2017 PMPM

Case study for illustration purposes only

Case study for illustration purposes only

Case Study: Diagnostic Analytics

20

Investigating and identifying trends & relationship

Relationship between potential predictors (independent variable), relationships between predictors and target, potential transformed variables relationships

Claim costs are lognormally distributed: fitting normal distribution to log of PMPM costs for current and prior years

visually there is a linear relationship butcorrelation is only 0.54

Diagnostics focused on uncovering

patterns, relationships, trends, and potentially

engineering predictive features

0 2 4 6 8 100

50

100

150

200

250

300

350

400

450

0 2 4 6 8 10 12

2016 log PMPM

0

2

4

6

8

10

12

2017

log

PM

PM

Case study for illustration purposes onlyCase study for illustration purposes only

Case Study: Start Simple!

21

Simple approach: variation on “stacking” concept

Off-the-shelf HCC (test data): 𝑅𝑅2 is 0.24, and correlation of predicted values versus target is 0.65Linear regression on three variables (test data):𝑅𝑅2 is 0.42, and correlation of predicted values versus target is 0.65

Prefect prediction at 100%

109%

131%117%

97% 94%

122%

98% 96% 100% 102%

0

0.2

0.4

0.6

0.8

1

1.2

1.4

child 18-34 35-49 50-64 65+

Linear regression on three variablePredictive Ratios on Test Data by Age Group

Male Female Total

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

baby child 18-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65+

Predictive Ratios by Age Group (Test Data)

Off-the-shelf HCC Linear regression

Comparison of age-group predictive ratios: Off-the-shelf versus linear regression with HCC diagnosis severity as input

Case study for illustration purposes only

Estimate SE t Stat p ValueIntercept 170.5993 26.20668 6.509764 7.87E-11gender -17.7672 22.63822 -0.78483 0.432571age 4.697473 0.58525 8.026445 1.11E-15Diagnosis HCC severity 292.9169 3.590215 81.58757 0

Case study for illustration purposes only

Case Study: Complexity versus Interpretability

22

Gradient Boosted Trees or Random Forest: More Accurate-Hard to Interpret

Two models (linear and “bagged trees”) fit to the same variables, but the scatter shown against just one predictor (𝑅𝑅2 = 0.2 for linear and 0.3 for random forest)

Feature importance allows for easier

interpretation but also predictive power analysis

Many machine learning models are hard to explain/interpretRandom forest model

Linear regression model

Case study for illustration purposes only

Case Study: Predictive Analytics

23

Calibrating residual using bagged trees and additional features

On Test Data:𝑅𝑅2 is 0.24, correlation 0.65,

MAE=67% for off-the-shelf HCC

𝑅𝑅2 is 0.48, correlation 0.70, MAE = 73% for residual

custom-recalibrated HCC

Case study for illustration purposes only

0

1000

2000

3000

4000

5000

6000

0 1000 2000 3000 4000 5000 6000

Residual recalibrated HCC-based Risk Scores

0

1000

2000

3000

4000

5000

6000

0 1000 2000 3000 4000 5000 6000

Off-the-shelf HCC Risk Scores

0.62

0.74

0.88

0.991.03

1.10

1.01

1.08

1.161.12

1.17

0.93

1.10

1.011.04

0.991.03

1.05

0.971.00 0.99 0.98

1.02 1.03

0.6

0.7

0.8

0.9

1

1.1

1.2

baby child 18-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65+

Predictive Ratios by Age Group

Perfect fit HCC Recalibrated HCC-based

Actual cost

Actual cost

Pred

icte

d c

ost

Pred

icte

d c

ost

Case Study: Decision-informing Analytics

24

Various uses of risk scoring in health care: population health and care management

From low to high

Identifying best cases for care management

Acute illness, trauma, accidents

Chronic deceases, high cost and risk

HealthyRising cost?

Risk

Cost

Assess characteristics of high risk/low

cost group: potential for

care management

Prevention and wellness

programs

Low cost

High cost

Case study for illustration purposes only

Questions? Thoughts… Comments?