1
Jan 18, 2017
1
Using Data Science to Design Effective Precision Preventative Behavioral Medicine
Ryan QuanData ScientistOmada Health
Outline Omada Health
• Context! Who are we? What do we do? Data science?
Building a Data Science Culture from the Ground Up• Winning hearts & minds with data
The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works
The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way
• Impactful experimentation: beyond the sub-group analysis
3
Outline Omada Health
• Context! Who are we? What do we do? Data science?
Building a Data Science Culture from the Ground Up• Winning hearts & minds with data
The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works
The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way
• Impactful experimentation: beyond the sub-group analysis
4
OMADA HEALTH 5OMADA HEALTH
THE PROBLEM
6
7
8
THE SOLUTION
9
Lifestyle Change
10
58% Reduction
11
Large Body of Clinical Evidence for DPP
THE PROBLEMTHE SOLUTION
WHAT’S THE DEAL?
12
13
Behavior Change is Hard
EDUCATION
TRACKERS
CALORIE COUNTING
HEALTH COACHING
SOCIAL SUPPORT
One Dimension is Not Enough
19
20
INTRODUCINGThe Omada Program
OMADA HEALTH
PARTICIPANT PEERGROUP
Participants are matched into peer groups for
encouragement and healthy competition with the entire group progressing along a
shared timeline.
EASY-TO-USETECHNOLOGY
A wireless scale, pedometer, mobile apps, and interactive
portal are used to track weight loss, activity, meals,
and program accomplishments.
ONLINE INTERACTIVE LESSONS
Participants learn how to eat healthier, increase activity
levels, and overcome challenges through fun lessons
and games accessed via an online portal or mobile app.
FULL-TIME PROFESSIONALPREVENT COACH
A dedicated coach provides participants with support,
motivation, and personalized recommendations along their
Prevent journey.
22OMADA HEALTH
OVER 19 POINTS OF ENGAGEMENT PER WEEK
Group DiscussionPrevent Coach ConversationSkill ChallengeSuccess Updates
Weigh-InsExercise LogMeal TrackingLesson Completions
23
OMADA HEALTH 24
OMADA
25
26Data Science
Data Science @ Omada
27
• Use analytics, machine learning and experimentation to parameterize what works (and what doesn’t) for behavior change
• Deploy the right intervention, at the right time, for the right patient
Precision Preventative Medicine
The Data (one participant out of 100k+)
28
5
0
Weight Loss (%
)
15
10
5
0
Weight Loss (%
)
15
10
29
Private Messages• Private messages between
HC and participants• Over 1.2M messages• Data:
- Emotionally charged raw text (NLP)
The Data (one participant out of 100k+)
5
0
Weight Loss (%
)
15
10
30
Lessons / Curriculum• Based on original DPP
curriculum• Basic nutrition/activity
information• Interactive w/ games, etc.• Data:
- Completion rates- Reading times- Comprehension
The Data (one participant out of 100k+)
5
0
Weight Loss (%
)
15
10
31
Physical Activity Tracking• Personalized adaptive daily step goals• MyFitnessPal and auto-step tracking
integrations• Over 3.6M activities recorded• Data:
- Steps entered- Minutes of activity- Location (physical proximity triggers)
The Data (one participant out of 100k+)
5
0
Weight Loss (%
)
15
10
32
Group Messaging / Social Support• Group communication • participant <> participant• participant <> health coach• Data:
- Emotionally charged text communication (NLP)
The Data (one participant out of 100k+)
5
0
Weight Loss (%
)
15
10
33
Meal Tracking
• Daily food tracking• 6.2M meals tracked• Active feedback from coach• Data:
- Raw text (food)- Healthiness- Portion size- Time of day
The Data (one participant out of 100k+)
5
0
Weight Loss (%
)
15
10
34
Weighing-in / Progress • Low-power 3G scale• Automatically uploads to
profile on progress page• Weighing in as habit
formation• 6M weigh-ins• Data:
- Weight value- Time of day
The Data (one participant out of 100k+)
The Data
Outline Omada Health
• Context! Who are we? What do we do? Data science?
Building a Data Science Culture from the Ground Up
• Winning hearts & minds with data
The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works
The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way
• Impactful experimentation: beyond the sub-group analysis
36
Building a data science cultureScenario:
You’re the first/only data scientist at your company. You have amazing data with a lot of potential, but your company has no data science
team or experience with data driven product development.
What do you do? Two choices:
Building a data science cultureScenario:
You’re the first/only data scientist at your company. You have amazing data with a lot of potential, but your company has no data science
team or experience with data driven product development.
What do you do? Two choices:Choice 1: Data Police
Building a data science cultureScenario:
You’re the first/only data scientist at your company. You have amazing data with a lot of potential, but your company has no data science
team or experience with data driven product development.
What do you do? Two choices:Choice 2: Build a data
science culture
Building a data science culture
Lots of ways to kick this off:• Brown bag lunch presentations / journal club• Deploy visualization tools• Data education/proselytization• Pro-active analytics as a service• Start a data blog…
Building a data science culture“Plot of the Week (PotW)”
• Internal weekly email data blog • ‘Buzz-feedy’ stories using our data in interesting ways• Exposed company to opportunities with data• Mobilized support and excitement for data science
Building a data science culture“Plot of the Week (PotW)” examples:
1. All in the Family Provides view into raw data
2. Minds over Matter Illustrates potential uses of our data
3. Just Breathe Educates subject matter with data
Building a data science culture“Plot of the Week (PotW)” examples:
1. All in the Family Provides view into raw data
2. Minds over Matter Illustrates potential uses of our data
3. Just Breathe Educates subject matter with data
Our scales become ‘part of the family’ sending weight data of anyone (anything) that steps
on it
PotW: All in the Family
44
Family of two, losing weight together
PotW: All in the Family
45
Another family of two, losing weight together, a few trips to Sandals in there too?
PotW: All in the Family
46
Family of three, growing child
PotW: All in the Family
47
Family of three, neighborhood party
PotW: All in the Family
48
???
Outline Omada Health
• Context! Who are we? What do we do? Data science?
Building a Data Science Culture from the Ground Up• Winning hearts & minds with data
The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works
The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way
• Impactful experimentation: beyond the sub-group analysis
49
The Data Science & Product Relationship
Product
50
Sub-optimal - data science as ‘advisor’
Data-Science
• Exploration/data mining
• Modeling• Proto-typing• Lean-experimentation
• Prioritizes engineering work
• Creates data-models• Implements experiments• Requests analysis
Recommendations/insights
Requests
The Data Science & Product Relationship
Product
51
Sub-optimal - data science as ‘advisor’
Data-Science
Recommendations/insights
Requests
The Data Science & Product Relationship
Data Science
52
ProductOptimal - data science as ‘driver’
• Has own backlog/engineers• Drives data-model design• Experimentation built into
Product• Data scientists can launch
production experiments
Hypothesis
Generation
Experimentation
IterationOptimization
Outline Omada Health
• Context! Who are we? What do we do? Data science?
Building a Data Science Culture from the Ground Up• Winning hearts & minds with data
The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works
The Paths to Personalized (Precision) Prevention
• How to target the right patient, at the right time, in the right way• Impactful experimentation: beyond the sub-group analysis
53
Outline Omada Health
• Context! Who are we? What do we do? Data science?
Building a Data Science Culture from the Ground Up• Winning hearts & minds with data
The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works
The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way
• Impactful experimentation: beyond the sub-group analysis
54
Impactful Experimentation
55
Pros Cons• Causality !=
Correlation• Common language of
scientific discovery
• Low N• Labor intensive, expensive• Slow• Bias in patient populations• Results often not generalizable
For over a century, the Randomized Control Trial (RCT) has been the gold standard for scientific discovery
Impactful Experimentation
56
Largest meta-analysisof DPP ~5.5k participants
>60k Omada participants
vs
At Omada, we have built Randomized Control Experiments into the product to take advantage of the measurement power of experimentation, while the digital nature of our program mitigates the cons (e.g. high N, quick iteration, inexpensive, generalized population, deeper data).
Real-time, Randomized Controlled Experiments (RCEs)
35 in-product experiments & growing quicklyThis enables:
• Rapid exploration of product hypotheses• Structured innovation and evolution of
Prevent• Massive personalization of sub-specifics of
the intervention • Centralized intervention delivery allows
coordinated optimization across all demographics and sectors
• Asking fundamental about what works, and what doesn’t, in behavior change
Personalization through ExperimentationSubgroup Analysis
Answers questions like “Do older people respond differently to Tx than younger?”, or “Do males respond better than females?”
Broad cuts lose resolution to subtle changes
Guided by heuristics (e.g. demographics) and variables available – can lead to biases ITT
Age > 65
Uplift Modeling• Statistical model, based on experimental
data• Predicts who the intervention is likely to
be most impactful for• Don’t need heuristics (other than
choosing inputs) – let data do the personalization for you
59
Subgroup Analysis: Physical Activity Engagement
• Physical activity is a large part of a healthy lifestyle• The second phase of the Omada Program focuses on
increasing patient’s physical activity:1. Providing pedometers to patients to collect physical activity (“steps”)
data2. Educational components and health-coach interaction about exercise3. Setting daily “step goals” for participants
60
Subgroup Analysis: Personalization of Step Goals
Challenge: can we provide personalized step goals to increase physical activity behaviors?
Patients are challenged with daily ‘step goals’ to increase physical activity
61
What does the data tell us:
Subgroup Analysis: Personalization of Step Goals
62
Historical steps recorded, segmented by age/BMI
Assign each participant a personalized step goal based on: similar age/BMI historical mean + 20%
Subgroup Analysis: Personalization of Step Goals
63
Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)
Personalized
Static (7500)
Step
s Tra
cked
/ W
eek
Program Week Program Week
Subgroup Analysis: Personalization of Step Goals
64
Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)
Adaptive goals may be more impactful for younger [18-30] males
PersonalizedStatic (7500)
Step
s Tra
cked
/ W
eek
Program Week Program Week
Subgroup Analysis: Personalization of Step Goals
65
PersonalizedStatic (7500)
Step
s Tra
cked
/ W
eek
Program Week Program Week
No apparent effect in middle-aged adults
Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)Subgroup Analysis: Personalization of Step Goals
Personalization through ExperimentationSubgroup Analysis
Answers questions like “Do older people respond differently to Tx than younger?”, or “Do males respond better than females?”
Broad cuts lose resolution to subtle changes
Guided by heuristics (e.g. demographics) and variables available – can lead to biases ITT
Age > 65
Uplift Modeling• Statistical model, based on experimental
data• Predicts who the intervention is likely to
be most impactful for• Don’t need heuristics (other than
choosing inputs) – let data do the personalization for you
Personalization through ExperimentationSubgroup Analysis
Answers questions like “Do older people respond differently to Tx than younger?”, or “Do males respond better than females?”
Broad cuts lose resolution to subtle changes
Guided by heuristics (e.g. demographics) and variables available – limiting and can lead to biases ITT
Age > 65
Uplift Modeling• Statistical model, based on experimental
data• Predicts who the intervention is likely to
be most impactful for• Don’t need heuristics (other than
choosing inputs) – let data do personalization for you
Experiment: Health coach feedback on participant food tracking
Context:• Building awareness of nutrition through food tracking is a
core part of the Prevent program• Ran experiment to understand the effects of coach feedback
on participant food tracking• Participants in the experiment received feedback about their
food choices from their coaches, those in the control arm did not.
Experiment: Providing impactful food feedback
Experimental design
RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weigh gain – please provide feedback
Experiment: Providing impactful food feedback
1. Candidate Selection• Participants who track a meal & predicted to gain weight
2. Randomization• 50% are eligible to receive feedback
3. Prescription Generation• Coaches are notified of tracking behavior, with suggested
feedback4. Intervention Delivery
• Coach reaches out to participant with feedback on the flagged meal
5. Monitor outcomes
RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weigh gain – please provide feedback
Experiment: Providing impactful food feedback
1. Candidate Selection• Participants who track a meal & predicted to gain weight
2. Randomization• 50% are eligible to receive feedback
3. Prescription Generation• Coaches are notified of tracking behavior, with suggested
feedback4. Intervention Delivery
• Coach reaches out to participant with feedback on the flagged meal
5. Monitor outcomes
Experimental design
RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weight gain – please provide feedback
Experiment: Providing impactful food feedback
1. Candidate Selection• Participants who track a meal & predicted to gain weight
2. Randomization• 50% are eligible to receive feedback (other 50% in ‘control’)
3. Prescription Generation• Coaches are notified of tracking behavior, with suggested
feedback4. Intervention Delivery
• Coach reaches out to participant with feedback on the flagged meal
5. Monitor outcomes
Experimental design
1. Candidate Selection• Participants who track a meal & predicted to gain weight
2. Randomization• 50% are eligible to receive feedback
3. Prescription Generation• Coaches are notified of tracking behavior, with suggested
feedback4. Intervention Delivery
• Coach reaches out to participant with feedback on the flagged meal
5. Monitor outcomes
Experiment: Providing impactful food feedback
RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weight gain – please provide feedback
Experimental design
RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weight gain – please provide feedback
Experiment: Providing impactful food feedback
1. Candidate Selection• Participants who track a meal & predicted to gain weight
2. Randomization• 50% are eligible to receive feedback
3. Prescription Generation• Coaches are notified of tracking behavior, with suggested
feedback4. Intervention Delivery
• Coach reaches out to participant with feedback on the flagged meal
5. Monitor outcomes
Experimental design
5. Monitor outcomes 10-15% increase in days with meals tracked
Experiment: Providing impactful food feedback
8-12% increase in meals with healthiness tracked5. Monitor outcomes
Experiment: Providing impactful food feedback
+8% in relative week-16 weight loss
5. Monitor outcomes
Experiment: Providing impactful food feedback
Using experimental data for personalizationUplift modeling, built on experimental data, can tell you probability of incremental response for an individual, given a hypothetical intervention
When delivering interventions:
• The Sure Things: response regardless of intervention• The Lost Causes: no response regardless of
intervention• The Sleeping Dogs: less likely to respond with
intervention• The Persuadables: only respond to because of
intervention
Uplift modeling targets The Persuadables and leaves the Sleeping Dogs alone.
Uplift modeling Example: Food Feedback
• Random forest uplift model trained on likelihood of participant continuing to track meals after receiving food feedback
• Model accounts for participant demographics, program behavior and response to previous feedback
• Preliminary results: targeting ~80% of eligible population with uplift model increases response ~2% (from 5% to 7%)
Outline
79
Omada Health• Context! Who are we? What do we do? Data science?
Building a Data Science Culture from the Ground Up• Winning hearts & minds with data
The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works
The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way
• Impactful experimentation: beyond the sub-group analysis
WHAT’S NEXT?
80
THANKS!@omadahealth
@sciencethedata@ryancquan
82