Going Beyond AB Testing
How to Connect Customers to
Experiences
1. Decision Rules as a Framework
2. Picking Rules -> Optimization
3. Multi-Armed Bandit
twitter: @mgershoff
Blog: www.conductrics.com/blog
Overview
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
Decision Rules
Decision Rule
IF [THIS]
Decision Rule
IF [THIS]
THEN [THAT]
A Logic that links
Observations to Actions
Why Decision Rules?
Decision Rule: General Form
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Observed Data]
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Observed Data]
Then [Causal Action]
Decision Rule: General Form
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Anyone]
Decision Rule: Simple Test
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Anyone]
Then [A or B]
Decision Rule: Simple Test
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Unbounce User]
Decision Rule: Targeted Test
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Unbounce User]
Then [ or ]
Decision Rule: Targeted Test
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
Two Types of Data
• Day Part
• Weekend/Weekday
• Mobile/DeskTop
• Browser Type
• User Age
• Geo/Census
• Weather
• Tenure/RFM Score
Observed Data
• The Button – Ha!
• Price
• Sales Offers
• Shipping Type
• Layout/UX
• Which Products
• Suggested
Quantities
Causal Action
Learning Efficiently
+
Apply ‘Best’ Rules
Optimization is Balance
How do we pick the rules?
But …
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
Learn From Domain Experts …
* Image Source http://en.wikipedia.org/wiki/Anna_Wintour
…and from Experiments
* Image Source http://en.wikipedia.org/wiki/Long_Ashton_Research_Station
22 [email protected]; www.conductrics.com
Multi-Armed Bandit Experiment
Which is Better to Experiment?
Vs
"DevonLarratt-TarasIvakin" by Above99 - Own work. Licensed under CC BY 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:DevonLarratt-TarasIvakin.jpg#/media/File:DevonLarratt-TarasIvakin.jpg
Source: http://www.startrek.com/article/ten-for-ward-10-favorite-kirk-fight-scenes
Experimentation Review
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
The A/B
TEST
Anyone
A
B
Simple AB: Data Collection
Anyone
A
B
Simple AB: Evaluation Phase
1) Calculate Average/Means of A & B
2) Calculate Spread/Variance in A & B
+
Mean(B) – Mean(A) t=
4) Use evidence to make a selection
3)
28 [email protected]; www.conductrics.com
(Or favorite Test)
Simple AB: Evaluation Phase
B A
29
Simple AB: Apply Rule
If [Anyone]
Offer B Then
A measure of Uncertainty
Mean - 2*Std +2*Std
32 [email protected]; www.conductrics.com
Recall Confidence Interval
Mean +Bonus
Score each option using the upper
portion of the interval as a Bonus
33 [email protected]; www.conductrics.com
Upper Confidence as Bonus
0
Conversion Rate
5% 10%
A
B
C
1) Use upper portion of CI as ‘Bonus’
Select A
2) Select based on highest ‘Score’
34 [email protected]; www.conductrics.com
Upper Confidence as Bonus
0 5% 10%
A
B
C
1) Selecting Action ‘A’ reduces uncertainty
bonus (because more data)
Select C
2) Action ‘C’ now has highest score
35 [email protected]; www.conductrics.com
Upper Confidence as Bonus
Conversion Rate
When MAB?
• Headlines
• Flash Sales
Adaptive/Bandit AB Test
37 [email protected]; www.conductrics.com
Is Learning Perishable?
Yes! Not Really
• Site Redesign
• Many Options
• Many Tests
• Personalized
• Top ‘k of M
38 [email protected]; www.conductrics.com
Complex Problem?
Yes! Not Really
• Few Options
• Few Tests
• No/Simple Targeting
• Single Option
Pfizer Use Case The ‘IF‘ Part
• Three Different User Segments
• Wyeth Pharma Search Segment
• King Pharma Search Segment
• All others Segment
Conductrics Confidential
Pfizer Use Case
The ‘Then’ Part
• Best Content Topic for each Segment
• Best Navigation UI
The ‘IF‘ Part
• Three Different User Segments
• Wyeth Pharam Search Segment
• King Pharma Search Segment
• All others Segment
Conductrics Confidential
Pfizer Use Case
The ‘Then’ Part
• Best Content Topic for each Segment
• Best Navigation UI
The ‘IF‘ Part
• Three Different User Segments
• Wyeth Pharam Search Segment
• King Pharma Search Segment
• All others Segment
The ‘How’ Part
• Test Bandit Approach
Conductrics Confidential
Pfizer Use Case
Conductrics Confidential
Topics
1)Popular: Default 2) Wyeth Related 3) King Related
Conductrics Confidential
Investor Navigation
1) NavA: Default
2) NavB: Box
3) NavC: Tab
Create Agent
Create Segments
Wyeth Interested
King Interested
Top Option 8 times more often than the default
47 [email protected]; www.conductrics.com
Results: Wyeth Segment
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
48 [email protected]; www.conductrics.com
Confidence Intervals MAB & AB C
on
vers
ion
Rate
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
49 [email protected]; www.conductrics.com
Co
nve
rsio
n R
ate
High Value Option-> Smaller CIs
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
50 [email protected]; www.conductrics.com
Low Value -> Higher Uncertainty C
on
vers
ion
Rate
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
51 [email protected]; www.conductrics.com
Low Value -> Higher Uncertainty C
on
vers
ion
Rate
All Others
Wyeth Searchers
King Searchers
Content Options
Navigation Options
• For the segments - the content selection had a greater impact
• For ‘all others’ – the Navigation selection had the most impact
Conductrics Confidential
Nav or Topics: Impact
1. Automation
2. Changing World/Perishable
3. Complexity / Scale
53 [email protected]; www.conductrics.com
When to Use Bandits
See Russell & Norvig ‘s Artificial Intelligence: A Modern Approach
Optimization
Just 5 Parts
Goal
Measureable Goals
Ability to ‘Sense’ the Environment
Ability to Act/Execute
A B
Observe Outcomes
Play ‘Better’ Options More Often
If Then Receive
Learn Decision Logic
Wake up. We are Done!
Twitter:mgershoff Email:[email protected]
Image source: http://www.washingtonpost.com/blogs/london-2012-olympics/wp/tag/soccer/ (Andrew Medichini — Associated Press)