Share of Matches
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
With Sunglasses
Sunglasses
Share of Matches
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
With Sunglasses Control
Sunglasses
Share of Premium Conversion
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
Variation A Control
Sunglasses
User Retention
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
Variation A Control
Sunglasses
Example
button.color = [UIColor spotifyGreen];
BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) {
// alternate path here: button.color = [UIColor spotifyPink];
}
Key Value
charts Enabled
pink-buttons 0
buffer-quality low
gallery-artist Control
Key Value
charts Enabled
pink-buttons 0
buffer-quality low
gallery-artist Control
Example
button.color = [UIColor spotifyGreen];
BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) {
// alternate path here: button.color = [UIColor spotifyPink];
}
Key Value
charts Enabled
pink-buttons 1
buffer-quality low
gallery-artist Control
pink-button 1
Variation A Control
2.2 Test Group
more test bandwith, less confidence, smaller effect in business
less test bandwith, more confidence, bigger effect in business
Small
Large
Watch out for seasonality:i.e. Music habits are different on weekdays and weekends.
Purchasing habits are different near paydays.
1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n
Test
A
Test
A
Test
B
Test
B
Test
A /
Test
B
Watch out for conflicting tests!
Test: Tab Bar Navigation
Hypothesis:
By switching to TBN, we
expect an increase in the
share of users who click
at least one menu item.
1
Test: Tab Bar Navigation
Hypothesis:
By switching to TBN, we
expect an increase in the
share of users who click
at least one menu item.
2.1
4
Clicks on Menu Items
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
Control Tab Bar
130%
100%
Test: Home
Hypothesis:
We will measure an
increased second week
retention by using the
new ‘Home’ as start
page. 1
Test: Home
Hypothesis:
We will measure an
increased second week
retention by using the
new ‘Home’ as start
page. 2.1
1. Is the localization good? 2. Are recommendations relevant for
the market? 3. Technical restrictions we are not
aware of?
4
Test: Home
Hypothesis:
By compressing the data
sent for Home, more
users will play on day
one, two and be active
during their second
week. 1
1. Is the localization good? 2. Are recommendations relevant for
the market? 3. Technical restrictions we are not
aware of?
4
1. Is the localization good? 2. Are recommendations relevant for
the market? 3. Technical restrictions we are not
aware of?
4
1. Is the localization good? 2. Are recommendations relevant for
the market? 3. Technical restrictions we are not
aware of?
4