Top Banner
102

Héctor Zárate "AB Testing"

Jan 22, 2017

Download

Education

Mobicode
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Héctor Zárate "AB Testing"
Page 2: Héctor Zárate "AB Testing"
Page 3: Héctor Zárate "AB Testing"

Spotify Numbers

60 markets Over 30 million Songs100 million Active Users

(40 million Subscribers)

Page 4: Héctor Zárate "AB Testing"

iOS Client Numbers

50 monthly contributors 0.5 million lines of code +100 AB Tests Running

Page 5: Héctor Zárate "AB Testing"

Hector Zarate (@ChocoChipset)Software Engineer at Spotify

Boxun ZhangData Scientist at Spotify

Page 6: Héctor Zárate "AB Testing"

1. What?

Page 7: Héctor Zárate "AB Testing"
Page 8: Héctor Zárate "AB Testing"

😱

Page 9: Héctor Zárate "AB Testing"

😱

Page 10: Héctor Zárate "AB Testing"

YOU ARE PART OF AN EXPERIMENT

Page 11: Héctor Zárate "AB Testing"

AB Testing

YOU ARE PART OF AN EXPERIMENT

Page 12: Héctor Zárate "AB Testing"
Page 13: Héctor Zárate "AB Testing"

Samson, 3

Page 14: Héctor Zárate "AB Testing"

Max, 30Samson, 3

+5% matches

Page 15: Héctor Zárate "AB Testing"

Share of Matches

20%

23.75%

27.5%

31.25%

35%

Week 0 Week 1 Week 2 Week 3 Week 4

With Sunglasses

Sunglasses

Page 16: Héctor Zárate "AB Testing"

Share of Matches

20%

23.75%

27.5%

31.25%

35%

Week 0 Week 1 Week 2 Week 3 Week 4

With Sunglasses Control

Sunglasses

Page 17: Héctor Zárate "AB Testing"

Share of Premium Conversion

20%

23.75%

27.5%

31.25%

35%

Week 0 Week 1 Week 2 Week 3 Week 4

Variation A Control

Sunglasses

Page 18: Héctor Zárate "AB Testing"

User Retention

20%

23.75%

27.5%

31.25%

35%

Week 0 Week 1 Week 2 Week 3 Week 4

Variation A Control

Sunglasses

Page 19: Héctor Zárate "AB Testing"
Page 20: Héctor Zárate "AB Testing"

spotify:user:chocochipset92b2976bb15d26c9008

Page 21: Héctor Zárate "AB Testing"

1 2 3 4 5 6 7 8 9 10

p ( x )

x

1 / n

Page 22: Héctor Zárate "AB Testing"

1 2 3 4 5 6 7 8 9 10

p ( x )

x

1 / n

Page 23: Héctor Zárate "AB Testing"

1 2 3 4 5 6 7 8 9 10

p ( x )

x

1 / n 0% 60%

Page 24: Héctor Zárate "AB Testing"

Markets

Page 25: Héctor Zárate "AB Testing"

Demographics

Page 26: Héctor Zárate "AB Testing"

User Attributes

Page 27: Héctor Zárate "AB Testing"
Page 28: Héctor Zárate "AB Testing"

Login

resolveABFlags()

response(ABFlags)

ABBA

Cache AB Values

Load Cached or Default AB Flags

Page 29: Héctor Zárate "AB Testing"

Example

button.color = [UIColor spotifyGreen];

BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) {

// alternate path here: button.color = [UIColor spotifyPink];

}

Key Value

charts Enabled

pink-buttons 0

buffer-quality low

gallery-artist Control

Page 30: Héctor Zárate "AB Testing"

Key Value

charts Enabled

pink-buttons 0

buffer-quality low

gallery-artist Control

Example

button.color = [UIColor spotifyGreen];

BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) {

// alternate path here: button.color = [UIColor spotifyPink];

}

Key Value

charts Enabled

pink-buttons 1

buffer-quality low

gallery-artist Control

pink-button 1

Variation A Control

Page 31: Héctor Zárate "AB Testing"

Analytics

impressions and interactions

ControlVariation

Page 32: Héctor Zárate "AB Testing"

* just a personal preference

Page 33: Héctor Zárate "AB Testing"

2. How?

Page 34: Héctor Zárate "AB Testing"

1. Hypothesis 2. Design 3. Run 4. Analysis

Page 35: Héctor Zárate "AB Testing"

1. Formulate a Hypothesis

Page 36: Héctor Zárate "AB Testing"

2. Design the Test

Page 37: Héctor Zárate "AB Testing"

# shuffle

plays

2.1 Target Metrics

Page 38: Héctor Zárate "AB Testing"

# matches

Samson, 3

# replies

# walks in the park

Page 39: Héctor Zárate "AB Testing"

2.2 Test Group

more test bandwith, less confidence, smaller effect in business

less test bandwith, more confidence, bigger effect in business

Small

Large

Page 40: Héctor Zárate "AB Testing"

2.3 Duration

Page 41: Héctor Zárate "AB Testing"

Watch out for seasonality:i.e. Music habits are different on weekdays and weekends.

Purchasing habits are different near paydays.

Page 42: Héctor Zárate "AB Testing"

3. Run the test

Page 43: Héctor Zárate "AB Testing"

1 2 3 4 5 6 7 8 9 10

p ( x )

x

1 / n

Test

A

Test

A

Test

B

Test

B

Test

A /

Test

B

Watch out for conflicting tests!

Page 44: Héctor Zárate "AB Testing"

Don’t cut them short!

Page 45: Héctor Zárate "AB Testing"

0m 90mStandard Match Duration

0m 20mInconclusive

0m +95mUnnecessary

Page 46: Héctor Zárate "AB Testing"

0m +95m

Bruno Cesar 48’: 0

- 1

Page 47: Héctor Zárate "AB Testing"

0m +95m

Bruno Cesar 48’: 0

- 1

Ronaldo 89’: 1 - 1

Page 48: Héctor Zárate "AB Testing"

0m +95m

Bruno Cesar 48’: 0

- 1

Ronaldo 89’: 1 - 1

Morata 90’ + 4’: 2 - 1

2 - 1

Page 49: Héctor Zárate "AB Testing"

0

25

50

75

100

D1 D2 D3 D4 D5 D6 D7 D8

Don’t cut them short!

Page 50: Héctor Zárate "AB Testing"

4. Analyze the results

Page 51: Héctor Zárate "AB Testing"

1. Formulate a Hypothesisnew

Page 52: Héctor Zárate "AB Testing"

3. Case Studies

Page 53: Héctor Zárate "AB Testing"

3.1. Losing Calories

Page 54: Héctor Zárate "AB Testing"
Page 55: Héctor Zárate "AB Testing"
Page 56: Héctor Zárate "AB Testing"
Page 57: Héctor Zárate "AB Testing"
Page 58: Héctor Zárate "AB Testing"
Page 59: Héctor Zárate "AB Testing"
Page 60: Héctor Zárate "AB Testing"

Test: Tab Bar Navigation

Hypothesis:

By switching to TBN, we

expect an increase in the

share of users who click

at least one menu item.

1

Page 61: Héctor Zárate "AB Testing"

Test: Tab Bar Navigation

Hypothesis:

By switching to TBN, we

expect an increase in the

share of users who click

at least one menu item.

2.1

Page 62: Héctor Zárate "AB Testing"

10% New Users

1% Existing Users

2.2

Page 63: Héctor Zárate "AB Testing"

2.3

1 week period

Page 64: Héctor Zárate "AB Testing"

4

Clicks on Menu Items

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

Control Tab Bar

130%

100%

Page 65: Héctor Zárate "AB Testing"

4

Clicks Overall

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

Control Tab Bar

109%100%

Page 66: Héctor Zárate "AB Testing"

4

1. Increased clicks in Tab Items

2. Decreased clicks in non-tab items.

Page 67: Héctor Zárate "AB Testing"

WHY AB TEST?

QUANTIFY THE IMPACT OF A SPECIFIC CHANGE

Page 68: Héctor Zárate "AB Testing"

3.2. Home in Mexico

Page 69: Héctor Zárate "AB Testing"

My Rock Your Rock

Page 70: Héctor Zárate "AB Testing"

My Home Your Home

Page 71: Héctor Zárate "AB Testing"

My Home Your Home

Page 72: Héctor Zárate "AB Testing"

Test: Home

Hypothesis:

We will measure an

increased second week

retention by using the

new ‘Home’ as start

page. 1

Page 73: Héctor Zárate "AB Testing"

Test: Home

Hypothesis:

We will measure an

increased second week

retention by using the

new ‘Home’ as start

page. 2.1

Page 74: Héctor Zárate "AB Testing"

10% New Users

1% Existing Users

2.2

Page 75: Héctor Zárate "AB Testing"

2.3

1 week period

Page 76: Héctor Zárate "AB Testing"

US UK Germany Austria Mexico

4

Page 77: Héctor Zárate "AB Testing"

1. Is the localization good? 2. Are recommendations relevant for

the market? 3. Technical restrictions we are not

aware of?

4

Page 78: Héctor Zárate "AB Testing"

12.34 Mbps US

13.70 Mbps UK

13.42 Mbps Germany

15.48 Mbps Austria

7.4 Mbps Mexico* 3.7 Mb

4

Page 79: Héctor Zárate "AB Testing"

1. Hypothesis 2. Design 3. Run 4. Analysis

Page 80: Héctor Zárate "AB Testing"

Test: Home

Hypothesis:

By compressing the data

sent for Home, more

users will play on day

one, two and be active

during their second

week. 1

Page 81: Héctor Zárate "AB Testing"

1. Is the localization good? 2. Are recommendations relevant for

the market? 3. Technical restrictions we are not

aware of?

4

Page 82: Héctor Zárate "AB Testing"

1. Is the localization good? 2. Are recommendations relevant for

the market? 3. Technical restrictions we are not

aware of?

4

Page 83: Héctor Zárate "AB Testing"

1. Is the localization good? 2. Are recommendations relevant for

the market? 3. Technical restrictions we are not

aware of?

4

Page 84: Héctor Zárate "AB Testing"

4. Pitfalls

Page 85: Héctor Zárate "AB Testing"

ANALYSIS PARALYSIS

Page 86: Héctor Zárate "AB Testing"

NO DATA

Page 87: Héctor Zárate "AB Testing"

Max, 30

TOO MANY SMALL CHANGES

Page 88: Héctor Zárate "AB Testing"

Max, WOLF AT WALL STREET

TOO MANY SMALL CHANGES

Page 89: Héctor Zárate "AB Testing"

DONTLISTEN TO

YOUR HEART

(LISTEN TO THE DATA)

Page 90: Héctor Zárate "AB Testing"

5. Wrap Up

Page 91: Héctor Zárate "AB Testing"
Page 92: Héctor Zárate "AB Testing"
Page 93: Héctor Zárate "AB Testing"
Page 94: Héctor Zárate "AB Testing"
Page 95: Héctor Zárate "AB Testing"
Page 96: Héctor Zárate "AB Testing"

KNOWLEDGE IS YOUR

ROI

Page 97: Héctor Zárate "AB Testing"

A test is an investment

Design to maximize learning

Page 98: Héctor Zárate "AB Testing"

NEGATIVE RESULTS ARE

STILL POSITIVE(DON’T GIVE UP!)

Page 99: Héctor Zárate "AB Testing"

TRY AB TESTING

Page 100: Héctor Zárate "AB Testing"
Page 101: Héctor Zárate "AB Testing"

Вопросов?Hector Zarate @ChocoChipset

spotify.com/jobs

Page 102: Héctor Zárate "AB Testing"

Спасибо!Hector Zarate @ChocoChipset

spotify.com/jobs