YOU ARE DOWNLOADING DOCUMENT

Please tick the box to continue:

Transcript
  • Spotify Numbers

    60 marketsOver 30 million Songs100 million Active Users

    (40 million Subscribers)

  • iOS Client Numbers

    50 monthly contributors 0.5 million lines of code +100 AB Tests Running

  • Hector Zarate (@ChocoChipset)Software Engineer at Spotify

    Boxun ZhangData Scientist at Spotify

  • 1. What?

  • YOU ARE PART OF AN EXPERIMENT

  • AB Testing

    YOU ARE PART OF AN EXPERIMENT

  • Samson, 3

  • Max, 30Samson, 3

    +5% matches

  • Share of Matches

    20%

    23.75%

    27.5%

    31.25%

    35%

    Week 0 Week 1 Week 2 Week 3 Week 4

    With Sunglasses

    Sunglasses

  • Share of Matches

    20%

    23.75%

    27.5%

    31.25%

    35%

    Week 0 Week 1 Week 2 Week 3 Week 4

    With Sunglasses Control

    Sunglasses

  • Share of Premium Conversion

    20%

    23.75%

    27.5%

    31.25%

    35%

    Week 0 Week 1 Week 2 Week 3 Week 4

    Variation A Control

    Sunglasses

  • User Retention

    20%

    23.75%

    27.5%

    31.25%

    35%

    Week 0 Week 1 Week 2 Week 3 Week 4

    Variation A Control

    Sunglasses

  • spotify:user:chocochipset92b2976bb15d26c9008

  • 1 2 3 4 5 6 7 8 9 10

    p ( x )

    x

    1 / n

  • 1 2 3 4 5 6 7 8 9 10

    p ( x )

    x

    1 / n

  • 1 2 3 4 5 6 7 8 9 10

    p ( x )

    x

    1 / n 0% 60%

  • Markets

  • Demographics

  • User Attributes

  • Login

    resolveABFlags()

    response(ABFlags)

    ABBA

    Cache AB Values

    Load Cached or Default AB Flags

  • Example

    button.color = [UIColor spotifyGreen];

    BOOL isButtonPink = ([abFlags[@pink-buttons"] isEqual:@"1"]); if (isButtonPink) {

    // alternate path here: button.color = [UIColor spotifyPink];

    }

    Key Value

    charts Enabled

    pink-buttons 0

    buffer-quality low

    gallery-artist Control

  • Key Value

    charts Enabled

    pink-buttons 0

    buffer-quality low

    gallery-artist Control

    Example

    button.color = [UIColor spotifyGreen];

    BOOL isButtonPink = ([abFlags[@pink-buttons"] isEqual:@"1"]); if (isButtonPink) {

    // alternate path here: button.color = [UIColor spotifyPink];

    }

    Key Value

    charts Enabled

    pink-buttons 1

    buffer-quality low

    gallery-artist Control

    pink-button 1

    Variation A Control

  • Analytics

    impressions and interactions

    ControlVariation

  • * just a personal preference

  • 2. How?

  • 1. Hypothesis 2. Design 3. Run 4. Analysis

  • 1. Formulate a Hypothesis

  • 2. Design the Test

  • # shu

    ffle pl

    ays

    2.1 Target Metrics

  • # matches

    Samson, 3

    # replies

    # walks in the park

  • 2.2 Test Group

    more test bandwith, less confidence, smaller effect in business

    less test bandwith, more confidence, bigger effect in business

    Small

    Large

  • 2.3 Duration

  • Watch out for seasonality:i.e. Music habits are different on weekdays and weekends.

    Purchasing habits are different near paydays.

  • 3. Run the test

  • 1 2 3 4 5 6 7 8 9 10

    p ( x )

    x

    1 / n

    Test

    A

    Test

    A

    Test

    B

    Test

    B

    Test

    A /

    Test

    B

    Watch out for conflicting tests!

  • Dont cut them short!

  • 0m 90mStandard Match Duration

    0m 20mInconclusive

    0m +95mUnnecessary

  • 0m +95m

    Bruno

    Cesar

    48: 0

    - 1

  • 0m +95m

    Bruno

    Cesar

    48: 0

    - 1

    Rona

    ldo 89

    : 1 - 1

  • 0m +95m

    Bruno

    Cesar

    48: 0

    - 1

    Rona

    ldo 89

    : 1 - 1

    Morat

    a 90

    + 4: 2

    - 1

    2 - 1

  • 0

    25

    50

    75

    100

    D1 D2 D3 D4 D5 D6 D7 D8

    Dont cut them short!

  • 4. Analyze the results

  • 1. Formulate a Hypothesisnew

  • 3. Case Studies

  • 3.1. Losing Calories

  • Test: Tab Bar Nav

    igation

    Hypothesis:

    By switching to T

    BN, we

    expect an increa

    se in the

    share of users w

    ho click

    at least one men

    u item.

    1

  • Test: Tab Bar Nav

    igation

    Hypothesis:

    By switching to T

    BN, we

    expect an increa

    se in the

    share of users w

    ho click

    at least one men

    u item.

    2.1

  • 10% New Users

    1% Existing Users

    2.2

  • 2.3

    1 week period

  • 4

    Clicks on Menu Items

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%

    110%

    120%

    130%

    Control Tab Bar

    130%

    100%

  • 4

    Clicks Overall

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%

    110%

    120%

    130%

    Control Tab Bar

    109%100%

  • 4

    1. Increased clicks in Tab Items

    2. Decreased clicks in non-tab items.

  • WHY AB TEST?

    QUANTIFY THE IMPACT OF A SPECIFIC CHANGE

  • 3.2. Home in Mexico

  • My Rock Your Rock

  • My Home Your Home

  • My Home Your Home

  • Test: Home

    Hypothesis:

    We will measure

    an

    increased second

    week

    retention by usin

    g the

    new Home as st

    art

    page. 1

  • Test: Home

    Hypothesis:

    We will measure

    an

    increased second

    week

    retention by usin

    g the

    new Home as st

    art

    page. 2.1

  • 10% New Users

    1% Existing Users

    2.2

  • 2.3

    1 week period

  • US UK Germany Austria Mexico

    4

  • 1. Is the localization good? 2. Are recommendations relevant for

    the market? 3. Technical restrictions we are not

    aware of?

    4

  • 12.34 Mbps US

    13.70 Mbps UK

    13.42 Mbps Germany

    15.48 Mbps Austria

    7.4 Mbps Mexico* 3.7 Mb

    4

  • 1. Hypothesis 2. Design 3. Run 4. Analysis

  • Test: Home

    Hypothesis:

    By compressing t

    he data

    sent for Home, m

    ore

    users will play on

    day

    one, two and be a

    ctive

    during their seco

    nd

    week. 1

  • 1. Is the localization good? 2. Are recommendations relevant for

    the market? 3. Technical restrictions we are not

    aware of?

    4

  • 1. Is the localization good? 2. Are recommendations relevant for

    the market? 3. Technical restrictions we are not

    aware of?

    4

  • 1. Is the localization good? 2. Are recommendations relevant for

    the market? 3. Technical restrictions we are not

    aware of?

    4

  • 4. Pitfalls

  • ANALYSIS PARALYSIS

  • NO DATA

  • Max, 30

    TOO MANY SMALL CHANGES

  • Max, WOLF AT WALL STREET

    TOO MANY SMALL CHANGES

  • DONTLISTEN TO

    YOUR HEART

    (LISTEN TO THE DATA)

  • 5. Wrap Up

  • KNOWLEDGE IS YOUR

    ROI

  • A test is an investment

    Design to maximize learning

  • NEGATIVE RESULTS ARE

    STILL POSITIVE(DONT GIVE UP!)

  • TRY AB TESTING

  • ?Hector Zarate @ChocoChipset

    spotify.com/jobs

    http://spotify.com/jobs

  • !Hector Zarate @ChocoChipset

    spotify.com/jobs

    http://spotify.com/jobs


Related Documents