Fallstudie: A/B Testing mit Optimizely bei Westwing

AB  Testing:  Getting  Started

Jayme  Suravicz

15.  October  2015

Agenda• Introduction

• Team  Setup

• Understanding  your  traffic

• Things  to  consider  when  AB  testing

• Establishing  a  reporting  system

• Key  Takeaways

#OPTEX2015

3

4

5

6

7

Why  do  AB  testing?

Test  Hypotheses

Learn  more  about  user  behavior

Keep  our  site  “fresh”

Measure  the  impact  of  changes

Drive  Conversion  &  Engagement

#OPTEX20158

Agenda• Introduction

• Team  Setup

• Understanding  your  traffic

• Things  to  consider  when  AB  testing

• Establishing  a  reporting  system

• Key  Takeaways

#OPTEX2015

Identify  your  AB  testing  team  and  responsibilities

10

What  do  you  want  to  test?

Who  is  the  business  owner?

How  can  you  share  synergies?

Project  Manager

TrackingSupport

Tester Tester Tester

Tester Tester Tester

Department  A

Department  C

Department  B

11

Identify  your  AB  testing  team  and  responsibilities

Agenda• Introduction

• Team  Setup

• Understanding  your  traffic

• Things  to  consider  when  AB  testing

• Establishing  a  reporting  system

• Key  Takeaways

#OPTEX2015

Understanding  the  different  traffic  sources  to  your  site  will  help  you  better  address  the  needs  of  visitors

Westwing  receives  visitors  from  a  variety  of  different   traffic  sources  and  assumes  these  sources  would  perform  differently  and  therefore  require  different  landing  pages  (LPs)

Identify  your main  traffic  buckets:Example:

Consider   if  buckets  can  be  further  broken  down   into  subgroups:Examples:

SEM SEO Social Display Affiliate

SEMbrand

SEMnon  brand

DisplayPublisher  X

Display  Publisher  Y

SocialFB

SocialTwitter

SocialInstagram

13

Understanding  the  different  traffic  sources  to  your  site  will  help  you  better  analyze  your  test  results

In  addition   to  traffic  sources,  devices  should   also  be  considered  as  users  have  different  experiences  on  different  devices  and  therefore   require  different  landing  pages  (LPs)

Distinguish  between  your main  devices:Example:

As  well  as  your  different  platforms:Examples:

Desktop Tablet Mobile

Desktop  Site Mobile  Site Mobile  app Tablet  app

14

Understanding  the  different  traffic  sources  to  your  site  will  help  you  better  analyze  your  test  results

15

Agenda• Introduction

• Team  Setup

• Understanding  your  traffic

• Things  to  consider  when  AB  testing

• Establishing  a  reporting  system

• Key  Takeaways

#OPTEX2015

1. Consider  Primary  and  Secondary  Goals

3  main  points  to  consider  when  AB  testing  Landing  Pages

17

3  main  points  to  consider  when  AB  testing  Landing  Pages

18

Email  Only Email  +  Password Name  +  Email  +  Password

+37% +145%

+84%

3  main  points  to  consider  when  AB  testing  Landing  Pages

19

USPs  with  discount   [-­‐70%]

USPs  without  discount

+15% +50%+30%

2.  Think  Big,  Optimize  Small

3  main  points  to  consider  when  AB  testing  Landing  Pages

20

3  main  points  to  consider  when  AB  testing  Landing  Pages

21

3  main  points  to  consider  when  AB  testing  Landing  Pages

22

3  main  points  to  consider  when  AB  testing  Landing  Pages

23

3  main  points  to  consider  when  AB  testing  Landing  Pages

24

3.  Don’t  ignore  the  different  user  experiences  in  devices  and  environments

3  main  points  to  consider  when  AB  testing  Landing  Pages

25

Agenda• Introduction

• Team  Setup

• Understanding  your  traffic

• Things  to  consider  when  AB  testing

• Establishing  a  reporting  system

• Key  Takeaways

#OPTEX2015

Take  advantage  of  the  Optimizely  API  for  reporting

27

Export  test  performance  via  the  Optimizely  API

Build  your  custom  report

Use  a  consistent  namingconvention for  all  tests:Team  /  Country   /  Traffic  Group  /  Subgroup   /  Device

Organize  your  projects in  Optimizely

Agenda• Introduction

• Team  Setup

• Understanding  your  traffic

• Things  to  consider  when  AB  testing

• Reporting  and  Analyzing  AB  tests

• Key  Takeaways

#OPTEX2015

Key  Takeaways• Identify  your  team  &  establish  responsibilities

• Understand  your  traffic  &  build  your  testing  strategy  from  day  1

• Set  clear  goals,  both  primary  and  secondary

• Think  big  &  optimize  small  

• Invest  the  time  to  setup  proper  reporting

#OPTEX2015

[email protected]

#OPTEX2015