The 5 Stages of A/B Testing

The 5 Stages of A/B Testing

Byron JonesProduct Manager

What is A/B Testing?

Credit: Jonathan Rosenberg's post from 21 Aug 2012: Hippos are among the most dangerous animals in Africa. Conference rooms too.

conducting experiments to optimize your user experience and making data driven decisions

A/B Testing is…

Examples of A/B Tests

• Business Objective:Teach the world to code

• Goal: Increase participation in Hour of Code campaign

• Hypothesis: Changing the hero messaging to emphasize a sense of community around the movement would result in an increase in signups.

• Business Objective: Increase revenue

• Goal: Maximize sales on new product launches

• Hypothesis: Moving the call-to-action higher on the page by changing the way the promotional offer was displayed could drive more purchases and increase revenue generated from SimCity.com.

• Business Objective:Drive engagement and traffic

• Goal: Build follower base on social media

• Experiment: Which design of social media call-to-action buttons maximizes followers?

5 Stages of A/B Testing

MEASURE

ANALYZE

HYPOTHESIZE

BUILDRUN

of

A/B TESTING

MEASURE

ANALYZE

HYPOTHESIZE

BUILDRUN

Analyze your Data

• User Tests & Research

• Surveys

• Sales Team Feedback

• Heat Maps

• Web Traffic

• Product Usage

• Order History

• Support Tickets

QualitativeQuantitative

Analyze - Best Practices

• Look for high impact areas of your business

• Use the sample size calculator to predict volume and duration

MEASURE

ANALYZE

HYPOTHESIZE

BUILDRUN

"If _____[I do this] _____, then _____[this]_____ will happen because ___[rationale]____."

Create Your Hypothesis

Hypothesize - Best Practices

• Change one variable at a time, otherwise you won’t know which change influenced the results

• Be specific and descriptive in your hypothesis

Business Goal

Hypothesis

Idea 1 Idea 2

MEASURE

ANALYZE

HYPOTHESIZE

BUILDRUN

Build Your Experiment

1. Create a variation

2. URL targeting

3. Add an Audience

4. Traffic Allocation

5. Setup your goals

6. Preview

Example - Adding Customer Quotes

A customer testimonial increases the accounts created because it increases the perceived value of the product.

Analysis & Sample Size Hypothesis

Demo - Building an Experiment in Optimizely

Best Practices

• You can use Query string parameters to target your ad spend or to QA your test live

For exampleutm_campaign=spring15

qa_testing=true

MEASURE

ANALYZE

HYPOTHESIZE

BUILDRUN

• 95% Statistical Significance

“How likely is it that my test results will say I have a winner when I actually don’t?”

• Optimizely’s Stats Engine powers our statistical significance calculations

The goal of Stats Engine is to make interpreting results as easy as possible for anyone, at any time

Optimizely Results

MEASURE

ANALYZE

HYPOTHESIZE

BUILDRUN

of

A/B TESTING