Optimizing website conversions through A/B split testing

OPTIMIZINGCONVERSIONSTHROUGHA/B SPLIT TESTING

Javid JamaeCTO / Co-founder CamperooPresented at the Houston Lean Startup CircleJune 18, 2013

Special thanks to - Emiliano Villareal

‘‘The goal of a Lean Startup is to inform our riskiest business model assumptions through

empirical testing with customers – not rhetorical reasoning on a white board.

~ ASH MAURYA

AirBNB

Zappos

Users coming in through a Zappos-style homepage will be 10% more likely

to go from the homepage into a product search results page

EXPERIMENTDESIGN

FALSIFIABLE HYPOTHESIS

EXPERIMENT SCOPE40 man-hours to design and code750 unique visitors per variant

UNIQUE VISITORS RANDOMIZER ANALYTICS

CONTROL

VARIANT

EXPERIMENTIMPLEMENTATION

50%

50%

THEWINNERWAS?

5.86% ??

Percentage of people going from the homepage to a product search results page

Control Variant

THEWINNERWAS?

5.86% 6.84%

16.89% increase!Percentage of people going from the homepage

to a product search results page

Control Variant

BUTWHY?

Was it the:- Filters- Category browsing- Different slider- The combination

?

In Lean Startups, success is defined by learning, not by test results

X

A/B TOOL √

ValidatedLearning

Loop

√Variant

+

Users coming into an AirBNB-style homepage with categories will be 5%

more likely to go into a product search results page than without categories

EXPERIMENTDESIGN

FALSIFIABLE HYPOTHESIS

EXPERIMENT SCOPE1 man-hours to design and code750 unique visitors per variant

THEWINNERWAS?

5.90% ??

Percentage of people going from the homepage to a product search results page

Control Variant

THEWINNERWAS?

5.90% 5.98%

Percentage of people going from the homepage to a product search results page

Control Variant

1.35% increase

THELEANPROCESS

IDEA

HYPOTHESIS

EXPERIMENT DESIGN

EXPERIMENTATION

MEASUREMENT

Scientific= Process

IDEA

HYPOTHESIS

EXPERIMENT DESIGN

EXPERIMENTATION

MEASUREMENT

Prioritize

Define

Design

Implement

Track

How do I...

PRIORITIZINGEXPERIMENTS

‘‘If you measure something and it’s not attached to a goal, in turn changing your

behavior, you’re wasting your time.

ALISTAIR CROLL & BENJAMIN YOSKOVITZ BOOK: LEAN ANALYTICS

GOALORIENTEDPLANNING

Landing PageChanges

SEO Rankings

Goal: IncreaseAcquisition

InboundChannels

AcquisitionActivationReferralRevenueRetention

PIRATEMETRICS

OneMetricThat

Matters

Lean Analytics (Lean Series). O’Reilly Media.

Stage

RISKSACROSSSTAGESPr

oduc

tC

usto

mer

Mar

ket

Problem/Solution

Identify

Competition/Positioning

Early / Inception

MVP

Narrow/Outbound

Pricing

Product

Scale

Inbound

Cost Structure

Scale

RisksQualitative,

Activation / RetentionQuantitative,

Revenue / Growth

Favor experiments that

make sense for your current stage & goals

Favor experiments that

are innovative

Favor experiments that

have high risk

Favor experiments that

you can measure

Favor experiments that

engaged customers seem to need or are asking for

Theory of Constraints

A chain is no stronger than its weakest link.

The Goal - Eliyahu M. Goldratt

DEFININGHYPOTHESES

Is Manchester United causingglobal warming?

http://theconversation.com/how-david-beckham-caused-global-warming-the-man-u-climate-model-4548

‘‘If you torture the data long enough, it will confess to anything.

~UNKNOWN

Don’t take Leaps of Faith

Leap of faith: A Zappos-style front page will increase browsability on our website

Falsifiable hypothesis: Users coming in through a Zappos-style homepage will be 10% more likely to go from the homepage into a product search results page

(vs. the AirBNB-style homepage)

Formula:

[Repeatable Action] will [Expected Result]

Writing a blog post about Houston activities will result in 25 new accounts

Users entering Variation B of the store page will be 2% more likely to purchase

Putting an item on sale will increase the chances that it will be added to a cart by 10%

Draw a linein the sand...

...before you start gathering data.

DESIGNINGEXPERIMENTS

Are you measuringimpact on a

Goal or a Metric

LEARNINGFROMDATA

A/B Testing

Cohort AnalysisAnalysis

Testing

Longitudinal Cross-Sectional

Multivariate Testing

Segmentation

(Conditional)Feature Rollout

Decide on a exit criteria up front

Cost It will take 5-10 hours to implement, if

it takes longer, we will abandon.

ScopeWe need 1,000 unique visitors per variant

before the data has any meaning to us.

TimeboxIf the test doesn’t end by July 2, then we

will abandon.

Follow upIf the test fails, then try experiment C. If

the test passes, then experiment B.

Pick the type of analysis

TYPESOFANALYSIS

JanuaryFebruary

MarchApril

May

segmentation(cross-sectional)

cohortanalysis

(longitudinal)

Pick the type of test

UNIQUE VISITORS RANDOMIZER

CONTROL

VARIANT

Version A

Version B

A/B

UNIQUE VISITORS RANDOMIZER

CONTROL

VARIANT 1

A/B/N

VARIANT 2

VARIANT 3

MULTIVARIATETESTING

Enroll Sign Up Join Now

Color:

Text:

Size:

27 Combinations

MULTIVARIATETESTINGEnroll Sign Up Join Now

Enroll Sign Up Join Now

Enroll Sign Up Join Now

Enroll Sign Up Join Now

Enroll Sign Up Join Now

Enroll Sign Up Join Now

Enroll

Enroll

Enroll

Sign Up

Sign Up

Sign Up

Join Now

Join Now

Join Now

IMPLEMENTINGEXPERIMENTS

Amazon A/B Testing Service (beta)

FREETOOLS

Google Analytics - Experiments APIGoogle Analytics - Content Experiments

WYSIWYGTOOLS

ENTERPRISETOOLS

http://www.roidna.com/tools/ab-testing-tool/

THECAMPEROOWAY?

github.com/andrew/split

SPLIT + MIXPANELOpen Source, Ruby

Freemium,SAAS

mixpanel.com

TRACKINGEXPERIMENTS

Goal Build Measure Learn

IncreaseAcquisition

ZapposRedesign

AirBNBwith category

navigation

Search engine display of

customer ratings

IncreaseActivation

Draconian modal dialog

box

KANBANBOARD

VALIDATEDLEARNINGBOARD

http://www.ashmaurya.com/2012/06/the-lean-stack/

EXPERIMENTREPORTING

http://spark59.s3.amazonaws.com/presentations/All%20About%20Experiments.pdf

WEEKLYPLANNING

Weekly meeting:

Review experiment results

Challenge interpretation

Kill or restart overdue experiments

Decide next experiments

THEEND

...to be continued?