A/B Tests for 6 Stages of Your Company's Growth

Brendan O’Rourke, Optimizely Dan McGaw, KISSmetrics

A/B Tests for 6 Stages of Your Company’s Growth May 1, 2014

Dan McGaw - KISSmetrics - @danielmcgaw !Dan is the Director of Marketing at KISSmetrics, where he get’s the opportunity to help the best marketers in the world measure their marketing and business efforts. Prior to KISSmetrics, Dan was VP of growth at Code School, where he helped the company grow to over 500k users and $400k+ in monthly revenue.

Brendan O’Rourke - Optimizely - @optimizely !Brendan is an Enterprise Solutions Engineer at Optimizely. He joined Optimizely two years ago, a"er realizing his dream of becoming an actuary was very... dry. He now spends his time helping enterprises like Microso", NBC Universal, and Yelp turn data into action through testing and conversion rate optimization.

@optimizely #KISSwebinar

Join us on Twi#er

@danielmcgaw #KISSwebinar

Follow-me and join the conversation

Why A/B Tests Ma#er for Growth

… it’s all about the customers

1 Product Development

2 Early Revenue

3 Online Expansion

Six Stages

4 First Overhaul

5 Defining Audiences

6 Optimizing for Market Changes

WATCH WEBINAR RECORDING NOW

Product Development

1

“Just run an experiment.”

Stage 1: Testing Through Product Development

• Experimental mindset

• Quantitative > Qualitative

• Start with what won’t scale

• User testing

• Customer interviews

• Always create variations

• Practice documenting

hypothesesWireframing Optimizely.com

Early Stages:

Early Customers

2

Stage 2: Optimizing for Early Customers

• Where is your most important conversion?

• Pricing page

• Checkout experience

• Product pages

• What keeps your soon-to-be customer from converting?

• Prime area for impactful A/B tests

What does your pricing page look like?

Where do you acquire users/customers?

Stage 2: Optimizing for Early Customers

An Optimizely experiment:

Pricing page circa 2012

Stage 2: Optimizing for Early Customers

Variation

An Optimizely experiment:

Stage 2: Optimizing for Early Customers

• High-impact test

• Understand audience triggers

• Social proof

• Recommendations

• Promotional offers

• Increase visibility of the page

Variation = 40% increase in Gold subscriptions

An Optimizely experiment:

Online Expansion

3

Where are my customers?

Stage 3: Click Paths and Online Expansion

Analytics can help to determine where visitors are

clicking and when they are dropping off

What is my bounce rate?

Stage 3: Click Paths and Online Expansion

• Why do visitors come to my site?

• What content is being added to my site? At what rate?

• How do you expect visitors will navigate to the final conversion?

Website Navigation:

Stage 3: Click Paths and Online Expansion

Original

An Optimizely Experiment:

Stage 3: Click Paths and Online Expansion

An Optimizely Experiment:

Variation

Stage 3: Click Paths and Online Expansion

Variation = +20% traffic to blog

+25% traffic to help center 3X increase in demo requests

An Optimizely Experiment:

• Maximize inbound traffic and interest on your site

• Choose your conversion goals based on business objectives:

• Educate and inform

• Solicit signups or orders

• Engage and socialize

• Keep funnels and click paths short

Pivotal Redesign

4

Stage 4: A Pivotal Redesign

Stage 4: A Pivotal Redesign

Optimizely website circa 2012

An Optimizely Experiment:

Stage 4: A Pivotal Redesign

Original Redesign

An Optimizely Experiment:

Stage 4: A Pivotal Redesign

Redesign = +28.3% engagement with editor

+4.6% accounts created

• Choose key metrics before redesign process starts

• Measure new design against the current one

• Validates a major investment

• Even a draw can be a win

An Optimizely Experiment:

Segmenting Audiences

5

Stage 5: Segmenting Audiences

• Do you have a marketing program with many channels?

• What types of traffic does your site receive?

• New vs. Returning

• Customers vs. Non-Customers

• Paid vs. Organic

• How do you communicate with your audiences at scale?

Engaging Audiences

Stage 5: Segmenting Audiences

Example: Li"opia

Original SEM Landing Page

Breadcrumb

navigation

Search fields

Suggestions for

similar resorts

Stage 5: Segmenting Audiences

Example: Li"opia

Variation SEM Landing Page

No breadcrumb

navigation

Removed

search fields

No recommended

resorts

Stage 5: Segmenting Audiences

Example: Li"opia

Variation = +23.7% Li" ticket purchases

• Match your visitor’s expectation and intent

• Use analytics tools to follow a click path and identify meaningful audiences

• Use A/B testing to determine your strategy for engaging different audiences

• Messaging and imagery

• Custom landing page experience

New Markets

5

Stage 5: New Markets

• What big initiatives are you tackling this year?

• New marketing campaigns

• Product launch

• Mobile experience

• Personalization

• Internationalization

• What role does your online marketing play?

Optimization and Expansion

Step 6: New Markets

Step 6: New Markets

Product page Cart page

Example: Veggie Tales

Step 6: New Markets

Product page Cart page

Example: Veggie Tales

Step 6: New Markets

Variation = +28.1% Revenue per visitor

• Use information from previous tests to inform your next initiative • Create strong hypotheses

• Use qualitative information to determine what to test and optimize • User testing

• Survey feedback

• Reapply Explore-Refine strategy

Optimization and Expansion

1 Product Development

2 Early Revenue

3 Online Expansion

Today, we’ve covered:

4 First Overhaul

5 Defining Audiences

6 Optimizing for Market Changes

But Wait, There Is More!

Measure All The Results Of Your Test

KISSmetrics A/B Test Report

Measuring the first conversion is just part of the equation

KISSmetrics Funnel Report

Make sure to evaluate the entire funnel when running and A/B Test

KISSmetrics Cohort Report

All A/B tests affect your retention rates

LOG IN WITH GOOGLE

Start Your Free KISSmetrics Trial

Optimize for your customers

… and always be testing

Questions?

Brendan O’Rourke Enterprise Solutions Engineer

Optimizely @optimizely

Dan McGaw Director of Marketing

KISSmetrics @danielmcgaw

THANK YOU