Always Be Testing - Learn from Every A/B Test (Hiten Shah)

Always Be Testing Learn from every thing you do!

Hiten [email protected]

@hnshah #FILIVE

Product, Design, Engineering, Marketing, Growth, Sales

What do you do?

How do you make all the things

be!er?!

Figuring this out is your job.

A/b testing

Only  1  out  of  5  tests  win.

4  out  of  5  tests  cost  you  money.

So, why experiment if so many tests fail?

Go Where Customers Are!Long, slow, SaaS ramp of death from 2006…

Go Where Customers Are!Now…

Go Where Customers Are!Growth can be sped up!

Increase Hello Bar’s installation rate

Improve the KISSmetrics features page

Increase signups from Crazy Egg’s homepage

Grow Crazy Egg’s revenue

Results from 3 months of focused growth effort

Results: KISSmetrics signups over time

Simple 4-step experimentation process

Data Figure out what to improve

Hypothesis Make an educated guess

Experiment Test your guess

Act Decide what’s next

Data Figure out what to improve

Data doesn’t make decisions for you.

Data informs your decision making.

Data helps you identify the most painful problems people have. (not the ones that bother you the most)

Find problems worth solving by starting with quantitative data.

When in doubt, look at your top pages. Usually the homepage.

A simple framework for choosing what to improve

Find pages with high traffic and high bounce rate

HTTP://GOOGLE.COM/ANALYTICS

Find pages with high traffic and low conversion rate

HTTP://UNBOUNCE.COM

Find sources with high traffic and low conversion rate

HTTP://HUBSPOT.COM

Find opportunities based on user paths

HTTP://KISS.LY/KMPATH

Increase Conversions By A/B Testing

Do a Happy Dance But Watch it Closely

Get More Traffic So You Can Start A/B Testing

Get More Traffic So You Can Do a Happy Dance

High Traffic

Low Traffic

High ConversionsLow Conversions

How To Figure Out What To Improve

What percentage of your

customers or users have

adopted each feature? Intercom on Product Management

When you’ve already got usage, analyze it.

Where do you have the most visitors, users, customers, emails, or notifications?

More volume helps you test faster.

๏ Website Traffic / App Installs

๏ Welcome Emails

๏ Web / Mobile Onboarding

๏ Email Digests

๏ Triggered Notifications

๏ Cart / Funnel Abandonment

๏ Dormant / Churned users

Common areas with large volume and high impact potential

Find out why by ge!ing qualitative data.

A few qualitative tools

HTTP://USERTESTING.COM HTTP://SURVEYMONKEY.COM

A few more qualitative tools

HTTP://QUALAROO.COM HTTP://OLARK.COM

Use exit surveys to learn why people bounce

HTTP://QUALAROO.COM

Use post-conversion surveys to understand why

HTTP://QUALAROO.COM

Discover if your web pages are effective at their goal

HTTP://QUALAROO.COM

What frustrates people?

EMAIL

Find people who don’t do things and email them

EMAIL + SURVEYMONKEY

User Research gives you a headstart

Hypothesis Make an educated guess

Ideas Designs Solutions

Assumptions(A guess about the best way to solve a problem)}

Everything can start with a testable hypothesis.

Before you create anything, make an educated guess.

Use simple equations to assess potential impact of each idea

DAN MCKINLEY HTTP://KISS.LY/DANETSY

visits  x  conversion  rate  x  order  value  x  li4  =  ?

users  x  order  value  x  resurrec7on  rate  =  ?

ESTIMATED

ESTIMATED

MODELED

MODELED

Assess the potential impact,

difficulty and priority of

each test idea you have.

Prioritize your tests

Experiment Test your guess

Test your hypothesis by starting an experiment.

Not bu!on colors…

Don’t copy “best practices” blindly

Test specific bu!on copy

Sensitivity tests help you

learn what your visitors care

about and what they don’t.

Run sensitivity tests on your web pages

Remove graphical elements and see what happens

Does social proof and logos ma!er for your visitors?

Remove social proof and learn what happens

Act Decide what’s next

You’ve probably seen an A/B testing report before

1. Your starting hypothesis.

2. Dates for when you ran the test.

3. Screenshots of control and all variations used during the test.

4. Expected change in conversion.

5. Probability of change.

6. Screenshot of the raw data used in your analysis.

7. A screenshot to the A/B Test Report in KISSmetrics.

8. The decision you've made from the test.

9. What you learned from the test.

Document every A/B test and create a playbook

What learnings do we want to carry over from the past?

Helps you avoid unknowingly decreasing your conversions.

✓ Keep the footer links

✓ Autofocus on the first form field

✓ Do not include social proof

✓ Link to the features page

✓ Use “Log in” copy on the bu"on

Create experiment checklists based on learnings

If you’re not testing, you’re not learning.

☀

Always have tests running, no ma!er how small.

⌾

If you don't have a process, you won't continuously improve.

∞

Work on things that ma!er. Make those things be!er. Be happier ☺

Thank Y0u =) Got questions? Need help? Contact me any time!

Hiten [email protected]