How one company implements an entire testing methodology every day The Nuts and Bolts Ryan Hutchings Director of Marketing VacationRoost Daniel Burstein Director of Editorial Content MECLABS
The Nuts and Bolts: How one company implements an entire testing methodology every day
May 13, 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
How one company implements an entire testing methodology every day
The Nuts and Bolts
Ryan HutchingsDirector of MarketingVacationRoost
Daniel BursteinDirector of Editorial ContentMECLABS
Session speaker
2
Ryan Hutchings
Director of Marketing
VacationRoost
Hutchings has more than 10 years of online marketing experience. He leads the marketing team for VacationRoost, a $120 million private company in the travel industry. From advertising to analytics, he organizes a comprehensive and robust online marketing strategy focused on tracking and optimization. His specialties include SEO, PPC, testing, conversion rate optimization, advanced data modeling, analytics and every technical aspect related to online marketing.
Hutchings is a Google AdWords certified professional, SEOmoz Pro member, nationally recognized industry speaker and participant of top online marketing communities. He also help found the SLCSEM organization, which is Utah’s largest
nonprofit group of online marketing professionals.
@HutchSEO
About VacationRoost
About VacationRoost
• 25 websites • Always running 2 to 3 tests • 100+ tests run across landing pages, homepages,
lead forms and shopping carts• Goal of 10% incremental lift in total revenue
generated by testing
Fundamental Question:
How can marketers systematically run tests to achieve consistent lifts?
and hundreds of other tests….
An overall testing process …
Build
… to support an iterative optimization cycle
Collecting data
Planning experiments
Preparing treatments
Conducting tests
Analyzing results
The dual testing track
A/B split test• Internal system with server-side
splitting, custom variables• Full manual reporting –
spreadsheets and database reports
Larg
e te
st
Search results
Cart
Products
Test
ing
Landing page optimization• Third-party software• Internal analysis determines
what to testSmal
l tes
t PPC and SEO landing pages
Headlines/CTAs
Images
The dual testing track
A/B split test• Internal system with server-side
splitting, custom variables• Full manual reporting –
spreadsheets and database reports
Larg
e te
st
Search results
Cart
Products
Test
ing
Landing page optimization• Third-party software• Internal analysis determines
what to testSmal
l tes
t PPC and SEO landing pages
Headlines/CTAs
Images
How to test small
Decide Identify Hypothesize Design Analyze
Decide what to test
90%Replace “our recommendations” section with content snippet for value proposition banner usage
Restyle “most popular destinations” change hierarchy 40%
How to test small
Decide Identify Hypothesize Design Analyze
Identify the target conversion goal
What’s a good goal for bounce rates?
> 50%
40%-50% 30%-40%
< 30%
What’s a good goal for CTR to the next page or step?
> 60%
50%-60%30%-50%
< 30%
How to test small
Decide Identify Hypothesize Design Analyze
Create a hypothesis
Small Test: PPC landing pages
17
Small Test: PPC landing pages
18
Small Test: PPC landing pages
19
How can we reduce friction and
anxiety on this page?
How to test small
Decide Identify Hypothesize Design Analyze
Wireframe, design and launch treatment
Small Test: PPC landing pages
21
Small Test: PPC landing pages
22
Small Test: PPC landing pages
23
Small Test: PPC landing pages
24
How to test small
Decide Identify Hypothesize Design Analyze
Analyze results and calculate ROI
Small Test: PPC landing pages
26
427% increase in CTR
Small test validation tool
95.2%
ROI template
Let’s take a closer look…
ROI templateAverage ticket
JanFebMar.AprMayJunJul.AugSeptOctNov.Dec
Close rate
JanFebMar.AprMayJunJul.AugSeptOctNov.Dec
Lead conversion
JanFebMar.AprMayJunJul.AugSeptOctNov.Dec
Contribution% of sales
ROI template
Incremental IncreasesNew Bounce RateLeads Sales $ Contribution $
53$ 40,307$ 3,225
Incremental IncreasesNew Bounce RateLeads Sales $ Contribution $
20%7
5,246 $420 $
At 10%
At 30%
158$ 119,256$ 9,541
25 %2 d
1,749 $140 $
How do you get the resources to run large tests?
The dual testing track
A/B split test• Internal system with server-side
splitting, custom variables• Full manual reporting –
spreadsheets and database reports
Larg
e te
st
Search results
Cart
Products
Test
ing
Landing page optimization• Third-party software• Internal analysis determines
what to testSmal
l tes
t PPC and SEO landing pages
Headlines/CTAs
Images
How to test big
Decide Collaborate Identify Create Design Analyze
Decide on an element to optimize
Looking at the same elements as the small
tests.
The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Purchase
3.5%
1.7%
“Rising tide”A/B
(large test)
Single page LPO(small test)
40%
1.7%
Purchase
Determining a “good” conversion rate
Source: MarketingSherpa.com/ecommerce
How to test big
Decide Collaborate Identify Create Design Analyze
Work with IT to define variables or events
Large Testing: Variables are the key
38
ConditionsSegment your users and/or their visits to single or multi-visit conditions.
inlineLeadForm1mrSearchPage1NotestsimpleModalLeadFormsimpleModalLeadForm4
controltest
How to test big
Decide Collaborate Identify Create Design Analyze
Identify target conversion goal for control
The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Book now, Web lead, interactions/events
Checkout, add “cross sell”
Book, fill out info
Bounce rates, CTR
Purchase
“Rising tide”A/B
(large test)
Single page LPO(small test)
The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Book Now, web lead, interactions/events
Checkout, add “cross sell”
Book, fill out info
Bounce rates, CTR
Purchase
Book Now, web lead, interactions/events
“Rising tide”A/B
(large test)
Single page LPO(small test)
10% - 30% minimum targeted gains
Conversion goal results
How to test big
Decide Collaborate Identify Create Design Analyze
Create hypothesis and set up validation sheet
Manual testing database
Number of Days
40
Control Treatment
Large Test: Lead forms 1
Security seals
Control
Fail – no lift.
Large Test: Lead forms 1
Treatment
Control Treatment
Large Test: Lead forms 2
Shorter form
Control
Fail – no lift.
Large Test: Lead forms 2
Treatment
Control Treatment
Large Test: Lead forms 3
Option to call
Control Treatment
Fail – no lift.
Large Test: Lead forms 3
How to test big
Decide Collaborate Identify Create Design Analyze
Wireframe, design and launch treatment
Large Test: Lead forms
Large Test: Lead forms
Large Test: Lead forms
19% Lift
How to test big
Decide Collaborate Identify Create Design Analyze
Analyze results and calculate ROI
Manual testing database
Testing roadmap – it’s really:
Search Results
Property Details
Cart
Billing info
Booking
ROI template
Same as a small test.
Setting up your own programmatic testing
1
2
3
4
5
Custom variable tracking system, spreadsheet templates are vital building blocks
Dedicate a person and tie performance to testing
“Top of the funnel” to the “bottom conversion” may have too much noise in
between – keep it simple and isolated
Build a reporting system that ties front end and back end together
Get buy-in from the management team
Questions?
Thank you
Ryan Hutchings
Marketing Director
VacationRoost
@HutchSEO
Daniel Burstein
Director of Editorial Content
MECLABS
@DanielBurstein
Black & White Headshot
Black & White Headshot
Related Documents
