1 www.onetooneinteractive.com Content Testing & Optimization (CTO) April 29, 2010 One to One Interactive, 2010. All rights reserved.
May 13, 2015
1www.onetooneinteractive.com
Content Testing & Optimization (CTO)April 29, 2010
One to One Interactive, 2010. All rights reserved.
2www.onetooneinteractive.com
Agenda
3www.onetooneinteractive.com
What Is Content Testing & Optimization?
%?
(But how many made it through?)
Sale!
4www.onetooneinteractive.com
The Main Question
“Can we rapidly test new landing page elements -- or
entirely new pages -- to continue to lift our site’s
conversion rate?”
5www.onetooneinteractive.com
Introduction
Who should consider Content Testing & Optimization?
Marketers looking to increase online conversion, revenue and ROI by leveraging the unique opportunities offered by the digital space to improve visitor response
What will we cover today?
The philosophy, planning and implementation that drives a Content Testing and Optimization (CTO) strategy and share a few of One To One’s successes
What will be my takeaways?● The digital marketing strategy behind Content Testing and Optimization● How A/B/N and multivariate test strategies are designed● How CTO can lead to significant lifts in conversion, revenue and ROI● How optimization learnings can improve ongoing marketing strategy
6www.onetooneinteractive.com
One To One’s Approach To Content Testing and Optimization(CTO)
7www.onetooneinteractive.com
Approach To Content Testing and Optimization
Test Formats
Content development (A/B/N testing) Content elements (multivariate optimization testing)
Approach
Conduct test in waves against specific objectives Design tests to always achieve statistical significance with rapid turnaround times (2-6 weeks)
Objectives
Create ROI positive sustainable improvements in content performance against desired actions Build ongoing optimization engine
8www.onetooneinteractive.com
A/B/N Redesign Test
Champion Page Redesign A Redesign B
• The “A/B/…N” approach provides the simplest approach to Content Testing and Optimization
9www.onetooneinteractive.com
A/B/N Redesign Test
Champion Redesign A Redesign B
• Present highly varied, competing page concepts designed to demonstrate as widely different designs as possible
10www.onetooneinteractive.com
A/B/N Redesign Test
Champion Redesign A Redesign B
● The greater potential to affect overall results temporarily supersedes your ability measure the effect of specific elements on the page
11www.onetooneinteractive.com
A/B/N Redesign Test
Champion Redesign A Redesign B
● Don’t get trapped trying to keep any elements “consistent”. Your visitors will only ever see one version!
12www.onetooneinteractive.com
Most popular complaint to A/B/N testing
“But I don’t know why that page won!”
Solution?
Eye Tracking
13www.onetooneinteractive.com
● Understanding which elements on your
winning page generated the most fixation
and longest gaze duration can help provide
the “why” of A/B/N testing
Using Eye Tracking To Identify Areas of Interest
B
A
Area of Interest Fixation Count Fixation Count Duration (ms) Duration (ms)
Image 22.40 14.60 253.99 307.42
Header 32.60 18.20 228.39 267.60
Body 17.60 14.80 292.14 307.98
CTA 39.20 36.40 295.97 247.64
Feature 17.20 30.80 363.31 252.54
BA BA
14www.onetooneinteractive.com
Multivariate Testing (MVT) Completion
● Once an A/B/N test’s winning design is determined, the losing designs are turned off and the winner is declared the new Champion
● Any findings are determined via methods such as eye tracking or link analysis, though the specific “what’s most important on this page?” answer may not be available… yet
● At this point “multivariate” or “MVT” testing of specific page elements on the new Champion is recommended
15www.onetooneinteractive.com
Multivariate Testing (MVT)● Elements within a layout such as imagery, copy, calls to action are varied and
tested against each other in differing combinations, or “recipes”
SecondaryCTAsCTA 1
CTA 2
CTA 3
CTA 4
CTAs
CTA 1
CTA 2
CTA 3
CTA 4
Messages
Message 1
Message 2
Message 3
Message 4
Headline/Subhead
Headline/Subhead 1
Headline/Subhead 2
Headline/Subhead 3
Headline/Subhead 4
Imagery
Image 1
Image 2
Image 3
Image 4
16www.onetooneinteractive.com
Multivariate Testing (MVT)● The best performing recipe (for example, Headline “1” + Image “3” + Call To
Action “2,”) becomes the winner and is rolled out into the next ongoing campaign
SecondaryCTAsCTA 1
CTA 2
CTA 3
CTA 4
CTAs
CTA 1
CTA 2
CTA 3
CTA 4
Messages
Message 1
Message 2
Message 3
Message 4
Headline/Subhead
Headline/Subhead 1
Headline/Subhead 2
Headline/Subhead 3
Headline/Subhead 4
Images
Image 1
Image 2
Image 3
Image 4
17www.onetooneinteractive.com
Multivariate Test Dimensions
Sample elements for test consideration
• Quantity
• Design style
• Text surrounding, “prompting”
CTAs
• Color
• Location
• Size
• Prominence
• Highlighted feature content
Calls To Action
• Size
• Theme
• Location
• Quantity
• Prominence
• Style – lifestyle, product
• Headline within imagery
• Clickability of image
Imagery
• Style – story v. straight
• Size
• Location
• Color
• Length
• Case sensitivity
Headlines
• Style – story v. straight
• Search keyword matching
• Banner ad image matching
• Search / banner message or
offer matching
• Trust message
• Testimonials/Statistics
• Geo/demo message
• Feature information in pop-ups
• Offers
Content/Message
18www.onetooneinteractive.com
Test Wave Approach
Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 61.0%
1.3%
1.5%
1.8%
2.0%
2.3%
2.5%
2.8%
3.0%
Conv
ersi
on R
ate
Base action rate before testing: 1.60%
Impacting Success Metrics
Redesign Tests MVT Optimizations
● Try to project out ongoing lift prior to launching any test, then benchmark against your projections to ensure a continued positive return on investment
Action rate after Wave 6: 2.60%
Lift: 63%
19www.onetooneinteractive.com
● Maybe surprisingly, the sum of the winning MVT elements is often NOT the best possible page
Accounting For Interaction Effect
Imagery
Image 1
Image 2
Image 3
Image 4
Test 1
Theoretical Best Recipe
Image 1 Headline 3 Message 4
CTA 4 Secondary CTA 2
Messages
Message 1
Message 2
Message 3
Message 4
Test 3
Headline/Subhead
Headline/Subhead 1
Headline/Subhead 2
Headline/Subhead 3
Headline/Subhead 4
Test 2
CTAs
CTA 1
CTA 2
CTA 3
CTA 4
Test 4
Secondary CTAs
CTA 1
CTA 2
CTA 3
CTA 4
Test 5
#2 Headline
20www.onetooneinteractive.com
● Tests can be completed more rapidly using the popular “one element at a time” approach… but at the risk of missing better pages
Accounting For Interaction Effect
Sign Up To Hear Our Latest Offers!• Lorem ipsum dolor sit amet, consectetur• Adipisicing elit, sed do eiusmod tempor• Incididunt ut labore et dolore magna• Aliqua. Ut enim ad minim veniam
Recipe AAll Winning Elements Options From Wave 1
Conversion Rate: 16.7%
Find Out If Service Is Available In Your Area!• Lorem ipsum dolor sit amet, consectetur• Adipisicing elit, sed do eiusmod tempor• Incididunt ut labore et dolore magna• Aliqua. Ut enim ad minim veniam
Recipe DAll Winning Element Options except Headline #2
Conversion Rate: 23.5%
21www.onetooneinteractive.com
OTOi’s Content Testing & Optimization Philosophy● Determine test objectives, e.g. Clickthrough From Page vs. Action Rate vs. Conversion Rate
● Start with the ideal optimization process
Optimization Wave: Ideal
Redesign Optimize Optimize Redesign Optimize Optimize
Optimization Wave: Lower-Volume Traffic Source
Redesign Removed Optimize Removed
! The greatest growth in conversion is usually seen with alternating waves of redesign and optimization
● Estimate actionable volume based on projected traffic
● Extend and/or remove tests due to insufficient volume or time
22www.onetooneinteractive.com
Typical testing success metrics include:● Conversion rate (share of visitors completing a sale)● Revenue per visitor● Average shopping cart value
Secondary success metrics include:
● Bounce Rate
● Action Rate (percentage of visitors completing a valued non-sale action)
● Funnel Depth (average number of steps through the purchase funnel taken)
Recommended Success Metrics
23www.onetooneinteractive.com
● Some tests may be rightfully set up to only measure clicks off the test page
● Be careful that success isn’t determined by a point too early in the funnel
Beware Incomplete Fallout
Landing Page A40% Clickthrough
Landing Page B30% Clickthrough
Step 1
Step 1
Reach Step40%
Reach Step30%
Step 2
Step 2
Reach Step34%
Reach Step27%
Step 3
Step 3
Reach Step28%
Reach Step25%
Conversion
Conversion
Convert18%
Convert23%
24www.onetooneinteractive.com
Key Questions
1. How many visits does your tested page receive per month?
2. How long will you continue to award credit to the new, winning page? 12 months?
3. How many conversions or other desired outcomes is your winning page expected to provide over your Champion?
4. What’s the value of each desired outcome -- in terms of average lifetime revenue if available?
Forecasting Incremental Revenue
100,000
1-Mo Visits
1,200,000
Projected Annual Visits
x 12 =
2.00%Winning Design
24,000=
1.80%Champion Design
21,600
x
=
Conversion Rate
Projected Conversions
+2,400Projected Additional Annual
Conversions
$600Estimated Lifetime
Value Per Conversion
xSample Return Estimate
$1,440,000Projected Total Incremental
Revenue
Revenue - CostCost = ROI
25www.onetooneinteractive.com
Case Studies
26www.onetooneinteractive.com
Case Studies: Comcast 2009 Results
11% Lift In Conversion RateWave 1
66% Lift In Conversion RateWave 3
24% Lift In Conversion RateWave 2
Test Champions Test WinnersConversion Rate Metric
36% Lift In Action Rate
(over lowest performer)
Wave 4
27www.onetooneinteractive.com
Case Study: 2009 Wave 3 Multivariate Test● A 2009 Comcast multivariate study focused on testing:
1. Image
2. Headline /
3. Call To Action
● The test audience was all visitors who had clicked on a Comcast Paid Search ad
● All versions of the new page’s design were tested against the prior Champion page
28www.onetooneinteractive.com
Case Study: Comcast 2009 Wave 3 Results
● The redesign’s template seemed to significantly beat the Champion no matter which combination of Image and Headline/Copy were used
● Among the winning Images, those with the people facing toward the headline and next step button seemed to perform the best (see above)
Headline/Copy Results% Lift In Conversion Over Champion
“Married Man”
“Orange Woman”
Image Results% Lift In Conversion Over Champion
29www.onetooneinteractive.com
“Champion” Landing PageTest Hypothesis:“Versioning a banner ad’s landing page to match the design of the banners will increase clicks to the next page”
Methodology:Version “Champion” elements against variations designed to match the banner ad creative more closely
Elements Matched:
● Primary Headline Viewed
● Primary Image
● Call To Action Colors
2
1
3
Case Study: Comcast Business Class
30www.onetooneinteractive.com
“Champion” Elements“Small vs. Big”
Elements“Dream Big” Elements “Compare” Elements
He
ad
line
Pri
ma
ry Im
ag
eC
all
To A
cti
on
Co
lors
Case Study:Variable Elements
31www.onetooneinteractive.com
winner
Question: “Does matching landing page content to the campaign creative shown make a difference in clickthrough?”
Answer: For all creatives tested: yes
Case Study: Winning Pages
Creative Clicked OnMixed Landing Pages
Fully Matching Champion
32www.onetooneinteractive.com
Building Your Library
33www.onetooneinteractive.com
● Firms should view Content Testing & Optimization as an ongoing process which yields an ongoing lift in conversion
● Never forget that you’re looking to build out a library of best practices to inform future designs
Building Your Library
34www.onetooneinteractive.com
Past studies have produce insights which have radically changed the prior approach to content, as ongoing testing has driven substantial lift in conversion via such findings such as:
Images perform far more successfully on the right side of the page, potentially due to the natural “inverse C” path of eye tracking
Building Your Library
35www.onetooneinteractive.com
Past studies have produce insights which have radically changed the prior approach to content, as ongoing testing has driven substantial lift in conversion via such findings such as:
For consumer products, images of people are far more successful in driving conversions than images of products – especially of faces
Building Your Library
36www.onetooneinteractive.com
Click To Sign Up Now!
Past studies have produce insights which have radically changed the prior approach to content, as ongoing testing has driven substantial lift in conversion via such findings such as:
The newer an offering, the less successful hard-sell messaging such as “Sign Up Now!” is to drive page interaction
Building Your Library
Check Out Products And Features!
37www.onetooneinteractive.com
Past studies have produce insights which have radically changed the prior approach to content, as ongoing testing has driven substantial lift in conversion via such findings such as:
Images of happy consumers looking in the direction of Calls To Action rather than out of the screen generally generate higher conversion
Building Your Library
Offer! Offer!
38www.onetooneinteractive.com
Past studies have produce insights which have radically changed the prior approach to content, as ongoing testing has driven substantial lift in conversion via such findings such as:
Matching landing page messaging to the copy in a paid search advertisement clicked is far more important when the copy includes an offer
Building Your Library
39www.onetooneinteractive.com
In Conclusion
● A Content Testing & Optimization strategy effectively leverages the unique opportunities offered by the digital marketplace and provides a foundation of best practices for ongoing content development
● By engaging in the ongoing process of content testing, firms can rapidly implement new content to generate increased response, resulting in steady lifts in conversion, revenue and return on investment (ROI)
● GIVE IT A TRY!