How Do You Know… Designing Multi-Channel Campaign Tests That Optimize Your Budget Jim DeLash November 18, 2014
Jul 26, 2015
How Do You Know…Designing Multi-Channel
Campaign Tests That Optimize Your Budget
Jim DeLashNovember 18, 2014
Moving registration to the right side form increased the site’s Conversion Rate by
24.6%
Page layout
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan
Establishing hypotheses
Statements to be proven
All tests must support a hypothesis
Extrapolating the results to other brands
Low unaided recall scores for advertising creative will require
additional frequency….
Establishing hypotheses
Low unaided recall scores for advertising creative will require
additional frequency….
Establishing hypotheses
Low unaided recall scores for advertising creative will require
additional frequency….
BUT HOW MUCH MORE?
Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More direct mail and e-mail
Different levels of frequency
Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
How many times must th
e test
be run to
ensure enterprise
application
The best design is not always the best approach
Do not let design conflate the objective of an email or message
Every unnecessary piece of content is waste and reduces your chances of getting a click
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan
What will we learn from this test….
1. Best frequency by Specialty and Quintile2. One channel vs. Two channels (Segments 5,7,9)3. Total frequency: 10 vs. 12 vs. 14
Version B, the one with the ‘Save Now’ button, increased opt-ins for logged-in users by 333% at a 99.9% confidence level.
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan
There are only two ways to get fired from Harrah’s:
stealing from the company,
or
failing to include a proper control group in your business experiment
Control Groups…
Not a ‘holdout’ group
Must include all relevant variables
Large enough to be statistically
valid
Creating a Measurement Plan
Measurement Milestones…
Track progress against goal
Forecast test impact early
Connect KPIs with Rx impact
Creating a Measurement Plan
Headline
copy> 30%
Call to action
> 80%
Image
s>
100%
Video> 80%
Use of
Color
20-50%
Page layout> 25%
MICRO TESTS
Subject Line
Version B, the subject line broken into three consumable sections, increased opens by 73.7%.
Version B, the ‘Get My Free Paper’ Call to Action, increased downloads by 25% at a 95% confidence
level.
Call to Action
Version B, the ambiguous subject line, increased opens by 112% at a 95%
confidence rate.
Subject Line
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan