DV 2016: Building a Culture of Testing and Optimization

© 2016 Tealium Inc. All rights reserved. | 1

Building a Culture of Testing and Optimization

Joel Ring, HUB International

2000

2006

2015

2009

Joel RingDirector of Analytics, HUB International

15+ Years in digital analytics

Created and managed the Web Analytics department at bestbuy.com

Sr. Digital Marketing Consultant

working with some of the

biggest brands on the web

Created the Digital

Strategy & Solutions team helping clients

with their analytics and

digital marketing

Manage the Analytics team

focusing on digital

analytics, data sciences, and

testing & optimization

A lot has changed

2000

2006

2015

2009

The great thing about the

internet is you can measure

anything!

We’re ‘gonna fail-fast and

break things!Like Facebook!

The only way to know what will work best is to

test it!

With all the data generated through testing we will be able to understand our customers

like never before and learn

new ways to meet their

needs.

Every minute…• Facebook users share nearly 2.5 million pieces of

content.• Twitter users tweet nearly 300,000 times.• Instagram users post nearly 220,000 new photos.• YouTube users upload 72 hours of new video content.• Email users send over 200 million messages.• Amazon generates over $80,000 in online sales.

“Two-thirds of business are unable to extract value from their information.”

Source: 2015 Information Value Index from PwC and Iron Mountain

If we have access to all

this data, how do we create

an environment

that enables us to derive value

from that data?

Definition of an

effective test and

learn organization look like?

Implemented roles and procedures to create,

execute, and evaluate testsAND

are actively communicating results to drive change.

“The key to great analysisis not math,

it’s communication.”

Keys to creating a test and

learn organizatio

n

Sponsors Ideators

Executors Analyzers

The sponsor helps evangelize the testing organization and helps drive changes based on the results.

The ideators help develop testing ideas. These can be part of the testing team or stakeholders.

This is the group that executes the tests. This will generally encompass multiple departments and can include creative and technical resources.

The analyzers tabulate and communicate the results of the tests. They will be the team members who declare the test “winners”.

• Individuals can, and will, fall into multiple roles

• Include people from multiple departments

• Make sure to include stakeholders

• How many people in these roles?• Forrester estimates 5-15

How do I get the initial buy in?

Demonstrate value using existing

channels that already allow for testing

Email – Display Advertising – Paid Search

How do I get the initial buy in?

Conduct a kick off meeting

Conduct a kick off meeting

• Bring together people who will be either involved in or interested in the execution and effectiveness of your testing program

• Goal of meeting is to educate generate excitement through collaboration

• Kick off agendao Testing 101

• Types of test (A/B, multivariate, etc.)• What makes a good test• What is a testing plan

o Intro to our testing platformo Intro to our testing processo Ideation workshop

• Used the ideation ideas to start our testing prioritization

Develop a testing plan

Develop a testing plan

• Guides your testing prioritization, status and results

• What is in a testing plan?o List each test ideao Conversiono Status (Live, scheduled, complete, removed, on-hold)o Date launchedo Date completedo Results (winner)o Test type (A/B, A/B/C, multivariate)o Test location (site, email, SEM, display advertising)

• Keep in a central location accessible by the team

Test NameTest Cell Test Description Conversion Test Type Test Location Status

Launch Date

End Date Winner

Link to test variations Results

Control: Current quote first position Control Group

Test: The IRS is expecting “many filings” to be in error at some level. No one can just blow off the law, but there will be something of a pass on reporting penalties as long as it can be demonstrated that effort went into compliance. Walter Fagan

Test Group

Control: What’s dominating the Employee Benefits agenda in 2016?Get the answers to the big questions →

Control Group

Test A: What’s dominating the Employee Benefits agenda in 2016?Get expert insights and tools you can use in the year ahead →

Test Group A

Control: current template designTest: Test: plain text format that will appear personalized including 1. Dear [name] 2. Sincerely [regional EB president]

Control: no nurture emailTest: CM nurture series to those who completed ACA nurture Ems

Control: always onTest A: second panel of ACA siteTest B: new contact form to be designed for campaignControl: no button/SM buttons Control

Test: remove SM buttons and add contact us button that goes to contact a rep form TestControl: https://www.hubinternational.com/employee-benefits/contact-us/

Test: modal pop-upControl: Current copy Control

Test:Headline: Optimize your Employee Benefits in 2016

Box 1: ACA Reporting & Verification AuditsBox 2: Cost Management Strategies

Test

Q1 Campaign Site

email

Live 9-Feb TBD

Complete 4-Feb 1-Apr Control ResultsEB Always On HP Interstitial Link clicks A/B Q1 ACA Campaign Site

ACA Panel Jack Quote 1 Share button clicks A/B

TBD TBD

CM nurture to ACA list Increase lead pipeline

A/B email

Live 9-Feb TBD Results

CM nurture email #2 template Email CTR A/B On-Hold

Results

Contact Us button location Button clicks A/B Q1 ACA Campaign Site

ACA Ad words Landing page test Form Completion Rate

A/B/C Ad words

Live 7-Mar

Complete 22-Feb 11-Apr Test

TBD

Results

Contact Rep Form Design - sub pages

Form Completion Rate

A/B Always On Site Live 9-Mar

L1 page CTA Link clicks A/B Q1 L1 page Complete 9-Mar 6-Apr Test Results

Conduct regular testing/analytics meetings

Conduct regular testing/analytics meetings

• Include core team and stakeholders

• Use the testing plan as a guideline for the agendao Review tests in progress

• Determine any status changeso Review tests scheduled to launcho Discuss any new testso Reprioritize based on new tests (if necessary)o Present deep dives from previous meeting questions

• Use the meeting to generate new test and analysis ideas

Standardize the test process and embed it in business as usual

DK Cool Crayon

Test approved Build testAsset(s)

needed?Add to test

plan N

Create asset(s)

Y

Review test Launch test

Wait 7 days

DK Cool Crayon

Test approved Build testAsset(s)

needed? Review test

Create asset(s)

Launch testAdd to test plan

Y

N

Wait 7 days

Why wait 7 days?

• Early results show greater variance.

• Behavior often varies by day of week, want to ensure all days are represented.

DK Cool Crayon

Test approved Build testAsset(s)

needed? Review test

Create asset(s)

Launch testAdd to test plan

Y

N

Wait 7 days

Declare winner?

Y

Review meeting

N

Determine next steps

Update test plan End

Suspend test

Declare Winner

Continue testing

Update test plan

Update test plan

End

Revise test?

N

Y

Review results

Final thoughts

Start Slow

Start Slow

Use basic tests to establish your

process, then grow into complexity

Don’t panic if your tests don’t have significant results

You still learn from tests with no differences

Don’t panic if your tests don’t have significant results

Implemented roles and procedures to create,

execute, and evaluate testsAND

are actively communicating results to drive change.

Questions?

THANKYOU