A Complete Walkthrough of Multivariate Testing

© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.© 2009 WEBTRENDS INC. ALL RIGHTS RESERVED.

Webinar #3: March 18, 2010

Step-by-step Overview of Multivariate Testing

Frans KeylardDirector, Optimization Services

Bob Garcia Director, Product Marketing

© 2010 WEBTRENDS INC. ALL RIGHTS RESERVED.

Agenda

Introduction

Multivariate Testing Process

1. Strategy

2. Test Selection

3. Test Planning

4. Test Setup

5. Test Activation

6. Reporting & Analysis

Wrap-up

Sample Site


Rationale - Optimization as #1 Priority

Optimization is a quadruple win:

1. FAST TIME TO VALUE - produces results quickly

2. SELF FUNDING - pays for itself many times over

3. RELEVANCE - improves user experience and site “stickiness”

4. BROADER ROMI - maximizes conversions (success events)

Your audiences will reward you:

1. Loyalty

2. Customer satisfaction

3. Share of wallet

4. All the big metrics – revenue, conversions, leads, etc.


What Can be Optimized?pa

ges

Landing pages

Microsites

Sales funnels

Shopping cart pages

Credit card pages

E-tail templates

Forms

Interactive pages & templates

Web-based UIs

cont

ent

Copy

Images

Offers

Pricing

Promotions

Rich media

Headlines

Quotes

Graphics

Colors

Rich media

traf

fic Customer segments

Search traffic

Display traffic

Affiliate traffic

New & repeat visitors

Traffic by geography

Weekend vs. weekday

Demographic

Behavioral

Site types that can benefit:1. LEAD GENERATION Boost campaign performance

2. SELF-SERVICE Improve performance of processes

3. ECOMMERCE Maximize revenue & optimize funnels

4. CONTENT Improve relevance to drive advertising revenue

Examples of opportunities:


Site Optimization Continuum

Segment Testing General Testing Targeting Personalization

ROMI ROMI ROMI ROMI


Requirements for Success

Test PlanningTest PlanningStrategy Test

SelectionTest

PlanningTest Setup

Test Activation

Reporting & Analysis

PROCESS

Project Management Strategy/Planning Analysis & Reporting

Test development Implementation QA

Creative strategy Asset generation Usability

Optimization Analyst Creative Specialist Web Developer

PEOPLE

TECHNOLOGY


THE MULTIVARIATE TEST PROCESSUsing Azul Travel as a Sample Site

Sample Site


Agenda

Introduction

Multivariate Testing Process

1. Strategy

2. Test Selection

3. Test Planning

4. Test Setup

5. Test Activation

6. Reporting & Analysis

Wrap-up


Azul Travel Business Objective

Sample Site

Drive registrations for MyAzul Membership Program that will provide users with

travel incentives and help marketers build customer loyalty


Strategy

Evaluate - General opportunity assessment

• Use web analytics to ID areas of opportunity

• Understand your marketing investments

• Consider the business value of opportunities

Strategy | Test Selection | Test Planning | Test Setup | Test Activation | Reporting & Analysis

PPC, Social, Email, Display, etc.


Test Selection

Picking a page to optimize. Things to consider:

• Simple and easily measurable conversion objectives

• Clear marketing goals and shared understanding of importance

• Look to analytics to validate selection



Test Planning

How and what to test

• Test design options

• Evaluate test ideas & define page elements to test

• Create test recipe matrix & creative assets


A/B/n Multivariate Testing


Test Planning | Page Elements to Test



Test Planning | Recipe Matrix



Test Setup

Execute your test recipe matrix

• Set-up test in platform

• Implement tags

• 1 page/table set-up from RC

• 1-time

• QA

• Preview content

• Ensure conversion points are being recorded



Test Setup | Webtrends Optimize


Site Optimization Platform

Test Campaign Setup

Multivariate Test Setup

Segment Definition & Use Content Management


Test Setup | Preview Experiments


Preview Experiments 1 of 16

2 of 16

3 of 16

8 of 16

16 experiments using Webtrends Optimize Fractional Factorial approach versus 256 experiments using a Full Factorial Approach


Test Activation


Go!


Reporting & Analysis



Site Optimization Continuum

Segment Testing General Testing Targeting Personalization

ROMI ROMI ROMI ROMI


Bob Garcia(e) [email protected](t) @bob_garcia

Frans Keylard(e) [email protected]

blog: For more on site optimization check out Billy’s Bloghttp://blogs.webtrends.com/optimization/

FOR MORE INFO:

http://blogs.webtrends.com/optimization/

A Complete Walkthrough of Multivariate Testing

Business

test setupexecute

test selectionpicking

test development

test recipe matrixset

test planning page elements

analysis reporting

rationale optimization

site types