Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect your Experiment Process

#opticon2015

Hypothesize, Analyze, Optimize:

Using Hidden Insights to Perfect your Experiment Process

Abby Lokshin Mobile Product Manager, Rue La La

[email protected]

Abel Ramos Product Manager, Ecommerce Princess Cruises

@abelramos58 [email protected]

#opticon2015

Iteration – The Key to your Testing Success Abel Ramos Product Manager, Ecommerce Princess Cruises

@abelramos58 [email protected]

“Every strike brings me closer to the next homerun.” – Babe Ruth

“We Failed”

Our Hypothesis

The next greatest search UI ……or so we thought.

Oops, What Happened?

Let the fun begin. Where do we start with these types of results? Time to regroup.

Introducing the Next Greatest Search UI…… no, really

OK, Someone Want to Interpret These Results?

Could we be getting closer?

“Hey, what about our mobile traffic?”

– Joseph Espana, Web Designer Princess Cruises

Understanding your Business Segments

How is your Mobile Traffic affecting your results ?

“Life is a series of experiences, each one of which makes us

bigger, even though sometimes it is hard to realize this.”

– Henry Ford

Know your Audience

Segment your website traffic for your tests.

What are the Obstacles Preventing Testing Success? How are your promotional efforts affecting your testing?

Review all of your Data Points Constantly

Use all available data resources to help with your testing.

What Does the Future Hold for Testing at Princess Cruises?

Are your testing goals inline with larger company initiatives?

“We Won” Testing is an iterative process and it takes time.

#opticon2015

Thank you!

Abel Ramos Product Manager, Ecommerce Princess Cruises

@abelramos58 [email protected]

#opticon2015

What should we test (again)?

The testing lifecycle

Abby Lokshin Mobile Product Manager, Rue La La

[email protected]

#opticon2015

RUE LA LA

LEADING E-COMMERCE LIFESTYLE DESTINATION

15M+ MEMBERS

NEW BOUTIQUES DAILY

EXPIRE IN 2-3 DAYS

AN OVERVIEW

RUE LA LA USER FLOW

Kicking off the testing process

TEST IDEATION WHAT HAPPENS

MULTIPLE STAKEHOLDERS

UNIQUE KPIS

DISORGANIZED BACKLOG

Determine common themes to define primary goals

TESTING BUCKETS GROUPED BY PRIMARY GOALS

Prioritization

TEST PRIORITIZATION 60 SECOND BUSINESS CASE

Coordination Effort Creative Effort

Initial Development

EffortRollout Effort

Strategic Alignment

Revenue Impact

Customer Experience

Value

Lowers Cost PM Priority

RANK: HIGH, MEDIUM, LOW

SCORE: 100 PTS. TOTAL

+

EFFORT

BUSINESS VALUE

TEST PRIORITIZATION 60 SECOND BUSINESS CASE

An organized backlog leads to a stronger sprint

EXPIRING FREE SHIPPING TEST

TESTING GOALS:•  USER SEES QUICK TIP

•  USER ADDS TO CART

•  USER CHECKS OUT

•  PRIMARY GOAL: REVENUE

RESULTS: 4.2%+ CONVERSION

NEXT STEPS: ROLL OUT!

QUICK TIP ON PRODUCT DETAIL

LEVEL OF EFFORT: LOW

BUSINESS VALUE: HIGH

What if the test is inconclusive?

MEN’S SUIT SHOP MIX & MATCH BOUTIQUE

TESTING GOALS:•  USER BUILDS A SUIT

•  USER ADDS TO CART

•  USER CHECKS OUT

•  PRIMARY GOAL: REVENUE

RESULTS: NO CLEAR WINNER

What impacted the primary goal?

MEN’S SUIT SHOP SEVERAL ISSUES

TOO MANY STEPS

CONFUSING UI

AUDIENCE TOO SMALL

STATISTICALLY INSIGNIFICANT

MEN’S SUIT SHOP NEXT STEPS

RE-RUN, WITH MODIFICATIONS:•  FEMALE AUDIENCE

•  FEWER STEPS

•  LONGER TIME FRAME

•  MORE BOUTIQUES

What if converting isn’t the problem?

CATEGORY SORT TEST SHOPPING PREFERENCES

TESTING GOALS:•  USER INTERACTS WITH

SORTING

•  USER CHECKS OUT

•  PRIMARY GOAL: REVENUE

RESULTS: NO CLEAR WINNER

VAR. A VAR. B

CATEGORY SORT TEST WHAT HAPPENED

NO IMPACT ON THE WHOLE

MANY CATEGORY COMBINATIONS

CAN WE FIND ANY TRENDS?

VAR. A VAR. B

CATEGORY SORT TEST INTERESTING TRENDS

TREND IN VARIATION A: FEMALES RANK WOMEN’S FIRST

•  4%+ IN $/VISIT

TREND IN VARIATION B: MALES RANK MEN’S FIRST

•  31%+ IN $/VISIT

•  32%+ IN PURCHASES

VAR. A VAR. B

CATEGORY SORT TEST INTERESTING TRENDS

NEW TESTS:•  DEFAULT MEN TO MEN’S

SECTION

•  RE-OPEN TO LAST VIEWED CATEGORY

VAR. A VAR. B

REMEMBER… TESTING IS NEVER ONE AND DONE

CONSIDER THE DATA

COLLECT FEEDBACK

REMEMBER THE PRIMARY GOAL

KEEP ITERATING

#opticon2015

Abby Lokshin Mobile Product Manager, Rue La La

Thank you!

[email protected]

#opticon2015

Questions