How to Benchmark Your Online Customer Experience Against Competition

1

How to Benchmark Your Online Customer

Experience Against Competition

2

Conducting Remote Unmoderated Usability Studies: Part 1

Ann Rochanayon

Sr. Director UX/CX Research

UserZoom

[email protected]

Speaker

Meet the Speakers

Elijah Degen

VP, Head of Digital Marketing Practice

Interakt

[email protected]

Speaker

3

The #1 Platform for Agile Usability Testing & User Experience Analytics

Our all-in-one enterprise solution makes research & testing

cost-effective, agile and easy

4

• Chat box is available if you have any questions

• There will be time for Q&A at the end

• We will be recording the webinar for future viewing

• All attendees will receive a copy of the slides/recording

Quick Housekeeping

Continue the discussion using #uzwebinar

Continue the discussion using: #uzwebinar

5

UserZoom Retail Benchmark | December 2014

• How CE/UX Benchmarks Add Value to Your Brand

• About Remote Unmoderated User Testing

• Case Study: Retail Competitive Benchmark Study

Study Methodology

Participant Profile

Summary

Brand, Conversion, and NPS Findings

Contents

6

How can CE/UX benchmarks add to value to your brand?

7

Perceived

CredibilityProfitability

Intent to

Return

Intent to

Purchase

User

Satisfaction

Word of

Mouth

A Positive

Experience

A Negative

Experience

Expectations User Experience

Online User experience – the defining momentThe content and quality of each individual visitor’s interaction — the user experience — collectively defines your digital brand, communicates your value proposition, and creates the foundation for future brand interactions — positive and negative.

8

• Traffic, user demographics, media spend, etc…

• Customer Experience & User Experience have varying

definitions between companies

• Usability benchmarks are generally custom

• Competitors vary depending on who you ask

• Info from UX benchmarking is so actionable, it is rarely

shared

Lots of macro benchmarks are out there…

Why are there few benchmarks for CE/UX?

9

Where can UX benchmarking impact decisions?

Area of Discovery Example value

Best Practices in service/UX design Leverage competitors findings and savecosts

Service/UX Design Decisions Justify investments/costs via VOCOr test prototypes

Know your customer Know what keeps them from converting

Quantify to customer feedback Give hard numbers to show impact.

Example Question:For those who considered your brand, what are the top 5 reasons why not converting?

Online Brand Impact Help brand marketers understand the brand's impact in online experience

Eg Brand Perception can impact conversion as much as online task completion.

10

Web design components

• Navigation, content, site performance, interactive

elements.

• Example Metrics: Task Completion, Ease of Use, Time

on Task, etc.

Brand components

• Voice of Customer, Brand affinity/likeability, Attitudes

toward Brand

• Example Metrics: NPS, CSAT, Brand Attributes Matrices

What can be measured?

11

What is needed to benchmark properly?

Area of Discovery Example value

Correct tools • cross device• website independent• In-app• Video & audio & clicks

Correct Audiences • Target Audience• Counter Audiences

eg Lost Opportunity Audience, Control Group

Correct Proctor / Partner • Impartial = Let the numbers speak for themselves

• Specialists w/ Broad Experience

Correct Type of Testing • Moderated In-Person (powerful exploratory / directional tool, but EXPENSIVE)

• Unmoderated Remote Testing(great deep dive analysis and very cost scalable globally)

12


Remote Unmoderated User Testing

• ‘In-the-wild’ testing (natural context)

• Geographic representation

• No moderation needed

• Software collects data, stores securely in the cloud,

accessible real time

•Test any web-based UI, live or prototype

• Output: Usability + Behavior + Video & Audio +

Satisfaction+NPS+SurveyFeedback

• Fast data collection

• Cost efficient

13


• Goals

• Determine which retail site presents the best tablet shopping experience

• Assess the impact of the tablet shopping experience on brand perception, Net

Promoter Score and conversion

• Sites evaluated

• Participants tested one of four sites in Nov - Dec 2014 on a tablet

• N=202

• Ages 18+

• Shopped for clothing and accessories online at least once per year

Methodology

14


• Between-subjects design

• N=202 (~50 per site)

Methodology: Study Design

Screener

Introductory Questions

Macy’s1. Find a jacket

2. Return policy

3. Checkout

Nordstrom1. Find a jacket

2. Return policy

3. Checkout

Urban Outfitter1. Find a jacket

2. Return policy

3. Checkout

Lands’ End1. Find a jacket

2. Return policy

3. Checkout

Final Questions

15


Participant Profile: Purchase experience

Please indicate your gender.

84%

16%

Female

Male

5%

42%

17%

14%

21%

18 - 25

26 - 35

36 - 45

46 - 55

Over55

Please indicate your age range.

15%

39%

38%

8%

Weekly

Monthly

Several times per year

Once per year

Which of the following best describes how frequently you

purchase clothes and accessories online?

N=202

16


Participant Profile: Sites Browse / Purchased

• Kohl’s and Macy’s were the top two sites participants visited to purchase

clothing and accessoriesWhich of the following websites have you browsed for clothes and accessories using your tablet? / Which of the

following websites have you purchased clothes and accessories using your tablet? [Check all that apply]

20%

14%

14%

34%

13%

62%

21%

14%

52%

26%

10%

11%

6%

7%

18%

5%

38%

9%

6%

37%

10%

5%

Aeropostle

Banana Republic

Dillard’s

GAP

J. Crew

Kohl’s

L.L.Bean

Land’sEnd

Macy’s

Nordstrom

Urban OutfittersBrowsed Purchased N=202

17

Summary

18


To assess the overall user experience ranking of the sites, UserZoom captured usability

metrics for three key tasks on each tablet site. Participants were asked to find an item

meeting specific criteria, find return policy information and asked to evaluate the checkout

process. The usability metrics captured include key performance indicators such as task

effectiveness, ease of use, and clarity of information. The rankings were derived according

to how each site performed on each measure.

User Experience Rankings

79

72

70

55

1

2

3

4

User Experience

Score*

*Score based out of 100 possible points

19


Summary: Task Effectiveness

• Participants were most successful in completing the study tasks on the Lands’

End site

• Urban Outfitters had the lowest overall task success rates

96% 98%

86%88% 90%94%

79%

69%

85%

98% 98% 98%

Task 1: Find a Jacket Task 2: Return Policy Task 3: Checkout

Macy's N=50 Nordstrom N=49 Urban Outfitters N=52 Lands' End N=51

20


Summary: Task Ease of Use

• Lands’ End performed relatively well overall on task ease of use

• Urban Outfitters had the low ease of use ratings, consistent with the lower task

success rates

72%

48%

68%65% 67%

78%

37% 38%

62%

69%

61%

86%

Task 1: Find a Jacket Task 2: Return Policy Task 3: Checkout

Macy's N=50 Nordstrom N=49 Urban Outfitters N=52 Lands' End N=51

21


• Lands’ End led all sites with

the highest ratings in the

usability metrics

• Nearly all participants

succeeded in all three tasks,

more than any other site

• Lands’ End also had relatively

high ease of use ratings and

was rated highest on post-

study brand perception

• Participants found the Lands’

End site fast, simple, easy to

use and understand; Lands’

End also provided clear return

policy information

Summary: Lands’ End

• Although Lands’ End rated highest on task success and fewest issues encountered

for the Return Policy task, one area of improvement to consider would be cleaning up

the look and feel / organization of the Return policy page

22


Summary: Lands’ End

• Study participants appreciated the fast, simple, easy to navigation with clear

information available on Lands’ End

Simple and easy to use

“Very easy to navigate”

“Overall a very easy pleasant experience”

“Very easy, big selection”

“A lot of ways to narrow your search. Easy

checkout process”

“Everything was so simple and easy to do.”

Clear Information, Easy to Understand

“Easy to navigate site. Information clear and easy

to understand. No issues.”

“The site was very easy to navigate, all necessary

information was given, pictures and descriptions

of merchandise was clear”

“Very easy to maneuver around, easy to

understand return policy”

Fast

“Easy and fast to find what I was looking for”

“Your website does not lag at all which made it an

enjoyable experience.”

23


• Nordstrom edged out Macy’s for second place in the user experience rankings

• A vast majority of Nordstrom participants succeeded in all tasks with 88% or more succeeding

• Task ease of use were fair with 65% of the participants or higher finding the site easy to use

while performing the assigned tasks

• Participants overall found the Nordstrom site easy to navigate and several mentioned the

helpfulness of the filters

• However, a sizeable percentage of participants encountered technical issues on the Nordstrom

site with 27% reporting site slowness and 24% had problems with pages jumping around when

looking for a product

• Another opportunity for improvement for Nordstrom would be providing clear return shipping

cost information; 27% reported frustration not understanding the cost to return an item by mail

Summary: Nordstrom

24


Summary: Nordstrom

• The ease of navigation and product filters were cited as being the most helpful

aspects of the Nordstrom site

Easy to Navigate

“It was easy to use their website and find the

information I needed to make a purchase”

“Very easy to navigate and checkout...also great

return policy with free returns”

“How relatively easy it was to make purchases

and to be able to ship those items by mail.”

“Easy to navigate. Inviting layout. Easy to add

items to shopping bag and to checkout.”

Easy-to-use Filters

“The website was set up for easy purchases. And

the filters made my shopping very easy and fast.

Very impressed!”

“The whole thing was very smooth and easy to

understand. The filters, the check out, customer

service- it’s all very accessible.”

“The site was pretty easy to navigate with lots of

filtering options. Most of the information I looked

for was easily found”

“Easy to use the filters to narrow a search.”

25


• Macy’s finished in third place based on the user experience rankings

• Nearly all participants in succeeded in finding a jacket meeting specific criteria and

finding the return policy while 86% succeeded with the checkout task

• Despite the high success rates in finding the return policy, only 48% found the task easy

with 36% reporting difficulty understanding the shipping cost and 18% struggling to find

return policy information

• 68% found it easy to check out on the Macy’s site; technical issues with the site were

the primary problems encountered

• 16% mentioned the pages were too slow during checkout and 12% had problems with

the page jumping around when clicking

Summary: Macy’s

26


Summary: Macy’s

• Participants found Macy’s easy to navigate, but also mentioned that the site

was fast and offered a good tablet experience

Easy to Navigate

“Fast. Easy to find things and search”

“Ease of the website, clear self explanatory instructions

on how to check out and add things to my bag”

“Site was very easy to navigate, simple links and the

filter/sort feature was very fast.”

Fast Site

“Many products, fast loading”

“The site was easy to navigate, loading time was fast and

almost everything was very easy to locate.”

“Very easy and clear to navigate. Loaded quickly and

responded in a good amount of time.”

Good Tablet Site

“It worked really well on my tablet. I do most of my online

shopping on a tablet and some sites work better then

other. I would say this one was easy to use its not a site

that I would have to switch to another computer to use.”

“I like the mobile version better than desktop”

27


• Urban Outfitters lagged behind the

other sites in fourth place

• Participants were least successful

finding a jacket, finding return policy

information and checking out

• Ease of use ratings were also lower

than the other sites

• Only 37% found it easy to find a jacket

with 23% struggling with the amount of

scrolling, 17% struggling with the filters

and 17% having problems narrowing

down results among other issues

• With the return policy, 37% had

problems finding it and another 13% felt

there was too much scrolling required

Summary: Urban Outfitters

28


Summary: Urban Outfitters

• Participants had greater difficulty navigating Urban Outfitters due to the

scrolling required along with technical problems with the site

Difficult to Navigate

“Everything so complicated to find. Too much scrolling to

find an item. Info text too small to read”

“Not easy to navigate thru to see items. Need more

groupings etc., men’s coats, women’s coats, shoes men’s or

women’s.”

“It is hard to maneuver to categories because the pictures

are so big for the categories.”

“Images were too big, I had to scroll around too much”

Unclear Icons

“Also need to identify the BAG icon.”

“Icons are not easily identifiable.”

Technical Issues

“The checkout kept asking for my shipping address and

then would reload the same page. I did this 4 times.”

“The page did not fully load”

“The page would not bring up the items; the page kept going

blank”

29

Brand, Conversion, and NPS

Findings

30


• After interacting with the site, Lands’ End had the biggest increase in positive

brand perception

Brand Perception

62%

57%

23%

18%

62%

69%

44%

76%

Macy's Nordstroms Urban Outfitters Lands' End

Pre-site evaluation Post-site evaluation

N=37 N=50 N=37 N=49 N=31 N=52 N=39 N=51

Site Delta

Macy’s 0%

Nordstrom 14%

Urban Outfitters 21%

Lands’ End 58%

* Selected 6-7 on a 7-point scale where 1 = Negative and 7 = Positive

31


Call to Action

• Participants were most likely to make a purchase from Macy’s or Nordstrom

and least likely to to make a purchase on the Urban Outfitters tablet site

52%47%

15%

39%

38%

35%

43%

57%

10%18%

42%

4%

Macy's N=50 Nordstrom N=49 Urban OutfittersN=52

Lands End N=51

Unlikely (1-2)

Neutral (3-5)

Likely (6-7)

Based only on your experience today, how likely are you to

make a purchase on [SITE] on your tablet in the future?

32


Net Promoter Score

• Participants were most likely to recommend Macy’s, followed by Nordstrom

• Urban Outfitters had far more detractors than promoters with a -56% NPS

10%

-6%

-56%

- 23%

NPS

36%

43%

69%

45%

18%

20%

17%

33%

46%

37%

14%

22%

Macy's N=50

Nordstrom N=49

Urban Outfitters N=52

Lands End N=51

Detractor (0-6) Passive (7-8) Promoter (9-10)

How likely are you to recommend [SITE] to a friend or

colleague?

33


Post Site Interaction Brand Attributes*

• Nordstrom was associated with ‘Customer-oriented’ and ‘High quality,’ while

Macy’s was most associated with ‘Value’ and ‘Variety’

42%

10%

54%

40%

52%

51%

24%

63%

18%

41%

19%

23%

40%

12%

35%

37%

8%

51%

31%

33%

Customer-oriented

Exclusive

High quality

Value

Variety

Macy's Post N=50

Nordstrom Post N=49

Urban Outfitters Post N=52

Lands End Post N=51

Based only on your experience today, which of the following

attributes do you associate with [COMPANY]?

* Partial list displayed

34


Change in Brand Attributes*

• Perceptions of ‘Customer-oriented’ and ‘Variety’ increased for all sites from

pre- to post-study

• Macy’s and Nordstrom had decreases in brand association with ‘Exclusive’

and ‘High quality’

10%

-10%

-26%

6%

4%

27%

-15%

-17%

0%

19%

6%

11%

13%

-3%

23%

21%

0%

12%

15%

25%

Macy's N=50

Nordstrom N=49

Urban Outfitters N=50

Lands End N=51

Value

Customer-oriented

Exclusive

High quality

Variety

Which of the following attributes do you associate with

[COMPANY]?

* Partial list displayed

35

Usability Testing

• Web & Mobile Task-based Testing

• UX & Usability Benchmarking

• Competitive Research

• Concept Testing

IA & UX Design Validation

• Navigation Optimization - Card Sorting

• Findability - Tree Testing

• Design Research - Click Testing

• Impression Testing - Timeout Testing

Customer Experience Measurement

• Voice of Customer- Live Intercept surveys

• Customer Feedback & Sat- Online Surveys

• Web Analytics Integration

Suite of tools For

All stages of your product

life cycle

Introduction to UserZoom

36

UserZoom Data | Results

37

• Benchmarking is valuable to investment decisions

• Benchmarking can give you unparalleled insight and a

competitive edge

• Nearly every customer perspective can be measured

• Partner, Tool & Audience selection is key to success

• Retail is audiences are very measurable for online user

experience & offline in-store experience

• Task Metrics are key to conversion improvement

• Brand Metrics are very measurable via Remote Testing

• Lift in Brand Perception by a website can be HUGE

• Know if your website is presenting the desired Brand

Attributes

Summary

38

Questions

How to Benchmark Your Online Customer Experience Against Competition

Documents

online experience

digital brand

brand affinitylikeability

brand perception

online customer experience

brand attributes matrices

ceux benchmarks

user demographics