What analytics can tell you and what it cant!

What analytics can tell you and what it can’t!

Paul & Aran - Evosite

#DigitalExeter

#DigitalExeter

In WWII, Allied bomber planes were constantly being shot down. Armour-plating the whole plane would have made it too heavy, so extra plating could only be put in the areas where planes were being shot at the most. A mathematician, named Abraham Wald was asked to look into this problem.

#DigitalExeter

1. He began by creating a diagram that outlined the plane and marked where the bullet holes were found on the planes that had returned.

2. The pink areas are where the plane was shot at the most, so based on that you would say that plating these areas would protect them. But the thing is, when looking at the averages, people were missing the real weakness.

3. Wald argued that instead of plating the areas where the planes had been shot, that instead the plane needed reinforcement where the plane didn’t have bullet holes.

Don’t look at general averages hoping to find answers.

#DigitalExeter

Improvements won’t jump out at you, they have to be uncovered

#DigitalExeter

Averages...

#DigitalExeter

When looking at analytics, our dashboards are filled with averages: Average bounce rate, average time on site, average conversion rate. It’s easy to make assumptions with generalised data, but to understand the problem you have to look a little deeper.

Averages...

#DigitalExeter

The problem is, averages hide variations and anomalies. One event can skew the average and your perception of the information.

Weaknesses!

#DigitalExeter

By digging into data, you can find areas which aren't performing as well as you would have expected. Compare paid traffic to organic traffic, identify dropout points or break down your conversion rate by device to uncover these weaknesses.

Step One:

Analytics tells you what, but it doesn’t tell you why

#DigitalExeter

Step Two:

Conduct usability tests to validate your hypothesis

#DigitalExeter

Step Three:

Use A/B testing to find the most effective improvement

#DigitalExeter

Identify

Find areas to improve

#DigitalExeter

1

Identify Understand

Find areas to improve

Observe user behaviours

#DigitalExeter

1 2

Identify Understand Test

Find areas to improve

Observe user behaviours

Define the best solution

#DigitalExeter

1 2 3

How to identify website improvements

● Look at your conversion rates

● Review the pages with the top 5 highest bounce rates

● Conduct usability testing that includes these pages and devices

#DigitalExeter

Are you truly understanding what your data is telling you?

Here’s a case study that reflects on the first partand provides a few actionable ideas.

#DigitalExeter

Gadsbywww.gadsby.co.uk

#DigitalExeter

Some important background: Gadsby is a B2B wholesaler of high quality wicker baskets, hampers, display and gift packaging. They work together with a wide range of highly regarded national brands to provide high quality retail display products.

Evosite has a long running partnership with Gadsby, our company had designed and developed the previous website for the client.

In 2014 we undertook the next major update to the Gadsby website and tried to capitalise on what we knew about their customer base to boost their conversion rate.

We had a LOT of tracking in place on the existing website.

This would allow us to easily diagnose any potential issues when the new version of

the site launched.

#DigitalExeter

Following the launch of the NEW improved website, our tracking showed an increase across the

whole site!

#DigitalExeter

Well…...Almost every area!

#DigitalExeter

On the product list pages, we observed a decrease in click-throughs

16%Decrease in product

list click-throughs

#DigitalExeter

The product list pagesOne of the main differences between the two designs of these pages was that we used a larger header on the pages with eye-catching lifestyle imagery to make the products more appealing to the customers.

What had caused the 16% drop in CTR?We analysed the pages and concluded with a high level of certainty the most likely cause was that the increased height of the header image on the new version (shown on the right) had pushed too much of the products below the fold and out of view.

We quickly launched an A/B test to test our hypothesis...

#DigitalExeter

The Control

Our control version was the original, newly launched product list page with the full height header imagery.

#DigitalExeter

Variation #1

The first variation kept the lifestyle imagery but narrowed the header height by around 30% allowing ~1.5 rows of products to now appear on the page.

#DigitalExeter

Variation #2

The second variation completely removed the header image and the higher header, so almost two complete rows of products could been seen.

#DigitalExeter

What were we expecting to win?

#DigitalExeter

What were we expecting to win?

We expected Variation #1 to win due as we know for Gadsby’s customers, the lifestyle imagery was important.

Variation #1 offered a good blend of both options in our opinion.

#DigitalExeter

Here were our test results

Large header: ---

As we had expected, the skinny header won

#DigitalExeter

Here were our test results

Large header: ---

No header: -1.15% CTR

As we had expected, the skinny header won

#DigitalExeter

Here were our test results

Large header: ---

No header: -1.15% CTR

Skinny header: +9.31% CTR

As we had expected, the skinny header won

#DigitalExeter

As we predicted, Variation #1 had won!

#DigitalExeter

As we reviewed our A/B data against Google Analytics we learnt something

more important...

BUT...

#DigitalExeter

Revenue Per VisitorLarge header (Control): ----

Slim header (Variation #1): -20.98%

#DigitalExeter

But, why did this happen?Let’s quickly recap...

#DigitalExeter

#DigitalExeter

1. By focusing on our goal (raising the click-through rate), we had almost forgotten that revenue per visitor was also key for Gadsby, if not even more important. Our “bullet holes” didn’t tell us first that even though the CTR had dropped, the changes had a positive effect on revenue.

2. Showing the products in situ was more important for Gadsby customers as it encouraged customers to buy more. Outlining the ultimate goal for the website from the start and making sure there was a bottom line for business benefits helped us turn the website significantly more profitable.

Define and ALWAYS refer back to your critical KPIs

As this is an eCommerce site; revenue and orders will always be their #1 Priority

#DigitalExeter

Thank youfor your time

Aran ReeksHEAD OF CLIENT [email protected]: @EvositeAran

Paul RandallSENIOR UX [email protected]: @paulrandall

@Evosite

Get in touch:

#DigitalExeter