UX by the numbers: The meaning and value of numbers in UX

The meaning and value of numbers in UX. Dr Simone Stumpf

Centre for HCI Design@DrSimoneStumpf

Simone.Stumpf.1@city.ac.uk

Everyone loves numbers.

My background.

Industry

BT– Fraud detection– Product management– Marketing– Project management

White Horse– UX Architect

Academia

University College London– BSc Computer Science

with Cognitive Science– PhD Computer Science– Research Fellow

Oregon State University– Research Manager

City University London– Senior Lecturer

That makes me years old.3876

How old was Methuselah when he died?

670 969 12542756

Cognitive bias and heuristics.

Anchoring – any number has a priming effect on number estimates.

People, even researchers, are bad at probability, predictions and statistics.

[Daniel Kahneman – Thinking, Fast and Slow]

Quantitative approaches in UX.

Quantitative data – numbers.

Quantitative analysis – statistics.

For statistical tests you have a hypothesis.

Quantitative data and/or analysis?

How many problems does a user have using my snazzy new design?

What kind of problems does a user have using my snazzy new design?

Do you like my snazzy new design?

Is this snazzy new design better than the old boring design?

Quantitative

approaches.

3

Do you like my snazzy new design?

Let’s ask the user.

How much do you like the design on a scale of 1 to 5 (where 5 is best).

Average of ratings across all users.

Then, er, do some stats?

Way around?

NASA Task Load Index (TLX) to assess user’s perceptions of

– Mental Demand– Physical Demand– Temporal Demand– Performance– Effort– Frustration

Mea culpa!“Responses to TLX questions (Mental Demand, Temporal Demand, Success of Performance, Effort, Frustration) were all around the mid-point of the scale.”

On an interface which was truly hateful!

“However, the [Condition 1] participants showed no significant difference to [Condition 2] participants’ TLX scores.” – 62 participants

At least our sample size wasn’t shabby and we did some stats.

What kinds of problems does a user have with my snazzy new

design?

Hold on – is that a trick question?

Surely, that’s qualitative analysis!

Yes, but no.

It starts out that way but then I expect frequencies to back this up. No stats though, thanks.

“4 out of 5 users could not find the Purchase button.”

Count them!

Visual environment101 positive – 37 negative

Textual environment62 positive – 69 negative

16

Is this new design better than my old design?

Easy-peasy.

I’ll use a between-subject design using objective measures.

Like…eye tracking! What could be more objective than where people look.

Lots of numbers – First Fixation Duration, Fixation Duration, Time to First Fixation, …

[http://uxmag.com/articles/eye-tracking-the-best-way-to-test-rich-app-usability]

Well, that was fun.

Yay – we did stats! There were results!

Oh bum…

There was a highly significant difference in the number of fixations between versions (Χ2(2,N=4257)=22.25, p<0.001). Each participant on average fixated 240.83 times in version 1, 259.83 times in version 2 yet only 209.33 times in version 3. The average fixation duration between versions was also different (ANOVA, F(2,4257)=13.30, p=<0.001), with

participants in version 1 spending on average 0.57 seconds per fixation, 0.56 seconds in version 2 but 0.69 seconds in version 3.

The total fixation duration is the sum of all individual fixation’s durations. There was no statistical significance between participants’ length of total fixations (Kruskal-Wallis, H(2,N=18), p=0.236).

To summarise.

Try and quantify as much as possible but be clear about limitations of what you can measure.

Descriptive statistics are good but are relatively meaningless without context.

If you must use statistical tests, please make sure they are appropriate.

Numbers are

awesome.

Be clear about your questions and the best way to answer them.

@DrSimoneStumpf

Simone.Stumpf.1@city.ac.uk