The meaning and value of numbers in UX. Dr Simone Stumpf Centre for HCI Design @DrSimoneStumpf [email protected] .uk
Oct 28, 2014
The meaning and value of numbers in UX. Dr Simone Stumpf
Centre for HCI Design@DrSimoneStumpf
Everyone loves numbers.
My background.
Industry
BT– Fraud detection– Product management– Marketing– Project management
White Horse– UX Architect
Academia
University College London– BSc Computer Science
with Cognitive Science– PhD Computer Science– Research Fellow
Oregon State University– Research Manager
City University London– Senior Lecturer
That makes me years old.3876
How old was Methuselah when he died?
670 969 12542756
Cognitive bias and heuristics.
Anchoring – any number has a priming effect on number estimates.
People, even researchers, are bad at probability, predictions and statistics.
[Daniel Kahneman – Thinking, Fast and Slow]
Quantitative approaches in UX.
Quantitative data – numbers.
Quantitative analysis – statistics.
For statistical tests you have a hypothesis.
Quantitative data and/or analysis?
How many problems does a user have using my snazzy new design?
What kind of problems does a user have using my snazzy new design?
Do you like my snazzy new design?
Is this snazzy new design better than the old boring design?
Quantitative
approaches.
3
Do you like my snazzy new design?
Let’s ask the user.
How much do you like the design on a scale of 1 to 5 (where 5 is best).
Average of ratings across all users.
Then, er, do some stats?
Way around?
NASA Task Load Index (TLX) to assess user’s perceptions of
– Mental Demand– Physical Demand– Temporal Demand– Performance– Effort– Frustration
Mea culpa!“Responses to TLX questions (Mental Demand, Temporal Demand, Success of Performance, Effort, Frustration) were all around the mid-point of the scale.”
On an interface which was truly hateful!
“However, the [Condition 1] participants showed no significant difference to [Condition 2] participants’ TLX scores.” – 62 participants
At least our sample size wasn’t shabby and we did some stats.
What kinds of problems does a user have with my snazzy new
design?
Hold on – is that a trick question?
Surely, that’s qualitative analysis!
Yes, but no.
It starts out that way but then I expect frequencies to back this up. No stats though, thanks.
“4 out of 5 users could not find the Purchase button.”
Count them!
Visual environment101 positive – 37 negative
Textual environment62 positive – 69 negative
16
Is this new design better than my old design?
Easy-peasy.
I’ll use a between-subject design using objective measures.
Like…eye tracking! What could be more objective than where people look.
Lots of numbers – First Fixation Duration, Fixation Duration, Time to First Fixation, …
[http://uxmag.com/articles/eye-tracking-the-best-way-to-test-rich-app-usability]
Well, that was fun.
Yay – we did stats! There were results!
Oh bum…
There was a highly significant difference in the number of fixations between versions (Χ2(2,N=4257)=22.25, p<0.001). Each participant on average fixated 240.83 times in version 1, 259.83 times in version 2 yet only 209.33 times in version 3. The average fixation duration between versions was also different (ANOVA, F(2,4257)=13.30, p=<0.001), with
participants in version 1 spending on average 0.57 seconds per fixation, 0.56 seconds in version 2 but 0.69 seconds in version 3.
The total fixation duration is the sum of all individual fixation’s durations. There was no statistical significance between participants’ length of total fixations (Kruskal-Wallis, H(2,N=18), p=0.236).
To summarise.
Try and quantify as much as possible but be clear about limitations of what you can measure.
Descriptive statistics are good but are relatively meaningless without context.
If you must use statistical tests, please make sure they are appropriate.
Numbers are
awesome.
Be clear about your questions and the best way to answer them.
@DrSimoneStumpf