Remote User Research Methods, tools and practicalities Breakfast Briefing 1 October 2014 @UserVision
Remote User Research Methods, tools and practicalities Breakfast Briefing 1 October 2014
@UserVision
Remote User Research
What do we mean by remote user research?
Types of remote usability testing
Practicalities and tips
What to use when
Usability testing is the best way to get empirical evidence of the likely appeal or success of your product
What is remote usability testing?
• Participant far away – can’t interact in person
• It is NOT
…collecting analytics, Split testing or MVT VWO, Optimisely, Google Experiments
… where user is unaware they are being tested Background playback tools – Mouseflow, Sessioncam
Heatmaps of where user clicked – Clicktale, Crazy egg
… only a survey of opinion Eg. Surveymonkey, primarily based on opinion
… remote ethnography, diary study
Some things that we’re not talking about
• Various ways to record clicks, mouse movements, scrolling and secretly record users using your site.
• Crazy Egg
• Clicktale
• Inspectlet
• Sessioncam
• Mouseflow
• Anyone using these?
Remote research
• Advantages
Access users anywhere in the world
Remove travel costs
Faster turnaround – results within an hour
Natural environment –at home
Can ‘live recruit’ to capture ‘in the moment’ experience
More honest / representative?
• Disadvantages
No chance to talk with user (unless remote moderated)
Less rich results compared to standard F2F tests
Types of tasks may be constrained
Potential for technical glitches
Getting the right audience may be difficult
Other considerations
• Still need a good research plan
What do you want to find out?
Who participates? Are they familiar with your product?
Do you want qualitative or quant insights?
• Make it personal, friendly
• Be clear in task scenarios, and success states
• Consider the audience:
Will technology overly complicate the process?
How can we best replicate real-life circumstances?
Online research audience may be more web savvy or ‘professional testers’
Remote User Research
What do we mean by remote user research?
Types of remote usability testing
Practicalities and tips
What to use when
Types of remote research
1. Remote moderated – Like normal testing but aided by screenshare / audio share – live discussion
2. Remote unmoderated – User goes through tasks to collect behavioural & opinion data, no discussion
Remote Moderated Testing
Remote moderated testing
• Like a face to face test – over distance
• See their screen and hear the audio
• Captures experience from where they normally use the web
• Can allow 3rd party observation
• Technical issues may occur
Remote Moderated Testing - example
Edinburgh,
London
Mexico
Brazil
Nigeria
Egypt
Pakistan Bangladesh
Indonesia
Remote observation – with translation Tests conducted in
China – in Chinese Simultaneous
translation to English
Observ ed by UX
consultant in the UK, listening to the English
translation
Observ ed by the
client in Dubai Observ ed by the
agency in NY
Practicalities and tips for remote moderated tests
• Five common practical questions:
What can you test?
Who should you test with?
Where can participants be located?
How should you conduct the testing?
When will the sessions happen?
• Already covered why…
What can you test?
What can you test?
• Website/web services
Most common
‘Best fit’ for this form of testing
What can you test?
• Website/web services
• Corporate IT systems/Intranets
Can present some security/firewall issues
Some participants can feel constrained when taking part from their employer’s environment
What can you test?
• Website/web services
• Corporate IT systems/Intranets
• Highly graphical/animated interfaces e.g. Flash
Can suffer in transmission due to frame-rate/image quality limitations
What can you test?
• Website/web services
• Corporate IT systems/Intranets
• Highly graphical/animated interfaces e.g. Flash
• Mobile
Much bigger challenge
No single, cross-platform solution
Many security-related restrictions (and risks) at OS level regarding screen-sharing
What can you test?
• Mobile: ‘Laptop hugging’ technique
Who should you test
with?
Who should you test with?
• Anyone…
…with some caveats
Who should you test with?
IT Confidence
Context
High
Low
Unrepresentative Representative
Where can
participants be located?
Where can participants be located?
• Anywhere
…with some caveats:
Language requirements
Where can participants be located?
• Anywhere
…with some caveats:
Language requirements
Geographical regions with ‘endemic infrastructure limitations’
How should you
conduct the testing?
How should you conduct the testing?
• Many screen sharing apps available
Join.me
Webex
GoToMeeting
Adobe Connect
Skype
How should you conduct the testing?
How should you conduct the testing?
• Skype:
Good audio quality
How should you conduct the testing?
• Skype:
Good audio quality
Relatively low bandwidth requirements
How should you conduct the testing?
• Skype:
Good audio quality
Relatively low bandwidth requirements
‘Set up’ already done by the participant
How should you conduct the testing?
• Skype:
Good audio quality
Relatively low bandwidth requirements
‘Set up’ already done by the participant
Implies some level of IT comfort/capability
How should you conduct the testing?
• Skype:
Good audio quality
Relatively low bandwidth requirements
‘Set up’ already done by the participant
Implies some level of IT comfort/capability
Gets the communications technology ‘out of the way’
How should you conduct the testing?
• Skype:
Good audio quality
Relatively low bandwidth requirements
‘Set up’ already done by the participant
Implies some level of IT comfort/capability
Gets the communications technology ‘out of the way’
Risk that it restricts pool of potential participants
How should you conduct the testing?
• Skype:
Good audio quality
Relatively low bandwidth requirements
‘Set up’ already done by the participant
Implies some level of IT comfort/capability
Gets the communications technology ‘out of the way’
Risk that it restricts pool of potential participants
Shifting sands of features/account types
How should you conduct the testing?
• Skype:
Good audio quality
Relatively low bandwidth requirements
‘ ‘Set up’ already done by the participant
Implies some level of IT comfort/capability
Gets the communications technology ‘out of the way’
Risk that it restricts pool of potential participants
Shifting sands of features/account types
And…
ALWAYS have a back-up plan!
When will the
sessions happen?
When will the sessions happen?
• Time.is:
When will the sessions happen?
• Time.is:
When will the sessions happen?
• Time.is
• Calendly:
When will the sessions happen?
• Time.is
• Calendly:
When will the sessions happen?
• Time.is
• Calendly:
When will the sessions happen?
• Time.is
• Calendly
• Google docs:
Calendar
Spreadsheet
Remote Un-moderated Testing
Self-moderated (video) testing
• Participants video themselves doing task & talking
• Can’t discuss in real time with the participant
• Recruited from provider’s panel, crowdsourcing / social media or your own list
• Participants set up own web cam
• Still need time to review the videos, analyse , recommend solution
• Better for general audiences rather than specialist
Self-moderated (video) testing
• Usertesting.com
Panel focused on US, Canada, UK
Can annotate, edit video clips
Peek as a free trial
• WhatUsersDo
Panel covering 26 countries, can be filtered
• Both can cover mobile platforms
Example - mobile - http://youtu.be/zb0HigU_rys
Remote unmoderated testing
• Various tools, most capturing quantitative & qual
• Track success by specifying success pages
• May be able to test competitor sites
• Both can test on mobile
Other Mobile variations
• Forward facing camera captures user’s face
• Apsee
a mobile iOS app analytics platform
Requires line of code inserted to the App
Captures user recordings, creates touch heatmaps
• Lookback
• UX Recorder
Mobile website (not Apps) testing
Quick Exposure tests
• Five Second Tests
• UI Tests – 10 people for 10 Seconds for $9
www.uitests.com/t/aXmgpXE
• Usability Tools
Click Testing
Web Testing
Card Sorting
Optimal workshop
Verify
• Test screenshots / designs to gain insights on expectations and reactions
1. Preference test – which of 2 options prefer and why
2. Yes / no test –Mark areas, see if click right place
3. Click test – See where people click based on instructions
4. Multiclick test – As above but can compare different sites
5. Memory test – like the 5 seconds test – what you remember
6. Annotate test – allow free form notes to be marked on page
7. Mood test –show screenshot, select from emotion options
8. Label test – Check users understanding of what link will do
Example output - Verify
Usabilla
• Usabilla Visual Survey
Collect visual feedback on wireframes, mockups or any other visual
Feedback shows where people clicked in a heatmap
Choosing remote methods
Qualitative
Concrete
Quantitative
Conceptual
Video Self-moderated
Remote Moderated
IA, Short tests, Verify
Remote Unmoderated
Resources
• http://remoteresear.ch/
• Books
Beyond the Usability Lab – Bill Albert
Remote research – Nate Bolt
55 North Castle Street Edinburgh EH2 3QA Tel: 0131 225 0853
www.uservision.co.uk
@UserVision
Contact us