A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014

A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014

Arindam Dey, Mark Billinghurst, Robert W. Lindeman, J. Edward Swan II

[email protected]

• Earlier Surveys • Methodology • Results • Key Insights


Earlier Surveys of AR Usability Studies

Time Venues Papers#

Swan & Gabbard (2005) 1992 - 2004 ISMAR,ISWC,

VR, Presence 21

Dünser et al. (2007) 1992 - 2007 ACM, IEEE

digital libraries 165

Bai & Blackwell (2012) 2001 - 2010 IEEE ISMAR 71

Our Survey 2005 - 2014 Scopus 291



Methodology

Scopus Keyword Search Unique (1,147)

Reported User Study (604)

Quality Check (396)

Norming (5) Review / Data Collection

Impact (291)

Methodology


Quality Check (396)

Impact (291)

Keywords from Denser et al. (2007):"augmented reality" AND "user evaluation(s)" "augmented reality" AND "user study/-ies" "augmented reality" AND "feedback" "augmented reality" AND "experiment(s)" "augmented reality" AND "pilot study" "augmented reality" AND participant AND study "augmented reality" AND participant AND experiment "augmented reality" AND subject AND study "augmented reality" AND subject AND experiment

Additional keywords for our survey: "augmented reality" AND "user experience(s)” "augmented reality" AND "usability” “augmented reality” AND “evaluation(s)” AND “participant” “augmented reality” AND “evaluation(s)” AND “user” “augmented reality” AND “empirical” AND “participant” “augmented reality” AND “empirical” AND “user” “augmented reality” AND “empirical”

Reported User Study(604)

Methodology


Description of the user study included • Task • Design • Participant demographics

Quality Check(396)

Impact (291)


Methodology


Average Citation Count (ACC) = Total Lifetime Citation / Lifetime in Years

ACC >= 1.5 included

Quality Check (396)

Impact(291)


Methodology

Norming(5) Review / Data Collection

• Five papers were randomly selected • All reviewers reviewed the papers • Discussed the process in a group • Agreed on reviewing consistency • Started reviewing papers individually

Methodology

• Distributed papers equally among four • Regular meetings (Skype) • Doubt resolution • Data collected through Google Form

291 papers with 369 studies were reviewed

Norming (5) Review / Data Collection

Methodology

• Application areas & keywords • Experimental design & data collection • Participant demographics • Experimental tasks & environments • Types of experiment (pilot, formal etc.) • Types of senses augmented and displays used

Focused Attributes



Results

Publication Year2014

20132012

20112010

20092008

20072006

2005

Num

ber o

f Pap

ers

60

50

40

30

20

10

0

InteractionCollaborationTourism and ExplorationNavigation and DrivingIndustrial ApplicationEntertainment and GamingEducationMedicalPerception

Application Areas

Page 1

• Most papers in Interaction (67) and Perception (51)

• Tourism (8), Collaboration (12), and Entertainment (14) are among the least

Results

• HMD (102) and Handheld (100) almost equal

• 2010: Handheld overtook

Num

ber o

f Pap

ers

30

25

20

15

10

5

0


20132012

20112010

20092008

20072006

2005

…SpatialDesktopHMDHandheld

Displays Used

Page 1

Results

• Primarily lab-based / controlled

Num

ber o

f Pap

ers

60

50

40

30

20

10

0


20132012

20112010

20092008

20072006

2005

Heuristic EvaluationPilot TestField StudyFormal User Study

Experiment Type

Page 1

Results

• Tourism and Collaboration most cited • Medical had most authors per paper on average • Most collected both quantitative and qualitative data

Results

• Majority indoor studies • Within-subjects studies are most common • Overall participant median 16 (12-28) • Young, male, and university-based participants

-> results generelizable?

Results

perception

depth

displayevaluation

uservisual

optical outdoor

cuesdevices

interfacemobile

see-through

study text

virtual

x-ray

contrast

distance drawing

feedback

handheld head-mounted

information

interaction

legibility

presence

styles

vision

blur calibration character color computing

design

empirical environment

focus hand-

held haptic

immersion

label

mr

presentation

rendering saliency

sensitivity

user-perspective view

accommodation activity acuity attention aug autostereoscopy awareness background bci

bone

consistency context depth-

of-field discrimination disparity

eeg embodied

exocentric expectations experience geo

human illusion image

layout

measures mented methodology metric mirror

navigation online

peripheral placement polarity predictive

projected proprio-perception quantitative

realism rubber saltation

sensation simulation situation stereoscopic

stiffness subtle systems three-dimensional

video

Perception

user

museum

humaninterface

virtual

gesture guide

interaction

mobile

study

activity audio aware behavioral collaborative computing context detection environment

expectations experience following

human-robot identification immersion information

investigation learning lunar

modeling multi-user ontologies

presence public recognition requirements robot rule-based science screen sensory

services social surface systems tabletop tour-guide tracking ubiquitous

usability visiting

Tourism

mobilenavigation

displays

studies

uservisual

indoor

virtual

acceptance

camera

computing devices

driver

interface

maps

audio

driving

field

handheld

interaction

localization

outdoor

phones processes rendering

spatial

activity-based aging ambience arrow artificial

authoring aware band binaural camera-based

controls cues design differences digital

disorientation distraction

dynamic editing electronic evaluation gap geospatial gps graphic guided

heads-up highlighting human inattention individual

input intelligent interac legged

locomotion magic management maneuver manipulation

movement

object occlusion output path

peephole perceptual presence roads

screen search selection sensory slam small super-

imposed system tags tangible techniques technology tion topometric transportation trust

turn ultar-wide urban user-centered view visibility

wayfinding wearable windshield-based

Navigation

rehabilitation

surgery

medical

simulation

trainingvirtual

feedback

laparoscopy

navigation

computer control

phobia

device endoscopy

forcegait guidance

laparoscopic

learning

motor needle

visual

animal biopsy box

ct

cues education

exposure

game haptic image

medicine mixed

neurosurgery none patient

phone physical

registration

small stroke

systems therapy

tracking trainers

treatmentuser

acrophobia augmented-reality closed-loop

computer-assisted

exercise fmri

image-

guided index insertion intravenous

load

mobile model

mri nasa

objective parkinson

placement pretend radio-

frequency

sinus skull soft

therapeutic

tissue tomography

variability

Medical

interactionmobile

user

displaydevice

interface

techniques

camera

computing

environments

gesture

lens

selection

tracking

virtual

magic

spatial study

tangible

visual

design

evaluation experience field

handheld

manipulation

multimodal

phone projection

attention based

communicative

head head-

mounted

menus

modeling mounted none

view wearable

acquisition active annotation application

conflict context depth

dynamic end

eye

finger fitts force gaze hand

human human-computer illusion information input intent

law

map

natural object online orientation oz

peephole point prototype pseudo-haptics recognition see-

through service survey system

target touch

usability wizard

Interaction

user

computing

guidance

interactioninterfacesmaintenance

mobile

studies

virtual

awareness

device

model

order

picking

task

assembly

building

display engineering

ergonomics

industrial

information instruction

multimedia none

real-time repair

service

tracked

aided algorithms analysis applications architecture attention audio-

haptic augmentedreality compass

computer-assisted contex data design

environments

everyday expectations fabrication facility geometries

geospatial gis gps handheld haptic hardware head-

mounted heart-rate helmet hmd hypermedia

interfaces--input large learning life logistics long-term

management methodology micro-projection

military mixed monocular mounted

non-visual optical participation

planning prototypes readability

scenarios screen see-through sensor signal sketching spatial strategies

styles surveying systems systems--artificial

techniques techniques--interaction text tree urban

video vision visualization wearable workpiece

Industrial

games

mobile

study

handheld

play

co-

located collaboration

experiences

interactive interface

phone

social sound

virtual

ar audio audio-augmented auditory avoidance based board capture cd

competitive computer computer-supported

cooperative cross-media cscp cscw depth devices displays drama empirical engagement

environments exploratory eyes-free field

garden imaginary

interference intergenerational lens linear

location magic mixed motion music narratives orientation

peephole perception performance physical presence

probabilistic qualitative spatial static

tabletop task trial user work

Gaming

learninginteractive

user

computing

education environments

interface

systems

teaching

collaborative

evaluation experience

information

mobile

multimedia

training

academic

human

tangible

virtual

acceptance achievement applications areas attitude authoring cal

classroom cognitive cooperative

design elementary

game guideline human-computer

improving

method motivation

none rehabilitation sensor simulations subject

task technology ubiquitous

visualization

Education

collaborativecommunication

interaction

video-mediated

computer

mobile

cscw depthhuman

telepresence

analysis ar around artificial automotive awareness camera

design device

games gesture handheld

interfaces interpretation markerless

mediated mixed mr out-of-body person phones presence

recognition referencing remote sensors social spatial streaming study support system

tangible telecollaboration touch tracking user

view virtual visual

vr

Collaboration



Key Insights

• Nine primary application areas emerged• Primarily young, male, and university-based participants used• Handheld displays had a rapid uptake and equalled HMDs• More user studies needed in collaboration and entertainment• Median of 12-28 participants used in studies (overall median 16)• Not enough execution of pilot studies• Heuristics for AR applications need to be developed• Subjective questionnaires are most widely used instrument• Medical interfaces less focused on usability• Work in progress

List of all papers reviewed are available on poster!

Arindam Dey [email protected]

Empathic Computing Lab University of South Australia

www.empathiccomputing.org

A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014

Technology