CHI 2013 DARE Course

Billinghurst and Duh 1

Designing Augmented Reality Experiences Mark Billinghurst

University of Canterbury Christchurch, New Zealand

Henry B.L. Duh National University of Singapore

Singapore, Singapore

[email protected] http://chi2013.acm.org/

Copyright is held by Billinghurst & Duh CHI 2013, April 27–May 2, 2013, Paris, France. ACM 13/04


Introduction


Instructors

 Mark Billinghurst •  Director of HIT Lab NZ, University of Canterbury •  Degrees in Electrical Engineering, Applied Mathematics •  Research on collaborative AR, mobile AR, AR usability •  More than 250 papers in AR, VR, interface design

 Henry Duh •  Co-director Keio-NUS Joint International Research (CUTE) Center •  Degrees in Psychology, Industrial design and Engineering •  Research on interaction design and AR applications •  More than 80 papers in HCI, AR and Design

Introduction


How Would You Design This?

 Put nice AR Picture here – and video


Or This?


  How to design effective AR experiences   Understanding AR interaction design possibilities   Hardware and software tools for rapid prototyping of AR applications   Effective evaluation methods for AR applications   Current areas of AR research that will contribute to future AR experiences   Hands on experiences with AR applications   Resources for your own research

What You Will Learn

Introduction


  Introduction [Mark]  AR and the Interaction Design Process [Mark]  Design Guidelines and Interaction Metaphors for AR [Mark]  AR Development/Prototyping Tools [Mark]  Afternoon Tea – Demos [Mark and Henry]  AR Evaluation Methods [Henry]  AR Design Case Studies [Henry]  AR Research Directions [Mark]

Course Agenda

Introduction


Course Demos  AR Authoring

BuildAR, Metaio Creator

 AR Browers •  Junaio, Layar, Wikitude

 AR Gaming •  Elite CommandAR, Transformers, etc..

 Marker Based Handheld AR •  NASA and CCDU

 Outdoor AR •  CityViewAR

 Displays •  Vuzix, Google Glass


Course Motivation

 AR Needs Good Interaction Design   AR increasingly popular but ergonomics, design and social

issues need to be addressed   There is a need for deeper understanding of how to uncover,

design build and evaluate effective AR experiences   AR authoring tools are making it easier than ever before to build

an AR experience, but there are few design guidelines   Many AR applications are being developed, but there is little

formal evaluation being conducted   AR experiences are being delivered without an understanding of

the interaction design/experience design process

Introduction


What is Augmented Reality?

 Defining Characteristics (Azuma 97) •  Combines Real and Virtual Images

– Both can be seen at the same time •  Interactive in real-time

– The virtual content can be interacted with •  Registered in 3D

– Virtual objects appear fixed in space

Introduction

Azuma, R., A Survey of Augmented Reality, Presence, Vol. 6, No. 4, August 1997, pp. 355-385.


From Science Fiction to Fact

1977 – Star Wars

2008 – CNN

Introduction


AR Part of MR Continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

"...anywhere between the extrema of the virtuality continuum."

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.


AR History

 1960’s – 80’s: Early Experimentation •  Military, Academic labs

 1980’s – 90’s: Basic Research •  Tracking, Displays

 1995 – 2005: Tools/Applications •  Interaction, Usability, Theory

 2005 - : Commercial Applications •  Games, Medical, Industry, Mobile

Introduction


Core Technologies

 Combining Real and Virtual Images •  Display technologies

 Interactive in Real-Time •  Input and interactive technologies

 Registered in 3D •  Viewpoint tracking technologies

Introduction

Display

Processing

Input Tracking


Display Technologies

 Types (Bimber/Raskar 2003)  Head attached

•  Head mounted display/projector  Body attached

•  Handheld display/projector  Spatial

•  Spatially aligned projector/monitor

 HMD Optical vs. Video see-through  Optical: Direct view of real world -> safer, simpler  Video: Video overlay -> more image registration options

Introduction


Display Taxonomy


Input Technologies

 Tangible objects •  Tracked items

 Touch (HHD) •  Glove, touch

 Gesture •  Glove, free-hand

 Speech/Multimodal  Device motion

•  HHD + sensors

Introduction


Tracking Technologies

 Active •  Mechanical, Magnetic, Ultrasonic •  GPS, Wifi, cell location

 Passive •  Inertial sensors (compass, accelerometer, gyro) •  Computer Vision

•  Marker based •  Natural feature tracking

 Hybrid Tracking •  Combined sensors (eg Vision + Inertial)

Introduction


 Web Based AR •  Flash, HTML 5 based AR •  Marketing, education

 Outdoor Mobile AR •  GPS, compass tracking •  Viewing Points of Interest in real world •  Eg: Junaio, Layar, Wikitude

 Handheld AR •  Vision based tracking •  Marketing, gaming

 Location Based Experiences •  HMD, fixed screens •  Museums, point of sale, advertising

Typical AR Experiences

Introduction


AR Becoming Big Business

 Marketing •  Web-based, mobile

 Mobile AR •  Geo-located information and service •  Driving demand for high end phones

 Gaming •  Mobile, Physical input (Kinect)

 Upcoming areas •  Manufacturing, Medical, Military

 Rapid Growth •  Market projected to grow 53% 2012 – 2016 •  Over $5 Billion USD in Mobile AR alone by 2017


Mobile AR Market Size


Commercial AR Companies

 ARToolworks (http://www.artoolworks.com/) •  ARToolKit, FLARToolKit, SDKs

 Metaio (http://www.metaio.com/) •  Marketing, Industry, SDKs

 Total Immersion (http://www.t-immersion.com/) •  Marketing, Theme Parks, AR Experiences

 Qualcomm (http://developer.qualcomm.com/dev/augmented-reality) •  Mobile AR, Vuforia SDK

 Many small start-ups (String, Ogmento, etc)


The Interaction Design Process


“The product is no longer the basis of value. The

experience is.”

Venkat Ramaswamy The Future of Competition.

Interaction Design


experiences

services

products

components

Valu

e

Gilmore + Pine: Experience Economy

Function

Emotion

Interaction Design


experiences

applications

tools

components

Designing AR Experiences

Tracking, Display

Authoring

Interaction

Usability

Interaction Design


The Value of Good User Experience

20c

50c

$3.50

Interaction Design


Good Experience Design

 Reactrix •  Top down projection •  Camera based input •  Reactive Graphics •  No instructions •  No training

Interaction Design


Apple: The Value of Good Design

 Good Experience Design Dominates Markets

iPod Sales 2002-2007


Nokia N-Gage

 Great idea – bad experience design  See - http://www.sidetalkin.com

Good: Handheld Gaming + Phone Bad: Look like a dork using it


Interaction Design

 Answering three questions: •  What do you do? - How do you affect the world? •  What do you feel? – What do you sense of the world? •  What do you know? – What do you learn?

 The Design of User Experience with Technology

“Designing interactive products to support people in their everyday and working lives”

Preece, J., (2002). Interaction Design

Interaction Design


Interaction Design is All About You

 Users should be involved throughout the Design Process

 Consider all the needs of the user •  Especially context of use

Interaction Design


Interaction Design Process

Interaction Design


Gabbard Model for AR Design

1. user task analysis 2. expert guidelines-based evaluation 3. formative user-centered evaluation 4. summative comparative evaluations

Gabbard, J.L.; Swan, J.E.; , "Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design,” Visualization and Computer Graphics, IEEE Transactions on, vol.14, no.3, pp.513-525, May-June 2008


Gabbard Model in Context


Design Guidelines for AR

Design Guidelines




AR Interaction Design

 Designing AR System = Interface Design •  Using different input and output technologies

 Objective is a high quality of user experience •  Ease of use and learning •  Performance and satisfaction


Design Considerations

 Combining Real and Virtual Images •  Perceptual issues

 Interactive in Real-Time •  Interaction issues

 Registered in 3D •  Technology issues

Introduction


 Interface Components •  Physical components • Display elements

– Visual/audio •  Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Design Elements


AR UI Design

 Consider your user  Follow good HCI principles  Adapt HCI guidelines for AR  Design to device constraints  Using Design Patterns to Inform Design  Design for you interface metaphor  Design for evaluation


Consider Your User

 Consider context of user •  Physical, social, emotional, cognitive, etc

 Mobile Phone AR User •  Probably Mobile •  One hand interaction •  Short application use •  Need to be able to multitask •  Use in outdoor or indoor environment •  Want to enhance interaction with real world


Good HCI Principles

 Affordance  Reducing cognitive overload  Low physical effort  Learnability  User satisfaction  Flexibility in use  Responsiveness and feedback  Error tolerance


Norman’s Principles of Good Practice •  Ensure a high degree of visibility

– allow the user to work out the current state of the system and the range of actions possible.

•  Provide feedback – continuous, clear information about the results of actions.

•  Present a good conceptual model – allow the user to build up a picture of the way the system

holds together, the relationships between its different parts and how to move from one state to the next.

•  Offer good mappings – aim for clear, natural relationships between actions the

user performs and the results they achieve.


Adapting Existing Guidelines

 Mobile Phone AR •  Phone HCI Guidelines •  Mobile HCI Guidelines

 HMD Based AR •  3D User Interface Guidelines •  VR Interface Guidelines

 Desktop AR •  Desktop UI Guidelines


iPhone Guidelines

 Make it obvious how to use your content.  Avoid clutter, unused blank space, and busy

backgrounds.  Minimize required user input.  Express essential information succinctly.  Provide a fingertip-sized target area for all links

and controls.  Avoid unnecessary interactivity.  Provide feedback when necessary


Applying Principles to Mobile AR

 Clean  Large Video View  Large Icons  Text Overlay  Feedback


AR vs. Non AR Design

 Design Guidelines •  Design for 3D graphics + Interaction •  Consider elements of physical world •  Support implicit interaction

Characteristics Non-AR Interfaces AR Interfaces Object Graphics Mainly 2D Mainly 3D

Object Types Mainly virtual objects Both virtual and physical objects

Object behaviors Mainly passive objects Both passive and active objects

Communication Mainly simple Mainly complex

HCI methods Mainly explicit Both explicit and implicit


Maps vs. Junaio

 Google Maps •  2D, mouse driven, text/image heavy, exocentric

 Junaio •  3D, location driven, simple graphics, egocentric


Design to Device Constraints

 Understand the platforms used and design for limitations •  Hardware, software platforms

 Eg Handheld AR game with visual tracking •  Use large screen icons •  Consider screen reflectivity •  Support one-hand interaction •  Consider the natural viewing angle •  Do not tire users out physically •  Do not encourage fast actions •  Keep at least one tracking surface in view

50

Art of Defense Game


Handheld AR Constraints/Affordances   Camera and screen are linked

•  Fast motions a problem when looking at screen •  Intuitive “navigation”

  Phone in hand •  Two handed activities: awkward or intuitive •  Extended periods of holding phone tiring •  Awareness of surrounding environment

  Small screen •  Extended periods of looking at screen tiring •  In general, small awkward platform

  Vibration, sound •  Can provide feedback when looking elsewhere

  Networking - Bluetooth, 802.11 •  Collaboration possible

  Guaranteed minimum collection of buttons   Sensors often available

•  GPS, camera, accelerometer, compass, etc


Design Patterns

“Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice.”

– Christopher Alexander et al.

Use Design Patterns to Address Reoccurring Problems

C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.


Handheld AR Design Patterns Title Meaning Embodied Skills Device Metaphors Using metaphor to suggest available player

actions Body A&S Naïve physics

Control Mapping Intuitive mapping between physical and digital objects

Body A&S Naïve physics

Seamful Design Making sense of and integrating the technological seams through game design

Body A&S

World Consistency Whether the laws and rules in physical world hold in digital world

Naïve physics Environmental A&S

Landmarks Reinforcing the connection between digital-physical space through landmarks

Environmental A&S

Personal Presence The way that a player is represented in the game decides how much they feel like living in the digital game world

Environmental A&S Naïve physics

Living Creatures Game characters that are responsive to physical, social events that mimic behaviours of living beings

Social A&S Body A&S

Body constraints Movement of one’s body position constrains another player’s action

Body A&S Social A&S

Hidden information The information that can be hidden and revealed can foster emergent social play

Social A&S Body A&S


Example: Seamless Design

 Design to reduce seams in the user experience •  Eg: AR tracking failure, change in interaction mode

 Paparazzi Game •  Change between AR tracking to accelerometer input

Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--Arts, Media, and Humanities, p.19-28, October 26-29, 2011


Example: Living Creatures

 Virtual creatures should respond to real world events •  eg. Player motion, wind, light, etc •  Creates illusion creatures are alive in the real world

 Sony EyePet •  Responds to player blowing on creature

55


Physical Elements

Design Guidelines


AR Design Space

Reality Virtual Reality

Augmented Reality

Physical Design Virtual Design


Design of Objects

 Objects •  Purposely built – affordances •  “Found” – repurposed •  Existing – already at use in marketplace

 Affordance •  The quality of an object allowing an action-

relationship with an actor •  An attribute of an object that allows people to

know how to use it – e.g. a door handle affords pulling


Norman on Affordances

"...the term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. [...] Affordances provide strong clues to the operations of things. Plates are for pushing. Knobs are for turning. Slots are for inserting things into. Balls are for throwing .. " (Norman, The Psychology of Everyday Things 1988, p.9)


Physical vs. Virtual Affordances

 Physical affordances -  Physical and material aspects of real object

 Virtual affordance -  Visual and perceived aspects of digital objects

 AR is mixture of physical and virtual affordances •  Physical

– Tangible controllers and objects •  Virtual

– Virtual graphics and audio

- 


Affordance Framework

William W. Gaver. 1991. Technology affordances. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '91), Scott P. Robertson, Gary M. Olson, and Judith S. Olson (Eds.). ACM, New York, NY, USA, 79-84.


Affordance Led Design

 Make affordances perceivable •  Provide visual, haptic, tactile, auditory cues

 Affordance Led Usability •  Give feedback •  Provide constraints •  Use natural mapping •  Use good cognitive model


Example: AR Chemistry

 Tangible AR chemistry education (Fjeld) Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible

interaction approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich, Switzerland.


Input Devices

 Form informs function and use


Picking up an Atom


AR Interaction Metaphors

Design Guidelines


 Interface Components •  Physical components • Display elements

– Visual/audio •  Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Design Principles


Interaction Tasks

 2D (from [Foley]): •  Selection, Text Entry, Quantify, Position

 3D (from [Bowman]): •  Navigation (Travel/Wayfinding) •  Selection •  Manipulation •  System Control/Data Input

 AR: 2D + 3D Tasks and.. more specific tasks?

[Foley] The Human Factors of Computer Graphics Interaction Techniques Foley, J. D., V. Wallace & P. Chan. IEEE Computer Graphics and Applications (Nov.): 13-48. 1984. [Bowman]: 3D User Interfaces: Theory and Practice D. Bowman, E. Kruijff, J. Laviola, I. Poupyrev Addison Wesley 2005


AR Interaction Metaphors

 Viewpoint Control   Information Browsing

•  establish shared meaning

 3D AR Interfaces •  establish shared meaning

 Augmented Surfaces •  serve as cognitive artifacts

 Tangible AR •  serve as cognitive artifacts


1. Viewpoint Control

 2D/3D virtual objects are registered in 3D •  “VR in Real World”

 Interaction •  2D/3D virtual viewpoint control

 Applications •  Visualization, training


2. Information Browsering

 Information is registered to real-world context •  Hand held AR displays

 Interaction •  Manipulation of a window

into information space  Applications

•  Context-aware information displays

Rekimoto, et al. 1997


3. 3D AR Interfaces

 Virtual objects displayed in 3D physical space and manipulated •  HMDs and 6DOF head-tracking •  6DOF hand trackers for input

 Interaction •  Viewpoint control •  Traditional 3D user interface

interaction: manipulation, selection, etc.

Kiyokawa, et al. 2000


4. Augmented Surfaces

 Basic principles •  Virtual objects are projected on a surface •  Physical objects are used as controls for

virtual objects •  Support for collaboration

 Rekimoto, et al. 1998 •  Front projection •  Marker-based tracking •  Multiple projection surfaces


5. Tangible User Interfaces

 Create digital shadows for physical objects

 Foreground •  graspable UI

 Background •  ambient interfaces


Lessons from Tangible Interfaces

 Physical objects make us smart •  Norman’s “Things that Make Us Smart” •  encode affordances, constraints

 Objects aid collaboration •  establish shared meaning

 Objects increase understanding •  serve as cognitive artifacts


TUI Limitations

 Difficult to change object properties •  Can’t tell state of digital data

 Limited display capabilities •  projection screen = 2D •  dependent on physical display surface

 Separation between object and display •  Augmented Surfaces


Tangible AR Metaphor

 AR overcomes limitation of TUIs •  enhance display possibilities •  merge task/display space •  provide public and private views

 TUI + AR = Tangible AR •  Apply TUI methods to AR interface design


 Space-multiplexed •  Many devices each with one function

– Quicker to use, more intuitive, clutter – Real Toolbox

 Time-multiplexed •  One device with many functions

–  Space efficient – mouse


Tangible AR: Tiles (Space Multiplexed)

 Tiles semantics •  data tiles •  operation tiles

 Operation on tiles •  proximity •  spatial arrangements •  space-multiplexed


Tangible AR: Time-multiplexed Interaction

 Use of natural physical object manipulations to control virtual objects

 VOMAR Demo •  Catalog book:

–  Turn over the page •  Paddle operation:

–  Push, shake, incline, hit, scoop


Object Based Interaction: MagicCup

  Intuitive Virtual Object Manipulation on a Table-Top Workspace

•  Time multiplexed •  Multiple Markers

– Robust Tracking •  Tangible User Interface

–  Intuitive Manipulation •  Stereo Display

– Good Presence



Tangible AR Design Principles

 Tangible AR Interfaces use TUI principles •  Physical controllers for moving virtual content •  Support for spatial 3D interaction techniques •  Time and space multiplexed interaction •  Support for multi-handed interaction •  Match object affordances to task requirements •  Support parallel activity with multiple objects •  Allow collaboration between multiple users


Interaction with Handheld AR

 Embodied Interaction •  Focuses on the device itself

•  Touch, gesture, orientation, etc  Tangible Interaction

•  Direct manipulation of known objects •  Tracking objects

 Egocentric vs. Exocentric Interaction •  Egocentric – inside out (eg outdoor AR browsing) •  Exocentric – outside in (eg marker based AR)


Handheld AR Metaphors

HandHeld AR Wearable AR

Output: Display

Input

Input & Output


Handheld Interface Metaphors

 Tangible AR Lens Viewing •  Look through screen into AR scene •  Interact with screen to interact with

AR content –  Eg Invisible Train

 Tangible AR Lens Manipulation •  Select AR object and attach to device •  Use the motion of the device as input

–  Eg AR Lego


Case Study 1: 3D AR Lens

Goal: Develop a lens based AR interface

 MagicLenses •  Developed at Xerox PARC in 1993 •  View a region of the workspace differently to the rest •  Overlap MagicLenses to create composite effects


3D MagicLenses

MagicLenses extended to 3D (Veiga et. al. 96)   Volumetric and flat lenses


AR Lens Design Principles

 Physical Components •  Lens handle

–  Virtual lens attached to real object  Display Elements

•  Lens view – Reveal layers in dataset

  Interaction Metaphor •  Physically holding lens


Case Study 2: LevelHead

 Physical Components •  Real blocks

 Display Elements •  Virtual person and rooms

  Interaction Metaphor •  Blocks are rooms


AR Perceptual + Cognitive Issues

Design Guidelines


AR and Perception

 Creating the illusion that virtual images are seamlessly part of the real world •  Must match real and virtual cues

•  Depth, occlusion, lighting, shadows..


AR as Perception Problem

 Goal of AR to fool human senses – create illusion that real and virtual are merged

 Depth •  Size •  Occlusion •  Shadows •  Relative motion •  Etc..


Central goal of AR systems is to fool the human perceptual system

 Display Modes •  Direct View •  Stereo Video •  Stereo graphics

 Multi-modal display •  Different objects with different display modes •  Potential for depth cue conflict

Perceptual Issues

D. Drascic and P. Milgram. Perceptual issues in augmented reality. In M. T. Bolas, S. S. Fisher, and J. O. Merritt, editors, SPIE Volume 2653: Stereoscopic Displays and Virtual Reality Systems III, pages 123-134, January/February 1996.


Perceptual Issues

 Combining multiple display modes •  Direct View, Stereo Video View, Graphics View

 Conflict between display modes •  Mismatch between depth cues


Perceptual Issues

 Static and Dynamic registration mismatch  Restricted Field of View  Mismatch of Resolution and Image clarity  Luminance mismatch  Contrast mismatch  Size and distance mismatch  Limited depth resolution  Vertical alignment mismatches  Viewpoint dependency mismatch


Types of Perceptual Issues

 Environment: Issues related to the environment itself.  Capturing: Issues related to digitizing the environment  Augmentation: Issues related to the design, layout, and

registration or AR content  Display device: Technical issues associated with the

display device.  User: Issues associated with user perceiving content.

E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited. 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2010, pp. 3--12.


Depth Cues

 Pictorial: visual cues •  Occlusion, texture, relative brightness

 Kinetic: motion cues •  Relative motion parallax, motion perspective

 Physiological: motion cues •  Convergence, accommodation

 Binocular disparity: two different eye images



Depth Perception


Occlusion Handling


Cognitive Issues in AR

 Three categories of issues •  Information Presentation – displaying virtual

information on the real world •  Physical Interaction – content creation,

manipulation and navigation in AR •  Shared Experience – collaboration and

supporting common experiences in AR

Li, Nai, and Henry Been-Lirn Duh. "Cognitive Issues in Mobile Augmented Reality: An Embodied Perspective." Human Factors in Augmented Reality Environments. Springer New York, 2013. 109-135.


Information Presentation

 Information Presentation •  Amount of information

•  Clutter, complexity •  Representation of information

•  Navigation cues, POI representation •  Placement of information

•  Head, body, world stabilized •  View combination

•  Multiple views


Twitter 360

 www.twitter-360.com   iPhone application  See geo-located tweets in real world  Twitter.com supports geo tagging


Wikitude – www.mobilizy.com

Blah

Blah

Blah Blah Blah

Blah Blah

Blah

Blah Blah Blah

Blah Blah

Blah

Blah Blah

Blah

Blah Blah

Blah

Blah Blah


Information Filtering


Information Filtering


Physical Interaction

 Physical Interaction •  Navigation •  Direct Manipulation

•  Embodied vs. Tangible •  Multimodal interaction •  Content creation


Outdoor AR: Limited FOV


Possible solutions

 Overview + Detail •  spatial separation; two views

 Focus + Context •  merges both views into one view

 Zooming •  temporal separation


 TU Graz – HIT Lab NZ - collaboration •  Zooming panorama •  Zooming Map

Zooming Views


Gesture Based Interaction

 HMD-based AR frees the users hands •  Natural hand based interaction •  Intuitive manipulation – low cognitive load

 Example •  Tinmith-Hand Two hand manipulation of 3D models

112


Shared Experiences

 Shared Experience •  Social context •  Bodily configuration •  Artifact manipulation •  Display space


TAT Augmented ID




Designing for Children

 Development Psychology Factors •  Motor Abilities •  Spatial Abilities •  Logic Abilities •  Attention Abilities

Radu, Iulian, and Blair MacIntyre. "Using children's developmental psychology to guide augmented-reality design and usability." Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on. IEEE, 2012.


Motor Abilities

Skill Type Challenging AR Interaction Multiple hand coordination Holding phone in one hand and

using another hand to move marker

Hand-eye coordination Using a marker to intercept a moving object

Fine motor skills Moving a marker on a specified path

Gross motor skills and endurance Turning body around to look at a panorama


Spatial Abilities

Skill Type Challenging AR Interaction Spatial memory Remembering the configuration of a large

virtual space while handheld screen shows a limited view

Spatial Perception Understanding when a virtual item is on top of a physical item

Spatial Visualization Predict when virtual objects are visible by other people or virtual characters


Attention and Logic

Skill Type Challenging AR Interaction Divided attention Playing an AR game, and making sure to

keep marker in view so tracking is not lost

Selective and executive attention

Playing an AR game while moving outdoors

Skill Type Challenging AR Interaction Remembering and reversing Remembering how to recover from tracking

loss

Abstract over concrete thinking

Understanding that virtual objects are computer generated, and they do not need to obey physical laws

Attention Abilities

Logic and Memory

Billinghurst and Duh 121 121

AR Development Tools


AR Authoring Tools

 Low Level Software Libraries •  osgART, Studierstube, MXRToolKit

 Plug-ins to existing software •  DART (Macromedia Director), mARx, Unity,

 Stand Alone •  AMIRE, BuildAR, Metaio Creator etc

 Rapid Prototyping Tools •  Flash, OpenFrameworks, Processing, Arduino, etc

 Next Generation •  iaTAR (Tangible AR)


ARToolKit (Kato 1998)

 Open source – computer vision based AR tracking  http://artoolkit.sourceforge.net/


ARToolKit Structure

 Three key libraries: •  AR32.lib – ARToolKit image processing functions •  ARgsub32.lib – ARToolKit graphics functions •  ARvideo.lib – DirectShow video capture class

DirectShow

ARvideo.lib


Software

 Cross platform •  Windows, Mac, Linux, IRIX, Symbian, iPhone, etc

 Additional basic libraries •  Video capture library (Video4Linux, VisionSDK) •  OpenGL, GLUT

 Requires a rendering library •  Open VRML, Open Inventor, osgART, etc


OSGART Programming Library

  Integration of ARToolKit with a High-Level Rendering Engine (OpenSceneGraph) OSGART= OpenSceneGraph + ARToolKit

 Supporting Geometric + Photometric Registration


osgART Approach: AR Scene Graph

Video Geode

Root

Transform

3D Object

Virtual Camera

Projection matrix from tracker calibration

Transformation matrix updated from marker tracking in realtime

Video Layer

Full-screen quad with live texture updated from Video source

Orthographic projection


osgART:Features

 C++ (but also Python, Lua, etc).  Multiple Video Input supports:

•  Direct (Firewire/USB Camera), Files, Network by ARvideo, PtGrey, CVCam, VideoWrapper, etc.

 Benefits of Open Scene Graph •  Rendering Engine, Plug-ins, etc


ARToolKit Family

ARToolKit ARToolKit NFT

ARToolKit (Symbian)

NyToolKit - Java, C#, - Android, WM

JARToolKit (Java)

FLARToolKit (Flash)

FLARManager (Flash)


Why Browser Based AR?

 High impact •  High marketing value

 Large potential install base •  1.6 Billion web users

 Ease of development •  Lots of developers, mature tools

 Low cost of entry •  Browser, web camera


FLARToolkit

Papervision 3D

Adobe Flash

AR Application Components


FLARToolKit Example

 Boffswana Living Sasquatch  In first month

•  100K unique visits •  500K page views •  6 minutes on page


Low Level Mobile AR Tools

 Vuforia Tracking Library (Qualcomm) •  Vuforia.com •  iOS, Android •  Computer vision based tracking •  Marker tracking, 3D objects, frame markers

 Integration with Unity •  Interaction, model loading, game logic


Junaio - www.junaio.com


Junaio Key Features

 Content provided in information channels •  Over 2,000 channels available

 Two types of AR channels •  GLUE channels – visual tracking •  Location based channels – GPS, compass tracking

 Simple to use interface with multiple views •  List, map, AR (live) view

 Point of Interest (POI) based •  POIs are geo-located content



AREL

 Augmented Reality Environment Language •  Overcomes limitations of XML by itself •  Based on web technologies; XML, HTML5, JavaScript

 Core Components 1. AREL XML: Static file, specifies scene content 2. AREL JavaScript: Handles all interactions and animation. Any

user interaction send an event to AREL JS 3. AREL HTML5: GUI Elements. Buttons, icons, etc

 Advantages •  Scripting on device, more functionality, GUI customization





Result


BirdsView

 Location Based CMS •  Add content, publish to Layar or Junaio •  http://www.birdsview.de/


BirdsView on Junaio


BirdsView on Junaio


BuildAR

 http://www.buildar.co.nz/  Stand alone application  Visual interface for AR model viewing application  Enables non-programmers to build AR scenes


Metaio Creator

 Drag and drop Junaio authoring


Total Immersion D’Fusion Studio

 Complete commercial authoring platform •  http://www.t-immersion.com/ •  Multi-platform •  Markerless tracking •  Scripting •  Face tracking •  Finger tracking •  Kinect support


Others

 AR-Media •  http://www.inglobetechnologies.com/ •  Google sketch-up plug-in

 LinceoVR •  http://linceovr.seac02.it/ •  AR/VR authoring package

 Libraries •  JARToolKit, MXRToolKit, ARLib, Goblin XNA


Research in AR Authoring

  iaTAR (Lee 2004) •  Immersive AR Authoring •  Using real objects to create AR applications


Rapid Prototyping

 Speed development time by using quick hardware mockups •  handheld device connected to PC •  LCD screen •  USB phone keypad •  Camera

 Can use PC development tools for rapid application


Build Your Own Google Glass

 Rapid Prototype Glass-Like HMD  Myvu HMD + headphone + iOS Device + basic glue skills

•  $300 + less than 3 hours construction   http://www.instructables.com/id/DIY-Google-Glasses-AKA-the-Beady-i/



BUNRATTY FOLK PARK

 Irish visitor attraction run by Shannon Heritage

 19th century life is recreated

 Buildings from the mid-west have been relocated to the 26-land surrounding Bunratty Castle

 30 buildings are set in a rural or village setting there.


AUGMENTED REALITY

154

In Bunratty Folk Park:  Allows the visitor to point a camera at an exhibit, the

device recognises its by it’s location and layers digital information on to the display

 3- dimensional virtual objects can be positioned with real ones on display

 Leads to dynamic combination of a live camera view and information


ITERATIVE DESIGN PROCESS

Prototyping and User Testing  Low Fidelity Prototyping

• Sketches • Paper Prototyping • Post-It Prototyping • PowerPoint Prototyping

 High Fidelity Prototyping • Wikitude


Storyboarding

156


INITIAL SKETCHES

Pros: •  Good for idea genera/on •  Cheap •  Concepts seem feasible

Cons: •  Not great feedback gained •  Photoshop not fast enough for making changes


Post-it Note Prototyping Camera View with 3D Annota/on

•  Selec/on highlighted in blue •  Home buBon added for easy naviga/on to main menu


POWERPOINT PROTOTYPING

Benefits •  Used for User Tes/ng •  Interac/ve •  Func/onali/es work •  Quick •  Easy arrangement of slides

User Tes/ng •  Par/cipants found •  15 minute sessions screen captured

•  ‘Talk Allowed’ technique used •  Notes taken •  Post-‐Interview


WIKITUDE PROTOTYPE

User Testing  Application well received  Understandable  Participants playful with the

technology


FINAL VIDEO PROTOTYPE

 Flexible tool for capturing the use of an interface

 Elaborate simula/on of how the naviga/onal aid will work

 Does not need to be realis/c in every detail

 Gives a good idea of how the finished system will work


AR Evaluation Methods




Why Evaluate AR Applications?

 To test and compare interfaces, new technologies, interaction techniques

 To validate the efficiency and efficient the AR interface and system

 Test Usability (learnability, efficiency, satisfaction,...)  Get user feedback  Refine interface design  Better understand your end users   ...


Survey of AR Papers  Edward Swan (2005)  Surveyed major conference/journals (1992-2004)

– Presence, ISMAR, ISWC, IEEE VR  Summary

•  1104 total papers •  266 AR papers •  38 AR HCI papers (Interaction) •  21 AR user studies

 Only 21 from 266 AR papers had a formal user study •  Less than 8% of all AR papers


HIT Lab NZ Usability Survey

  A Survey of Evaluation Techniques Used in Augmented Reality Studies •  Andreas Dünser, Raphaël Grasset, Mark

Billinghurst

 reviewed publications from 1993 to 2007 •  Extracted 6071 papers which mentioned

“Augmented Reality” •  Searched to find 165 AR papers with User Studies




Types of Experimental Measures Used

 Types of Experimental Measures •  Objective measures •  Subjective measures •  Qualitative analysis •  Usability evaluation techniques •  Informal evaluations


Types of Experimental Measures Used


Types of Experiments and topics

  Sensation, Perception & Cognition •  How is virtual content perceived ? •  What perceptual cues are most important ? •  How to visualize augmented/overlay information on real environment? •  Visual search/attention/salience issues of human performance

  Interaction •  How can users interact with virtual content ? •  Which interaction techniques are most efficient in certain context ?

  Collaboration & Social issues •  How is collaboration in AR interface different ? •  Which collaborative cues can be conveyed best ? •  Privacy and security issues of AR interface


Types of AR User Studies


Summary

 Over last 10 years •  Most user studies focused on user performance •  Fewest user studies on collaboration

– MobileAR was not popular before 2009 •  Objective performance measures most used •  Qualitative and usability measures least used


Sample Size

 … the more the better •  for quantitative analysis:

•  rule of thumb approx. 15-20 or more (for cognitive and lab type of experiment)

•  absolute minimum of 8-10 per cell

 Ideal sample size can be calculated - power analysis •  Power (1- beta) => the chance to reject the null hypothesis

when the null hypothesis is false •  Power is the probability of observing a difference when it really

exists •  Power increases with sample size •  Power decreases with variance

 Large effects can be detected with smaller samples •  e.g. to discriminate mean speed between turtles and a rabbits


Data Collection and Analysis

 The choice of a method is dependent on the type of data that needs to be collected

  In order to test a hypothesis the data has to be analysed using a statistical method

 The choice of a statistical method depends on the type of collected data

 All the decisions about an experiment should be made before it is carried out


Observe and Measure

 Observations are gathered… •  manually (human observers) •  automatically (computers, software, cameras, sensors, etc.)

 A measurement is a recorded observation  Objective metrics  Subjective metrics


Typical objective metrics

 task completion time  errors (number, percent,…)  percent of task completed  ratio of successes to failures  number of repetitions  number of commands used  number of failed commands  physiological data (heart rate,…)  …


Typical subjective metrics

 user satisfaction  subjective performance  ratings  ease of use   intuitiveness   judgments  …


Data Types

 Subjective •  Subjective survey

–  Likert Scale, condition rankings

•  Observations –  Think Aloud

•  Interview responses

 Objective •  Performance measures

–  Time, accuracy, errors

•  Process measures –  Video/audio analysis

How easy was the task

1 2 3 4 5 Not very easy Very easy


Experimental Measures Measure What does it tell us? How is it

measured? Timings Performance Via a stopwatch, or

automatically by the device. Errors Performance, Particular sticking

points in a task By success in completing the task correctly. Through experimenter observation, examining the route walked.

Perceived Workload Effort invested. User satisfaction Through NASA TLX scales and other questionnaires.

Distance traveled and route taken

Depending on the application, these can be used to pinpoint errors and to indicate performance

Using a pedometer, GPS or other location-sensing system. By experimenter observation.

Percentage preferred walking speed

Performance By finding average walking speed, which is compared with normal walking speed.

Comfort User satisfaction. Device acceptability

Comfort Rating Scale and other questionnaires.

User comments and preferences

User satisfaction and preferences. Particular sticking points in a task.

Through questionnaires, interviews and think-alouds.

Experimenter observations

Different aspects, depending on the experimenter and on the observations

Through observation and note-taking


Statistical Analysis

 Once data is collected statistics can be used for analysis  Typical Statistical Techniques

•  Comparing between two results –  Unpaired T-Test (for between subjects – assumes normal distribution) –  Paired T-Test (for within subjects – assumes normal distribution) –  Mann–Whitney U (independent samples)

•  Comparing between > two results –  Followed by post-hoc analysis – Bonferroni Test –  Analysis of Variance – ANOVA –  Kruskal–Wallis (does not assume normal distribution)


Case Study: A Wearable Information Space

Head Stabilized Body Stabilized

An AR interface provides spatial audio and visual cues Does a spatial interface aid performance?

– Task time / accuracy

M. Billinghurst, J. Bowskill, Nick Dyer, Jason Morphett (1998). An Evaluation of Wearable Information Spaces. Proc. Virtual Reality Annual International Symposium.


Task Performance

 Task •  find target icons on 8 pages •  remember information space

 Conditions A - head-stabilized pages B - cylindrical display with trackball C - cylindrical display with head tracking

 Subjects •  Within subjects (need fewer subjects) •  12 subjects used


Experimental Measures

 Objective •  spatial ability (pre-test) •  time to perform task •  information recall •  workload (NASA TLX)

 Subjective •  Post Experiment Survey

–  rank conditions (forced choice) –  Likert Scale Questions

-  “How intuitive was the interface to use?”

Many Different Measures


Post Experiment Survey

For each of these conditions please answer: 1) How easy was it to find the target? 1 2 3 4 5 6 7 1=not very easy 7=very easy

For the head stabilised condition (A): For the cylindrical condition with mouse input (B): For the head tracked condition (C):

Rank all the conditions in order on a scale of one to three 1) Which condition was easiest to find target (1 = easiest, 3 = hardest)

A: B: C:


Results

 Body Stabilization Improved Performance •  search times significantly faster (One factor ANOVA)

 Head Tracking Improved Information recall •  no difference between trackball and stack case

 Head tracking involved more physical work


Subjective Impressions

  Subjects Felt Spatialized Conditions (ANOVA): •  More enjoyable •  Easier to find target


Subjective Impressions

  Subject Rankings (Kruskal-Wallis) •  Spatialized easier to use than head stabilized •  Body stabilized gave better understanding •  Head tracking most intuitive


AR Evaluation  Field, Field, Field –

•  Field studies vs. Lab studies •  Contextual design and evaluation

 Combined methods (qualitative and quantitative studies) •  Weakness of each method should be considered

 New/modified evaluation methods may need to be developed

 Seek for more new evaluation case studies in AR


AR Design Case Study


“The Jackson Plan” An Educational Location-based Handheld AR Game

Learning while in travel

Mobile AR Entertainment for Children


The Jackson Plan

 Overview

‘The Jackson Plan’ is an educational discovery Mobile Augmented Reality game that is set on the historical urban plan of the same name (also known as the “Plan of the Town of Singapore”)

Using multi-modality features on an Apple iPad2, players collaboratively experience this location-based Mobile Augmented Reality game around the several important historical sites and events that revolve around Sir Thomas Stamford Raffles and his founding of the island of Singapore in 1819.

The Jackson Plan 1822, is on display at the Singapore History Gallery, National Museum of Singapore


 Learning Goals/Objectives

Unit Learning objectives

Jackson Plan

【Knowledge】 1. To acquire a better understanding of the key developments of the Raffles’s arrival, its early settlers and Raffle’s town plan.

【Skills】 1. To explains the reasons for the founding of Singapore (1819). 2. To explain the importance of trade to Singapore. 3. To describe the contributions of key personalities and immigrants to the growth and development of Singapore.

【Values & Attitudes】 1. To develop an interest in the past. 2. To appreciate culture heritage as well as to instill a sense of courage, diligence and perseverance to Singapore.

History Syllabus for Lower Secondary, Year of Implementation: 2006. ISBN 981-05-1669-X. Source: Curriculum Planning and Development Division, Ministry of Education, Singapore

Learning Content


Consideration

 How can a new technology help new learning experience in cultural heritage?

 Interdisciplinary research (Design, Technology, Education and Learning)

 System building, a single application or  Recognition in each field  Real deployment in schools

194


Theoretical Framework

“Situated cognition via scaffolding mechanisms

([Vygotsky, 1978])”

Distinct HAR technology pairings available in a game, (0=No, 1=Yes), resulting in four possible eHAR game types and play styles, each with an implementation process.

Y.-N. Chang, R. K. C. Koh, and H. B.-L. Duh, "Handheld AR games - A triarchic conceptual design framework," in Mixed and Augmented Reality - Arts, Media, and Humanities (ISMAR-AMH), 2011 IEEE International Symposium On, Basel, Switzerland, 2011, pp. 29-36.

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.


 Triarchic conceptual design framework

•  GPS navigation: Location-based implementation for Cultural & Historical (contextual) explorations

•  Overlaying options: ‘Binoculars’ metaphor (i.e., Panoramic Map)

•  Virtual properties (game inventory)

•  Geo-tagging / (diary) •  Blended mini games

(i.e. puzzles) •  Tasks may exploit

platform’s hardware features

(GPS, Accelerometer)

•  Visual identification of past and present imagery

•  History comes to life by exploiting location-dependent contexts

•  Backend confirmation with server connectivity (‘Wizard of Oz’ possibility) for dynamic situational exchanges, i.e. messages, images, induce player-behaviors, etc

•  Promote Contextual Inquiry & Collaboration

(Learning Strategies)

Theoretical Framework


The Jackson Plan

Textbook: SINGAPORE: FROM SETTLEMENT TO NATION - PRE-1819 TO 1971 (Marshall Cavendish Education) Theme: Chapter 3 - What Part Did the Different Immigrant Communities Play in Singapore’s Development?

The

Jack

son

Pla

n

Prior Knowledge:

The settlement of Singapore

-Why Raffles chose Singapore

Singapore’s central location

Central location

Excellent port

Good supply of drinking water

The Dutch had not occupied the island

Immigrants

Why immigrants came

Singapore’s town plan

Lieutenant Philip Jackson,1822

Improve the haphazard

building plan

Segregated population groups

Populations from trading goods /

countries

Chinese

Coolies

Samsui women

Indians

Labourers

Coolies

Malays Shipbuilders

Europeans Merchants

Arabs Traders


•  Ideation: Use of Historical Illustrations / Images in Situated Augmented Views

•  Panoramic / Still - Visual Imageries of the Past + GPS


Game Features, Mechanisms & Platform - Ideations

•  ‘Civic District Trail’ - A tourist’s DIY exploration experience promoted by the Singapore Tourism Board


Virtual & physical interaction

Manipulate knowledge-collect trading materials

(i.e. spices)

Geo-Tagging the “right” locations

Taking pictures (Wizard of Oz) Blended casual

mini-games with physical interaction and collaboration

Game Features, Mechanisms & Platform




Chinese: Chinatown

Indians

Europeans &

Rich Asians

Malays & Muslims

“Plan of the town of Singapore” by Lieutenant Phillip Jackson,1822

Commercial Square



Phase Learning Objectives Learning Task(s) Time

1 Understanding of activities

To understand the Gameplay and manipulation of iPad2 devices

- Introduction to Gameplay - Game introduction / Mission Briefing

15 min

2 Constructing Knowledge

To understand the background of Singapore settlement

- Information Collection: Know who these immigrants are

15 min

3 Mastering

To analyze how did the immigrants contribute to Singapore as a trading centre

- Experience the entrepot trade

20 min

4 Knowledge Application

To make comparisons and organize information of the different contributions of immigrants

- Make an accusation by evidence (Gain a summative feedback)

15 min

The Jackson Plan: Planned Gaming Activities


The Trail


Game Design


Game Design


The Jackson Plan - Features


Evaluation

•  72 students (36 pairs) took part in the evaluation

•  Secondary One classes (~12-13 years old)

•  They were equally divided into 2 main groups, Location-based and Digital Book versions

Digital Book Location-based AR

Platform Apple iPad2 Apple iPad2 Collaboration Yes Yes Interaction Type Non-AR Location-based AR

Play Space Indoors Outdoors


 The structure of knowledge

Evaluation


The Jackson Plan


The Jackson Plan


The Jackson Plan


The Jackson Plan


The Jackson Plan


The Jackson Plan

Theory into Practice: Domain-Centric Handheld Augmented Reality Game Design Study 3 - Co-creativity fusions in interdisciplinary AR game developments


AR Research Directions

Billinghurst and Duh 218 The Vision of AR


To Make the Vision Real..

 Hardware/software requirements • Contact lens displays •  Free space hand/body tracking •  Speech/gesture recognition •  Etc..

 Most importantly • Usability


Natural Interaction

 Automatically detecting real environment •  Environmental awareness •  Physically based interaction

 Gesture Input •  Free-hand interaction

 Multimodal Input •  Speech and gesture interaction •  Implicit rather than Explicit interaction

Environmental Awareness


AR MicroMachines

 AR experience with environment awareness and physically-based interaction •  Based on MS Kinect RGB-D sensor

 Augmented environment supports •  occlusion, shadows •  physically-based interaction between real and virtual objects


Operating Environment


Architecture

 Our framework uses five libraries:

•  OpenNI •  OpenCV •  OPIRA •  Bullet Physics •  OpenSceneGraph


System Flow

 The system flow consists of three sections: •  Image Processing and Marker Tracking •  Physics Simulation •  Rendering


Physics Simulation

 Create virtual mesh over real world

 Update at 10 fps – can move real objects

 Use by physics engine for collision detection (virtual/real)

 Use by OpenScenegraph for occlusion and shadows


Rendering

Occlusion Shadows

Gesture Input


Architecture

5. Gesture

• Static Gestures • Dynamic Gestures • Context based Gestures

4. Modeling

• Hand recognition/modeling • Rigid-body modeling

3. Classification/Tracking

2. Segmentation

1. Hardware Interface

HITLabNZ’s Gesture Library


Architecture

5. Gesture


4. Modeling

• Hand recognition/modeling

• Rigid-body modeling


2. Segmentation



o  Supports PCL, OpenNI, OpenCV, and Kinect SDK.

o  Provides access to depth, RGB, XYZRGB.

o  Usage: Capturing color image, depth image and concatenated point clouds from a single or multiple cameras

o  For example:

Kinect for Xbox 360

Kinect for Windows

Asus Xtion Pro Live


Architecture 5. Gesture


4. Modeling




2. Segmentation


o  Segment images and point clouds based on color, depth and space.

o  Usage: Segmenting images or point clouds using color models, depth, or spatial properties such as location, shape and size.

o  For example:


Skin color segmentation

Depth threshold


Architecture 5. Gesture


4. Modeling




2. Segmentation


o  Identify and track objects between frames based on XYZRGB.

o  Usage: Identifying current position/orientation of the tracked object in space.

o  For example:


Training set of hand poses, colors represent unique regions of the hand.

Raw output (without-cleaning) classified on real hand input (depth image).


Architecture

5. Gesture


4. Modeling




2. Segmentation


o  Hand Recognition/Modeling   Skeleton based (for low resolution

approximation)   Model based (for more accurate

representation) o  Object Modeling (identification and tracking

rigid-body objects) o  Physical Modeling (physical interaction)

  Sphere Proxy   Model based   Mesh based

o  Usage: For general spatial interaction in AR/VR environment



Method Represent models as collec1ons of spheres moving with the

models in the Bullet physics engine


Method Render AR scene with OpenSceneGraph, using depth map

for occlusion

Shadows yet to be implemented


Results


Architecture

5. Gesture


4. Modeling




2. Segmentation


o  Static (hand pose recognition) o Dynamic (meaningful movement

recognition) o  Context-based gesture

recognition (gestures with context, e.g. pointing)

o  Usage: Issuing commands/anticipating user intention and high level interaction.


Multimodal Interaction


Multimodal Interaction

 Combined speech input  Gesture and Speech complimentary

•  Speech – modal commands, quantities

•  Gesture –  selection, motion, qualities

 Previous work found multimodal interfaces intuitive for 2D/3D graphics interaction


Free Hand Multimodal Input  Use free hand to interact with AR content  Recognize simple gestures  No marker tracking

Point Move Pick/Drop


Multimodal Architecture


Multimodal Fusion


Hand Occlusion


User Evaluation

 Change object shape, colour and position  Conditions

•  Speech only, gesture only, multimodal

 Measure •  performance time, error, subjective survey


Experimental Setup

Change object shape and colour


Results

 Average performance time (MMI, speech fastest) •  Gesture: 15.44s •  Speech: 12.38s •  Multimodal: 11.78s

 No difference in user errors  User subjective survey

•  Q1: How natural was it to manipulate the object? – MMI, speech significantly better

•  70% preferred MMI, 25% speech only, 5% gesture only

Future Directions


Natural Gesture Interaction on Mobile

 Use mobile camera for hand tracking •  Fingertip detection


Evaluation

 Gesture input more than twice as slow as touch  No difference in naturalness


Intelligent Interfaces

 Most AR systems are stupid •  Don’t recognize user behaviour •  Don’t provide feedback •  Don’t adapt to user

 Especially important for training •  Scaffolded learning •  Moving beyond check-lists of actions


Intelligent Interfaces

 AR interface + intelligent tutoring system •  ASPIRE constraint based system (from UC) •  Constraints

–  relevance cond., satisfaction cond., feedback


Domain Ontology


Intelligent Feedback

 Actively monitors user behaviour •  Implicit vs. explicit interaction

 Provides corrective feedback



Evaluation Results  16 subjects, with and without ITS   Improved task completion

  Improved learning


Intelligent Agents

 AR characters •  Virtual embodiment of system •  Multimodal input/output

 Examples •  AR Lego, Welbo, etc •  Mr Virtuoso

–  AR character more real, more fun – On-screen 3D and AR similar in usefulness


Contact Lens Display

 Babak Parviz •  University Washington

 MEMS components •  Transparent elements •  Micro-sensors

 Challenges •  Miniaturization •  Assembly •  Eye-safe


Contact Lens Prototype


Conclusion


Conclusion

 There is need for better designed AR experiences  Through

•  use of Interaction Design principles •  understanding of the technology •  use of rapid prototyping tools •  rigorous user evaluation

 There a number of important areas for future research •  Natural interaction •  Multimodal interfaces •  Intelligent agents •  Novel displays


More Information

•  Mark Billinghurst – [email protected]

•  Websites – www.hitlabnz.org

•  Henry Duh – [email protected]


Resources


Websites

 Meta List of AR SDKs •  http://www.icg.tugraz.at/Members/gerhard/augmented-reality-sdks

 ARToolKit Software Download •  http://artoolkit.sourceforge.net/

 ARToolKit Documentation •  http://www.hitl.washington.edu/artoolkit/

 ARToolKit Forum •  https://www.artoolworks.com/community/forum/

 ARToolworks Inc •  http://www.artoolworks.com/


 ARToolKit Plus •  http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php

 osgART •  http://www.osgart.org/

 FLARToolKit •  http://www.libspark.org/wiki/saqoosha/FLARToolKit/

 FLARManager •  http://words.transmote.com/wp/flarmanager/


AR Labs

 Europe •  TU Graz, Cambridge U, TU Munich, FraunhoferIGD

 USA •  Columbia U, Georgia Tech, USC

 Asia •  KIST, KAIST •  AIST, Kyoto U, NAIST, U of Tsukuba •  NUS, UniSA, HITLab NZ

 Companies •  Qualcomm, Nokia, Layar, Wikitube, Metaio


Books

  Interactive Environments with Open-Source Software: 3D Walkthroughs and Augmented Reality for Architects with Blender 2.43, DART 3.0 and ARToolKit 2.72 by Wolfgang Höhl

 A Hitchhikers Guide to Virtual Reality by Karen McMenemy and Stuart Ferguson

 Bimber, Raskar. Spatial Augmented Reality (2005)


 Books Mobile Interaction Design Matt Jones and Gary Marsden Designing for Small Screens Studio 7.5 Handheld Usability Scott Weiss Designing the Mobile User Experience Barbara Ballard


Publication venues   Conference

•  IEEE/ACM International Symposium in Mixed and Augmented Reality (IEEE/ACM ISMAR) (ismar.net)

•  IEEE Virtual Reality (IEEE VR) •  Korean-Japan Mixed Reality Workshop (KJMR)

  Journal •  IEEE Transaction on Visualization and Computer Graphics (IEEE) •  Computer & Graphics (Elsevier) •  PRESENCE (MIT Press)

  Papers •  Zhou, F., Duh, H.B.L., and Billinghurst, M. (2008). Trends in Augmented Reality Tracking,

Interaction and Display: A Review of Ten Years of ISMAR. in IEEE International Symposium on Mixed and Augmented Reality (IEEE/ACM ISMAR) 193-202

•  Azuma, R., Baillot, Behringer, R., Feiner, S., Julier, S., MacIntyre, B., (2001). Recent Advances in Augmented Reality, IEEE Computer Graphics and Applications, 34-47


More Papers

  E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited. 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2010, pp. 3--12.

  D. Drascic and P. Milgram. Perceptual issues in augmented reality. In M. T. Bolas, S. S. Fisher, and J. O. Merritt, editors, SPIE Volume 2653: Stereoscopic Displays and Virtual Reality Systems III, pages 123-134, January/February 1996.

270


Developer Guidelines

 Palm http://www.access-company.com/developers/documents/docs/ui/

UI_Design.html  Zen of Palm guidelines http://www.access-company.com/developers/documents/docs/

zenofpalm.pdf

 Motorola http://developer.motorola.com/docstools/developerguides/

  iPhone Human Interface Guidelines http://developer.apple.com/documentation/iPhone/Conceptual/iPhoneHIG/


Handheld HCI Design Websites

Do’s and Don’ts of PocketPC design http://www.pocketpcmag.com/_archives/Nov04/Commandements.aspx

Usability special interest group – handheld usability http://www.stcsig.org/usability/topics/handheld.html

Usable Mobile website http://www.smartgroups.com/groups/usablemobile

Mobile Coders Website http://www.mobilecoders.com/Articles/mc-01.asp

Univ of Waikato Handheld Group http://www.cs.waikato.ac.nz/hci/pdas.html

Mobile Interaction Website http://www.cs.waikato.ac.nz/~mattj/mwshop.html

CHI 2013 DARE Course

Technology

ar usability

collaborative ar

ar history1960s

based ar marketing

good interaction design

ar applications resources

wikitudehandheld ar

handheld ar nasa