Augmented and mixed reality (AR & MR) Doug Bowman CS 5754 Based on original lecture notes by Ivan Poupyrev
Augmented and mixed reality(AR & MR)
Doug BowmanCS 5754
Based on original lecture notes by Ivan Poupyrev
(C) 2008 Doug Bowman, Virginia Tech 2
AR/MR example
(C) 2008 Doug Bowman, Virginia Tech 3
Definitions
Augmented reality: Refers to a system inwhich the user views and acts within anenhanced version of the real world. Theenhancements are virtual (computer-generated), and can include objects orinformation.
Mixed reality: Refers to a system thatcombines real and virtual objects andinformation.
(C) 2008 Doug Bowman, Virginia Tech 4
Mixed reality continuum
Reality VirtualityAugmentedReality (AR)
AugmentedVirtuality (AV)
Mixed Reality (MR)
Milgram (1994)
(C) 2008 Doug Bowman, Virginia Tech 5
AR/MR application areas
MaintenanceTrainingTourism / Cultural heritageDesign / constructionBattlefield information display
(C) 2008 Doug Bowman, Virginia Tech 6
AR/MR technology - displays
See-through HMDs:Video see-throughOptical see-through
Handheld displays Projection
(C) 2008 Doug Bowman, Virginia Tech 7
AR/MR technology - tracking
Optical / vision-basedtrackingAR toolkitEnsures portabilityLarge number of
tracked objects
Registration and lowlatency are crucial forAR systems
(C) 2008 Doug Bowman, Virginia Tech 8
AR/MR technology - tracking
Sourceless inertial orientation trackingGPS position trackingEnables mobile outdoor AR
(C) 2008 Doug Bowman, Virginia Tech 9
Mobile outdoor AR
“Backpack systems” User wears/carries:
ComputerHMD Inertial trackerGPS unit/antenna Input device(s)
(C) 2008 Doug Bowman, Virginia Tech 10
AR video examples
(C) 2008 Doug Bowman, Virginia Tech 11
Mixed Reality Interfaces
KARMA, Feiner, et al. 1993
Azuma (1997)combine real and virtual
objects interactive in real timevirtual objects are
registered in 3Dphysical world
(C) 2008 Doug Bowman, Virginia Tech 12
Challenges in AR Interfaces
Conflict between realworld and virtualNot neatly separated
anymore Limitations of displays
Precise, fast registration &tracking
Spatially seamless display Limitations of controllers
Precise, fast registration &tracking
Spatially seamless interactivityImage Copyright Sony CSL
(C) 2008 Doug Bowman, Virginia Tech 13
AR interfaces as 3D informationbrowsers (I)
3D virtual objects areregistered in 3D See-through HMDs, 6DOF
optical, magnetic trackers “VR in Real World”
Interaction 3D virtual viewpoint
control Applications
Visualization, guidance,training State, et al. 1996
(C) 2008 Doug Bowman, Virginia Tech 14
AR interfaces as context-basedinformation browsers (II) Information is registered to real-world
context Hand held AR displaysHand held AR displays
Video-see-through (Video-see-through (RekimotoRekimoto,,1997) or non-see through1997) or non-see through((Fitzmaurice, et al. 1993Fitzmaurice, et al. 1993))
Magnetic trackers or computerMagnetic trackers or computervision basedvision based
Interaction Manipulation of a window
into information space Applications
Context-aware information displays Rekimoto, et al. 1997
(C) 2008 Doug Bowman, Virginia Tech 15
AR Info Browsers (III):Pros and Cons
Important class of ARinterfacesWearable computersAR simulation, training
Limited interactivityModification and
authoring virtual contentis difficult Rekimoto, et al. 1997
(C) 2008 Doug Bowman, Virginia Tech 16
3D AR Interfaces (I)
Virtual objects are displayed in 3Dspace and can be alsomanipulated in 3D See-through HMDs and 6DOF
head-tracking for AR display 6DOF magnetic, ultrasonic, or other
hand trackers for input Interaction
Viewpoint control 3D user interface interaction:
manipulation, selection, etc.Kiyokawa, et al. 2000
(C) 2008 Doug Bowman, Virginia Tech 17
3D AR Interfaces (II):Information Displays
How to move information in ARcontext dependent informationbrowsers?
InfoPoint (1999)Hand-held deviceComputer-vision 3D trackingMoves augmented data
between marked locationsHMD is not generally
needed, but desired sincethere are little displaycapabilities
Khotake, et al. 1999
(C) 2008 Doug Bowman, Virginia Tech 18
3D AR Interfaces (III):Pros and Cons
Important class of AR interfacesEntertainment, design, training
AdvantagesSeamless spatial interaction: User can interact with 3D
virtual object everywhere in physical spaceNatural, familiar interfaces
DisadvantagesUsually no tactile feedback and HMDs are often required Interaction gap: user has to use different devices for virtual
and physical objects
(C) 2008 Doug Bowman, Virginia Tech 19
Tangible interfaces andaugmented surfaces (I)
Basic principlesVirtual objects are projected on
a surface back projection overhead projection
Physical objects are used ascontrols for virtual objects Tracked on the surface Virtual objects are registered to
the physical objects Physical embodiment of the
user interface elementsCollaborative
Digital Desk. 1993
(C) 2008 Doug Bowman, Virginia Tech 20
Tangible Interfaces and AugmentedSurfaces (II)
Graspable interfaces, Bricks system(Fitzmaurice, et al. 1995) and Tangibleinterfaces, e.g. MetaDesk (Ullmer’97): Back-projection, infrared-illumination computer
vision tracking Physical semantics, tangible handles for virtual
interface elements
metaDesk. 1997
(C) 2008 Doug Bowman, Virginia Tech 21
Tangible Interfaces and AugmentedSurfaces (III) Rekimoto, et al.
1998Front projectionMarker-based trackingMultiple projection
surfacesTangible, physical
interfaces+ AR interaction withcomputing devices
Augmented surfaces, 1998
(C) 2008 Doug Bowman, Virginia Tech 22
Tangible Interfaces and AugmentedSurfaces (IV)Advantages
Seamless interaction flow – user hands areused for interacting with both virtual andphysical objects.
No need for special purpose input devicesDisadvantages
Interaction is limited only to 2D surfaceSpatial gap in interaction - full 3D interaction
and manipulation is difficult
(C) 2008 Doug Bowman, Virginia Tech 23
Orthogonal nature of AR interfaces(Poupyrev, 2001)
Nosame devices for
physical andvirtual objects
Yesseparate devices forphysical and virtual
objects
Interaction gap
Yesinteraction is onlyon 2D surfaces
Nointeraction iseverywhere
Spatial gap
Augmentedsurfaces
3D AR
(C) 2008 Doug Bowman, Virginia Tech 24
Tangible AR interfaces (I)
Virtual objects are registered to markedphysical “containers” HMD Video-see-through tracking and registration
using computer vision tracking Virtual interaction by using
3D physical container Tangible, physical interaction 3D spatial interaction
Collaborative
Shared Space, 1999
(C) 2008 Doug Bowman, Virginia Tech 25
Tangible AR (II): generic interfacesemantics
Tiles semantics data tiles operation tiles
menu clipboard trashcan help
Operation on tiles proximity spatial arrangements space-multiplexed Tiles, 2001
(C) 2008 Doug Bowman, Virginia Tech 26
DO NOT REMOVE!!!THIS IS A PLACEHOLDER FOR EXTRANOTES PAGE!!!!!!!
(C) 2008 Doug Bowman, Virginia Tech 27
Tangible AR (III):Space-multiplexed
Data authoring in Tiles (Poupyrev, et al. 2001). Left, outsideview of the system; right, view of the left participant.
(C) 2008 Doug Bowman, Virginia Tech 28
Tangible AR (IV): Time-multiplexedinteraction
Data authoring in WOMARinterfaces (Kato et al. 2000). Theuser can pick, manipulate andarrange virtual furniture using aphysical paddle.
(C) 2008 Doug Bowman, Virginia Tech 29
Tangible AR (V): AR - VR TransitoryInterfaces
Magic Book (Billinghurst,et al. 2001)3D pop-up book: a
transitory interfacesAugmented Reality
interfacePortal to Virtual
Reality Immersive virtual
reality experienceCollaborative
Augmented Reality
Virtual Reality
(C) 2008 Doug Bowman, Virginia Tech 30
Tangible AR (VI):ConclusionsAdvantages
Seamless interaction with both virtual andphysical toolsNo need for special purpose input devices
Seamless spatial interaction with virtual objects3D presentation of and manipulation with virtual objects
anywhere in physical space
DisadvantagesRequired HMDMarkers should be visible for reliable tracking
(C) 2008 Doug Bowman, Virginia Tech 31
Interfaces for mobile outdoor AR
Devices must behandheld
No tracking or limitedtracking for devices
Interaction at-a-distance
Tinmith project
(C) 2008 Doug Bowman, Virginia Tech 32
Challenges in AR/MR
Occlusion and depth perceptionText display and legibilityVisual differences between real and virtual
objectsRegistration and trackingBulky HMDs and other equipment