Virtual Reality David Johnson. What is Virtual Reality?

Virtual Reality Virtual Reality

David JohnsonDavid Johnson

What is Virtual Reality?What is Virtual Reality?

What is Virtual Reality?What is Virtual Reality?► Replace sensory input from Replace sensory input from

real world with computer real world with computer generated sensationsgenerated sensations SightSight HearingHearing TouchTouch SmellSmell TasteTaste PropriocepticProprioceptic

► Create a sense of immersion Create a sense of immersion in that computer-generated in that computer-generated worldworld presencepresence

1956 – Heilig1956 – Heilig

VR as a ParadigmVR as a Paradigm

►VR is a paradigm for thinking about VR is a paradigm for thinking about human-computer interfaceshuman-computer interfaces What is the paradigm for a Windows PC?What is the paradigm for a Windows PC? What is the paradigm for an Xbox 360?What is the paradigm for an Xbox 360? Others?Others?

►Augmented realityAugmented reality►Ubiquitous computingUbiquitous computing

VR as a ParadigmVR as a Paradigm

►VR is a paradigm for thinking about VR is a paradigm for thinking about human-computer interfaceshuman-computer interfaces What is the paradigm for a Windows PC?What is the paradigm for a Windows PC? What is the paradigm for an Xbox 360?What is the paradigm for an Xbox 360? Others?Others?

►VR paradigm is VR paradigm is Immerse the user in dataImmerse the user in data Natural, emergent interfaceNatural, emergent interface

VR HistoryVR History► 70’s and 80’s – early work70’s and 80’s – early work► Early 1990’s – immense enthusiasmEarly 1990’s – immense enthusiasm

Virtuality arcade gamesVirtuality arcade games► Late 90’s – immense disappointmentLate 90’s – immense disappointment

Researchers complain of the attention www is Researchers complain of the attention www is gettinggetting

► Current – commodity elements availableCurrent – commodity elements available WiiWii Touch screens – iPhone, etc.Touch screens – iPhone, etc. Personal video displaysPersonal video displays Second Life, MMORPGSecond Life, MMORPG 3D IMAX3D IMAX G-speakG-speak

VR and Sensory InputVR and Sensory Input

►Much effort has been in developing Much effort has been in developing technologies to immerse the usertechnologies to immerse the user

VR and SightVR and SightHead-mounted displayHead-mounted display PowerwallPowerwall CAVE CAVE

(HMD)(HMD) CAVE movieCAVE movie

Ivan Sutherland developed Ivan Sutherland developed the modern HMD at Utah the modern HMD at Utah (1969)(1969)

CurrentCurrent

► Better HMDsBetter HMDs SensicsSensics

► Tiled OLED displaysTiled OLED displays► 4K x 2K resolution4K x 2K resolution► PanoramicPanoramic

► Autostereoscopic Autostereoscopic displaysdisplays

► Higher-res CAVEsHigher-res CAVEs

VR and TouchVR and Touch

►Two aspectsTwo aspects Know where the user’s body isKnow where the user’s body is

►TrackingTracking

Apply contact sensations to userApply contact sensations to user►HapticsHaptics►Tactile interfacesTactile interfaces

Position TrackingPosition Tracking

► Use sensors to track Use sensors to track hand, head, bodyhand, head, body PositionPosition OrientationOrientation

► Sensors areSensors are MagneticMagnetic InertialInertial UltrasoundUltrasound OpticalOptical

Head TrackingHead Tracking

►Very important forVery important for HMDHMD CAVECAVE Fishtank VRFishtank VR

►Can be done with a WiimoteCan be done with a Wiimote Johnny LeeJohnny Lee

►VR class project to use laptop cameraVR class project to use laptop camera Sensitive to IR lightSensitive to IR light

Hand TrackingHand Tracking

► Allow natural Allow natural interaction with interaction with virtual worldvirtual world GrabGrab GesturesGestures

► CybergloveCyberglove

Body TrackingBody Tracking

►More often used More often used in motion capture in motion capture systemssystems FilmFilm TVTV GamesGames

HapticsHaptics

►Haptics = touchHaptics = touch►Virtual objects apply real forcesVirtual objects apply real forces

ExampleExample

Sound in VRSound in VR

►Very important for immersionVery important for immersion►Study of PTSD veterans using VR Study of PTSD veterans using VR

therapy found sound critical for therapy found sound critical for creating sense of presencecreating sense of presence

►Use headphonesUse headphones►Render soundRender sound►ExampleExample

Olfactory DisplaysOlfactory Displays

► DIVEpakDIVEpak Computer controlled Computer controlled

scratch and sniffscratch and sniff► Smell-o-Vision (K Smell-o-Vision (K

Opticom)Opticom) 6 gels, USB controlled6 gels, USB controlled

► Fire TrainerFire Trainer Odors range from burning Odors range from burning

wood, grease and rubber wood, grease and rubber to sulfur, oil and diesel to sulfur, oil and diesel exhaust.exhaust.

Smell CannonSmell Cannon► Track target with Track target with

cameracamera► Aim below eyesAim below eyes► Prerecorded smellsPrerecorded smells

Gustatory DisplaysGustatory Displays

► Simulate the Simulate the experience of eatingexperience of eating

► Iwata food simulatorIwata food simulator► Capture forces of Capture forces of

eatingeating► PlaybackPlayback► Combine flavorsCombine flavors

five basic taste five basic taste sensations: sweet, sensations: sweet, sour, bitter, salty sour, bitter, salty and umami and umami

VR InterfacesVR Interfaces

►How do we tell the computer to do How do we tell the computer to do things?things?

►How do we select things?How do we select things?►How do we navigate around?How do we navigate around?

VR Interface ChallengesVR Interface Challenges

► IntuitiveIntuitive Make interaction work like the real worldMake interaction work like the real world Minimize cognitive overheadMinimize cognitive overhead

►AugmentationAugmentation Give users new capabilitiesGive users new capabilities

From the BeginningFrom the Beginning

► Sutherland and Sutherland and VickersVickers Sorcerer’s Sorcerer’s

Apprentice (1972)Apprentice (1972)

► Track stylusTrack stylus► Selection of verticesSelection of vertices

Intersection of cube Intersection of cube at tip of stylusat tip of stylus

► Physical menu with Physical menu with see-through HMD!see-through HMD!

Pointing: Put That TherePointing: Put That There

►19791979►Ray from tracked handRay from tracked hand►Speech interfaceSpeech interface►moviemovie

Pointing: JD-CAD (1993)Pointing: JD-CAD (1993)

► ““Laser gun” from Laser gun” from handhand Tracker noiseTracker noise Harder to select far Harder to select far

awayaway► SpotlightSpotlight

Add a cone to the rayAdd a cone to the ray Select objects based Select objects based

onon► Distance from cone Distance from cone

axisaxis► Distance from handDistance from hand

Silk CursorSilk Cursor

► Replace wireframe Replace wireframe selection box with selection box with translucent boxtranslucent box Visual cues to Visual cues to

containmentcontainment

Hand: GoGo Interaction Hand: GoGo Interaction (1996)(1996)

► Go-Go uses Non-Linear Go-Go uses Non-Linear mapping between mapping between virtual and real handvirtual and real hand► Control-display ratioControl-display ratio► Stretch go-go variationStretch go-go variation► Pros:Pros:

Extended reach when neededExtended reach when needed Direct manipulationDirect manipulation

► Cons:Cons: Reach still limited by arm Reach still limited by arm lengthlength Precision suffers when reach isPrecision suffers when reach is extended (low level of control)extended (low level of control)

MovieMovie

Image Plane TechniquesImage Plane Techniques

► Point or gesture at an objects Point or gesture at an objects projection onto the viewing projection onto the viewing planeplane ““head-crusher”head-crusher”

► Kids in the hallKids in the hall ““Sticky finger”Sticky finger”

► Similar to ray castingSimilar to ray casting► Pros:Pros:

Very intuitiveVery intuitive Allows user to reach objects Allows user to reach objects

at an arbitrary distanceat an arbitrary distance► Cons:Cons:

Limited by the need for line Limited by the need for line of sightof sight

Can be fatiguingCan be fatiguing Virtual hand may obscure Virtual hand may obscure

small objectssmall objects

World in MiniatureWorld in Miniature

► User holds dynamic User holds dynamic map in one handmap in one hand

► Objects can be Objects can be moved in mapmoved in map

Exploring Large Virtual Exploring Large Virtual SpacesSpaces

►Tracked spaces Tracked spaces are expensiveare expensive

►Redirected Redirected walkingwalking MovieMovie

►TreadmillsTreadmills bicyclesbicycles

Movie

Menus in Virtual SpaceMenus in Virtual Space

►Ring MenusRing Menus JDCAD 1993JDCAD 1993

►LiangLiang

ISAAC 1995 ISAAC 1995 ►Mark MineMark Mine

Rotate hand to moveRotate hand to moveselection pointselection point

Menus in Virtual SpaceMenus in Virtual Space

► Pen and TabletPen and Tablet Track a tablet and Track a tablet and

penpen Put 2D menus on Put 2D menus on

tablettablet

Lindeman 1999

Physical PropsPhysical Props

►Tablet and penTablet and pen►Passive hapticsPassive haptics

Neurosurgical propsNeurosurgical props

►Haptic HandHaptic Hand

Menus in Virtual SpaceMenus in Virtual Space

► BowmanBowman► Pinch GlovesPinch Gloves

Select with thumb to Select with thumb to fingerfinger

High-level menu on High-level menu on ND handND hand

Secondary menu of Secondary menu of D handD hand

First triesFirst tries► Scrolling menu using Scrolling menu using

pinches orpinches or► More items on pinkieMore items on pinkie

Menus in Virtual SpacesMenus in Virtual Spaces

► ‘‘Virtual tricorder’Virtual tricorder’►Wloka 1995Wloka 1995

GesturesGestures

►GIVEN (1992)GIVEN (1992) Neural net recognitionNeural net recognition 20 gestures20 gestures

►Fly, grab, etc.Fly, grab, etc.

►MineMine ““Physical mnemonics”Physical mnemonics”

►Pull-down menus from near headPull-down menus from near head►Delete by throwing over shoulderDelete by throwing over shoulder

Numerical InputNumerical Input

►No keyboards!No keyboards!►Mark MineMark Mine

A digit at a timeA digit at a time Sliders too Sliders too

impreciseimprecise

Who is using VR?Who is using VR?

►MilitaryMilitary►GovernmentGovernment►Big businessBig business

►Common theme?Common theme?

MilitaryMilitary

► TrainingTraining► TelepresenceTelepresence

Scientific visualizationScientific visualization

► Virtual Wind TunnelVirtual Wind Tunnel NASANASA

Stress TherapyStress Therapy

► VR therapy movieVR therapy movie

Business infrastructureBusiness infrastructure

► telecollaborationtelecollaboration

MedicalMedical

► Medical trainingMedical training Surgical playbackSurgical playback

► Medical planning Medical planning and rehearsaland rehearsal

► Surgical assistanceSurgical assistance ultrasoundultrasound BiopsyBiopsy

Car companiesCar companies

► EvaluationEvaluation StyleStyle ErgonomicsErgonomics ManufacturabilityManufacturability

► TrainingTraining

ConclusionConclusion

►VR is like the space programVR is like the space program Lots of side benefitsLots of side benefits Still no clear victory on the overall goalStill no clear victory on the overall goal

►Device technology is much improvedDevice technology is much improved►UI ideas are still primitive and UI ideas are still primitive and

inelegantinelegant