Immersive Rendering. General Idea ► Head pose determines eye position Why not track the eyes? ► Eye position determines perspective point ► Eye properties.

Immersive RenderingImmersive Rendering

General IdeaGeneral Idea►Head pose determines eye positionHead pose determines eye position

Why not track the eyes?Why not track the eyes?►Eye position determines perspective Eye position determines perspective

pointpoint►Eye properties determine what part of Eye properties determine what part of

the real world can be seenthe real world can be seen►Screen determines how the virtual Screen determines how the virtual

world can be seenworld can be seen Screen is a window into the virtual worldScreen is a window into the virtual world

Virtual World

Head Pose

Eye Position

Eye Properties

Display Pose

Head-Eye-Screen-World Head-Eye-Screen-World RelationshipsRelationships

Depth effectsDepth effects

►Most displays are Most displays are 2D (pixels)2D (pixels)

► Yet we want to Yet we want to show a 3D world!show a 3D world!

►How can we do this?How can we do this? We can include We can include

‘cues’ in the image ‘cues’ in the image that give our brain that give our brain 3D information about 3D information about the scenethe scene

These cues are These cues are visual depth cuesvisual depth cues

Visual Depth CuesVisual Depth Cues

► Cues about the 3Cues about the 3rdrd dimension – total of 10dimension – total of 10

► Monoscopic Depth Cues Monoscopic Depth Cues (single 2D image) [6](single 2D image) [6]

► Stereoscopic Depth Stereoscopic Depth Cues (two 2D images) Cues (two 2D images) [1][1]

► Motion Depth Cues Motion Depth Cues (series of 2D images) [1](series of 2D images) [1]

► Physiological Depth Physiological Depth Cues (body cues) [2]Cues (body cues) [2]

Monoscopic Depth CuesMonoscopic Depth Cues► InterpositionInterposition

An occluding object is closerAn occluding object is closer► ShadingShading

Shape and shadowsShape and shadows► SizeSize

The larger object is closerThe larger object is closer► Linear PerspectiveLinear Perspective

Parallel lines converge at a Parallel lines converge at a single pointsingle point

Higher the object is Higher the object is (vertically), the further it is(vertically), the further it is

► Surface Texture GradientSurface Texture Gradient More detail for closer More detail for closer

objectsobjects► Atmospheric effects Atmospheric effects

Further away objects are Further away objects are blurrier and dimmerblurrier and dimmer

► Images from Images from http://ccrs.nrcan.gc.ca/resource/tutor/stereo/chap2/chapter2_5_e.phphttp://ccrs.nrcan.gc.ca/resource/tutor/stereo/chap2/chapter2_5_e.php

Physiological Physiological Depth CuesDepth Cues

► Accommodation Accommodation – – focusing adjustment focusing adjustment made by the eye to made by the eye to change the change the shapeshape of of the lens. (up to 3 m)the lens. (up to 3 m)

► ConvergenceConvergence – – movement of the eyes movement of the eyes to bring in the an to bring in the an object into the same object into the same location on the retina location on the retina of each eye.of each eye.

Stereoscopic Depth CueStereoscopic Depth Cue

► StereopsisStereopsis► Stereoscopic Display Stereoscopic Display

TechnologyTechnology► Computing Computing

Stereoscopic ImagesStereoscopic Images► Stereoscopic Display Stereoscopic Display

and HTDs.and HTDs.►Works for objects < Works for objects <

5m. 5m. Why?Why?

StereopsisStereopsisThe result of the two slightly different views of the world that our laterally-displaced eyes receive.

Screen ParallaxScreen Parallax

Pleft – Point P projected screen location as seen by left eyePright – Point P projected screen location as seen by right eye

Screen parallax - distance between Pleft and Pright

P

Left eye position

Right eye position

Pleft

Pright

Pright

Pleft

P

Display Screen

Object with positive parallax

Object with negative parallax

How to create correct left- How to create correct left- and right-eye viewsand right-eye views

► What do you What do you need to specify need to specify for most for most rendering rendering engines?engines? EyepointEyepoint Look-at PointLook-at Point Field-of-View Field-of-View

or location of or location of Projection Projection PlanePlane

View Up View Up DirectionDirection

P

Left eye position

Right eye position

Pleft

Pright

Pright

Pleft

P

Display Screen

Object with positive parallax

Object with negative parallax

Basic Perspective Projection Basic Perspective Projection Set Up from Viewing Set Up from Viewing

ParamentersParamentersY

Z

X

Projection Plane is orthogonal to one of the major axes (usually Z). That axis is along the vector defined by the eyepoint and the look-at point.

What doesn’t usuallyworkWhat doesn’t usuallywork

•Each view has a different projection plane

•Each view will be presented (usually) on the same plane

What Does WorkWhat Does Work

i i

What if you don’t have 2 What if you don’t have 2 displays?displays?

Look at pointEyeLocations

Look at point

EyeLocations

No

Yes

Asymmetric Camera FrustumAsymmetric Camera Frustum

Images from: http://local.wasp.uwa.edu.au/~pbourke/miscellaneous/stereographics/stereorender/

Cue Mismatch: Cue Mismatch: Accommodation/ Accommodation/

ConvergenceConvergence

Display Screen

Position Dependence Position Dependence (without head-tracking)(without head-tracking)

Interocular DependanceInterocular Dependance

F

Modeled Point

Perceived Point

Projection Plane

True EyesModeled Eyes

Two View Points with Head-TrackingTwo View Points with Head-Tracking

Projection Plane

Modeled Point

Perceived Points

Modeled Eyes

True Eyes

Stereoscopic DisplayStereoscopic Display

►Stereoscopic images are easy to do Stereoscopic images are easy to do badly, hard to do well, and impossible badly, hard to do well, and impossible

to do correctly.to do correctly.

Stereoscopic Stereoscopic DisplaysDisplays

► Stereoscopic display Stereoscopic display systems presents systems presents each eye with a each eye with a slightly different slightly different view of a scene.view of a scene. Time-parallel – 2 Time-parallel – 2

images same timeimages same time Time-multiplexed – 2 Time-multiplexed – 2

images one right images one right after anotherafter another

Time Parallel Stereoscopic Time Parallel Stereoscopic DisplayDisplay

Two ScreensTwo Screens► Each eye sees a Each eye sees a

different screendifferent screen► Optical system directs Optical system directs

correct viewcorrect view► HMD stereoHMD stereo

Single ScreenSingle Screen► Two different images Two different images

projectedprojected► Images are colored or Images are colored or

polarized “differently” polarized “differently” ► User wears glasses to User wears glasses to

filter out L image for L eye filter out L image for L eye and R image for R eyeand R image for R eye

Passive Passive Polarized Polarized ProjectionProjection

► Linear PolarizationLinear Polarization Ghosting increases Ghosting increases

when you tilt headwhen you tilt head Reduces brightness of Reduces brightness of

image by about ½image by about ½ Potential Problems Potential Problems

with Multiple Screenswith Multiple Screens

► Circular PolarizationCircular Polarization Reduces ghosting Reduces ghosting Reduces brightness Reduces brightness Reduces crispnessReduces crispness

Problem with Polarization Problem with Polarization Technology for Multiple Technology for Multiple

ScreensScreens► With linear polarization, With linear polarization,

the separation of the left the separation of the left and right eye images is and right eye images is dependent on the dependent on the orientation of the glasses orientation of the glasses with respect to the with respect to the projected image. projected image.

► The floor image cannot The floor image cannot be aligned with both the be aligned with both the side screens and the front side screens and the front screens at the same time.screens at the same time.

► Solution? Solution?

Time Multiplexed DisplayTime Multiplexed Display

► Left and right-eye Left and right-eye views of an image views of an image are computed are computed

► Alternately displayed Alternately displayed on the screenon the screen

► A shuttering system A shuttering system occludes the right occludes the right eye when the left-eye when the left-eye image is being eye image is being displayeddisplayed

Shutter GlassesShutter Glasses

GhostingGhosting

►Some of L image is visible to R eye Some of L image is visible to R eye and vice versaand vice versa

►Not a problem for HMDs, why?Not a problem for HMDs, why?►About equal problem for polarized and About equal problem for polarized and

shuttered glassesshuttered glasses►Pixel persistencePixel persistence►Vertical screen position of the image.Vertical screen position of the image.

Other stereo limiting factorsOther stereo limiting factors

►Right and left-eye images do not match Right and left-eye images do not match in color, size, vertical alignment.in color, size, vertical alignment.

►Distortion caused by the optical systemDistortion caused by the optical system►ResolutionResolution►HMDs interocular settingsHMDs interocular settings►Computational model does not match Computational model does not match

viewing geometry.viewing geometry.

SummarySummary

►Monoscopic – Interposition is Monoscopic – Interposition is strongest.strongest.

►Stereopsis is very strong.Stereopsis is very strong.►Relative Motion is also very strong (or Relative Motion is also very strong (or

stronger).stronger).►Physiological is weakest (we don’t Physiological is weakest (we don’t

even use them in VR!)even use them in VR!)

Stereo Capable HMDsStereo Capable HMDs

►Each has some way to send two Each has some way to send two independent signalsindependent signals Usually dependent on both graphics card Usually dependent on both graphics card

and APIand API

►Lab has 4 different HMDs each with Lab has 4 different HMDs each with different requirements to produce different requirements to produce stereostereo

►Head Tracking is a separate conceptHead Tracking is a separate concept

Emagin Z800Emagin Z800

► Must use machine with Must use machine with Geforce 7900 GPU (on Geforce 7900 GPU (on back wall) to get quality back wall) to get quality stereostereo

► NVIDIA API (old version) to NVIDIA API (old version) to set convergence and set convergence and separationseparation

► Must be 800x600x60hzMust be 800x600x60hz► Controlled Internally Controlled Internally

LRLRLRLRLRLRLRLRLRLRLRLRLRLR► Best resolution, best color, Best resolution, best color,

best head-tracking, best best head-tracking, best comfort, worst hardware comfort, worst hardware requirementrequirement

Vuzix VR920Vuzix VR920

► Use their API Use their API ► Must hack Ogre or Must hack Ogre or

use OpenGL or use OpenGL or DirectX directlyDirectX directly

► 640x480, frame 640x480, frame interleaved interleaved LRLRLRLRLRLRLRLRLRLRLRLRLRLRLRLR….….

► Very uncomfortable, Very uncomfortable, bad FOV, but bad FOV, but internal head internal head trackingtracking

Vuzix Wrap 920Vuzix Wrap 920

►Use Internal API (like Use Internal API (like VR920)VR920)

►Use side-by-side stereo Use side-by-side stereo (two viewports) (two viewports) settings in control boxsettings in control box

►640x480, two 320x480 640x480, two 320x480 viewportsviewports

►Coolest looking, worst Coolest looking, worst FOVFOV

VR Research V6VR Research V6

►Dual Input (L and R Dual Input (L and R connections)connections) Must have two outputs on Must have two outputs on

graphics cardgraphics card►1280x480 in horizontal 1280x480 in horizontal

span window modespan window mode Two 640x480 viewports Two 640x480 viewports

side by sideside by side►2 640x480 fullscreen 2 640x480 fullscreen

windowswindows►Best FOV (60 degrees)Best FOV (60 degrees)

Other stereo optionsOther stereo options

► 2 Viewsonic projectors 2 Viewsonic projectors and Asus 3D monitorand Asus 3D monitor 1024x768x120hz 1024x768x120hz

frame interleavedframe interleaved Use new NVIDIA Api Use new NVIDIA Api

(convergence and (convergence and separation) or Use separation) or Use QuadBuffered OpenGL QuadBuffered OpenGL (or hacked ogre)(or hacked ogre)

Must use newer NVIDIA Must use newer NVIDIA (desktop) GPU or (desktop) GPU or hacked laptop driverhacked laptop driver

Use NVIDIA 3D Vision Use NVIDIA 3D Vision Shutter glassesShutter glasses

Anaglyph StereoAnaglyph Stereo

►Red Cyan glassesRed Cyan glasses►Render scene twiceRender scene twice

First pass – Left - First pass – Left - Convert to grayscale Convert to grayscale then to Red (R then to Red (R channel)channel)

Second pass – Right - Second pass – Right - Convert to grayscale Convert to grayscale then to Cyan (BG then to Cyan (BG channels)channels)

►Worst effectWorst effect►Easiest to deployEasiest to deploy