Realistic Reflections and Refractions on Graphics Hardware with Hybrid Rendering and Layered Environment Maps Ziyad S. Hakura John M. Snyder Ziyad S. Hakura.

Realistic Reflections and Refractionson Graphics Hardware with Hybrid Rendering

and Layered Environment Maps

Realistic Reflections and Refractionson Graphics Hardware with Hybrid Rendering

and Layered Environment Maps

Ziyad S. Hakura

John M. Snyder

Ziyad S. Hakura

John M. Snyder

Goal: Real-time rendering of Parameterized Image Spacesincluding Photorealistic, Reflective/Refractive ObjectsGoal: Real-time rendering of Parameterized Image Spacesincluding Photorealistic, Reflective/Refractive Objects

Viewpoint PositionViewpoint Position

ObjectMotionObjectMotion

Parameterized Image SpacesParameterized Image Spaces

• Space can be 1D, 2D, 3D or more

• Content author specifies parameters

• Space can be 1D, 2D, 3D or more

• Content author specifies parameters

Light motionLight motion

Viewpoint positionViewpoint position

Object motionObject motion

Parameterized Image Spaces: Application Examples

Parameterized Image Spaces: Application Examples

AnimatedMechanismAnimatedMechanism viewpoint x timeviewpoint x time

Cockpit LightingCockpit Lighting

Day lightDay light

Night skyNight sky

viewpoint x time of dayviewpoint x time of day

Interactive Toy Story

Interactive Toy Story

• Limited viewpoint motion Head motion parallax puts the user “in the scene”

• Character parameters e.g. happiness/sadness Rose98, Gleicher98, Popović99

• Limited viewpoint motion Head motion parallax puts the user “in the scene”

• Character parameters e.g. happiness/sadness Rose98, Gleicher98, Popović99

viewpoint x time x emotional stateviewpoint x time x emotional state

Overall ModelOverall Model

Fit EnvironmentalImpostors to Match

Hybrid Rendering

Render Ray-TracedImages Offline

PR

EP

RO

CE

SS

RU

N-T

IME

Overall ModelOverall Model

Fit EnvironmentalImpostors to Match

Hybrid Rendering

Render Ray-TracedImages Offline

PR

EP

RO

CE

SS

RU

N-T

IME

Overall ModelOverall Model

Fit EnvironmentalImpostors to Match

Hybrid Rendering

Render Ray-TracedImages Offline

PR

EP

RO

CE

SS

RU

N-T

IME

Overall ModelOverall Model

Fit EnvironmentalImpostors to Match

Hybrid Rendering

Render Ray-TracedImages Offline

PR

EP

RO

CE

SS

RU

N-T

IME

Ray Tracing vs. Hybrid RenderingRay Tracing vs. Hybrid Rendering

Ray TracingRay Tracing Hybrid RenderingHybrid Rendering

local lensobject

environmental impostor

local lensobject

environmental impostor

Related WorkRelated Work

• IBR Gortler96, Levoy96, Miller98, Wood00

• Reflections Diefenbach96, Ofek98, Cabral99, Lischinski98, Bastos99

• Refractions Heidrich99, Zongker99, Chuang00

• Ray Tracing Kajiya86, Pharr97

• IBR Gortler96, Levoy96, Miller98, Wood00

• Reflections Diefenbach96, Ofek98, Cabral99, Lischinski98, Bastos99

• Refractions Heidrich99, Zongker99, Chuang00

• Ray Tracing Kajiya86, Pharr97

Our Related WorkOur Related Work

• Parameterized Texture Maps Hakura00 “pasted on” look away from pre-rendered views

• Parameterized Environment Maps Hakura01 view samples must be close together no refractions

• Parameterized Texture Maps Hakura00 “pasted on” look away from pre-rendered views

• Parameterized Environment Maps Hakura01 view samples must be close together no refractions

OutlineOutline

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

Greedy Ray Path Shading ModelGreedy Ray Path Shading Model

NN

Reflective Path

Reflective Path

RefractivePath

RefractivePath

Trace two ray paths until rays exit object.Trace two ray paths until rays exit object.

Refractive Lens ObjectRefractive Lens Object

Greedy Ray Path Shading ModelGreedy Ray Path Shading Model

NN

Reflective Path

Reflective Path

RefractivePath

RefractivePath

Trace two ray paths until rays exit object.Propagate child ray of greatest Fresnel coefficient.Trace two ray paths until rays exit object.Propagate child ray of greatest Fresnel coefficient.

Refractive Lens ObjectRefractive Lens Object

Comparison of Shading ModelsComparison of Shading Models

Full binary ray treeFull binary ray tree Two-term greedyray path

Two-term greedyray path

OutlineOutline

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

Impostor Fitting AlgorithmImpostor Fitting Algorithm

For each point in parameter space

For each local lens object

• generate outgoing rays in ray tracer

• record intersections with environment

• cluster intersections into layers

• fit textured impostor to each layer

For each point in parameter space

For each local lens object

• generate outgoing rays in ray tracer

• record intersections with environment

• cluster intersections into layers

• fit textured impostor to each layer

Layered Environment Maps (EMs)Layered Environment Maps (EMs)

local lensobject L1

L2

L3environmental layers

EM Geometry TypesEM Geometry Types

• Sphere at infinity• Sphere• Box• Ellipsoid• Cylinder• Quadrilateral

• Sphere at infinity• Sphere• Box• Ellipsoid• Cylinder• Quadrilateral

Linear Hardware ModelLinear Hardware Model

A x = bA x = bUnknown

Texture PixelsUnknown

Texture PixelsRay-Traced

ImageRay-Traced

ImageHW Filter

CoefficientsHW Filter

Coefficients

HardwareRender

HardwareRender

ScreenScreenTextureTexture

Inferred EMsInferred EMs

original imageoriginal image

Inferred EMsInferred EMs

L1

cup

L1

cup

L2

cols

L2

cols

L3

walls

L3

walls

ReflectionTerm

ReflectionTerm

RefractionTerm

RefractionTerm Inferred Environment MapsInferred Environment Maps

OutlineOutline

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

Adaptive TessellationAdaptive Tessellation• Two criteria:

ray path “topology” outgoing ray distance

• Consider both terms of shading model

• Two criteria:

ray path “topology” outgoing ray distance

• Consider both terms of shading model

Texture Coordinate GenerationTexture Coordinate Generation

OutgoingRay

EM TextureMapping

EM Texture

(u,v)

EM Geometry Layers

Overlaying/Blending PassesOverlaying/Blending Passes

LR = L1

(uR1,vR1) over L2 (uR2,vR2) over L3

(uR3,vR3)

LT = L1 (uT1,vT1) over L2

(uT2,vR2) over L3 (uT3,vT3)

LR = L1

(uR1,vR1) over L2 (uR2,vR2) over L3

(uR3,vR3)

LT = L1 (uT1,vT1) over L2

(uT2,vR2) over L3 (uT3,vT3)

reflection termreflection term

refraction termrefraction term

LR = L1

(uR1,vR1) over L2 (uR2,vR2) over L3

(uR3,vR3)

LT = L1 (uT1,vT1) over L2

(uT2,vR2) over L3 (uT3,vT3)

Single viewpoint (i): Li = LRi FRGR + LT

i FTGT

Blended viewpoint: L = Li + Li+1 (1-)

Requires 4n passes, where n is number of layers

LR = L1

(uR1,vR1) over L2 (uR2,vR2) over L3

(uR3,vR3)

LT = L1 (uT1,vT1) over L2

(uT2,vR2) over L3 (uT3,vT3)

Single viewpoint (i): Li = LRi FRGR + LT

i FTGT

Blended viewpoint: L = Li + Li+1 (1-)

Requires 4n passes, where n is number of layers

reflection termreflection term

refraction termrefraction term

Overlaying/Blending PassesOverlaying/Blending Passes

OutlineOutline

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

• Greedy Ray Path Shading Model

• Fitting Environment Maps

• Hybrid Rendering Runtime

• Results and Conclusions

• 1D viewspace circling teapot

• 8º separation between view samples

• simultaneous solution over 5 viewpoints

• 3 layers for teapot, 2 for cup

• 1D viewspace circling teapot

• 8º separation between view samples

• simultaneous solution over 5 viewpoints

• 3 layers for teapot, 2 for cup

Experimental SetupExperimental Setup

Ray-Traced vs. HybridRay-Traced vs. Hybrid

Ray-Traced480 sec/frameRay-Traced

480 sec/frameHybrid Rendered

19 sec/frameHybrid Rendered

19 sec/frame

Benefit of Hybrid Renderingover Ray-Tracing

Benefit of Hybrid Renderingover Ray-Tracing

• Lower cost Adaptive ray-tracing algorithm

• Lower cost and higher predictability Greedy two-term shading model Environment substitution with layered shells

• Lower cost Adaptive ray-tracing algorithm

• Lower cost and higher predictability Greedy two-term shading model Environment substitution with layered shells

Future WorkFuture Work

• Compression with hybrid rendering

• Handling glossy objects

• Matching full binary tree renderings

• More efficient pre-rendering Multi-dimensional Ray-Tracing

• Accelerating ray traced animations

• Making it faster (local ray tracing hardware?)

• Compression with hybrid rendering

• Handling glossy objects

• Matching full binary tree renderings

• More efficient pre-rendering Multi-dimensional Ray-Tracing

• Accelerating ray traced animations

• Making it faster (local ray tracing hardware?)

EndEnd

Overall ModelOverall Model

Fit EnvironmentalImpostors to Match

Hybrid Rendering

Render Ray-TracedImages Offline

PR

EP

RO

CE

SS

RU

N-T

IME

Greedy Ray Path Shading ModelGreedy Ray Path Shading Model

==

resultresult

reflection termreflection termrefraction termrefraction term

** **++

T T FT GT FT GT RR FR GR FR GR

Surface Light Fields [Miller98,Wood00] Surface Light Fields [Miller98,Wood00]

Surface Light Field Surface Light Field

Dense sampling over surface points of

low-resolution lumispheres

Dense sampling over surface points of

low-resolution lumispheres

PEMPEM

Sparse sampling over viewpoints of

high-resolution EMs

Sparse sampling over viewpoints of

high-resolution EMs

Why Fit Impostors?Why Fit Impostors?

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

Why Fit Impostors?Why Fit Impostors?

• geometric error of impostor• geometric error of impostor

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

Fitting accounts for: Fitting accounts for:

Why Fit Impostors?Why Fit Impostors?

• geometric error of impostor• view-dependent shading in environment

• geometric error of impostor• view-dependent shading in environment

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

Fitting accounts for: Fitting accounts for:

Why Fit Impostors?Why Fit Impostors?

• geometric error of impostor• view-dependent shading in environment

• geometric error of impostor• view-dependent shading in environment

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

local lens object

outgoing ray

environmentalobject

impostor

intersection

impostor center of projection

Fitting accounts for: Fitting accounts for:

• simultaneous inference over multiple views nearby views of lens object

“direct” views (without lens object)

solve confidence-weighted least squares

• simultaneous inference over multiple views nearby views of lens object

“direct” views (without lens object)

solve confidence-weighted least squares

• ray propagation• ray propagation

incomingincoming outgoingoutgoing

Preventing Disocclusion ArtifactsPreventing Disocclusion Artifacts

withwith withoutwithout

Preventing Disocclusion ArtifactsPreventing Disocclusion Artifacts