Image-Based Rendering Image-Based Rendering to Accelerate to Accelerate Interactive Interactive Walkthroughs Walkthroughs Lucent Technologies Bell Labs Innovations Daniel G. Aliaga Daniel G. Aliaga Visual Communications Visual Communications Research Research Lucent Technologies Bell Lucent Technologies Bell Labs Labs
62
Embed
Image-Based Rendering to Accelerate Interactive Walkthroughs
Lucent Technologies. Bell Labs Innovations. Image-Based Rendering to Accelerate Interactive Walkthroughs. Daniel G. Aliaga Visual Communications Research Lucent Technologies Bell Labs . 3D Models. 2.0M tris. 1.7M tris. 1.0M tris. 0.9M tris. Why Use Images?. - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Image-Based Rendering Image-Based Rendering to Accelerate to Accelerate Interactive Interactive
• 30-60 Hz30-60 Hz– High visual fidelityHigh visual fidelity
E5
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• ConclusionsConclusions
E6
Replacing Geometry with Replacing Geometry with ImagesImages
• AlgorithmAlgorithm– Select subset of modelSelect subset of model– Create image of the subsetCreate image of the subset– Cull subset and replace with imageCull subset and replace with image
• Why?Why?– Image displayed in (approx.) constant Image displayed in (approx.) constant
timetime– Image reused for several framesImage reused for several frames
E7
Simple ExampleSimple Example
E8
Simple ExampleSimple Example
E9
Simple ExampleSimple Example
E10
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• ConclusionsConclusions
E11
Geometric Geometric DiscontinuityDiscontinuity
• If we move from the center-of-projection, If we move from the center-of-projection, discontinuities appear at the borderdiscontinuities appear at the border
E12
Temporal DiscontinuityTemporal Discontinuity
• While moving, if we switch between While moving, if we switch between geometry and image, a sudden geometry and image, a sudden poppop occurs occurs
E13
ApproachesApproaches
• Geometric and Temporal Geometric and Temporal ContinuityContinuity– Error metricsError metrics– Geometry warpingGeometry warping– Image warpingImage warping– Lightfield/LumigraphLightfield/Lumigraph
E14
Error MetricsError Metrics
• Use an Use an error metricerror metric to control to control amount of discontinuityamount of discontinuity
c1
c2
[Maciel95][Shade96][Schaufler96]
E15
Error MetricError Metric
• Relies on “angular-deviation” Relies on “angular-deviation” measuring the visual quality of measuring the visual quality of using the (same) imageusing the (same) image
– Display image as a (simplified, Display image as a (simplified, textured) meshtextured) mesh• [Darsa97][Sillion97][Darsa97][Sillion97]
E29
Image WarpingImage Warping
• A raster scan of each sheet produces a A raster scan of each sheet produces a back-to-front ordering of warped pixelsback-to-front ordering of warped pixels
eye
Reference COP
E30
Image WarpingImage Warping
• McMillan and Bishop’s Warping McMillan and Bishop’s Warping EquationEquation
~Texture mappingMove pixels based on distance to eye
xx22 = = (x(x11) P) P22-1-1
(c(c11 - c - c22) + P) + P22-1-1
PP1 1 xx11
x1c1
c2x2
d
E31
Example...Example...
• Image outlined in yellowImage outlined in yellow• Viewed from image’s center-of-projectionViewed from image’s center-of-projection
E32
3D Image Warp3D Image Warp
• Single sample per pixelSingle sample per pixel
E33
Layered Depth Image Layered Depth Image WarpWarp
• Multiple samples per pixel Multiple samples per pixel – Previous occlusions are filled-inPrevious occlusions are filled-in
[Popescu98]
E34
MeshesMeshes
• (Simplified) Textured Depth Mesh(Simplified) Textured Depth Mesh– Per-pixel depth creates mesh that Per-pixel depth creates mesh that
approximates 3D parallax effectsapproximates 3D parallax effects– Image is texture-mapped onto meshImage is texture-mapped onto mesh
E35
Video Segment IIIVideo Segment III
• 3D Image Warping3D Image Warping– McMillan95McMillan95
• Flow of light at all positions and Flow of light at all positions and directionsdirections– [Levoy96][Gortler96][Levoy96][Gortler96]
• Large number of images are used as Large number of images are used as 2D slices of a 4D light function2D slices of a 4D light function
(u,v)
(s,t)
E37
Video Segment IVVideo Segment IV
• Light fieldLight field– Levoy96Levoy96
• LumigraphLumigraph– Gortler96Gortler96
E38
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• Geometry is Geometry is replaced by replaced by image to limit image to limit the number of the number of primitives to primitives to renderrender
E44
Eye
Grid viewpoint
Key ObservationKey Observation
• Less geometry Less geometry is in the view is in the view frustum from frustum from the eye than the eye than the one from the one from the grid the grid viewpoint viewpoint
Ideal portal Ideal portal image would be image would be one sampled one sampled from the current from the current eye positioneye position
E54
Creating Portal ImagesCreating Portal Images
Reference COPs
Display one of a large number of pre-computed images (~120) portal
E55
Creating Portal ImagesCreating Portal Images
portal
Reference COPs
or…
Warp one of a much smaller number of reference images
E56
Brooks HouseBrooks House
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0 50 100 150 200
Frame
Ren
der
ing
Tim
e (s
ec.)
View Frustum Culling
Portal Culling
Static Portal Images
Onyx IR
E57
Brooks HouseBrooks House
0
0.1
0.2
0.3
0.4
0.5
0 100 200 300 400 500
Frame
Re
nd
erin
g T
ime
(sec
.)
Portal Culling
Warped Portal Images
Onyx IR
E58
Video Segment VIVideo Segment VI
• Architectural Walkthroughs using Architectural Walkthroughs using Portal ImagesPortal Images– Aliaga97, Rafferty98Aliaga97, Rafferty98
E59
OverviewOverview
• Replacing Geometry with ImagesReplacing Geometry with Images• Displaying ImagesDisplaying Images
– Texture-mapping and error metricsTexture-mapping and error metrics– Geometry and image warping Geometry and image warping – Meshes, Lightfield/LumigraphMeshes, Lightfield/Lumigraph
• Image PlacementImage Placement– Automatically Bounding Model ComplexityAutomatically Bounding Model Complexity– Cells and PortalsCells and Portals
• ConclusionsConclusions
E60
Image QualityImage Quality
• What about measuring quality? What about measuring quality? – Need a perceptual quality metric!Need a perceptual quality metric!
• We knowWe know– Texture-mapping: bad perspective, Texture-mapping: bad perspective,
small distortions believable (geometry small distortions believable (geometry warping)warping)
– IBR: correct perspective, disocclusionsIBR: correct perspective, disocclusions– Meshes: stretching of skinsMeshes: stretching of skins
• Cannot sample all visible surfacesCannot sample all visible surfaces– Smarter reconstruction/resampling?Smarter reconstruction/resampling?
• Can only sample surfaces at a fixed Can only sample surfaces at a fixed resolutionsresolutions– Multi-resolution reference images?Multi-resolution reference images?
E62
AcknowledgmentsAcknowledgments
• Authors of the Video SegmentsAuthors of the Video Segments• ModelsModels