Marc Levoy IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry.”

Marc Levoy

IBM / IBR

“The study of image-based modeling

and rendering is the study of

sampled representations of geometry.”

Marc Levoy

Shortcutting thevision/graphics pipeline

geometry

views

real world

graphicspipeline

image-basedrendering

visionpipeline

(from M. Cohen)

Marc Levoy

Apple QuickTime VR[Chen, Siggraph ’95]

• outward-looking– panoramic views taken at regularly spaced points

• inward-looking– views taken at points on the surface of a sphere

Marc Levoy

View interpolationfrom a single view

1. Render object

2. Convert Z-buffer to range image

3. Re-render from new viewpoint

4. Use depths to resolve overlaps

Q. How to fill in holes?

Marc Levoy

View interpolationfrom multiple views

1. Render object from multiple viewpoints

2. Convert Z-buffers to range images

3. Re-render from new viewpoint

4. Use depths to resolve overlaps

5. Use multiple views to fill in holes

Marc Levoy

Post-rendering 3D warping[Mark et al., I3D97]

• render at low frame rate

• interpolate to real-time frame rate– interpolate observer viewpoint using B-Spline

– convert reference images to polygon meshes

– warp meshes to interpolated viewpoint

– composite by Z-buffer comparison and conditional write

Marc Levoy

Results

• rendered at 5 fps, interpolated to 30 fps

• live system requires reliable motion prediction– tradeoff between accuracy and latency

• fails on specular objects

Marc Levoy

Light field rendering[Levoy & Hanrahan, Siggraph ’96]

• must stay outside convex hull of the object

• like rebinning in computed tomography

Marc Levoy

A light field is an array of images

Marc Levoy

Stanford multi-camera array(Horowitz, Levoy, Hanrahan)

• cheap imagers + cheap optics +fast networking + plentiful computation =

high-performance imagingusing an array of low-cost cameras

Marc Levoy

Applications for the array

• key issue is the spacing andarrangement of the cameras

Marc Levoy

Cameras tightly packed:high-X imaging

• high-resolution– by abutting the cameras’ fields of view

• high speed– by staggering their triggering times

• high dynamic range– mosaic of shutter speeds, apertures, density filters

• high depth of field– mosaic of differently focused lenses

Marc Levoy

Cameras widely spaced:a video-rate light field camera

• compression of (video) light fields– key is estimation of rough geometry

• autostereoscopic display of light fields– lens arrays, bumpy mirrored sheets

• shape from light fields– Do more images help?