Image-Based Rendering Computer Vision CSE576, Spring 2005 Richard Szeliski CSE 576, Spring 2005 Image-Based Rendering 2 Today’s lecture Image–Based Rendering • Light Fields and Lumigraphs • Panoramas and Concentric Mosaics • Environment Matting • Image-Based models CSE 576, Spring 2005 Image-Based Rendering 3 Today’s lecture Video-Based Rendering • Facial animation • Video matting and shadow matting • Video Textures and Animating Stills • Video-based tours CSE 576, Spring 2005 Image-Based Rendering 4 Readings • S. J. Gortler , R. Grzeszczuk , R. Szeliski and M. F. Cohen, The Lumigraph , SIGGRAPH'96. • M. Levoy and P. Hanrahan, Light field rendering , SIGGRAPH'96. • H.-Y. Shum and L.-W. He. Rendering with concentric mosaics, SIGGRAPH’99.
27
Embed
Image-Based Rendering - courses.cs.washington.edu• Facial animation • Video matting and shadow matting • Video Textures and Animating Stills ... Spring 2005 Image-Based Rendering
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Image-Based Rendering
Computer VisionCSE576, Spring 2005
Richard Szeliski
CSE 576, Spring 2005 Image-Based Rendering 2
Today’s lectureImage–Based Rendering• Light Fields and Lumigraphs• Panoramas and Concentric Mosaics• Environment Matting• Image-Based models
CSE 576, Spring 2005 Image-Based Rendering 3
Today’s lectureVideo-Based Rendering• Facial animation• Video matting and shadow matting• Video Textures and Animating Stills• Video-based tours
CSE 576, Spring 2005 Image-Based Rendering 4
Readings• S. J. Gortler , R. Grzeszczuk , R. Szeliski and M. F.
Cohen, The Lumigraph, SIGGRAPH'96. • M. Levoy and P. Hanrahan, Light field rendering,
SIGGRAPH'96.• H.-Y. Shum and L.-W. He. Rendering with concentric
mosaics, SIGGRAPH’99.
2
CSE 576, Spring 2005 Image-Based Rendering 5
Readings• D. E. Zongker et al. Environment matting and
compositing, SIGGRAPH'99.• Y.-Y. Chuang et al. Environment matting extensions:
Towards higher accuracy and real-time capture. SIGGRAPH'2000, pp.121-130, 2000.
• P. E. Debevec , C. J. Taylor and J. Malik, Modeling and rendering architecture from photographs:…, SIGGRAPH'96.
CSE 576, Spring 2005 Image-Based Rendering 6
Readings• Y.-Y. Chuang et al. Video matting of complex scenes.
ACM Trans. on Graphics, 21(3):243-248, July 2002• Y.-Y. Chuang et al. Shadow matting. ACM
Transactions on Graphics, 22(3):494-500, July 2003.• A. Schödl et al., Video textures. SIGGRAPH'2000,
pp. 489-498, 2000.• M. Uyttendaele et al. Image-based interactive
exploration of real-world environments. IEEE Comp. Graphics and Applications, 24(3), May/June 2004.
Lightfields and Lumigraphs
(with lots of slides from Michael Cohen)
CSE 576, Spring 2005 Image-Based Rendering 8
Modeling light
How do we generate new scenes and animations from existing ones?
Classic “3D Vision + Graphics”:• take (lots of) pictures• recover camera pose• build 3D model• extract texture maps / BRDFs• synthesize new views
3
CSE 576, Spring 2005 Image-Based Rendering 9
Computer Graphics
Image
Output
ModelSyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 10
Real Scene
Computer Vision
Real Cameras
Model
Output
CSE 576, Spring 2005 Image-Based Rendering 11
Combined
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 12
But, vision technology falls short
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
4
CSE 576, Spring 2005 Image-Based Rendering 13
… and so does graphics.
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 14
Image Based Rendering
Real Scene
Real Cameras-or-
Expensive Image Synthesis
Images+Model
Image
Output
SyntheticCamera
CSE 576, Spring 2005 Image-Based Rendering 15
Ray
Constant radiance• time is fixed
5D• 3D position• 2D direction
CSE 576, Spring 2005 Image-Based Rendering 16
All Rays
Plenoptic Function:• all possible images• too much stuff!
5
CSE 576, Spring 2005 Image-Based Rendering 17
Line
Infinite line
4D• 2D direction• 2D position• non-dispersive medium
CSE 576, Spring 2005 Image-Based Rendering 18
Ray
Discretize, then interpolate
Distance between 2 rays• Which is closer together?
CSE 576, Spring 2005 Image-Based Rendering 19
Image
What is an image?
All rays through a point• Panorama
CSE 576, Spring 2005 Image-Based Rendering 20
Panoramic MosaicsConvert panoramic image sequence into a
cylindrical image
+ + … + =
6
CSE 576, Spring 2005 Image-Based Rendering 21
Image
Image plane
2D• position in plane
CSE 576, Spring 2005 Image-Based Rendering 22
Light leaving towards “eye”
2D• just dual of image
Object
CSE 576, Spring 2005 Image-Based Rendering 23
Object
All light leaving object
CSE 576, Spring 2005 Image-Based Rendering 24
Object
4D• 2D position (on surface)• 2D direction
7
CSE 576, Spring 2005 Image-Based Rendering 25
Object
All images
CSE 576, Spring 2005 Image-Based Rendering 26
Lumigraph / Lightfield
Outside convex space
4D
StuffEmpty
CSE 576, Spring 2005 Image-Based Rendering 27
Lumigraph
Inside convex space
4D
EmptyStuff
CSE 576, Spring 2005 Image-Based Rendering 28
Lumigraph
How to ?• organize• capture• render
8
CSE 576, Spring 2005 Image-Based Rendering 29
Lumigraph - Organization
2D position2D direction
sθ
CSE 576, Spring 2005 Image-Based Rendering 30
Lumigraph - Organization
2D position2D position
2 plane parameterization
su
CSE 576, Spring 2005 Image-Based Rendering 31
Lumigraph - Organization
2D position2D position
2 plane parameterization
us
t s,tu,v
v
s,t
u,v
CSE 576, Spring 2005 Image-Based Rendering 32
Lumigraph - Organization
Hold s,t constantLet u,v varyAn image
s,t u,v
9
CSE 576, Spring 2005 Image-Based Rendering 33
Lumigraph - Organization
Discretization• higher res near object
– if diffuse– captures texture
• lower res away– captures directions
s,t u,v CSE 576, Spring 2005 Image-Based Rendering 34
Lumigraph - Capture
Idea 1• Move camera carefully over
s,t plane• Gantry
– see Light Field paper
s,t u,v
CSE 576, Spring 2005 Image-Based Rendering 35
Lumigraph - Capture
Idea 2• Move camera anywhere• Rebinning
– see Lumigraph paper
s,t u,v CSE 576, Spring 2005 Image-Based Rendering 36
Lumigraph - RenderingFor each output pixel
• determine s,t,u,v• either
– find closest discrete RGB– interpolate near values s,t u,v
10
CSE 576, Spring 2005 Image-Based Rendering 37
Lumigraph - Rendering
For each output pixel• determine s,t,u,v
• either• use closest discrete RGB• interpolate near values s u
CSE 576, Spring 2005 Image-Based Rendering 38
Lumigraph - Rendering
Nearest• closest s• closest u• draw it
Blend 16 nearest• quadrilinear interpolation
s u
CSE 576, Spring 2005 Image-Based Rendering 39
Lumigraph - Rendering
Depth Correction• closest s• intersection with “object”• best u• closest u
s u CSE 576, Spring 2005 Image-Based Rendering 40
Lumigraph - Rendering
Depth Correction • quadralinear interpolation• new “closest”• like focus
Pull dynamic α-mattefrom video withcomplex backgrounds
[Chuang et al. @ UW, SIGGRAPH’2002]
CSE 576, Spring 2005 Image-Based Rendering 86
Video Matting
CSE 576, Spring 2005 Image-Based Rendering 87
Shadow Matting
Transfer a shadow from one background to another:
• Extract and model photometry (darkening)• Extract and model geometry (deformation)
CSE 576, Spring 2005 Image-Based Rendering 88
Shadow Matting
23
CSE 576, Spring 2005 Image-Based Rendering 89
Shadow Matting
CSE 576, Spring 2005 Image-Based Rendering 90
Shadow MattingVideo
Video Textures
CSE 576, Spring 2005 Image-Based Rendering 92
Video Textures
How can we turn a short video clipinto an ∞ amount of continuous video?• dynamic elements in 3D games and presentations• alternative to 3D graphics animation?
[Schödl, Szeliski, Salesin, Essa, SG’2000]
24
CSE 576, Spring 2005 Image-Based Rendering 93
Video TexturesFind cyclic structure in the video
(Optional) region-based analysisPlay frames with random shuffleSmooth over discontinuities (morph)
CSE 576, Spring 2005 Image-Based Rendering 94
Region-based analysis
CSE 576, Spring 2005 Image-Based Rendering 95
Crossfading and morphing
Jump Cut Crossfade Morph
CSE 576, Spring 2005 Image-Based Rendering 96
Video portrait
25
CSE 576, Spring 2005 Image-Based Rendering 97
Dynamic scene element
Live waterfall in static panorama
Demo
CSE 576, Spring 2005 Image-Based Rendering 98
Interactive fish
CSE 576, Spring 2005 Image-Based Rendering 99
A complete animation
Video-Based Tours
26
CSE 576, Spring 2005 Image-Based Rendering 101
Video-Based Walkthroughs
Move camera along a rail (“dolly track”) and play back a 360° video
Applications:• Homes and architecture• Outdoor locations
(tourist destinations)
CSE 576, Spring 2005 Image-Based Rendering 102
Surround video acquisition system
OmniCam (six-camera head)
CSE 576, Spring 2005 Image-Based Rendering 103
OmniCamBuilt by Point Grey Research (Ladybug)
Six camera head
Portable hard drives, fiber-optic link
Resolution per image: 1024 x 768
FOV: ~100o x ~80o
Acquisition speed: 15 fps uncompressed
CSE 576, Spring 2005 Image-Based Rendering 104
Acquisition platforms
Robotic cart
Wearable
27
DemoA Virtual Home Tour
CSE 576, Spring 2005 Image-Based Rendering 106
Open issues
How to best sample and interpolate Light Field• (sub-?) pixel accurate stereo• reflections, refractions, …Compositing• how to insert Light Field into new environment• relighting• …?
CSE 576, Spring 2005 Image-Based Rendering 107
SummaryImage–Based Rendering• Light Fields and Lumigraphs• Panoramas and Concentric Mosaics• Matting: natural, environment, and shadows• Image-Based modelsVideo-Based Rendering• Facial animation• Video Textures and Animating Stills• Video-based tours
CSE 576, Spring 2005 Image-Based Rendering 108
SummaryImage–Based Rendering• Light Fields and Lumigraphs• Panoramas and Concentric Mosaics• Environment Matting• Image-Based modelsVideo-Based Rendering• Facial animation• Video matting• Video Textures and Animating Stills• Video-based tours