Multi-View Stereo for Static and Multi-View Stereo for Static and Dynamic Scenes Dynamic Scenes Wolfgang Burgard Wolfgang Burgard Jan 6, 2010 Jan 6, 2010 ● Main references Main references ● Yasutaka Furukawa and Jean Ponce, Yasutaka Furukawa and Jean Ponce, Accurate, Dense and Robust Multi-View Stereopsis, Accurate, Dense and Robust Multi-View Stereopsis, 2007 2007 ● C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, and R. Szeliski, High-quality Video View Interpolation using a High-quality Video View Interpolation using a Layered Representation, Layered Representation, 2004 2004
33
Embed
Multi-View Stereo for Static and Dynamic Scenestheobalt/courses/...Multi-View Stereo for Static and Dynamic Scenes Wolfgang Burgard Jan 6, 2010 Main references Yasutaka Furukawa and
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Multi-View Stereo for Static and Multi-View Stereo for Static and Dynamic ScenesDynamic Scenes
Wolfgang BurgardWolfgang BurgardJan 6, 2010Jan 6, 2010
● Main referencesMain references● Yasutaka Furukawa and Jean Ponce,Yasutaka Furukawa and Jean Ponce,
Accurate, Dense and Robust Multi-View Stereopsis, Accurate, Dense and Robust Multi-View Stereopsis, 20072007
● C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski,and R. Szeliski,High-quality Video View Interpolation using a High-quality Video View Interpolation using a Layered Representation, Layered Representation, 20042004
22
Stereo Reconstruction - Static SceneStereo Reconstruction - Static Scene
● SettingsSettings● Two images (2D) of the same sceneTwo images (2D) of the same scene● Static: Scene hasn't changed accross imagesStatic: Scene hasn't changed accross images● Acquisition from different viewpointsAcquisition from different viewpoints● Camera parameters known / estimated (Zhang)Camera parameters known / estimated (Zhang)
● GoalGoal● Reconstruct geometry (3D) of objects in sceneReconstruct geometry (3D) of objects in scene
● Uniform coverage:Uniform coverage:4 local maxima with4 local maxima withstrongest responsestrongest responseof each operatorof each operator
● Triangulation with feature pairs (f,f') => 3D pointsTriangulation with feature pairs (f,f') => 3D points● Tolerance of 2 pixels from epipolar lineTolerance of 2 pixels from epipolar line● Consider matches (f,f') if of same typeConsider matches (f,f') if of same type
77
Initial Patch Set (Match step) 2/2Initial Patch Set (Match step) 2/2
● Many 3D points for feature f. Optimal one ?Many 3D points for feature f. Optimal one ?Nearest to O which is photoconsistent:Nearest to O which is photoconsistent:● Patch candidate pPatch candidate p
– Center c(p): 3D pointCenter c(p): 3D point– Extension of p:Extension of p:
Projection into image inProjection into image in5x5 (7x7) axis aligned5x5 (7x7) axis alignedsquaresquare
● OptimiziationOptimiziation– MaximalMaximal
photometric consistencyphotometric consistency– Refine parameters: c(p) and n(p)Refine parameters: c(p) and n(p)
● p p photoconsistentphotoconsistent with 2-3 images with 2-3 images➔ Accept, otherwise next 3D pointAccept, otherwise next 3D point
opticalcenter
O
inconsistentpatch
consistentpatches
normal
88
Photometric ConsistencyPhotometric Consistency
● p photoconsistent with image I ?p photoconsistent with image I ?
● Normalized cross corelation (NCC)Normalized cross corelation (NCC)– Energy independent similarity measureEnergy independent similarity measure– Similarity of p in I with p in reference imageSimilarity of p in I with p in reference image
● NCC > treshold => photoconsistentNCC > treshold => photoconsistent● Correspondence problem not solved pointwise but Correspondence problem not solved pointwise but
for an areafor an area● Local surface area considered perspectivelyLocal surface area considered perspectively
Reference image
99
Expand 1/2Expand 1/2
● Initial patches too sparseInitial patches too sparse● For all patches p add patches p' to neighbour cellsFor all patches p add patches p' to neighbour cells● Conditions for addingConditions for adding
● No visible patch in cellNo visible patch in cell● No should-be-visible patch n-adjacent in cellNo should-be-visible patch n-adjacent in cell
Close patch centers
patch p p's neighbour patch p p's neighbour
Similar normals
1010
Expand 2/2Expand 2/2
● Initial parameters of new patch p'Initial parameters of new patch p'● n(p') = n(p)n(p') = n(p)● c(p') intersection of ray throughc(p') intersection of ray through
Cell (i',j') with plane of pCell (i',j') with plane of p
● OptimiziationOptimiziation● Maximal photometric consistencyMaximal photometric consistency● Refines c(p') and n(p')Refines c(p') and n(p')
● Accept if photometric consistent in 2-3 imagesAccept if photometric consistent in 2-3 images
plane of p
neighbour patch p'
1111
Filter 1/2Filter 1/2
● Remove patches outside of real surfaceRemove patches outside of real surface● Caused by e.g. obstaclesCaused by e.g. obstacles
● Condition to remove p:Condition to remove p:– p's Photometric consistency < photometric p's Photometric consistency < photometric
consistency of patches hidden by pconsistency of patches hidden by p– Intuition:Intuition:
● Projected outliers visible in less images than Projected outliers visible in less images than real surface patchesreal surface patches
outlier seen in oneimage conflicts 2
patches seen in otherimages
1212
Filter 2/2Filter 2/2
● Remove patches inside real surfaceRemove patches inside real surface● Caused by iterative scheme ( expand )Caused by iterative scheme ( expand )
– Patch added occludes "inside“ patchPatch added occludes "inside“ patch● Not visible in 2-3 images => remove pNot visible in 2-3 images => remove p● Visibility here defined via depth valuesVisibility here defined via depth values
outlier only visible in 1 image
1313
Mesh from Patches 1/2Mesh from Patches 1/2
● Once Once Match Match (initial patches)(initial patches)● ExpandExpand and and FilterFilter step until convergence step until convergence
(dense patches)(dense patches)● Mesh from patchesMesh from patches
● Initial phase:Initial phase:– Move towards photoconsistent patchesMove towards photoconsistent patches
● Later phases:Later phases:– Create patch at vertex;Create patch at vertex;– Optimize patch;Optimize patch;– Photoconsistency: c(p) - c*(p)Photoconsistency: c(p) - c*(p)
– Rim consistency Rim consistency - pull mesh towards visual cone- pull mesh towards visual cone● projected surface silhouette ~ silhouette in projected surface silhouette ~ silhouette in
imageimage
1515
ResultsResults
● Computation Time: Minutes to hoursComputation Time: Minutes to hours
1616
Multi-View Stereo for Dynamic ScenesMulti-View Stereo for Dynamic Scenes
● InputInput● Image sequences (video) of dynamic sceneImage sequences (video) of dynamic scene● Each sequence captured scene over timeEach sequence captured scene over time
● Goal:Goal:● Variable viewpoints in videoVariable viewpoints in video
( using geometric data )( using geometric data )● Additional challenges:Additional challenges:
● Object MovementObject Movement● Object DeformationObject Deformation● Much dataMuch data
● 'Light-field Rendering' for dynamic scences'Light-field Rendering' for dynamic scencesProblem: Requires many videosProblem: Requires many videos
● HomographyHomography
– Problem:Problem:● No parallax effect,No parallax effect,
i.e. foreground and background objects movei.e. foreground and background objects movewith same velocity - independent from depthwith same velocity - independent from depth
Novel view
Project & Blend
2121
OverviewOverview
Matte extraction( depth disconuity artifacts )
3D Reconstruction withImage Segmentation
Snchronized videos + Camera parameters
Compression
Rendering using temporal twolayered compression representation
– Similar disparity in each segment => no artifactsSimilar disparity in each segment => no artifacts1. Smooth & reduce noise1. Smooth & reduce noise2. Merge segments (initially: each pixel )2. Merge segments (initially: each pixel ) if average color similar if average color similar3. Split/merge too large/small segments3. Split/merge too large/small segments
● Initial disparity in each segmentInitial disparity in each segment– Photometric conditionsPhotometric conditions– Constant disparityConstant disparity
● Refine disparityRefine disparity– Relax constant assumptionRelax constant assumption– Average across imagesAverage across images– Average between segmentsAverage between segments– Smoothness in each segmentSmoothness in each segment
2525
Boundary Matting 1/4Boundary Matting 1/4
● Problem:Problem:At depth disconuities: Foreground pixels contain At depth disconuities: Foreground pixels contain background colorbackground color
background
Pixel contain foregroundand background color
foreground
Hairs from foreground object having blue color
from background
2626
Boundary Matting 2/4Boundary Matting 2/4
Novel view
Left camera image
Right camera image
2727
Boundary Matting 3/4Boundary Matting 3/4
● Matting at disparity disconuitiesMatting at disparity disconuities● Extract foreground and backgroundExtract foreground and background
1.1.Select 2 nearest camerasSelect 2 nearest cameras2.2.Render into 2 buffers separate ( 1 for each camera )Render into 2 buffers separate ( 1 for each camera )
● Project main & boundary layer into viewProject main & boundary layer into view● Depth map => 3D meshDepth map => 3D mesh● Remove triangles across depth disconuities from Remove triangles across depth disconuities from
main layer; Boundary mesh insteadmain layer; Boundary mesh instead
Main mesh at depth disconuity
Boundary mesh at depth disconuity
3030
Rendering 2/2Rendering 2/2
3.3.Blend buffersBlend buffers– Pixels with different depth => use frontmostPixels with different depth => use frontmost– Similar depthSimilar depth
=> average with camera distance to view as => average with camera distance to view as weightweight
Right camera nearer =>more influence of pixel colors
3131
ResultsResults
● Interactive renderingInteractive rendering● 1024x768 @ 5 fps1024x768 @ 5 fps● 512x384 @ 10 fps512x384 @ 10 fps● ATI 9800 PROATI 9800 PRO● If all images on GPU memory:If all images on GPU memory:
1024x768 @ 30 fps 1024x768 @ 30 fps
3232
SummarySummary
● Static scene approachStatic scene approach● Stereo reconstuction with featuresStereo reconstuction with features● Photoconsistent patchesPhotoconsistent patches● Geometry: Complete 3D patch & mesh modelGeometry: Complete 3D patch & mesh model
● Dynamic scene approachDynamic scene approach● Stereo reconstruction with segmented imagesStereo reconstruction with segmented images● Geometry:Geometry:
– 2 layers: Main layer + boundary layer ( matting )2 layers: Main layer + boundary layer ( matting )– 3D mesh for rendering3D mesh for rendering