From Images to Voxels From Images to Voxels Steve Seitz Steve Seitz Carnegie Mellon University Carnegie Mellon University University of Washington University of Washington http://www.cs.cmu.edu/~seitz http://www.cs.cmu.edu/~seitz SIGGRAPH 2000 Course on SIGGRAPH 2000 Course on 3D Photography 3D Photography
54
Embed
From Images to Voxels Steve Seitz Carnegie Mellon University University of Washington seitz SIGGRAPH 2000 Course on 3D Photography.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
From Images to VoxelsFrom Images to Voxels
Steve SeitzSteve Seitz
Carnegie Mellon UniversityCarnegie Mellon University
University of WashingtonUniversity of Washingtonhttp://www.cs.cmu.edu/~seitzhttp://www.cs.cmu.edu/~seitz
Reconstruction Contains the True SceneReconstruction Contains the True Scene• But is generally not the same But is generally not the same • In the limit get In the limit get visual hullvisual hull
> Complement of all lines that don’t intersect SComplement of all lines that don’t intersect S
Voxel Algorithm for Volume IntersectionVoxel Algorithm for Volume Intersection
Color voxel black if on silhouette in every imageColor voxel black if on silhouette in every image• O(MNO(MN33), for M images, N), for M images, N33 voxels voxels
• Don’t have to search 2Don’t have to search 2NN33 possible scenes! possible scenes!
Properties of Volume IntersectionProperties of Volume Intersection
ProsPros• Easy to implement, fastEasy to implement, fast
• Accelerated via octrees [Szeliski 1993]Accelerated via octrees [Szeliski 1993]
ConsCons• No concavitiesNo concavities
• Reconstruction is not photo-consistentReconstruction is not photo-consistent
• Requires identification of silhouettesRequires identification of silhouettes
Dinosaur ReconstructionDinosaur Reconstruction72 K 72 K voxels coloredvoxels colored7.6 M 7.6 M voxels testedvoxels tested7 min. 7 min. to compute to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K 70 K voxels coloredvoxels colored7.6 M 7.6 M voxels testedvoxels tested7 min. 7 min. to compute to compute on a 250MHz SGIon a 250MHz SGI
Limitations of Depth OrderingLimitations of Depth Ordering
A view-independent depth order may not existA view-independent depth order may not exist
p q
Need more powerful general-case algorithmsNeed more powerful general-case algorithms• Unconstrained camera positionsUnconstrained camera positions
• Unconstrained scene geometry/topologyUnconstrained scene geometry/topology
3. General Case3. General Case• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]
Voxel Coloring SolutionsVoxel Coloring Solutions
Space Carving AlgorithmSpace Carving Algorithm
Space Carving AlgorithmSpace Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true sceneInitialize to a volume V containing the true scene
• Repeat until convergenceRepeat until convergence
• Choose a voxel on the current surfaceChoose a voxel on the current surface
• Carve if not photo-consistentCarve if not photo-consistent
• Project to visible input imagesProject to visible input images
ConvergenceConvergence
Consistency PropertyConsistency Property• The resulting shape is photo-consistentThe resulting shape is photo-consistent
> all inconsistent points are removedall inconsistent points are removed
Convergence PropertyConvergence Property• Carving converges to a non-empty shapeCarving converges to a non-empty shape
> a point on the true scene is a point on the true scene is nevernever removed removed
V’
V
p
What is Computable?What is Computable?
The The Photo HullPhoto Hull is the UNION of all photo-consistent scenes in V is the UNION of all photo-consistent scenes in V
• It is a photo-consistent scene reconstructionIt is a photo-consistent scene reconstruction
• Tightest possible bound on the true sceneTightest possible bound on the true scene
• Computable via provable Computable via provable Space Carving AlgorithmSpace Carving Algorithm
True SceneTrue Scene
VV
Photo HullPhoto Hull
VV
Space Carving AlgorithmSpace Carving Algorithm
The Basic Algorithm is UnwieldyThe Basic Algorithm is Unwieldy• Complex update procedureComplex update procedure
Alternative: Multi-Pass Plane SweepAlternative: Multi-Pass Plane Sweep• Efficient, can use texture-mapping hardwareEfficient, can use texture-mapping hardware
• Converges quickly in practiceConverges quickly in practice
• Easy to implementEasy to implement
Results Algorithm
Space Carving AlgorithmSpace Carving Algorithm
• Step 1:Step 1: Initialize V to volume containing true scene Initialize V to volume containing true scene
• Step 2:Step 2: For every voxel on surface of V For every voxel on surface of V
> test test photo-consistencyphoto-consistency of voxel of voxel
> if voxel is inconsistent, carve itif voxel is inconsistent, carve it
• Step 3:Step 3: Repeat Step 2 until all voxels consistent Repeat Step 2 until all voxels consistent
Convergence: Convergence: • AlwaysAlways converges to a photo-consistent model (when converges to a photo-consistent model (when
all assumptions are met)all assumptions are met)
• Good results on difficult real-world scenes Good results on difficult real-world scenes
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
True Scene Reconstruction
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
• Consider cameras on only one side of planeConsider cameras on only one side of plane
• Repeat until convergenceRepeat until convergence
Other ApproachesOther Approaches
Level-Set Methods Level-Set Methods [Faugeras & Keriven 1998][Faugeras & Keriven 1998]• Evolve implicit function by solving PDE’sEvolve implicit function by solving PDE’s
Transparency and Matting Transparency and Matting [Szeliski & Golland 1998][Szeliski & Golland 1998]• Compute voxels with alpha-channelCompute voxels with alpha-channel
Max Flow/Min Cut Max Flow/Min Cut [Roy & Cox 1998][Roy & Cox 1998]• Graph theoretic formulationGraph theoretic formulation
Mesh-Based Stereo Mesh-Based Stereo [Fua & Leclerc 95][Fua & Leclerc 95]• Mesh-based but similar consistency formulationMesh-based but similar consistency formulation
Virtualized Reality Virtualized Reality [Narayan, Rander, Kanade 1998][Narayan, Rander, Kanade 1998]• Perform stereo 3 images at a time, merge resultsPerform stereo 3 images at a time, merge results
Advantages of VoxelsAdvantages of Voxels• Non-parametricNon-parametric
> can model arbitrary geometrycan model arbitrary geometry
> can model arbitrary topologycan model arbitrary topology
• Good reconstruction algorithmsGood reconstruction algorithms
• Good rendering algorithms (splatting, LDI)Good rendering algorithms (splatting, LDI)
DisadvantagesDisadvantages• Expensive to process hi-res voxel gridsExpensive to process hi-res voxel grids
• Large number of parametersLarge number of parameters
> Simple scenes (e.g., planes) require lots of voxelsSimple scenes (e.g., planes) require lots of voxels
ConclusionsConclusions
Volume IntersectionVolume Intersection• Martin & Aggarwal, “Volumetric description of objects from multiple views”, Martin & Aggarwal, “Volumetric description of objects from multiple views”,
Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.
• Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.
Voxel Coloring and Space CarvingVoxel Coloring and Space Carving• Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc. Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc.
Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073.Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073.
• Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, pp. 17-24.Vision (ICCV), 1998, pp. 17-24.
• Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp. Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp. 307-314.307-314.
BibliographyBibliography
Related ReferencesRelated References• Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set
methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344.pp. 336-344.
• Szeliski & Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. Szeliski & Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, 517-524.Conf. on Computer Vision (ICCV), 1998, 517-524.
• Roy & Cox, “A Maximum-Flow Formulation of the N-camera Stereo Roy & Cox, “A Maximum-Flow Formulation of the N-camera Stereo Correspondence Problem”, Proc. ICCV, 1998, pp. 492-499.Correspondence Problem”, Proc. ICCV, 1998, pp. 492-499.
• Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-image stereo and shading", Int. Journal of Computer Vision, 16, 1995, pp. 35-56.image stereo and shading", Int. Journal of Computer Vision, 16, 1995, pp. 35-56.
• Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3-10.Stereo”, Proc. ICCV, 1998, pp. 3-10.