Multiview ReconstructionMultiview Reconstruction
Why More Than 2 Views?Why More Than 2 Views?
• BaselineBaseline– Too short – low accuracyToo short – low accuracy
– Too long – matching becomes hardToo long – matching becomes hard
Why More Than 2 Views?Why More Than 2 Views?
• Ambiguity with 2 viewsAmbiguity with 2 views
Camera 1Camera 1 Camera 2Camera 2Camera 3Camera 3
Trinocular StereoTrinocular Stereo
• Straightforward approach to eliminate bad Straightforward approach to eliminate bad correspondencescorrespondences– Pick 2 views, find correspondencesPick 2 views, find correspondences
– For each matching pair, reconstruct 3D pointFor each matching pair, reconstruct 3D point
– Project point into 3Project point into 3rdrd image image
– If can’t find correspondence near predicted If can’t find correspondence near predicted location, rejectlocation, reject
Volumetric Multiview ApproachesVolumetric Multiview Approaches
• Goal: find a model consistent with imagesGoal: find a model consistent with images
• ““Model-centric” (vs. image-centric)Model-centric” (vs. image-centric)
• Typically use discretized volume (voxel Typically use discretized volume (voxel grid)grid)
• For each voxel, compute occupied / freeFor each voxel, compute occupied / free(for some algorithms, also color, etc.)(for some algorithms, also color, etc.)
Photo ConsistencyPhoto Consistency
• Result: not necessarily correct sceneResult: not necessarily correct scene
• Many scenes produce the same imagesMany scenes produce the same images
All scenesAll scenes
Photo-consistent scenesPhoto-consistent scenes
True sceneTrue scene ReconstructedReconstructedscenescene
Silhouette CarvingSilhouette Carving
• Find silhouettes in all imagesFind silhouettes in all images
• Exact version:Exact version:– Back-project all silhouettes, find intersectionBack-project all silhouettes, find intersection
Binary ImagesBinary Images
Silhouette CarvingSilhouette Carving
• Find silhouettes in all imagesFind silhouettes in all images
• Exact version:Exact version:– Back-project all silhouettes, find intersectionBack-project all silhouettes, find intersection
Silhouette CarvingSilhouette Carving
• Discrete version:Discrete version:– Loop over all voxels in some volumeLoop over all voxels in some volume
– If projection into images lies inside all If projection into images lies inside all silhouettes, mark as occupiedsilhouettes, mark as occupied
– Else mark as freeElse mark as free
Silhouette CarvingSilhouette Carving
Voxel ColoringVoxel Coloring
• Seitz and Dyer, 1997Seitz and Dyer, 1997
• In addition to free / occupied, store colorIn addition to free / occupied, store colorat each voxelat each voxel
• Explicitly accounts for occlusionExplicitly accounts for occlusion
Voxel ColoringVoxel Coloring
• Basic idea: sweep through a voxel gridBasic idea: sweep through a voxel grid– Project each voxel into each image Project each voxel into each image in whichin which
it is visibleit is visible
– If colors in images agree, mark voxel with colorIf colors in images agree, mark voxel with color
– Else, mark voxel as emptyElse, mark voxel as empty
• Agreement of colors based on comparing Agreement of colors based on comparing standard deviation of colors to thresholdstandard deviation of colors to threshold
Voxel Coloring and OcclusionVoxel Coloring and Occlusion
• Problem: which voxels are visible?Problem: which voxels are visible?
• Solution, part 1: constrain camera viewsSolution, part 1: constrain camera views– When a voxel is considered, necessary occlusion When a voxel is considered, necessary occlusion
information must be availableinformation must be available
– Sweep occluders before occludeesSweep occluders before occludees
– Constrain camera positions to allow this sweepConstrain camera positions to allow this sweep
Voxel Coloring Sweep OrderVoxel Coloring Sweep Order
LayersLayers
SceneScene
TraversalTraversal
SeitzSeitz
Voxel Coloring Camera PositionsVoxel Coloring Camera Positions
Inward-lookingCameras above scene
Outward-lookingCameras inside scene
SeitzSeitz
Panoramic Depth OrderingPanoramic Depth Ordering
• Cameras oriented in many different directionsCameras oriented in many different directions
• Planar depth ordering does not applyPlanar depth ordering does not apply
SeitzSeitz
Panoramic Depth OrderingPanoramic Depth Ordering
Layers radiate outwards from camerasLayers radiate outwards from camerasSeitzSeitz
Panoramic Depth OrderingPanoramic Depth Ordering
SeitzSeitzLayers radiate outwards from camerasLayers radiate outwards from cameras
Panoramic Depth OrderingPanoramic Depth Ordering
SeitzSeitzLayers radiate outwards from camerasLayers radiate outwards from cameras
Voxel Coloring and OcclusionVoxel Coloring and Occlusion
• Solution, part 2: per-image mask of which Solution, part 2: per-image mask of which pixels have been usedpixels have been used– Each pixel only used onceEach pixel only used once
– Mask filled in as sweep progressesMask filled in as sweep progresses
Image AcquisitionImage Acquisition
•Calibrated TurntableCalibrated Turntable
•360° rotation (21 images)360° rotation (21 images)
Selected Dinosaur ImagesSelected Dinosaur Images
Selected Flower ImagesSelected Flower Images SeitzSeitz
Voxel Coloring ResultsVoxel Coloring Results
Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
SeitzSeitz
Voxel Coloring ResultsVoxel Coloring Results
• With texture: good resultsWith texture: good results
• Without texture: regions tend to “bulge Without texture: regions tend to “bulge out”out”– Voxels colored at earliest time at which Voxels colored at earliest time at which
projection into images is consistentprojection into images is consistent
– Model good for re-rendering: image will look Model good for re-rendering: image will look correct for viewpoints near the original onescorrect for viewpoints near the original ones
Limitations of Voxel ColoringLimitations of Voxel Coloring
• A view-independent depth orderA view-independent depth ordermay not existmay not exist
• Need more powerful general-case Need more powerful general-case algorithmsalgorithms– Unconstrained camera positionsUnconstrained camera positions
– Unconstrained scene geometry/topologyUnconstrained scene geometry/topology
pp qq
Space CarvingSpace Carving
Image 1Image 1 Image NImage N
……......
Initialize to a volume V containing the true sceneInitialize to a volume V containing the true scene
Repeat until convergenceRepeat until convergence
Choose a voxel on the current surfaceChoose a voxel on the current surface
Carve if not photo-consistentCarve if not photo-consistentProject to visible input imagesProject to visible input images
Kutulakos & SeitzKutulakos & Seitz
Multi-Pass Plane SweepMulti-Pass Plane Sweep
• Faster alternative:Faster alternative:– Sweep plane in each of 6 principal directionsSweep plane in each of 6 principal directions
– Consider cameras on only one side of planeConsider cameras on only one side of plane
– Repeat until convergenceRepeat until convergence
Multi-Pass Plane SweepMulti-Pass Plane Sweep
True Scene Reconstruction
Multi-Pass Plane SweepMulti-Pass Plane Sweep
Multi-Pass Plane SweepMulti-Pass Plane Sweep
Multi-Pass Plane SweepMulti-Pass Plane Sweep
Multi-Pass Plane SweepMulti-Pass Plane Sweep
Multi-Pass Plane SweepMulti-Pass Plane Sweep
Space Carving Results: African Space Carving Results: African VioletViolet
Input Image (1 of 45) Input Image (1 of 45) ReconstructionReconstruction
ReconstructionReconstructionReconstructionReconstruction
Space Carving Results: HandSpace Carving Results: Hand
Input ImageInput Image(1 of 100) (1 of 100)
Views of ReconstructionViews of Reconstruction