Geo-consistency for Wide Multi-Camera Stereo Marc-Antoine Drouin, Martin Trudeau and S´ ebastien Roy D´ epartement d’Informatique et recherche op´ erationnelle, Universit´ e de Montr´ eal, Canada email: {drouim,trudeaum,roys}@iro.umontreal.ca Problem: 3D Reconstruction of a scene from a reference camera using multiple supporting ones For each pixel p we need to choose • a depth f (p) • a mask of used cameras g (p) so as to minimize the energy E (f,g )= p∈P e(p,f (p),g (p)) +smoothing. Handling Visibility Exact when g (p)= V (p|f (p),f ) (the real visibility) • Long range interaction ⇒ en- ergy minimization is hard Heuristic when g (p) = arg min m∈M e(p,f (p),m) (M is a set of plausible masks) • Photo-consistency ⇒ correct visibility Geo-consistency Defined as g (p) ≤ V (p|f (p),f ) • Occluded cameras are not used • Visible cameras are not always used Justified by Nakamura96: • Using an occluded camera ⇒ important artifacts • Not using a visible camera ⇒ no significant artifacts Bias in Border Localization Closest objects are enlarged by stan- dard algorithms. Supporting camera Reference camera GT occludees GT occluders DM occludees DM occluders 1 2 3 4 5 • Large number of occluders in depth map are occludees accord- ing to ground truth (zone 3) • Visibility from initial depth map is useless to iteratively improve the solution Pseudo-Visibility It compensates for the bias by label- ing both occluders and occludees as invisible. • Preserves Geo-consistency • Computed using rendering tech- niques (the depth map is repre- sented as a continuous mesh) Algorithm: Overview • Initialize with all cameras visible in each mask • Compute depth map • Update pseudo-visibility masks using depth map • Iterate until convergence depth map f initial masks g stereo matcher Pseudo-Visibility final depth map f final masks g masks changed ? new masks g History Yes No stereo matcher Ordering Constraint It is respected when the order in which 2 objects are encountered along an epipolar line does not change. • Not always true • Continuous mesh ⇒ ordering constraint is respected 2 1 2 1 2 1 2 1 1 2 2 1 History Once a camera is removed, it is never used again. • Guaranteed convergence to a Geo-consistent solution that re- spects the ordering constraint Results:Middlebury An error is a difference greater than 1 label from the ground truth. Middlebury sequence algorithms barn1 barn2 bull poster venus sawtooth Boykov99 3.5 % 3.1 % 0.7 % 3.7 % 3.4 % 3.3% Ours 0.8 % 0.6 % 0.4 % 1.1 % 2.4 % 1.1 % Kang01 1.4 % 1.5 % 0.9 % 1.1 % 4.0 % 1.5% Drouin05 0.7 % 3.9 % 0.8 % 4.0 % 5.3% 1.0 % Results:Baseline Test Depth maps recovered for the same viewpoint with different baselines should be identical. • baseline ↑⇒ occlusion ↑ 0 2 4 6 8 10 12 14% Boykov99 Sanfourche04 Kolmogorov02 Sanfourche04 Ours 1x vs 2x 2x vs 3x 3x vs 4x 1x vs 4x % differing pixels Ours (3x) Reference Kang01 (3x) Kolmogorov02 (3x) Conclusion • New framework to model occlu- sion in stereo by introducing Geo- consistency • Occlusion modeling is added to standard stereo algorithms Future Work • Better handling of regions break- ing the ordering constraint • Extending the framework to vol- umetric reconstruction