Accurate Depth and Normal Maps from Occlusion-Aware Focal Stack Symmetry Michael Strecke, Anna Alperovich, and Bastian Goldluecke Computer Vision and Image Analysis University of Konstanz Contributions Novel way to handle occlusions when constructing cost volumes based on focal stack symmetry Joint regularization of depth and normals for smooth normal maps consistent with depth estimate Light eld structure and focal stacks Light elds are dened as 4D function L: ! R on ray space, where rays are given by intersection points with the focal plane and the image plane. Refocusing to disparity: aperture lter over subaperture views v = ( s; t), ’ p ( )= Z (v)L(p + v;v) dv: (1) x y t s Lin et al. [6]:in absence of occlusion, the focal stack is symmetric around the ground truth disparity. Assignment cost for disparities measures symmetry, s ’ p ( )= Z max 0 (’ p ( ) ’ p ( + )) d: (2) 1 B B B d B d+ B d B d 1 B B B d B d+ B d Occlusion-aware focal stack symmetry Problem: Focal stack at occlusions not symmetric around true disparity Our assumption: occlusions only in one half-plane of view points Then we can prove symmetry in partial focal stacks , ’ e;p (d + )=’ + e;p (d ); where ’ e;p ( )= Z 0 1 L(p + s e; se) ds ’ + e;p ( )= Z 1 0 L(p + s e; se) ds: (3) Our new disparity cost encourages symmetry for partial horizontal and vertical stacks: s ’ p ( )= Z max 0 min (’ (1;0);p ( + ) ’ + (1;0);p ( )); (’ (0;1);p ( + ) ’ + (0;1);p ( )) d: (4) (a) slice through focal stack ’ from [6] (b) slice through ’ + (c) slice through ’ CV GT disp Lin et al. [6] Proposed Boxes 28.12 20.87 Cotton 6.83 3.37 Dino 10.01 3.52 Sideboard 26.80 9.23 This work was supported by the ERC Starting Grant "Light Field Imaging and Analysis" (LIA 336978, FP7-2014). Presented at the Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, July 2017. Joint depth and normal map optimization Problem: global optimal solution obtained with sublabel relaxation [7] locally at. Our approach: novel prior on normal maps which enforces correct relation to depth a well as smoothness of the normal eld: E(; n)=min > 0 Z (;x ) + kN nk 2 d x + R(n) d x (5) whereR(n) is a regularizer for the normal eld given as R(n ) = sup w 2C 1 c ( ;R n m ) Z kw Dnk + g kDwk F d x (6) and reparametrized depth := 1 2 z 2 is related to normals by a linear operator N( ) [2]. Optimization for depth: Terms not dependent onare removed, resulting in saddle point problem: min ;> 0 max kpk 2 ;jj 1 (p; N n)+ (; j 0 +( 0 )@ j 0 ) : (7) Solved using the primal-dual algorithm [1]. Optimization for normals: Removing all terms not depending on n: L 1 denoising problem min kn k=1 Z kN k kw nk d x + R(n ): (8) Nonconvex due to kn k = 1: adoption of ideas from [10] for solution (local parameterization of tangent space, eective linearization). Comparison of normal maps for dierent methods ground truth normals sublabel relaxation [7] normal smoothing [10] depth smoothing [2] proposed method Boxes 48.55 36.92 19.67 17.11 Cotton 33.64 31.54 12.87 9.78 Dino 43.55 29.91 2.67 1.68 Sideboard 49.59 65.48 12.49 8.58 Combining cost terms can improve robustness center view focal stack symmetry focal stack + stereo 37.67 21.61