The Variational Structure of Disparity and Regularization of 4D Light Fields Bastian Goldluecke and Sven Wanner Heidelberg Collaboratory for Image Processing, University of Heidelberg http://lightfield-analysis.net e-mail: {bastian.goldluecke,sven.wanner}@iwr.uni-heidelberg.de Contributions General variational framework for inverse problems on ray space Specialization to arbitrary convex data terms and spatial regularizers possible Currently implemented are denoising, disparity map regularization, inpainting, multi-label segmentation 4D Light Field Parametrization and Epipolar Plane Images (EPIs) Light field structure Disparity and epipolar plane images P =(X,Y,Z ) x 1 t Π Ω y f s 1 s 2 x 2 Δs x 2 - x 1 = f Z Δs Light field parametrization Epipolar plane image (EPI) Disparity equals local slope General inverse problems on ray space The general inverse problem Find a vector field U on ray space R which minimizes argmin U :R→R d J (U ) | {z } convex ray space regularizer + F (U ) | {z } convex data term encodes problem . Regularizer for an epipolar plane image Regularization needs to be performed in the direction of epipolar lines, which is given by the disparity field ρ: xxxxxx d =[ρ 1] This can be enforced by using an anisotropic total varia- tion J ρ (U y * ,t * ) := n X i=1 Z q (∇U i y * ,t * ) T D ρ ∇U i y * ,t * d(x, s), where the tensor D ρ encodes the direction information. Regularizer for ray space Sum of independent convex regularizers for epipolar plane images in (y,t) and (x, s) coor- dinates as well as pinhole views in (x, y ) coor- dinates: J λμ (U )= μJ xs (U )+ μJ yt (U )+ λJ st (U ) with J xs (U )= Z J ρ (U x * ,s * ) d(x * ,s * ) J yt (U )= Z J ρ (U y * ,t * ) d(y * ,t * ) J st (U )= Z J V (U s * ,t * ) d(s * ,t * ). Optimization scheme Constraints on disparity and disparity regularization Constraints on disparity maps Local constraints λ j λ i d i λ j λ i d i xxxAllowed if λ i <λ j Forbidden if λ i <λ j XXXy Local constraints are in general not sufficient d 0 d 1 d 2 Global constraints required, but too costly to optimize Unneccessary if baselines are small ! Variational energy for the constraints E ±d x * ,s * (ρ x * ,s * )= Z min(∇ ±d ρ x * ,s * , 0) 2 d(y,t) E ±d y * ,t * (ρ y * ,t * )= Z min(∇ ±d ρ y * ,t * , 0) 2 d(x, s) Constrained disparity map estimation disparity MSE in pixels ·10 2 Regularization local single view rayspace constrained Average 4.602 2.727 2.240 1.997 Inpainting for view interpolation Input view Linear interpolation Light field inpainting, Light field inpainting interpolated disparity inpainted disparity Input disparity Linear interpolation Disparity map inpainting Inpainting with constraints Light field inpainting for view interpolation. Intermediate views in the upsampled light field were marked as unknown regions before solving the inpainting model. IEEE International Conference on Computer Vision and Pattern Recognition 2013, Portland, Oregon, USA. Light field denoising Denoising data term First experiments were performed with a simple L 2 -norm dataterm F (U )= Z R (U - F ) 2 d(x, y, s, t), where F is the light field to be denoised. Extensions to more complex data terms and spatial regularizers are straight-forward. Results (total variation regularizer) Original (closeup) With Gaussian noise Single view denoising Ray space denoising Papillon σ =0.2, PSNR=15.35 PSNR=27.09 PSNR=28.72 Horses σ =0.2, PSNR=14.42 PSNR=24.62 PSNR=28.90 StillLife σ =0.2, PSNR=14.66 PSNR=22.61 PSNR=24.46 Plenoptic 1 σ =0.2, PSNR=14.95 PSNR=24.57 PSNR=26.84 Plenoptic 2 σ =0.2, PSNR=14.71 PSNR=26.70 PSNR=30.38 Light field inpainting Inpainting model Let Γ ⊂R be a region in ray space where the input light field F is unknown. The goal is to recover a function U which restores the missing values. For this, we find argmin U J λμ (U ) such that U = F on Ω \ Γ. Results (total variation regularizer) Damaged input Spatial inpainting (TV) Light field inpainting After 5 iterations After 10 iterations After 15 iterations After 20 iterations Optimization To solve the general inverse problem on ray space, we initialize the unknown vector-valued function with U =0 and iterate the following steps: data term descent: U ← U - 1 L ∇F (U ), EPI regularizer descent: U x * ,s * ← prox L -1 μJ ρ (U x * ,s * ) for all (x * ,s * ), U y * ,t * ← prox L -1 μJ ρ (U y * ,t * ) for all (y * ,t * ), spatial regularizer descent: U s * ,t * ← prox L -1 λJ V (U s * ,t * ) for all (s * ,t * ). All proximity operators above are two-dimensional problems with an L 2 dataterm. Bibliography B. Goldluecke and S. Wanner. The variational structure of disparity and regularization of 4D light fields. In Proc. International Conference on Computer Vision and Pattern Recognition, 2013. S. Wanner and B. Goldluecke. Globally consistent depth labeling of 4D light fields. In Proc. International Conference on Computer Vision and Pattern Recognition, pages 41–48, 2012. S. Wanner, C. Straehle, and B. Goldluecke. Globally consistent multi-label assignment on the ray space of 4D light fields. In Proc. International Conference on Computer Vision and Pattern Recognition, 2013.