Professor Horst Cerjak, 19.12.20051
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG
A Duality Based Approach for Realtime TV-L1 Optical Flow
Christopher Zach1, Thomas Pock2, and Horst Bischof2
1 VRVis Research Center, Graz2 Institute for Computer Graphics and Vision, TU Graz
E-mail: {zach, pock, bischof}@icg.tugraz.at
Professor Horst Cerjak, 19.12.20052
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG
time
Motivation
• Discontinuity preserving regularization of the flow field
• Robustness to occulsions
• Handle large displacements
• Realtime (> 30 fps) for large images (512x512)
Professor Horst Cerjak, 19.12.20053
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Outline
• (I) Variational Optical Flow
• (II) TV-L1 optical flow
• (III) Duality Based Approach
• (IV) Acceleration using the GPU
• (V) Performance Evaluation
• (VI) Conclusion & Demo
Professor Horst Cerjak, 19.12.20054
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Optical Flow
• Optical Flow (OF) is a major task of biological and artificial visual systems
• Relates the motion of pixel intensities between consecutive image frames
• Optical Flow Constraint:
• Gives only the normal flow• No OFC in untextured areas
0),(),(),( txItxIxu t
u1
u2 u
Professor Horst Cerjak, 19.12.20055
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Variational Optical Flow
• First studied by Horn and Schunck in 1981 [1]
• Quadratic regularization does not allow for discontinuities and occlusions
• Modifying the Horn and Schunck functional was pioneered by Black and Rangarajan [2]
[1] B.K. Horn and B.G. Schunck. Determinig Optical Flow. Artificial Intelligence, 1981[2] M.J. Black and P. Rangarajan. On the Unification of Line Processes, Outlier Rejection and Robust Statistics withApplications in Early Vision, IJCV, 1996
dxxIxuxIdxuuu
201
2
2
2
1 )())((min
)(2 ss
Professor Horst Cerjak, 19.12.20056
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG TV-L1 Optical Flow
• We use a robust variant of the Horn-Schunck formulation– Total Variation (TV) of Rudin Osher and Fatemi (ROF) [3]– L1 penalization of the OF constraint
• TV-L1 has been used in many approaches• Allows for discontinuities in the flow field and outliers in
the optical flow constraint• Sophisticated optimization techniques are needed• This is the major goal of this paper
dxxIxuxIdxuuu
)())((min 0121
[3] L. Rudin and S. Osher and E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, Physica D, 1992
Professor Horst Cerjak, 19.12.20057
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG An Approximative Formulation
dxvvudxuEd
ddd
dvu
2
, 2
1min
dxudxuEd
du
min
Eθ E as Θ 0
Main difficulty is induced by the TV term –> ROF model [3]Simple pointwise optimization problem -> Thresholding
[3] L. Rudin and S. Osher and E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, Physica D, 1992
Professor Horst Cerjak, 19.12.20058
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Primal Formulation
• Study of the ROF model
dxvudxuEROF2
2
1min
01
vuu
u
u
• Primal Euler Lagrange equations
• Degenerated if gradient vanishes
2uu• Simple solution: Replace by
• Disadvantage: Large ε smoothes edges!
Professor Horst Cerjak, 19.12.20059
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Dual Formulation
• Studied by Chan [4], and later by Chambolle [5]
0 uup
01
vuu
u
01
vup
0 vpvpp
p• One arrives at two new equations
[4] T. Chan and G. Golub and P. Mulet, A Nonlinear Primal Dual Method for TV-based Image Restoration, 1999[5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004
• Advantage: No regularization is needed!
• Dual Euler Lagrange Equations
|p| ≤ 1
u
Professor Horst Cerjak, 19.12.200510
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Primal vs. Dual• Convergence of the Primal and Dual formulation
– Primal: fixed-point scheme of Vogel & Oman [6]– Dual: fixed-point scheme of Chambolle [5]
ε=10-15ε=10-1ε=1ε=10EROF
iterations[5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004[6] C. R. Vogel and M. E. Oman. Iterative Methods For Total Variation Denoising. 1996
Professor Horst Cerjak, 19.12.200511
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Final Algorithm
dxvvudxuEd
ddd
dvu
2
, 2
1min
1.Fix v, minimize wrt. u (Chambolle‘s algorithm)
2.Fix u, minimize wrt. v (Thresholding)
3.Goto 1 until convergence
• Energy minimization is embedded into a coarse-to-fine approach to handle large displacements
• Solved via alternating optimization
Professor Horst Cerjak, 19.12.200512
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICGImplementation on Graphics
Hardware
• Particularly well suited to compute variational methods– High degree of parallelism– High performance processing units
• All features can be accessed via C-like languages• Performance of graphics cards is steadily increasing
G92
Nov2007
Professor Horst Cerjak, 19.12.200513
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Performance Evaluation
Image resolution 50 Iterations 100 Iterations 200 IterationsGraphics card 7900GTX 8800GTX 7900GTX 8800GTX 7900GTX 8800GTX
128x128 95 125.9 57.6 65.12 30.9 32.82
256x256 34.1 83.11 17.5 43.29 8.9 21.85
512x512 9.3 45.07 4.7 22.93 2.3 11.69
Error evaluation on the well known Yosemite without clouds sequence
HS [1] TV-L1
(50 it.)
Nir et al. [7]
AAE 32.43° 2.85° 0.85°
Frames per second
[1] B.K. Horn and B.G. Schunck. Determinig Optical Flow. Artificial Intelligence, 1981[7] T. Nir and A.M. Bruckstein and R. Kimmel, Over-Parameterized Variational Optical Flow, IJCV 2007
Professor Horst Cerjak, 19.12.200514
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Conclusion & Future work
• We have developed a duality based algorithm for TV-L1 optical flow computation
• We have implemented this algorithm on state-of-the-art graphics hardware
• In summary, we obtained an optical flow algorithm having a realtime performance of ~45 fps for 512x512 images
• Implementation in CUDA should give an additional speedup
• More sophisticated data terms for illumination changes• Multigrid techniques for the dual formulation
Professor Horst Cerjak, 19.12.200515
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG
Demo
Professor Horst Cerjak, 19.12.200516
Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow
ICG Solution of the ROF model
• Compute the minimizer of ROF model– Solution of a huge sytem of non-linear equations– Leads to iterative algorithms
• Primal formulation:– Fixed-point scheme of Vogel and Oman [6]
• Dual formulation– Fixed-point scheme of Chambolle [5]
,...3,2,1 ,01 1
1
nvuu
u nn
n
,...3,2,1 ,01 nvpvpp nnn
[5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004[6] C. R. Vogel and M. E. Oman. Iterative Methods For Total Variation Denoising. 1996