Iterative Image Registration: Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group Microsoft Research
Dec 19, 2015
Iterative Image Registration:
Lucas & Kanade Revisited
Kentaro Toyama
Vision Technology Group
Microsoft Research
Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future.
Jorge Luis Borges
History
• Lucas & Kanade (IUW 1981)
LK BAHH ST S BJ HB BL G SI CETSC
• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)
• Shi & Tomasi (CVPR 1994)
• Szeliski & Coughlan (CVPR 1994)
• Szeliski (WACV 1994)
• Black & Jepson (ECCV 1996)
• Hager & Belhumeur (CVPR 1996)
• Bainbridge-Smith & Lane (IVC 1997)
• Gleicher (CVPR 1997)
• Sclaroff & Isidoro (ICCV 1998)
• Cootes, Edwards, & Taylor (ECCV 1998)
Image Registration
Applications
Applications
• Stereo
LK BAHH ST S BJ HB BL G SI CETSC
Applications
• Stereo
• Dense optic flow
LK BAHH ST S BJ HB BL G SI CETSC
Applications
• Stereo
• Dense optic flow
• Image mosaics
LK BAHH ST S BJ HB BL G SI CETSC
Applications
• Stereo
• Dense optic flow
• Image mosaics
• Tracking
LK BAHH ST S BJ HB BL G SI CETSC
Applications
• Stereo
• Dense optic flow
• Image mosaics
• Tracking
• Recognition
LK BAHH ST S BJ HB BL G SI CETSC
?
Lucas & Kanade
#1
Derivation
L&K Derivation 1
I0(x)
)('0 xI
h
xIhxIh
)()(lim 00
0
)('0 xI
L&K Derivation 1
)('0 xI
h
xIhxI )()( 00
h I0(x)
I0(x+h)
L&K Derivation 1
h I0(x)
)('0 xI
h
xIxI )()( 0
I(x)
L&K Derivation 1
h I0(x)
h)(
)()('0
0
xI
xIxI
I(x)
L&K Derivation 1
I0(x)
h
Rx xI
xIxI
R )(
)()(
||
1'0
0
RI(x)
L&K Derivation 1
I0(x)
h
RxxxI
xIxIxw
xw )(
)]()()[(
)(
1'0
0
I(x)
L&K Derivation 1
h0 I0(x)
0h
I(x)
RxxxI
xIxIxw
xw )(
)]()()[(
)(
1'0
0
L&K Derivation 1
1h
Rxx
hxI
hxIxIxw
xwh
)(
)]()()[(
)(
1
0'0
000
I0(x+h0)
I(x)
L&K Derivation 1
2h
Rxx
hxI
hxIxIxw
xwh
)(
)]()()[(
)(
1
1'0
101
I0(x+h1)
I(x)
L&K Derivation 1
1kh
Rx k
k
x
k hxI
hxIxIxw
xwh
)(
)]()()[(
)(
1'0
0
I0(x+hk)
I(x)
L&K Derivation 1
1kh
Rx k
k
x
k hxI
hxIxIxw
xwh
)(
)]()()[(
)(
1'0
0
I0(x+hf)
I(x)
Lucas & KanadeDerivation
#2
L&K Derivation 2
• Sum-of-squared-difference (SSD) error
E(h) = [ I(x) - I0(x+h) ]2x R
E(h) [ I(x) - I0(x) - hI0’(x) ]2x R
L&K Derivation 2
2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] x Rh
E
I0’(x)(I(x) - I0(x))x R h I0’(x)2
x R
= 0
Comparison
I0’(x)[I(x) - I0(x)] h I0’(x)2
x
x
h
w(x)[I(x) - I0(x)]
w(x)x
x I0’(x)
Comparison
I0’(x)[I(x) - I0(x)] h I0’(x)2
x
h
x
w(x)[I(x) - I0(x)]
w(x)x
x I0’(x)
Generalizations
Original
h ) = x R
(E [I( x ) - (x ]2)+ h I
Original
• Dimension of image
h ) = x R
(E [I( x ) - (x ]2)+ h
1-dimensional
I
LK BAHH ST S BJ HB BL G SI CETSC
Generalization 1a
• Dimension of image
h ) = x R
(E [I( x ) - (x ]2)+ h
y
xx2D:
I
LK BAHH ST S BJ HB BL G SI CETSC
Generalization 1b
• Dimension of image
h ) = x R
(E [I( x ) - (x ]2)+ h
1
y
x
xHomogeneous 2D:
I
LK BAHH ST S BJ HB BL G SI CETSC
Problem A
LK BAHH ST S BJ HB BL G SI CETSC
Does the iteration converge?
Problem A
Local minima:
Problem A
Local minima:
Problem B
- I0’(x)(I(x) - I0(x))x R h I0’(x)2
x R
h is undefined if I0’(x)2 is zerox R
LK BAHH ST S BJ HB BL G SI CETSC
Zero gradient:
Problem B
Zero gradient:
?
Problem B’
- (x)(I(x) - I0(x))x R
hy 2
x R
y
I )(0 xy
I
)(0 x
Aperture problem:
LK BAHH ST S BJ HB BL G SI CETSC
Problem B’
No gradient along one direction:
?
Solutions to A & B
• Possible solutions:– Manual intervention
LK BAHH ST S BJ HB BL G SI CETSC
• Possible solutions:– Manual intervention– Zero motion default
LK BAHH ST S BJ HB BL G SI CETSC
Solutions to A & B
• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”
LK BAHH ST S BJ HB BL G SI CETSC
Solutions to A & B
• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features
LK BAHH ST S BJ HB BL G SI CETSC
Solutions to A & B
• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering
LK BAHH ST S BJ HB BL G SI CETSC
Solutions to A & B
• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation
LK BAHH ST S BJ HB BL G SI CETSC
Solutions to A & B
• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation– Higher-order terms
LK BAHH ST S BJ HB BL G SI CETSC
Solutions to A & B
Original
h ) = x R
(E [I( x ) - (x ]2)+ h I
Original
• Transformations/warping of image
h ) = x R
(E [I( x ) -I(x ]2)+ h
Translations:
y
x
h
LK BAHH ST S BJ HB BL G SI CETSC
Problem C
What about other types of motion?
Generalization 2a
• Transformations/warping of image
A, h) = x R
(E [I(Ax ) - (x ]2)+h
Affine:
dc
baA
y
x
h
I
LK BAHH ST S BJ HB BL G SI CETSC
Generalization 2a
Affine:
dc
baA
y
x
h
Generalization 2b
• Transformations/warping of image
A ) = x R
(E [I( A x ) - (x ]2)
Planar perspective:
187
654
321
aa
aaa
aaa
A
I
LK BAHH ST S BJ HB BL G SI CETSC
Generalization 2b
Planar perspective:
187
654
321
aa
aaa
aaa
A
Affine +
Generalization 2c
• Transformations/warping of image
h ) = x R
(E [I( f(x, h) ) - (x ]2)
Other parametrized transformations
I
LK BAHH ST S BJ HB BL G SI CETSC
Generalization 2c
Other parametrized transformations
Problem B”
-(JTJ)-1 J (I(f(x,h)) - I0(x)) h ~
Generalized aperture problem:
LK BAHH ST S BJ HB BL G SI CETSC
- I0’(x)(I(x) - I0(x))x R h I0’(x)2
x R
Problem B”
?
Generalizedaperture problem:
Original
h ) = x R
(E [I( x ) - (x ]2)+ h I
Original
• Image type
h ) = x R
(E [I( x ) - (x ]2)+ h
Grayscale images
I
LK BAHH ST S BJ HB BL G SI CETSC
Generalization 3
• Image type
h ) = x R
(E ||I( x ) -I(x ||2)+ h
Color images
LK BAHH ST S BJ HB BL G SI CETSC
Original
h ) = x R
(E [I( x ) - (x ]2)+ h I
Original
• Constancy assumption
h ) = x R
(E [I( x ) -I(x ]2)+ h
Brightness constancy
LK BAHH ST S BJ HB BL G SI CETSC
Problem C
What if illumination changes?
Generalization 4a
• Constancy assumption
h, )=x R
(E [I( x ) - I(x ]2)++ h
Linear brightness constancy
LK BAHH ST S BJ HB BL G SI CETSC
Generalization 4a
Generalization 4b
• Constancy assumption
h,) = x R
(E [I( x ) - B(x]2)+ h
Illumination subspace constancy
LK BAHH ST S BJ HB BL G SI CETSC
Problem C’
What if the texture changes?
Generalization 4c
• Constancy assumption
h,) = x R
(E [I( x ) - ]2+ h
Texture subspace constancy
B(x)
LK BAHH ST S BJ HB BL G SI CETSC
Problem D
Convergence is slower as #parameters increases.
• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc.
LK BAHH ST S BJ HB BL G SI CETSC
Solutions to D
• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization
Solutions to D
LK BAHH ST S BJ HB BL G SI CETSC
• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation
Solutions to D
LK BAHH ST S BJ HB BL G SI CETSC
• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation
• Difference decomposition
LK BAHH ST S BJ HB G SI CETSC
Solutions to D
BL
Solutions to D
• Difference decomposition
Solutions to D
• Difference decomposition
• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation
• Difference decomposition
– Improvements in gradient descent
LK BAHH ST S BJ HB G SI CETSC
Solutions to D
BL
• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization– Offline precomputation
• Difference decomposition
– Improvements in gradient descent• Multiple estimates of spatial derivatives
LK BAHH ST S BJ HB G SI CETSC
Solutions to D
BL
Solutions to D
• Multiple estimates / state-space sampling
Generalizations
x R
[I( x ) - (x ]2)+ h I
Modifications made so far:
Original
• Error norm
h ) = x R
(E [I( x ) -I(x ]2)+ h
Squared difference:
LK BAHH ST S BJ HB BL G SI CETSC
Problem E
What about outliers?
Generalization 5a
• Error norm
h ) = x R
(E (I( x ) -I(x ))+ h
Robust error norm:
22
2
)(uk
uuρ
LK BAHH ST S BJ HB BL G SI CETSC
Original
h ) = x R
(E [I( x ) - (x ]2)+ h I
Original
• Image region / pixel weighting
h ) = x R
(E [I( x ) -I(x ]2)+ h
Rectangular:
LK BAHH ST S BJ HB BL G SI CETSC
Problem E’
What about background clutter?
Generalization 6a
• Image region / pixel weighting
h ) = x R
(E [I( x ) -I(x ]2)+ h
Irregular:
LK BAHH ST S BJ HB BL G SI CETSC
Problem E”
What about foreground occlusion?
Generalization 6b
• Image region / pixel weighting
h ) = x R
(E [I( x ) -I(x ]2)+ h
Weighted sum:
w(x)
LK BAHH ST S BJ HB BL G SI CETSC
Generalizations
x R
[I( x ) - (x ]2)+ h I
Modifications made so far:
Generalizations: Summary
= x R
(I( ) - w(x) (x ))h )(E f(x, h)
h ) = x R
(E [I( x ) - (x ]2)+ h I
Foresight
• Lucas & Kanade (IUW 1981)
• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)
• Shi & Tomasi (CVPR 1994)
• Szeliski & Coughlan (CVPR 1994)
• Szeliski (WACV 1994)
• Black & Jepson (ECCV 1996)
• Hager & Belhumeur (CVPR 1996)
• Bainbridge-Smith & Lane (IVC 1997)
• Gleicher (CVPR 1997)
• Sclaroff & Isidoro (ICCV 1998)
• Cootes, Edwards, & Taylor (ECCV 1998)
LK BAHH ST S BJ HB BL G SI CETSC
Summary
• Generalizations– Dimension of image– Image transformations / motion models– Pixel type– Constancy assumption– Error norm– Image mask
L&K ?Y
Y
n
Y
n
Y
Summary
• Common problems:– Local minima– Aperture effect– Illumination changes– Convergence issues– Outliers and occlusions
L&K ?Y
maybe
Y
Y
n
• Mitigation of aperture effect:– Manual intervention– Zero motion default– Coefficient “dampening”– Elimination of poor textures– Temporal filtering– Spatial interpolation / hierarchical – Higher-order terms
Summary
L&K ?n
n
n
n
Y
Y
n
Summary
• Better convergence:– Coarse-to-fine, filtering, etc.– Selective parametrization – Offline precomputation
• Difference decomposition
– Improvements in gradient descent• Multiple estimates of spatial derivatives
L&K ?Y
nmaybe
maybe
maybe
maybe
Hindsight
• Lucas & Kanade (IUW 1981)
• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)
• Shi & Tomasi (CVPR 1994)
• Szeliski & Coughlan (CVPR 1994)
• Szeliski (WACV 1994)
• Black & Jepson (ECCV 1996)
• Hager & Belhumeur (CVPR 1996)
• Bainbridge-Smith & Lane (IVC 1997)
• Gleicher (CVPR 1997)
• Sclaroff & Isidoro (ICCV 1998)
• Cootes, Edwards, & Taylor (ECCV 1998)