Transcript
CSE486, Penn StateRobert Collins
Lecture 30:
Video Tracking: Lucas-Kanade
CSE486, Penn StateRobert Collins
Two Popular Tracking Methods
• Mean-shift color histogram tracking (last time)
• Lucas-Kanade template tracking (today)
CSE486, Penn StateRobert Collins
Lucas-Kanade Tracking
CSE486, Penn StateRobert Collins
Review: Lucas-Kanade
• Brightness constancy
• One equation two unknowns
• How to get more equations for a pixel?– Basic idea: impose additional constraints
• one method: pretend the pixel’s neighbors have the same (u,v)– If we use a 5x5 window, that gives us 25 equations per pixel!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
unknown flow vector
spatial gradienttemporal gradient
CSE486, Penn StateRobert Collins
Review: Lucas-Kanade (cont)
• Now we have more equations than unknowns
– The summations are over all pixels in the K x K window
– This technique was first proposed by Lucas & Kanade (1981)• described in Trucco & Verri reading
• Solution: solve least squares problem– minimum least squares solution given by solution (in d) of:
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
CSE486, Penn StateRobert Collins
Lucas Kanade Tracking
Traditional Lucas-Kanade is typically run on small, corner-like features (e.g. 5x5) to compute optic flow.
Observation: There’s no reason we can’t use the same approach on a larger window around the object being tracked.
80x50 pixels
CSE486, Penn StateRobert Collins
Basic LK Derivation for Templates
template(model)
current frameu,v = hypothesized location of template in current frame
CSE486, Penn StateRobert Collins
Basic LK Derivation for Templates
Take partial derivs and set to zero
First order approx
Form matrix equation
solve vialeast-squares
CSE486, Penn StateRobert Collins
One Problem with this...
Assumption of constant flow (pure translation) forall pixels in a larger window is unreasonable for long periods of time.
However, we can easily generalize Lucas-Kanadeapproach to other 2D parametric motion models (like affine or projective) by introducing a “warp”function W.
2])],([))];,([([ yxTPyxWIpatchimagewithin
y
x
generalize
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
PPyxWI ))];,([([
The key to the derivation is Taylor series approximation:
PP
WIPyxWI
))];,([([~~
We will derive this step-by-step. First, we need two background formula:
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
First consider the expansion for a single variable p
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
Note that each variable parameter pi contributes a term of the form
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
Now let’s rewrite the expression as a matrix equation. For each term,we can rewrite:
So that we have:
CSE486, Penn StateRobert Collins
Step-by-Step DerivationFurther collecting the dw/dpi terms into a matrix, we can write:
PPyxWI ))];,([([ PP
WIPyxWI
))];,([([~~
which are the terms in the matrix equation:
CSE486, Penn StateRobert Collins
Example: Jacobian of Affine Warp
11
1)];,([
642
531 y
x
ppp
pppPyxW
affine warp function (6 parameters)
1000
0100
642
531
yx
yxP
PyPyxP
PyPxPx
P
W
n
yyyy
n
xxxx
yx
P
W
P
W
P
W
P
WP
W
P
W
P
W
P
W
P
W
WWPyxWLet
321
321
],[)];,([
general equation of Jacobian
CSE486, Penn StateRobert Collins
Iterate Warp I to obtain I(W([x y];P))
Compute the error image T(x) – I(W([x y]; P))
Warp the gradient with W([x y]; P)
Evaluate at ([x y]; P)
Compute steepest descent images
Compute Hessian matrix
Compute
Compute P
Update P P + P
I
P
W
P
WI
)()(P
WI
P
WI T
)))];,([(),(()( PyxWIyxTP
WI T
Until P magnitude is negligible
Source: “Lucas-Kanade 20 years on: A unifying framework” Baker and Mathews, IJCV 04
Dr. Ng Teck Khim
(Jacobian)
CSE486, Penn StateRobert Collins
Source: “Lucas-Kanade 20 years on: A unifying framework” Baker and Mathews, IJCV 04
Algorithm At a Glance
CSE486, Penn StateRobert Collins
Baker, Matthews, CMU
State of the Art Lucas Kanade Tracking
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Tracking facial mesh models (piecewise affine)
top related