Lecture30

CSE486, Penn StateRobert Collins

Lecture 30:

Video Tracking: Lucas-Kanade

CSE486, Penn StateRobert Collins

Two Popular Tracking Methods

• Mean-shift color histogram tracking (last time)

• Lucas-Kanade template tracking (today)

CSE486, Penn StateRobert Collins

Lucas-Kanade Tracking

CSE486, Penn StateRobert Collins

Review: Lucas-Kanade

• Brightness constancy

• One equation two unknowns

• How to get more equations for a pixel?– Basic idea: impose additional constraints

• one method: pretend the pixel’s neighbors have the same (u,v)– If we use a 5x5 window, that gives us 25 equations per pixel!

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

unknown flow vector

spatial gradienttemporal gradient

CSE486, Penn StateRobert Collins

Review: Lucas-Kanade (cont)

• Now we have more equations than unknowns

– The summations are over all pixels in the K x K window

– This technique was first proposed by Lucas & Kanade (1981)• described in Trucco & Verri reading

• Solution: solve least squares problem– minimum least squares solution given by solution (in d) of:

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

CSE486, Penn StateRobert Collins

Lucas Kanade Tracking

Traditional Lucas-Kanade is typically run on small, corner-like features (e.g. 5x5) to compute optic flow.

Observation: There’s no reason we can’t use the same approach on a larger window around the object being tracked.

80x50 pixels

CSE486, Penn StateRobert Collins

Basic LK Derivation for Templates

template(model)

current frameu,v = hypothesized location of template in current frame

CSE486, Penn StateRobert Collins

Basic LK Derivation for Templates

Take partial derivs and set to zero

First order approx

Form matrix equation

solve vialeast-squares

CSE486, Penn StateRobert Collins

One Problem with this...

Assumption of constant flow (pure translation) forall pixels in a larger window is unreasonable for long periods of time.

However, we can easily generalize Lucas-Kanadeapproach to other 2D parametric motion models (like affine or projective) by introducing a “warp”function W.

2])],([))];,([([ yxTPyxWIpatchimagewithin

y

x

generalize

CSE486, Penn StateRobert Collins

Step-by-Step Derivation

PPyxWI ))];,([([

The key to the derivation is Taylor series approximation:

PP

WIPyxWI

))];,([([~~

We will derive this step-by-step. First, we need two background formula:

CSE486, Penn StateRobert Collins

Step-by-Step Derivation

First consider the expansion for a single variable p

CSE486, Penn StateRobert Collins

Step-by-Step Derivation

Note that each variable parameter pi contributes a term of the form

CSE486, Penn StateRobert Collins

Step-by-Step Derivation

Now let’s rewrite the expression as a matrix equation. For each term,we can rewrite:

So that we have:

CSE486, Penn StateRobert Collins

Step-by-Step DerivationFurther collecting the dw/dpi terms into a matrix, we can write:

PPyxWI ))];,([([ PP

WIPyxWI

))];,([([~~

which are the terms in the matrix equation:

CSE486, Penn StateRobert Collins

Example: Jacobian of Affine Warp

11

1)];,([

642

531 y

x

ppp

pppPyxW

affine warp function (6 parameters)

1000

0100

642

531

yx

yxP

PyPyxP

PyPxPx

P

W

n

yyyy

n

xxxx

yx

P

W

P

W

P

W

P

WP

W

P

W

P

W

P

W

P

W

WWPyxWLet

321

321

],[)];,([

general equation of Jacobian

CSE486, Penn StateRobert Collins

Iterate Warp I to obtain I(W([x y];P))

Compute the error image T(x) – I(W([x y]; P))

Warp the gradient with W([x y]; P)

Evaluate at ([x y]; P)

Compute steepest descent images

Compute Hessian matrix

Compute

Compute P

Update P P + P

I

P

W

P

WI

)()(P

WI

P

WI T

)))];,([(),(()( PyxWIyxTP

WI T

Until P magnitude is negligible

Source: “Lucas-Kanade 20 years on: A unifying framework” Baker and Mathews, IJCV 04

Dr. Ng Teck Khim

(Jacobian)

CSE486, Penn StateRobert Collins

Source: “Lucas-Kanade 20 years on: A unifying framework” Baker and Mathews, IJCV 04

Algorithm At a Glance

CSE486, Penn StateRobert Collins

Baker, Matthews, CMU

State of the Art Lucas Kanade Tracking

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Tracking facial mesh models (piecewise affine)