Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics

Class 3 1

Multi-linear Systems and Invariant Theory

in the Context of Computer Vision and Graphics

Class 3: Infinitesimal Motion

CS329Stanford University

Amnon Shashua

Class 3 2

Material We Will Cover Today

• Infinitesimal Motion Model

• Infinitesimal Planar Homography (8-parameter flow)

• Factorization Principle for Motion/Structure Recovery

• Direct Estimation

Class 3 3

]ˆ)[(sinˆˆ)cos1()(cos wwwIR T w

Infinitesimal Motion Model

0 d

1)cos( d

0)sin( d

][]ˆ)[( wIwdIR

Rodriguez Formula:

Class 3 4

Z

Y

X

PtRPP '


tPwI x )][(

PPdt

dPP '

tPwP x ][

Class 3 5

X

Y

Z

x

y

Z

Y

X

P

),( 00 yx

),0,0( f

0xZ

Xfx

0yZ

Yfy

Reminder:

Assume: 0,1 00 yxf

PZ

p1

Class 3 6


tPwP x ][

ZxXZZ

XZZX

Z

X

dt

d

dt

dxu

12

Txs ),0,1( Let PsZ

u T 1

pwstsZ

u xTT ][

1

Class 3 7


Txs ),0,1(

pwstsZdt

dxu x

TT ][1

Tyr ),1,0(

Tyxp )1,,(

pwrtrZdt

dyv x

TT ][1

Class 3 8

Infinitesimal Planar Motion(the 8-parameter flow)

1PnT 1 cZbYaX

Zcbyax

1 0Z

pwstscbyaxu xTT ][)(

pwrtrcbyaxv xTT ][)(

Class 3 9


pwstscbyaxu xTT ][)(

pwxxttcbyax x])[,0,1())(( 31

yctbtxwatctwv )()()( 323221

23213 )()( xatwxywbt

ywbtxctatctwu )()()( 313112

xyatwywbt )()( 322

13

Class 3 10


287321 xxyyxu

278654 yxyyxv

Note: unlike the discrete case, there is no scale factor

Class 3 11

Reconstruction of Structure/Motion(factorization principle)

pwstsZ

u xTT ][

1

pwrtrZ

v xTT ][

1

)()(][ spwpwspws TTx

T

Note:

][][][)( cabbcaabccbaT

][][][ acbcbabac

2 interchanges

1 interchanges

Class 3 12


pwstsZ

u xTT ][

1

pwrtrZ

v xTT ][

1

)(][ spwpws Tx

T

t

w

rZ

rp

sZ

sp

v

u

T

T

62

1

1

Class 3 13


Let ),( ijij vu be the “flow” of point i at image j (image 0 is ref frame)

mm

m

n

Tnnn

T

Tnnn

T

nmnn

m

nmnn

m

tt

ww

rZrp

rZrp

sZsp

sZsp

vvv

vvvuuu

uuu

61

1

62

111

111

21

11211

21

11211

...

...

)/1(

..

..

)/1(

)/1(

..

..

)/1(

.

.

...

.

.

.

.

.

....

.

.

...

.

.

.

.

.

....

SMMS

S

V

UW

y

x

Class 3 14


SMMS

S

V

UW

y

x

Given W, find S,M

Let KLW (using SVD)

SMLAKA ))(( 1 for some 66A

Goal: find 66A such that SKA

using the “structural” constraints on S

Class 3 15




Columns 1-3 of S are known, thus columns 1-3 of A can be determined.

Columns 4-6 of A contain 18 unknowns:

TT rZsZ )/1(,)/1( eliminate Z and one obtains 5 constraints

Class 3 16




62

ny

x

K

KKLet ],...,[ 61 AAA

0,0 45 AKAK yx because

),1,0)(/1(),,0,1)(/1( yzxZ

Class 3 17


0,0 45 AKAK yx because

),1,0)(/1(),,0,1)(/1( yzxZ

54 AKAK yx

iix

ix xAK

AK

)(

)(

4

6

iiy

iy yAK

AK

)(

)(

5

6

Each point provides 5 constraints,thus we need 4 points and 7 views

Class 3 18

Direct Estimation

),(2 yxgI

),(1 yxfI The grey values of images 1,2

vy

ux

y

x

Goal: find u,v per pixel

Ryx

yxfvyuxgvuS),(

2)],()ˆ,ˆ([)ˆ,ˆ(

Class 3 19

Direct Estimation

vy

ux

y

x

Ryx

yxfvyuxgvuS),(

2)],()ˆ,ˆ([)ˆ,ˆ(

Assume: )ˆ,ˆ(minarg),( ˆ,ˆ vuSvu vu

u

v

)ˆ,ˆ( vuS

v

u

We are assuming that (u,v) can befound by correlation principle (minimizingthe sum of square differences).

Class 3 20

Direct Estimation

Ryx

yxfvyuxgvuS),(

2)],()ˆ,ˆ([)ˆ,ˆ(

Taylor expansion:

)(),(),(),(),( 2 Oyxvgyxugyxgvyuxg yx

Ryx

tyx IgvguvuS),(

2]ˆˆ[)ˆ,ˆ(

),(),(),( yxgyxfyxI t

Class 3 21

Direct Estimation

Ryx

tyx IIvIuvuS),(

2]ˆˆ[)ˆ,ˆ(

tI

yx II ,

image 1 minus image 2

gradient of image 2

Ryx

tyxvu IvIuI),(

2, ][min

yx

yx

II

II

A

.

.

.

.

.

.

v

ux

t

t

I

I

b

.

.

.

Class 3 22

Direct Estimation

Ryx

tyxvu IvIuI),(

2, ][min

2||min bAxx

bAAxA TT

ty

tx

yyx

yxx

II

II

v

u

III

III2

2

“aperture problem” 1)( AArank T

Class 3 23

Direct Estimation

Estimating parametric flow:

287321 xxyyxu

278654 yxyyxv

0(....))( 287321 tyx IIIxxyyx

Every pixel contributes one linear equation for the 8 unknowns

Class 3 24

Direct Estimation

Estimating 3-frame Motion:

pwstsZ

u xTT ][

1

pwrtrZ

v xTT ][

1

Combine with: 0 tyx IvIuI

0][)()(1

tT

yxT

yx IpwrIsItrIsIZ

Class 3 25

Direct Estimation

0][)()(1

tT

yxT

yx IpwrIsItrIsIZ

Let

xy

yxx

yyx

yIxI

xyIIxI

IIyxyI

hpq 2

2

yx

y

x

yx

yIxI

I

I

rIsIh

01

tTT Iqwth

Z

Class 3 26

Direct Estimation

01

tTT Iqwth

Z

0'''1

tTT Iqwth

Z

image 1 to image 2

image 1 to image 3

0]''['' qwttwhthIthI TTTTt

Tt

Each pixel contributes a linear equation to the 15 unknown parameters

Class 3 27

Direct Estimation: Factorization

Let ),( ijij vu

mm

m

n

Tnnn

T

Tnnn

T

nmnn

m

nmnn

m

tt

ww

rZrp

rZrp

sZsp

sZsp

vvv

vvvuuu

uuu

61

1

62

111

111

21

11211

21

11211

...

...

)/1(

..

..

)/1(

)/1(

..

..

)/1(

.

.

...

.

.

.

.

.

....

.

.

...

.

.

.

.

.

....

SMMS

S

V

UW

y

x

be the “flow” of point i at image j (image 0 is ref frame)

Class 3 28


02

2

t

mnnnyx IV

UII

),...,(1 nxxx IIdiagI

),...,(1 nyyy IIdiagI

mn

mtt

mtt

t

nnII

II

I

....

.

.

.

.

.

.

.

....

1

1

11

Class 3 29

ty

tx

yyx

yxx

II

II

v

u

III

III2

2


Recall:

ij

ij

ij

ij

ii

ii

h

g

v

u

cb

ba

Class 3 30

mnmnnnH

G

V

U

CB

BA

2222


),...,( 1 naadiagA

),...,( 1 nbbdiagB

),...,( 1 nccdiagC mnmnn

m

gg

gg

G

....

.

.

.

.

.

.

.

....

1

111

mnmnn

m

hh

hh

H

....

.

.

.

.

.

.

.

....

1

111

Class 3 31

mnmnnnH

G

V

U

CB

BA

2222


Rank=6 Rank=6

Enforcing rank=6 constraint on the measurement matrix

H

G

removes errors in a least-squares sense.

Class 3 32

mnmnnnH

G

V

U

CB

BA

2222


H

G

CB

BA

V

U#

Once U,V are recovered, one can solve for S,M as before.

Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics

Documents

image j image

gradient of image

factorizationdirect

frame motion

flow of point

framedirect estimation

structural constraints

parameter flownote