Top Banner
EECS 442 Computer vision Multiple view geometry Affine structure from Motion Reading: [HZ] Chapters: 6,14,18 [FP] Chapter: 12 Some slides of this lectures are courtesy of prof. J. Ponce, prof FF Li, prof S. Lazebnik & prof. M. Hebert - Affine structure from motion problem - Algebraic methods - Factorization methods
42

EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Sep 22, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

EECS 442 – Computer vision

Multiple view geometry

Affine structure from Motion

Reading: [HZ] Chapters: 6,14,18

[FP] Chapter: 12

Some slides of this lectures are courtesy of prof. J. Ponce,

prof FF Li, prof S. Lazebnik & prof. M. Hebert

- Affine structure from motion problem

- Algebraic methods

- Factorization methods

Page 2: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Applications Courtesy of Oxford Visual Geometry Group

Page 3: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Structure from motion problem

x1j

x2j

xmj

Xj

M1

M2

Mm

Given m images of n fixed 3D points

•xij = Mi Xj , i = 1, … , m, j = 1, … , n

Page 4: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

From the mxn correspondences xij, estimate:

•m projection matrices Mi

•n 3D points Xj

x1j

x2j

xmj

Xj

motion

structure

M1

M2

Mm

Structure from motion problem

Page 5: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Affine structure from motion (simpler problem)

Image World

Image

From the mxn correspondences xij, estimate:

•m projection matrices Mi (affine cameras)

•n 3D points Xj

Page 6: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Finite cameras

p

q

r

R

Q

P

O

XTRKx

M

Page 7: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

??10

0100

0010

0001

TR

Question:

Page 8: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Finite cameras

p

q

r

R

Q

P

O

XTRKx

10

TR

0100

0010

0001

KM 3x3

Canonical perspective projection matrix

Affine homography

(in 3D)

Affine

Homography

(in 2D)

100

y0

xs

K oy

ox

Page 9: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Projective & Affine cameras

XTRKx

10

TR

0100

0010

0001

KM

100

y0

xs

K oy

ox

Projective case

Affine case

Page 10: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Scaling function of the distance (magnification)

myy

mxx

'

'

Weak perspective projection When the relative scene depth is small compared to its distance from the camera

Page 11: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Orthographic (affine) projection

y'y

x'x

When the camera is at a (roughly constant) distance from the scene

–Distance from center of projection

to image plane is infinite

Page 12: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Transformation in 2D

Affinities:

1

y

x

H

1

y

x

10

tA

1

'y

'x

a

Page 13: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Projective & Affine cameras

XTRKx

100

yα0

xsα

oy

ox

K

10

TR

1000

0010

0001

KM

10

TR

0100

0010

0001

KM

100

y0

xs

K oy

ox

Projective case

Affine case

Parallel projection matrix

(points at infinity are mapped as points at infinity) Magnification (scaling term)

Page 14: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Affine cameras

XTRKx

100

00

00

y

x

K

10

TR

1000

0010

0001

KM

10

bA

1000

]affine44[

1000

0010

0001

]affine33[ 2232221

1131211

baaa

baaa

M

12

1

232221

131211 XbAXx EucM

b

b

Z

Y

X

aaa

aaa

y

x

[Homogeneous]

[non-homogeneous

image coordinates] bAMMEuc

;1

PEucM

Page 15: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

P p

p’

;1

PbAPp M

v

u bAM

Affine cameras

M = camera matrix

To recap: from now on we define M as the camera matrix for the affine case

Page 16: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

The Affine Structure-from-Motion Problem

Given m images of n fixed points Pj (=Xi) we can write

Problem: estimate the m 24 matrices Mi and

the n positions Pj from the mn correspondences pij .

2m n equations in 8m+3n unknowns

Two approaches: - Algebraic approach (affine epipolar geometry; estimate F; cameras; points)

- Factorization method

How many equations and how many unknown?

N of cameras N of points

Page 17: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Algebraic analysis (2-view case)

- Derive the fundamental matrix FA for the

affine case

- Compute FA

- Use FA to estimate projection matrices

- Use projection matrices to estimate 3D

points

Page 18: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

1. Deriving the fundamental matrix FA

P p

p’

Homogeneous system

u

v

Dim= ? 4x4

Page 19: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

The Affine Fundamental Matrix!

where

Deriving the fundamental matrix FA

Are the epipolar lines parallel or converging?

Page 20: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped
Page 21: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Affine Epipolar Geometry

Page 22: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Estimating FA

• From n correspondences, we obtain a linear system on

the unknown alpha, beta, etc…

• Measurements: u, u’, v, v’

0

1vuvu

1vuvu

nnnn

1111

f

• Computed by least square and by enforcing |f|=1

• SVD

Page 23: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

P p

p’

Estimating projection matrices from FA

Page 24: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Affine ambiguity

PQQMPMp A

-1

A

Affine

Page 25: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

P p

p’

Estimating projection matrices from FA

Page 26: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Estimating projection matrices from FA

Choose Q such

that…

'~

M

Where a,b,c,d can be expressed as function of the parameters of FA

See HZ page 348

Page 27: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

A factorization method – Tomasi & Kanade algorithm

C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization

method. IJCV, 9(2):137-154, November 1992.

• Centering the data

• Factorization

Page 28: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Centering: subtract the centroid of the image points

ji

n

1k

kji

n

1k

ikiiji

n

1k

ikijij

ˆn

1

n

1

n

XAXXA

bXAbXAxxx

A factorization method - Centering the data

Xk

xik xi ^

Page 29: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Centering: subtract the centroid of the image points

ji

n

1k

kji

n

1k

ikiiji

n

1k

ikijij

ˆn

1

n

1

n

XAXXA

bXAbXAxxx

A factorization method - Centering the data

Page 30: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Centering: subtract the centroid of the image points

n

k

kji

n

k

ikiiji

n

k

ikijij

n

nn

1

11

1

11ˆ

XXA

bXAbXAxxx

jiij XAx ˆ

A factorization method - Centering the data

Assume that the origin of the world coordinate system is at the centroid of

the 3D points

After centering, each normalized point xij is related to the 3D point Xi by

Page 31: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

X x

x’

A factorization method - Centering the data

jiij XAx ˆ

Page 32: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Let’s create a 2m n data (measurement) matrix:

mnmm

n

n

xxx

xxx

xxx

D

ˆˆˆ

ˆˆˆ

ˆˆˆ

21

22221

11211

cameras

(2 m )

points (n )

A factorization method - factorization

Page 33: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Let’s create a 2m n data (measurement) matrix:

n

mmnmm

n

n

XXX

A

A

A

xxx

xxx

xxx

D

21

2

1

21

22221

11211

ˆˆˆ

ˆˆˆ

ˆˆˆ

cameras

(2 m × 3)

points (3 × n )

The measurement matrix D = M S has rank 3 (it’s a product of a 2mx3 matrix and 3xn matrix)

A factorization method - factorization

(2 m × n)

M

S

Page 34: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Factorizing the measurement matrix

Source: M. Hebert

Page 35: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Factorizing the measurement matrix

Singular value decomposition of D:

Source: M. Hebert

Page 36: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Factorizing the measurement matrix

Singular value decomposition of D:

Source: M. Hebert

Since rank (D)=3, there are only 3 non-zero singular values

Page 37: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Factorizing the measurement matrix

Obtaining a factorization from SVD:

M = Motion (cameras)

S = structure

What is the issue here?

D has rank>3 because of - measurement noise

- affine approximation

Page 38: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Factorizing the measurement matrix

Obtaining a factorization from SVD:

S = structure

D

D

M = motion

Page 39: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Affine ambiguity

The decomposition is not unique. We get the same D by

using any 3×3 matrix C and applying the

transformations M → MC, S →C-1S

We can enforce some Euclidean constraints to resolve

this ambiguity (more on next lecture!)

Page 40: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Algorithm summary

1. Given: m images and n features xij

2. For each image i, center the feature coordinates

3. Construct a 2m × n measurement matrix D:

• Column j contains the projection of point j in all views

• Row i contains one coordinate of the projections of all the n

points in image i

4. Factorize D:

• Compute SVD: D = U W VT

• Create U3 by taking the first 3 columns of U

• Create V3 by taking the first 3 columns of V

• Create W3 by taking the upper left 3 × 3 block of W

5. Create the motion and shape matrices:

• M = M = U3 and S = W3V3T (or U3W3

½ and S = W3½ V3

T )

6. Eliminate affine ambiguity

Page 42: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped

Next lecture

Multiple view geometry

Perspective structure from Motion