Computer Vision Lecture 17 · • Partial reconstruction property of SVD Let i i=1,…,N be the singular values of A. Let A p = U p D p V p T be the reconstruction of A when we set

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Computer Vision – Lecture 17

Uncalibrated Reconstruction & SfM

08.07.2019

Bastian Leibe

Visual Computing Institute

RWTH Aachen University

http://www.vision.rwth-aachen.de/

[email protected]

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Announcements

• No lecture tomorrow (Tuesday)

Due to a schedule conflict

• Last exercise will be offered on Monday, 15.07.

Optional, but recommended

Time slot & room to be announced…

• Repetition slides

I will provide a slide set (pdf) with summary slides for the

entire lecture

Idea: you can use this as an index to the lecture

2B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Course Outline

• Image Processing Basics

• Segmentation & Grouping

• Object Recognition

• Local Features & Matching

• Deep Learning

• 3D Reconstruction

Epipolar Geometry and Stereo Basics

Camera calibration & Triangulation

Uncalibrated Reconstruction & Active Stereo

Structure-from-Motion

3

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Recap: A General Point

• Equations of the form

• How do we solve them? (always!)

Apply SVD

Singular values of A = square roots of the eigenvalues of ATA.

The solution of Ax=0 is the nullspace vector of A.

This corresponds to the smallest singular vector of A.4

B. Leibe

Ax 0

11 11 1

1

T

N

T

NN N NN

d v v

d v v

A UDV U

SVD

Singular values Singular vectors

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Recap: Camera Parameters

• Intrinsic parameters

Principal point coordinates

Focal length

Pixel magnification factors

Skew (non-rectangular pixels)

Radial distortion

• Extrinsic parameters

Rotation 𝐑

Translation 𝐭(both relative to world coordinate system)

• Camera projection matrix

General pinhole camera: 9 DoF

CCD Camera with square pixels: 10 DoF

General camera: 11 DoF 5B. Leibe

0

0

1 1 1

x x x

y y y

m f p x

K m f p y

s s

P K R | t

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Recap: Calibrating a Camera

• Goal

Compute intrinsic and extrinsic parameters

using observed camera data.

• Main idea

Place “calibration object” with known

geometry in the scene

Get correspondences

Solve for mapping from scene to image:

estimate P=PintPext

6B. LeibeSlide credit: Kristen Grauman

? P

Xi

xi

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Recap: Camera Calibration (DLT Algorithm)

• P has 11 degrees of freedom.

• Two linearly independent equations per independent 2D/3D

correspondence.

• Solve with SVD (similar to homography estimation)

Solution corresponds to smallest singular vector.

• 5 ½ correspondences needed for a minimal solution.7

B. Leibe

• Solve with …?

Slide adapted from Svetlana Lazebnik

0pA 0

P

P

P

X0X

XX0

X0X

XX0

3

2

1111

111

T

nn

TT

n

T

nn

T

n

T

TTT

TTT

x

y

x

y

? P

Xi

xi

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

• Stack equations and solve with …?

Recap: Triangulation – Linear Algebraic Approach

• Two independent equations each in terms of

three unknown entries of X.

• Stack equations and solve with SVD.

• This approach nicely generalizes to multiple cameras.8

B. Leibe

XPx

XPx

222

111

0XPx

0XPx

22

11

0XP][x

0XP][x

22

11

Slide credit: Svetlana Lazebnik

O1O2

x1x2

X?

R1R2

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Topics of This Lecture

• Revisiting Epipolar Geometry Calibrated case: Essential matrix

Uncalibrated case: Fundamental matrix

Weak calibration

Epipolar Transfer

• Active Stereo Kinect sensor

Structured Light sensing

Laser scanning

• Structure from Motion (SfM) Motivation

Ambiguity

Projective factorization

Bundle adjustment

9B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Recap: Epipolar Geometry – Calibrated Case

10B. Leibe

X

x x’


Essential Matrix

(Longuet-Higgins, 1981)

0)]([ xRtx RtExExT ][with0

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Epipolar Geometry: Calibrated Case

• E x’ is the epipolar line associated with x’ (l = E x’)

• ETx is the epipolar line associated with x (l’ = ETx)

• E e’ = 0 and ETe = 0

• E is singular (rank two)

• E has five degrees of freedom (up to scale)12

B. Leibe

X

x x’


0)]([ xRtx RtExExT ][with0

Why?

Why?

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Epipolar Geometry: Uncalibrated Case

• The calibration matrices K and K’ of the two cameras are unknown

• We can write the epipolar constraint in terms of unknownnormalized coordinates:

15B. Leibe

X

x x’


0ˆˆ xExT xKxxKx ˆ,ˆ

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


16B. Leibe

X

x x’


Fundamental Matrix(Faugeras and Luong, 1992)

0ˆˆ xExT

xKx

xKx

ˆ

ˆ

1with0 KEKFxFx TT

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


• F x’ is the epipolar line associated with x’ (l = F x’)

• FTx is the epipolar line associated with x (l’ = FTx)• F e’ = 0 and FTe = 0

• F is singular (rank two)

• F has seven degrees of freedom17

B. Leibe

X

x x’


0ˆˆ xExT 1with0 KEKFxFx TT

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Estimating the Fundamental Matrix

• The Fundamental matrix defines the epipolar geometry

between two uncalibrated cameras.

• How can we estimate F from an image pair?

We need correspondences…

18B. Leibe

xi

Xi

x0i

P? P0?

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

The Eight-Point Algorithm

• Taking 8 correspondences:

19B. Leibe

x = (u, v, 1)T, x’ = (u’, v’, 1)T

This minimizes:

2

1

)( i

N

i

T

i xFx


[u0u; u0v; u0; uv0; vv0; v0; u; v; 1]

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

= 0

Af = 0

Solve using… ?Solve using… SVD!

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Excursion: Properties of SVD

• Frobenius norm

Generalization of the Euclidean norm to matrices

• Partial reconstruction property of SVD

Let i i=1,…,N be the singular values of A.

Let Ap = UpDpVpT be the reconstruction of A when we set

p+1,…, N to zero.

Then Ap = UpDpVpT is the best rank-p approximation of A in the

sense of the Frobenius norm

(i.e. the best least-squares approximation).20

B. Leibe

2

1 1

m n

ijFi j

A a

min( , )

2

1

m n

i

i

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


• Problem with noisy data

The solution will usually not fulfill the constraint that F only has

rank 2.

There will be no epipoles through which all epipolar lines pass!

• Enforce the rank-2 constraint using SVD

• As we have just seen, this provides the best least-squares

approximation to the rank-2 solution.

21B. Leibe

11 11 13

22

33 31 33

T

T

d v v

F d

d v v

UDV U

SVD

Set d33 to

zero and

reconstruct F

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Problem with the Eight-Point Algorithm

• In practice, this often looks as follows:

22B. Leibe

266666666664

u01u1 u01v1 u01 u1v01 v1v

01 v01 u1 v1 1

u02u2 u02v2 u02 u2v02 v2v

02 v02 u2 v2 1

u03u3 u03v3 u03 u3v03 v3v

03 v03 u3 v3 1

u04u4 u04v4 u04 u4v04 v4v

04 v04 u4 v4 1

u05u5 u05v5 u05 u5v05 v5v

05 v05 u5 v5 1

u06u6 u06v6 u06 u6v06 v6v

06 v06 u6 v6 1

u07u7 u07v7 u07 u7v07 v7v

07 v07 u7 v7 1

u08u8 u08v8 u08 u8v08 v8v

08 v08 u8 v8 1

377777777775

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

=

266666666664

0

0

0

0

0

0

0

0

377777777775


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Problem with the Eight-Point Algorithm

• In practice, this often looks as follows:

Poor numerical conditioning

Can be fixed by rescaling the data

23B. Leibe

266666666664

u01u1 u01v1 u01 u1v01 v1v

01 v01 u1 v1 1

u02u2 u02v2 u02 u2v02 v2v

02 v02 u2 v2 1

u03u3 u03v3 u03 u3v03 v3v

03 v03 u3 v3 1

u04u4 u04v4 u04 u4v04 v4v

04 v04 u4 v4 1

u05u5 u05v5 u05 u5v05 v5v

05 v05 u5 v5 1

u06u6 u06v6 u06 u6v06 v6v

06 v06 u6 v6 1

u07u7 u07v7 u07 u7v07 v7v

07 v07 u7 v7 1

u08u8 u08v8 u08 u8v08 v8v

08 v08 u8 v8 1

377777777775

26666666666664

F11

F12

F13

F21

F22

F23

F31

F32

F33

37777777777775

=

266666666664

0

0

0

0

0

0

0

0

377777777775


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

The Normalized Eight-Point Algorithm

1. Center the image data at the origin, and scale it so the

mean squared distance between the origin and the data

points is 2 pixels.

2. Use the eight-point algorithm to compute F from the

normalized points.

3. Enforce the rank-2 constraint using SVD.

4. Transform fundamental matrix back to original units: if T

and T’ are the normalizing transformations in the two

images, then the fundamental matrix in original coordinates

is TT F T’.24

B. Leibe [Hartley, 1995]Slide credit: Svetlana Lazebnik

11 11 13

22

33 31 33

T

T

d v v

F d

d v v

UDV U

SVD

Set d33 to

zero and

reconstruct F

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


• Meaning of error

Sum of Euclidean distances between points xi and epipolar

lines Fx’i (or points x’i and epipolar lines FTxi), multiplied by

a scale factor

• Nonlinear approach for refining the solution: minimize

Similar to nonlinear minimization

approach for triangulation.

Iterative approach (Gauss-Newton,

Levenberg-Marquardt,…)25

B. Leibe

:)( 2

1

i

N

i

T

i xFx

N

i

i

T

iii xFxxFx1

22 ),(d),(d

O1O2

x1x2

X?

x’1

x’2


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Comparison of Estimation Algorithms

26B. Leibe

8-point Normalized 8-point Nonlinear least squares

Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel

Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

3D Reconstruction with Weak Calibration

• Want to estimate world geometry without requiring calibrated cameras.

• Many applications: Archival videos

Photos from multiple unrelated users

Dynamic camera system

• Main idea:

Estimate epipolar geometry from a (redundant) set of point correspondences between two uncalibrated cameras.


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Stereo Pipeline with Weak Calibration

• So, where to start with uncalibrated cameras?

Need to find fundamental matrix F and the correspondences

(pairs of points (u’,v’) ↔ (u,v)).

• Procedure

1. Find interest points in both images

2. Compute correspondences

3. Compute epipolar geometry

4. Refine28

B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


1. Find interest points (e.g. Harris corners)

29B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


2. Match points using only proximity

30B. Leibe Example from Andrew ZissermanSlide credit: Kristen Grauman

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Putative Matches based on Correlation Search

31B. Leibe Example from Andrew Zisserman

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

RANSAC for Robust Estimation of F

• Select random sample of correspondences

• Compute F using them

This determines epipolar constraint

• Evaluate amount of support – number of inliers

within threshold distance of epipolar line

• Iterate until a solution with sufficient support

has been found (or for max #iterations)

• Choose F with most support (#inliers)


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Putative Matches based on Correlation Search


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Pruned Matches

• Correspondences consistent with epipolar geometry


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Resulting Epipolar Geometry


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Epipolar Transfer

• Assume the epipolar geometry is known

• Given projections of the same point in two images, how can

we compute the projection of that point in a third image?

36B. Leibe

x1 x2

? x3


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Extension: Epipolar Transfer

• Assume the epipolar geometry is known

• Given projections of the same point in two images, how can

we compute the projection of that point in a third image?

37B. Leibe

x1 x2 x3l32l31

l31 = FT13 x1

l32 = FT23 x2

When does epipolar transfer fail?


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19




Weak calibration

Epipolar Transfer



Laser scanning


Ambiguity


Bundle adjustment

38B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Microsoft Kinect – How Does It Work?

• Built-in IR

projector

• IR camera for

depth

• Regular camera

for color

39B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Recall: Optical Triangulation

40B. Leibe

O1

x1

X?

Image plane

Camera center

3D Scene point

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Recall: Optical Triangulation

• Principle: 3D point given by intersection of two rays.

Crucial information: point correspondence

Most expensive and error-prone step in the pipeline…

41B. Leibe

O1 O2

x1 x2

X

Image plane

Camera center

3D Scene point

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Active Stereo with Structured Light

• Idea: Replace one camera by a projector.

Project “structured” light patterns onto the object

Simplifies the correspondence problem

42B. Leibe

O1 O2

x1 x2

X

Image plane

Camera center

3D Scene point

Projector

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

What the Kinect Sees…

43B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

3D Reconstruction with the Kinect

44B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Laser Scanning

• Optical triangulation

Project a single stripe of laser light

Scan it across the surface of the object

This is a very precise version of structured light scanning

46B. Leibe

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Slide credit: Steve Seitz

http://graphics.stanford.edu/projects/mich/

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Laser Scanned Models

47B. Leibe

The Digital Michelangelo Project, Levoy et al.


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


48B. Leibe



Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


49B. Leibe



Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


50B. Leibe



Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19


51B. Leibe



Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Poor Man’s Scanner

52Bouget and Perona, ICCV’98

../../../Local Settings/ANGEL.WRL

../../../Local Settings/ANGEL.WRL

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Slightly More Elaborate (But Still Cheap)

53B. Leibe

http://www.david-laserscanner.com/

Software freely available from Robotics Institute TU Braunschweig

http://www.david-laserscanner.com/

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19




Weak calibration

Epipolar Transfer



Laser scanning


Ambiguity


Bundle adjustment

54B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Structure from Motion

• Given: m images of n fixed 3D points

xij = Pi Xj , i = 1, … , m, j = 1, … , n

• Problem: estimate m projection matrices Pi and

n 3D points Xj from the mn correspondences xij55

B. Leibe

x1j

x2j

x3j

Xj

P1

P2

P3


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Applications

• E.g., movie special effects

Video

56B. Leibe Video Credit: Stefan Hafeneger

../cv-ws08/videos/MotivationFilm.mov

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Structure from Motion Ambiguity

• If we scale the entire scene by some factor k and, at the

same time, scale the camera matrices by the factor of 1/k,

the projections of the scene points in the image remain

exactly the same:

It is impossible to recover the absolute scale of the scene!

57B. LeibeSlide credit: Svetlana Lazebnik

x = PX =

µ1

kP

¶(kX)

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Structure from Motion Ambiguity

• If we scale the entire scene by some factor k and, at the

same time, scale the camera matrices by the factor of 1/k,

the projections of the scene points in the image remain

exactly the same.

• More generally: if we transform the scene using a

transformation Q and apply the inverse transformation to the

camera matrices, then the images do not change


x =PX= (PQ¡1)QX

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Reconstruction Ambiguity: Similarity

59B. LeibeSlide credit: Svetlana Lazebnik Images from Hartley & Zisserman

x =PX= (PQ¡1S )QSX

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Reconstruction Ambiguity: Affine


Affine

x =PX= (PQ¡1A )QAX

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Reconstruction Ambiguity: Projective


x=PX= (PQ¡1P )QPX

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Projective Ambiguity


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

From Projective to Affine


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

From Affine to Similarity


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Hierarchy of 3D Transformations

• With no constraints on the camera calibration matrix or on the scene, we get a projective reconstruction.

• Need additional information to upgrade the reconstruction to affine, similarity, or Euclidean. 65

B. Leibe

vTv

tAProjective

15dof

Affine

12dof

Similarity

7dof

Euclidean

6dof

Preserves intersection

and tangency

Preserves parallellism,

volume ratios

Preserves angles, ratios

of length

10

tAT

10

tRT

s

10

tRT

Preserves angles,

lengths


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Structure from Motion


xij = Pi Xj , i = 1, … , m, j = 1, … , n

• Problem: estimate m projection matrices Pi and

n 3D points Xj from the mn correspondences xij66

B. Leibe

x1j

x2j

x3j

Xj

P1

P2

P3


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Projective Structure from Motion


• zij xij = Pi Xj , i = 1,… , m, j = 1, … , n

• Problem: estimate m projection matrices Pi and n 3D points

Xj from the mn correspondences xij

• With no calibration info, cameras and points can only be recovered up to a 4£4 projective transformation Q:

X → QX, P → PQ-1

• We can solve for structure and motion when

2mn >= 11m +3n – 15

• For two cameras, at least 7 points are needed.


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Projective SfM: Two-Camera Case

• Assume fundamental matrix 𝐅 between the two views

First camera matrix: [I|0]Q-1

Second camera matrix: [A|b]Q-1

• Let , then

• And

• So we have

68B. Leibe

[ ]z z x A I | 0 X b Ax b

QXX ~

z z x b Ax b

( ) ( )z z x b x Ax b x

0][T Axbx

AbF ][

[ ] , [ | ]z z x I | 0 X x Ab X

b: epipole (FTb = 0), A = –[b×]F


0 ( )z Ax b x

F&P sec. 13.3.1

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Projective SfM: Two-Camera Case

• Decomposing the Fundamental Matrix

This means that if we can compute the fundamental matrix between

two cameras, we can directly estimate the two projection matrices

from F.

Once we have the projection matrices, we can compute the 3D

position of any point X by triangulation.

• How can we obtain both kinds of information at the same

time?

69B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Projective Factorization

• If we knew the depths z, we could factorize D to estimate

M and S.

• If we knew M and S, we could solve for z.

• Solution: iterative approach

(alternate between above two steps).70

B. Leibe

n

mmnmnmmmm

nn

nn

zzz

zzz

zzz

XXX

P

P

P

xxx

xxx

xxx

D

21

2

1

2211

2222222121

1112121111

Cameras

(3m × 4)

Points (4 × n)

D = MS has rank 4


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Sequential Structure from Motion

• Initialize motion from two images

using fundamental matrix

• Initialize structure

• For each additional view:

Determine projection matrix

of new camera using all the

known 3D points that are

visible in its image –

calibration

71B. Leibe

Ca

me

ras

Points


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19










calibration

Refine and extend structure:

compute new 3D points,

re-optimize existing points

that are also seen by this camera –

triangulation

72B. Leibe

Ca

me

ras

Points


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19










calibration

Refine and extend structure:

compute new 3D points,

re-optimize existing points

that are also seen by this camera –

triangulation

• Refine structure and motion: bundle adjustment73

B. Leibe

Ca

me

ras

Points


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Bundle Adjustment

• Non-linear method for refining structure and motion

• Minimizing mean-square reprojection error

74B. Leibe

2

1 1

,),(

m

i

n

j

jiijDE XPxXP

x1j

x2j

x3

j

Xj

P1

P2

P3

P1Xj

P2Xj

P3Xj


Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

Bundle Adjustment

• Idea

Seek the Maximum Likelihood (ML) solution assuming the

measurement noise is Gaussian.

It involves adjusting the bundle of rays between each camera center

and the set of 3D points.

Bundle adjustment should generally be used as the final step

of any multi-view reconstruction algorithm.

– Considerably improves the results.

– Allows assignment of individual covariances to each measurement.

• However…

It needs a good initialization.

It can become an extremely large minimization problem.

• Very efficient algorithms available.

75B. Leibe

Perc

ep

tual

an

d S

en

so

ry A

ug

me

nte

d C

om

pu

tin

gC

om

pute

r V

isio

n S

um

mer‘

19

References and Further Reading

• Background information on camera models and calibration

algorithms can be found in Chapters 6 and 7 of

• Also recommended: Chapter 9 of the same book on

Epipolar geometry and the Fundamental Matrix and

Chapter 11.1-11.6 on automatic computation of F.

76B. Leibe

R. Hartley, A. Zisserman

Multiple View Geometry in Computer Vision

2nd Ed., Cambridge Univ. Press, 2004

Computer Vision Lecture 17 · • Partial reconstruction property of SVD Let i i=1,…,N be the singular values of A. Let A p = U p D p V p T be the reconstruction of A when we set

Documents