# 3D Photography: Structure from Motion - CVG @ ETHZ · PDF file 2014-03-17 · 3D Photography: Structure from Motion. Feb 17 Introduction Feb 24 Geometry, Camera Model, Calibration

Jul 11, 2020

## Documents

others

• 3D Photography: Structure from Motion

• Feb 17 Introduction

Feb 24 Geometry, Camera Model, Calibration

Mar 3 Features, Tracking / Matching

Mar 10 Project Proposals by Students

Mar 17 Structure from Motion (SfM) + 1 papers

Mar 24 Dense Correspondence (stereo / optical flow) + 1 papers

Mar 31 Bundle Adjustment & SLAM + 1 papers

Apr 7 Multi-View Stereo & Volumetric Modeling + 1 papers

Apr 21 Easter

Apr 28 3D Modeling with Depth Sensors + 1 papers

May 5 3D Scene Understanding

May 12 4D Video & Dynamic Scenes + 1 papers

May 19 Guest lecture: KinectFusion by Shahram Izadi

May 26 Final Demos

Schedule (tentative)

• Structure from Motion

• Two view reconstruction

• Epipolar geometry computation

• Triangulation

• Pose estimation

• (i) Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the

corresponding point x’ in the second image?

(ii) Camera geometry (motion): Given a set of corresponding

image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?

(iii) Scene geometry (structure): Given corresponding image

points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?

Three questions:

Two-view geometry

• The epipolar geometry

C,C’,x,x’ and X are coplanar

• The epipolar geometry

What if only C,C’,x are known?

• The epipolar geometry

All points on π project on l and l’

• The epipolar geometry

Family of planes π and lines l and l’ Intersection in e and e’

• The epipolar geometry

epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction

an epipolar plane = plane containing baseline (1-D family)

an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs)

• Example: converging cameras

• Example: motion parallel with image plane

(simple for stereo → rectification)

• The fundamental matrix F

algebraic representation of epipolar geometry

l'x a

we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F

• The fundamental matrix F

geometric derivation

xHx' π

=

x'e'l' ×= [ ] FxxHe' π == × mapping from 2-D to 1-D family (rank 2)

• The fundamental matrix F

algebraic derivation

( ) λCxPλX += + ( )IPP =+

[ ] +×= PP'e'F

xPP'CP'l' +×=

(note: doesn’t work for C=C’⇒ F=0)

xP+

( )λX

• The fundamental matrix F

correspondence condition

0Fxx'T =

The fundamental matrix satisfies the condition

that for any pair of corresponding points x↔x’ in the two images

( )0l'x'T =

• The fundamental matrix F - recap

F is the unique 3x3 rank 2 matrix that

satisfies x’TFx=0 for all x↔x’

(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)

(ii) Epipolar lines: l’=Fx & l=FTx’ (iii) Epipoles: on all epipolar lines, thus e’TFx=0, ∀x

⇒e’TF=0, similarly Fe=0 (iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2) (v) F is a correlation, projective mapping from a point x to

a line l’=Fx (not a proper correlation, i.e. not invertible)

• Computation of F

• Linear (8-point)

• Minimal (7-point)

• Calibrated (5-point) (Essential matrix)

• Practical two-view geometry computation

• Epipolar geometry: basic equation

0Fxx'T =

separate known from unknown

0'''''' 333231232221131211 =++++++++ fyfxffyyfyxfyfxyfxxfx

[ ][ ] 0,,,,,,,,1,,,',',',',',' T333231232221131211 =fffffffffyxyyyxyxyxxx (data) (unknowns)

(linear)

0Af =

0f 1''''''

1'''''' 111111111111 =

 

 

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxx

MMMMMMMMM

• 0

1´´´´´´

1´´´´´´

1´´´´´´

33

32

31

23

22

21

13

12

11

222222222222

111111111111

=

           

           

   

   

f

f

f

f

f

f

f

f

f

yxyyyyxxxyxx

yxyyyyxxxyxx

yxyyyyxxxyxx

nnnnnnnnnnnn

MMMMMMMMM

~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100

! Orders of magnitude difference

between column of data matrix

→ least-squares yields poor results

the NOT normalized 8-point algorithm

• Transform image to ~[-1,1]x[-1,1]

(0,0)

(700,500)

(700,0)

(0,500)

(1,-1)

(0,0)

(1,1)(-1,1)

(-1,-1)

     

     

1

1 500

2

10 700

2

normalized least squares yields good results (Hartley, PAMI´97)

the normalized 8-point algorithm

• the singularity constraint

0Fe'T = 0Fe = 0detF = 2Frank =

T

333

T

222

T

111

T

3

2

1

VσUVσUVσUV

σ

σ

σ

UF ++=  

 

 =

SVD from linearly computed F matrix (rank 3)

T

222

T

111

T

2

1

VσUVσUV

0

σ

σ

UF' +=  

 

 =

F F'-FminCompute closest rank-2 approximation

• the minimum case – 7 point correspondences

0f 1''''''

1''''''

777777777777

111111111111

=  

 

yxyyyxyxyxxx

yxyyyxyxyxxx

MMMMMMMMM

( ) T9x9717x7 V0,0,σ,...,σdiagUA =

9x298 0]VA[V =⇒ ( )T8T ] 000000010[Ve.g.V =

1...70,)xλFF(x 21 T

=∀=+ iii

one parameter family of solutions

but F1+λF2 not automatically rank 2

• F1 F2

F

σ3

F7pts

0λλλ)λFFdet( 01 2

2

3

321 =+++=+ aaaa

(obtain 1 or 3 solutions)

(cubic equation)

0)λIFFdet(Fdet)λFFdet( 1 -1

2221 =+=+

the minimum case – impose rank 2

Compute possible λ as eigenvalues of

(only real solutions are potential solutions) 1

-1

2 FF

( ) ( ) ( )( )B.detAdetABdet =

• • Linear equations for 5 points

• Linear solution space

• Non-linear constraints

Calibrated case: 5-point relative motion

E = xX + yY + zZ + wW

detE = 0 10 cubic polynomials

w = 1 scale does not matter, choose

(Nister, CVPR03)

′x1x1 ′x1y1 ′x11 ′y1x1 ′y1y1 ′y1 x1 y1 1

′x2x2 ′x2y2 ′x21 ′y2x2 ′y2y2 ′y2 x2 y2 1

′x3x3 ′x3y3 ′x31 ′y3x3 ′y3y3 ′y3 x3 y3 1

′x4x4 ′x4y4 ′x41 ′y4x4 ′y4y4 ′y4 x4 y4 1

′x5x5 ′x5y5 ′x51 ′y5x5 ′y5y5 ′y5 x5 y5 1

      

      

E11

E12

E13

E21

E22

E23

E31

E32

E33

            

            

= 0

(assumes normalized coordinates)

• Calibrated case: 5-point relative motion

• Perform Gauss-Jordan elimination on polynomials

-z

-z

-z

[n] represents polynomial of degree n in z

(Nister, CVPR03)

• Step 1. Extract features

Step 2. Compute a set of potential matches

Step 3. do Step 3.1 select minimal sample (i.e. 7 or 5 matches)

Step 3.2 compute solution(s) for F

Step 3.3 determine inliers

until Γ(#inliers,#samples)

• restrict search range to neighborhood of epipolar line (e.g. ±1.5 pixels)

relax disparity restriction (along epipolar line)

Finding more matches

• (i) Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the

corresponding point x’ in the second image?

(ii) Camera geometry (motion): Given a set of corresponding

image points {xi ↔x’i}, i=1,…,n, what are the cameras P

Related Documents See more >
##### Motion Capture in 3D Animation
Category: Documents
##### Casual 3D Photography - Casual 3D Photography PETER HEDMAN,....
Category: Documents
##### Bayesian Reconstruction of 3D Human Motion from ... Bayesian...
Category: Documents
##### Motion Photography Lecture
Category: Art & Photos
##### Physics 101 Lecture 4 Motion in 2D&3D Physics 101 Lecture 4....
Category: Documents
##### 3D Photography Project An Overview of 3D Photography and The...
Category: Documents