Multiple view computation
•Tensors (2,3,4-views)Tensors (2,3,4-views)
•FactorizationFactorization– OrthographicOrthographic– PerpectivePerpective
•SequentialSequential
•Bundle adjustmentBundle adjustment
Factorization
• Factorise observations in structure of the scene and Factorise observations in structure of the scene and motion/calibration of the cameramotion/calibration of the camera
• Use Use all pointsall points in in all imagesall images at the same time at the same time
Affine factorisationAffine factorisation Projective factorisationProjective factorisation
Affine camera
The affine projection equations are The affine projection equations are
1
j
j
j
yi
xi
ij
ij
Z
Y
X
P
Py
x
10001
1
j
j
j
yi
xi
ij
ij
Z
Y
X
P
P
y
x
~
~
4
4
j
j
j
yi
xi
ij
ij
yiij
xiij
Z
Y
X
P
Py
x
Py
Px
Orthographic factorization
The ortographic projection equations are The ortographic projection equations are
where where njmijiij ,...,1,,...,1,Mm P
All equations can be collected for all All equations can be collected for all ii and and jj
wherewhere
n
mmnmm
n
n
M,...,M,M,,
mmm
mmm
mmm
212
1
21
22221
11211
M
P
P
P
Pm
MPm
M ~
~m
j
j
j
jyi
xi
iij
ijij
Z
Y
X
,P
P,
y
xP
Note that P and M are resp. 2mx3 and 3xn matrices and
therefore the rank of m is at most 3
(Tomasi Kanade’92)
Orthographic factorization
Factorize Factorize mm through singular value decomposition through singular value decomposition
An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows
TVUm
TVMUP ~,
~
(Tomasi Kanade’92)
n
mmnmm
n
n
M,...,M,M
mmm
mmmmmm
min 212
1
21
22221
11211
P
P
P
Closest rank-3 approximation yields MLE!
0~~
1~~
1~~
1
1
1
TT
TT
TT
yi
xi
yi
yi
xi
xi
PP
PP
PP
AA
AA
AA
0~~
1~~
1~~
T
T
T
yi
xi
yi
yi
xi
xi
PP
PP
PP
C
C
C
A metric reconstruction is obtained as followsA metric reconstruction is obtained as follows
Where A is computed from Where A is computed from
Orthographic factorization
Factorize Factorize mm through singular value decomposition through singular value decomposition
An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows
TVUm
TVMUP ~,
~
MAMAPP~
,~ 1
0
1
1
T
T
T
yi
xi
yi
yi
xi
xi
PP
PP
PP 3 linear equations per view on
symmetric matrix C (6DOF)
A can be obtained from C through Cholesky factorisation
and inversion
(Tomasi Kanade’92)
Perspective factorization
The camera equations The camera equations
for a fixed image for a fixed image ii can be written in matrix form can be written in matrix form asas
where where
mjmijiijij ,...,1,,...,1,Mmλ P
MPm iii
imiii
mimiii mmm
λ,...,λ,λdiag
M,...,M,M,,...,,
21
2121
Mm
Perspective factorization
All equations can be collected for all All equations can be collected for all ii as as
wherewherePMm
mnn P
P
P
P
m
m
m
m...
,...
2
1
22
11
In these formulas In these formulas mm are known, but are known, but ii,,PP and and MM are unknown are unknown
Observe that Observe that PMPM is a product of a 3 is a product of a 3mmx4 matrix and a 4xx4 matrix and a 4xnn matrix, i.e. it is a rank 4 matrixmatrix, i.e. it is a rank 4 matrix
Perspective factorization algorithm
Assume that i are known, then PM is known.
Use the singular value decomposition PM=U VT
In the noise-free case
S=diag(1,2,3,4,0, … ,0)and a reconstruction can be obtained by setting:
P=the first four columns of U.M=the first four rows of V.
Iterative perspective factorization
When i are unknown the following algorithm can be used:
1. Set ij=1 (affine approximation).
2. Factorize PM and obtain an estimate of P and M. If 5 is sufficiently small then STOP.
3. Use m, P and M to estimate i from the camera equations (linearly) mi i=PiM
4. Goto 2.
In general the algorithm minimizes the proximity measure P(,P,M)=5Note that structure and motion recovered
up to an arbitrary projective transformation
Further Factorization work
Factorization with uncertaintyFactorization with uncertainty
Factorization for dynamic scenesFactorization for dynamic scenes
(Irani & Anandan, IJCV’02)
(Costeira and Kanade ‘94)
(Bregler et al. 2000, Brand 2001)
Sequential structure from motion
• Initialize structure and motion from two viewsInitialize structure and motion from two views• For each additional viewFor each additional view
– Determine poseDetermine pose– Refine and extend structureRefine and extend structure
• Determine correspondences robustly by jointly Determine correspondences robustly by jointly estimating matches and epipolar geometry estimating matches and epipolar geometry
Initial structure and motion
eeaFeP
0IPT
x
2
1
Epipolar geometry Projective calibration
012 FmmT
compatible with F
Yields correct projective camera setup(Faugeras´92,Hartley´92)
Obtain structure through triangulation
Use reprojection error for minimizationAvoid measurements in projective space
Compute Pi+1 using robust approachFind additional matches using predicted projectionExtend, correct and refine reconstruction
2D-2D
2D-3D 2D-3D
mimi+1
M
new view
Determine pose towards existing structure
Non-sequential image collections
4.8im/pt64 images
3792
po
ints
Problem:Features are lost and reinitialized as new features
Solution:Match with other close views
For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure
Relating to more views
Problem: find close views in projective frame
For every view For every view iiExtract featuresExtract featuresCompute two view geometry Compute two view geometry ii-1/-1/ii and matches and matches Compute pose using robust algorithmCompute pose using robust algorithmFor all For all closeclose views views kk
Compute two view geometry Compute two view geometry kk//ii and matches and matchesInfer new 2D-3D matches and add to listInfer new 2D-3D matches and add to list
Refine pose using all 2D-3D matchesRefine pose using all 2D-3D matchesRefine existing structureRefine existing structureInitialize new structureInitialize new structure
Determining close views
• If viewpoints are If viewpoints are closeclose then most image changes then most image changes can be modelled through a can be modelled through a planar homographyplanar homography
• Qualitative distance measureQualitative distance measure is obtained by is obtained by looking at the looking at the residual errorresidual error on the on the best possible best possible planar homographyplanar homography
Distance = m´,mmedian min HD
9.8im/pt
4.8im/pt
64 images
64 images
3792
po
ints
2170
po
ints
Non-sequential image collections (2)
Refining structure and motion
• Minimize reprojection errorMinimize reprojection error
– Maximum Likelyhood Estimation Maximum Likelyhood Estimation (if (if error zero-mean Gaussian noise)error zero-mean Gaussian noise)
– Huge problem but can be solved efficientlyHuge problem but can be solved efficiently(Bundle adjustment)(Bundle adjustment)
m
k
n
iikD
ik 1 1
2
kiM̂,P̂
M̂P̂,mmin