Parameter estimation class 6 Multiple View Geometry Comp 290-089 Marc Pollefeys
Parameter estimationclass 6
Multiple View GeometryComp 290-089Marc Pollefeys
Content
• Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.
• Single View: Camera model, Calibration, Single View Geometry.
• Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.
• Three Views: Trifocal Tensor, Computing T.• More Views: N-Linearities, Multiple view
reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality
Multiple View Geometry course schedule(subject to change)
Jan. 7, 9 Intro & motivation Projective 2D Geometry
Jan. 14, 16
(no class) Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry (no class)
Jan. 28, 30
Parameter Estimation Parameter Estimation
Feb. 4, 6 Algorithm Evaluation Camera Models
Feb. 11, 13
Camera Calibration Single View Geometry
Feb. 18, 20
Epipolar Geometry 3D reconstruction
Feb. 25, 27
Fund. Matrix Comp. Structure Comp.
Mar. 4, 6 Planes & Homographies Trifocal Tensor
Mar. 18, 20
Three View Reconstruction
Multiple View Geometry
Mar. 25, 27
MultipleView Reconstruction
Bundle adjustment
Apr. 1, 3 Auto-Calibration Papers
Apr. 8, 10
Dynamic SfM Papers
Apr. 15, 17
Cheirality Papers
Apr. 22, 24
Duality Project Demos
Parameter estimation
• 2D homographyGiven a set of (xi,xi’), compute H (xi’=Hxi)
• 3D to 2D camera projectionGiven a set of (Xi,xi), compute P (xi=PXi)
• Fundamental matrixGiven a set of (xi,xi’), compute F (xi’TFxi=0)
• Trifocal tensor Given a set of (xi,xi’,xi”), compute T
DLT algorithmObjective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.
(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A
(iii) Obtain SVD of A. Solution for h is last column of V
(iv) Determine H from h
Geometric distancemeasured coordinatesestimated coordinatestrue coordinates
xxx
2H
xH,xargminH iii
d Error in one image
e.g. calibration pattern
221-
HHx,xxH,xargminH iiii
i
dd Symmetric transfer error
d(.,.) Euclidean distance (in image)
Reprojection error
22
x,xH,x,xx,xargminx,x,H iiii
iii dd
ii
ii xHx subject to
Geometric interpretation of reprojection error
Estimating homography~fit surface to points X=(x,y,x’,y’)T in 4Hν
2H22 ,Xx,xx,x iiiii ddd
Analog to conic fitting CxxC,x T2
alg d
2C,xd
eedSampson
1TT2 JJCx,
Statistical cost function and Maximum Likelihood
Estimation• Optimal cost function related to noise
model• Assume zero-mean isotropic Gaussian
noise (assume outliers removed) 22 2/xx,2πσ2
1xPr de
22i 2xH,x /
2iπσ2
1H|xPr ide
i
constantxH,xH|xPrlog 2i2i
σ2
1id
Error in one image
Maximum Likelihood Estimate
2i xH,x id
Statistical cost function and Maximum Likelihood
Estimation• Optimal cost function related to noise
model• Assume zero-mean isotropic Gaussian
noise (assume outliers removed) 22 2/xx,2πσ2
1xPr de
22i
2i 2xH,xx,x /
2iπσ2
1H|xPr
ii dd
ei
Error in both images
Maximum Likelihood Estimate
2i2
i x,xx,x ii dd
Mahalanobis distance
• General Gaussian caseMeasurement X with covariance
matrix Σ
XXXXXX 1T2
22XXXX
Error in two images (independent)
22XXXX
iiii
iii
Varying covariances
Invariance to transforms ?
xTx~ Txx~ Hxx x~H
~x~
TH~
TH 1-?
TxH~
xT TxH
~Tx -1
will result change? for which algorithms? for which transformations?
Non-invariance of DLT
Given and H computed by DLT,
and
Does the DLT algorithm applied to yield ?
iiii xTx~,Txx~ ii xx
ii x~x~ -1HTTH
~
iiiiie TxHTTxTx~H~
x~~ -1 iii e** THxxT
hA,R~,~h~
A~
i2121i seesee TT
Effect of change of coordinates on algebraic error
10
tsRT
ss
Rt-
0RT T
*
for similarities
so
iiii sdd x~H~
,x~Hx,x algalg
Non-invariance of DLT
Given and H computed by DLT,
and
Does the DLT algorithm applied to yield ?
iiii xTx~,Txx~ ii xx
ii x~x~ -1HTTH
~
1H
~ subject tox~H
~,x~minimize
1H subject tox~H~
,x~minimize
1H subject toHx,xminimize
2
alg
2
alg
2alg
iii
iii
iii
d
d
d
Invariance of geometric error
ii
iiiiii
sd
ddd
Hx,x
HxT,xTTxHTT,xTx~H~
,x~ -1
Given and H,
and
Assume T’ is a similarity transformations
,xTx~,Txx~ iiii ii xx
,x~x~ ii -1HTTH~
Normalizing transformations
• Since DLT is not invariant,what is a good choice of coordinates?e.g.• Translate centroid to origin• Scale to a average distance to the
origin• Independently on both images
2
1
norm
100
2/0
2/0
T
hhw
whwOr
Importance of normalization
0
h
h
h
0001
1000
3
2
1
iiiiiii
iiiiiii
xyxxxyx
yyyxyyx
~102 ~102 ~102 ~102 ~104 ~104 ~10211
orders of magnitude difference!
Normalized DLT algorithmObjective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) Normalize points
(ii) Apply DLT algorithm to
(iii) Denormalize solution
,x~x~ ii inormiinormi xTx~,xTx~
norm-1
norm TH~
TH
Iterative minimization metods
Required to minimize geometric error (i) Often slower than DLT(ii) Require initialization(iii) No guaranteed convergence, local minima(iv) Stopping criterion requiredTherefore, careful implementation required:(i) Cost function(ii) Parameterization (minimal or not)(iii) Cost function ( parameters )(iv) Initialization(v) Iterations
Parameterization
Parameters should cover complete space and allow efficient estimation of cost
• Minimal or over-parameterized? e.g. 8 or 9 (minimal often more complex, also cost surface)
(good algorithms can deal with over-parameterization)
(sometimes also local parameterization)
• Parametrization can also be used to restrict transformation to particular class, e.g. affine
Function specifications
(i) Measurement vector XN with covariance Σ(ii) Set of parameters represented by vector P N
(iii) Mapping f : M →N. Range of mapping is surface S representing allowable measurements
(iv) Cost function: squared Mahalanobis distance
Goal is to achieve , or get as close as possible in terms of Mahalanobis distance
PXPXPX 1T2fff
XP f
Error in one image
2i xH,x id
nf Hx,...,Hx,Hxh: 21
hX f
221- Hx,xxH,x iiiii
dd Symmetric transfer error
nnf Hx,...,Hx,Hx,xH,...,xH,xHh: 21-1
2-1
1-1
hX f
Reprojection error
2i2
i x,xx,x ii dd nnf x,...,x,xx,...,x,xh,: 2121
hX f
Initialization
• Typically, use linear solution• If outliers, use robust algorithm
• Alternative, sample parameter space
Iteration methods
Many algorithms exist• Newton’s method• Levenberg-Marquardt
• Powell’s method• Simplex method
Gold Standard algorithmObjective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the Maximum Likelyhood Estimation of H
(this also implies computing optimal xi’=Hxi)
Algorithm
(i) Initialization: compute an initial estimate using normalized DLT or RANSAC
(ii) Geometric minimization of -Either Sampson error:
● Minimize the Sampson error
● Minimize using Levenberg-Marquardt over 9 entries of h
or Gold Standard error:
● compute initial estimate for optimal {xi}
● minimize cost over {H,x1,x2,…,xn}
● if many points, use sparse method
2i2
i x,xx,x ii dd
Robust estimation
• What if set of matches contains gross outliers?
RANSACObjective
Robust fit of model to data set S which contains outliers
Algorithm
(i) Randomly select a sample of s data points from S and instantiate the model from this subset.
(ii) Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.
(iii) If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate
(iv) If the size of Si is less than T, select a new subset and repeat the above.
(v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si
Distance threshold
Choose t so probability for inlier is α (e.g. 0.95)
• Often empirically• Zero-mean Gaussian noise σ then
follows distribution with m=codimension of model
2d
2m
(dimension+codimension=dimension space)
Codimension
Model t 2
1 l,F 3.84σ2
2 H,P 5.99σ2
3 T 7.81σ2
How many samples?
Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99
sepN 11log/1log
peNs 111
proportion of outliers es 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 117
7
Acceptable consensus set?
• Typically, terminate when inlier ratio reaches expected ratio of inliers
neT 1
Adaptively determining the number of samples
e is often unknown a priori, so pick worst case, e.g. 50%, and adapt if more inliers are found, e.g. 80% would yield e=0.2
• N=∞, sample_count =0• While N >sample_count repeat
• Choose a sample and count the number of inliers• Set e=1-(number of inliers)/(total number of
points)• Recompute N from e• Increment the sample_count by 1
• Terminate
sepN 11log/1log
Robust Maximum Likelyhood Estimation
Previous MLE algorithm considers fixed set of inliers
Better, robust cost function (reclassifies)
outlier
inlier ρ with ρ
222
222
i tet
teeed iR
Other robust algorithms
• RANSAC maximizes number of inliers
• LMedS minimizes median error
• Not recommended: case deletion, iterative least-squares, etc.
Automatic computation of HObjective
Compute homography between two imagesAlgorithm
(i) Interest points: Compute interest points in each image
(ii) Putative correspondences: Compute a set of interest point matches based on some similarity measure
(iii) RANSAC robust estimation: Repeat for N samples
(a) Select 4 correspondences and compute H
(b) Calculate the distance d for each putative match
(c) Compute the number of inliers consistent with H (d<t)
Choose H with most inliers
(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML cost function with Levenberg-Marquardt
(v) Guided matching: Determine more matches using prediction by computed H
Optionally iterate last two steps until convergence
Determine putative correspondences
• Compare interest pointsSimilarity measure:• SAD, SSD, ZNCC on small neighborhood
• If motion is limited, only consider interest points with similar coordinates
• More advanced approaches exist, based on invariance…
Example: robust computation
Interest points(500/image)
Putative correspondences (268)
Outliers (117)
Inliers (151)
Final inliers (262)
Assignment
• Take two or more photographs taken from a single viewpoint
• Compute panorama• Use different measures DLT, MLE
• Use Matlab • Due Feb. 13
Next class: Algorithm evaluation and error analysis
• Bounds on performance• Covariance propagation• Monte Carlo covariance
estimation