Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 1 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking.

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung

Professor Horst Cerjak, 19.12.20051

22.4.2008Augmented Reality VU 4 Algorithms + Tracking Axel Pinz

Algorithms

• Point correspondences– Salient point detection – Local descriptors

• Matrix decompositions– RQ decomposition– Singular value decomposition - SVD

• Estimation– Systems of linear equations– Solving systems of linear equations

• Direct Linear Transform – DLT• Normalization• Iterative error / cost minimization• Outliers Robustness, RANSAC

– Pose estimation• Perspective n-point problem – PnP




Relevant Issues in Practice

• Poor condition of A Normalization

• Algebraic error vs.geometric error, Iterative minimizationnonlinearities (lens dist.)

• Outliers Robust algorithms (RANSAC)

0 0 0

fph AAA




Normalization (1)

0

0

000

000:0

9

1

'11

'11

'11

'11

'11

'11

'11

'11

'11

'11

'11

'11

1

h

h

xwxyxxwwwywx

ywyyyxwwwywxh

A

Homography H:

• Entries of A are quadratic in point coordinates• SVD is not robust against coordinate transform !

– change of coordinate system (translation, scaling) will influence result !

– algebraic vs. geometric error !

• Normalization recommended, e.g.:– translate origin (0,0,1) image center

– “isotropic” scaling such that:• either average distance to (0,0,1) is ,• or “average point” is (1,1,1)

2




Normalization (2)

Fundamental Matrix F:

• “poor condition” of ATA• Normalization is mandatory• “normalized 8-point algorithm” to estimate F [Hartley’95]

“in defense of the 8-point algorithm”

)1,10,10,10,10,10,10,10,10(: of diagonal

)1,10,10,10,10,10,10,10,10(:in line

)1,100,100(' assume

44488488

22244244

AA

AT

TT uu

note: some algorithms useeigenvalues of ATA insteadof singular values (SVD) !




Iterative Minimization

• DLT minimizes “algebraic error”• “geometric distance” is more complex

• Lens distortion is non-linear

• “Standard” approach:– estimate linear parameters by DLT initialization for– subsequent iterative minimization over all parameters

• E.g.: “gold standard” for estimation of H




“Gold Standard” for Estimation of H (1)[Hartley+Zisserman]

ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.

(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A

(iii) Obtain SVD of A. Solution for h is last column of V

(iv) Determine H from h[adapted from Pollefeys’ course]

DLT algorithm to estimate H:





[adapted from Pollefeys’ course]

normalized DLT algorithm to estimate H:

ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) Normalize points

(ii) Apply DLT algorithm to

(iii) Denormalize solution

inorminormi 'x'x,xx

TT i~~

,xx ii~~

norm-

norm THTH~1





[adapted fromPollefeys’ course]

ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the Maximum Likelihood Estimation of H

Algorithm

(i) Initialization: compute an initial estimate using normalized DLT or RANSAC

(ii) Geometric minimization of either Sampson error:

● Minimize the Sampson error

● Minimize using Levenberg-Marquardt over 9 entries of h

or Gold Standard error:

● compute initial estimate for “subsidiary”

● minimize cost over

● if many points, use sparse method

22 ˆˆ iiii x',xdx,xd }ˆ{ ix

nixi 1,ˆ and ˆ H




Robust Estimation (RANSAC) [Hartley+Zisserman]

Handling of outliers !

“RANSAC” = “RANdom Sample Consensus”




RANSAC Algorithm [Hartley+Zisserman]

ObjectiveRobust fit of model to data set S which contains outliers

Algorithm

(i) Randomly select a sample of s data points from S and instantiate the model from this subset.

(ii) Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.

(iii) If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate

(iv) If the size of Si is less than T, select a new subset and repeat the above.

(v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

[adapted FromPollefeys’course]




RANSAC Algorithm [Hartley+Zisserman]

proportion of outliers es 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 35

4 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 1177

sample size vs. proportion of outliers:

[adapted from Pollefeys’ course]




More Problems“critical” cases ! e.g. [Torr+Murray, IJCV 1997]




Pose Estimation

X

Y

Z

• calibrated camera, known K

C

xC

yC

zC

R, t

• Camera

xV

yV

zV

• Visualization (screen, HMD)

R, t

• determine camera pose: R, t




Perspective n-Point Problem – PnP (1)

• Calibrated camera– Known K

• Known points Pi in the scene• Given n point correspondences

– pi ↔ Pi

• What can be measured with one calibrated camera?

angle θ between two lines of sight

pa

pb

Pb

Pa

dabθ

C




Perspective n-Point Problem – PnP (2)

• PnP uses just this information:

• P3P will give up to 4 solutions• P4P is already overdetermined

– Perform 4 x P3P– Find consensus

pa

pb

Pb

Pa

dabθ

C

abbabaab CPCPCPCPd cos2222




Pose Estimation Tracking

• In theory, tracking is simple !– Calibrate your camera (K)– Measure some points Pi in the scene (“ground truth”)– Perform pose estimation in real-time (for each frame)

• In practice, tracking is a hard problem !– Point detection – Correspondence– Motion prediction – Occlusion– Unknown scene– …

• Many solutions have been proposed !

“Tracking beyond 15 minutes of thought”SIGGRAPH 2001 Turorial #15[Allen, Bishop, Welch]

“An introduction to the Kalman filter”SIGGRAPH 2001 Turorial #8[Welch, Bishop]




Tracking Systems (vision-based / hybrid)some of my own contributions (1)

Hybrid“inside out”magnetic + stereo vision[Auer 1999]





stereo vision“outside in”[Ribo ca. 2000]





inertial“inside out”hybrid inertal + vision[many 2000-2004]





vision (stereo or mono)“inside out”speed solves correspondence ! [Mühlmann 2005]




Our Current View [Schweighofer 2008]

stereo vision“inside out”structure and motion




Summary

In these four lectures, I gave an introduction to:• Projective geometry• Perspective cameras• Homographies, camera projection matrices, fundamental and

essential matrices• Algorithms that are typically applied to solve for

– Camera calibration– Stereo reconstruction– Camera pose estimation

I consider this the basis for further reading in topics including:• Vision-based pose tracking• Structure and motion analysis (sometimes termed “SLAM”)

Many aspects were, of course, not covered, but would also be important !




What could not be covered ?

• Self calibration (see Pollefeys, absolute conic,…)• Bundle adjustment• Levenberg-Marquardt• The full presentation of algorithms for the estimation of H, P, K, F, …

– see the Hartley, Zisserman book for all about “multiple view geometry”• Tracking in general, Kalman filter (two UNC Siggraph 2001Tutorials)• Several prominent variants of vision-based tracking algorithms/systems:

– KLT– Rapid, RoRapid– Condensation, ICondensation– [Lu, Hager]– [Ansar, Daniilidis]– [Wunsch, Hirzinger]– [Klein, Murray]– …

Another reference to Pollefeys:http://www.cs.unc.edu/~marc/tutorial/node159.html

interested in more detail ?2 VO “Image based measurement” WS1 LU “Image based measurement” SSseminar, project, bachelor, diploma, PhD, …

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 1 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking.

Documents

tracking axel pinz algorithms

estimation of h slide

set s i

augmented reality vu

subset of s i

homography h

entries of h

set of data points s