Twenty Five Years of RANSAC: MLESAC a new cost function for RANSAC Philip Torr Oxford Brookes Code Available

Twenty Five Years of RANSAC:MLESAC a new cost function for

RANSAC

Philip TorrPhilip Torr

Oxford BrookesOxford Brookes

Code AvailableCode Available

http://cms.brookes.ac.uk/staff/PhilipTorr/http://cms.brookes.ac.uk/staff/PhilipTorr/

[email protected]@brookes.ac.uk

Robust estimation

What if set of matches contains gross outliers?What if set of matches contains gross outliers?

Previous work

Suprisingly, at the time, algorithmic work in Suprisingly, at the time, algorithmic work in statistics quite weak.statistics quite weak.

– L1 norms.L1 norms.

– M-estimators.M-estimators.

Quiz, L1 norm

Is the L1 norm robust, in the sense that it is Is the L1 norm robust, in the sense that it is resistant to outliers?resistant to outliers?

– A) for one dimension?A) for one dimension?

– B) for greater than one dimension?B) for greater than one dimension?

Quiz Solutions

Contrary to widely held belief L1 norm is not robust Contrary to widely held belief L1 norm is not robust for dimension greater than 1for dimension greater than 1

Dimension 1 it leads to the median so has some Dimension 1 it leads to the median so has some robustness.robustness.

Dimension 2+ L1 norm is as vulnerable to outliers as Dimension 2+ L1 norm is as vulnerable to outliers as L2 (this observation although it might appear obvious L2 (this observation although it might appear obvious has eluded many eminent vision researchers).has eluded many eminent vision researchers).

L1 Problem

M-Estimators

No good algorithm for obtaining themNo good algorithm for obtaining them

Not justified from a Bayesian perspectiveNot justified from a Bayesian perspective

What we would like to do

If we are rational we might be interested to know If we are rational we might be interested to know the maximum a posteriori solutionthe maximum a posteriori solution

This will lead to an energy minimization problem This will lead to an energy minimization problem where the energy might be arbitrarywhere the energy might be arbitrary

Example: Gaussian Error in Variable Model (EVM)

Noise on points is (possibly a robust mixture):Noise on points is (possibly a robust mixture):

Where points lie on a manifold defined by q Where points lie on a manifold defined by q implicit relationsimplicit relations

Parameters:

Non-Latent Variables αα parameters of manifold (i.e. 7 for parameters of manifold (i.e. 7 for FF))

Latent Variables ββparameters of location of data on manifoldparameters of location of data on manifold

γγindicator variables, one per datum, whether indicator variables, one per datum, whether inlier or outlierinlier or outlier

Total least squares

Bayesian Form of the Non-robust & Robust likelihood L

Problem

How to minimize this complex (or any other How to minimize this complex (or any other complex) posterior likelihood.complex) posterior likelihood.

Answer of course by data driven search of Answer of course by data driven search of the space (e.g. RANSAC)the space (e.g. RANSAC)

RANSAC

Repeat M times:Repeat M times:– Sample minimal number of matches to estimate Sample minimal number of matches to estimate

two view relation.two view relation.

– Calculate number of inliers or posterior likelihood Calculate number of inliers or posterior likelihood for relation.for relation.

– Choose relation to maximize number of inliers.Choose relation to maximize number of inliers.

RANSAC line fitting example

Task:Task:

Estimate best lineEstimate best line

RANSAC line fitting example

Sample two pointsSample two points

RANSAC line fitting example

Fit LineFit Line

RANSAC line fitting example

Total number of points Total number of points within a threshold of line.within a threshold of line.

RANSAC line fitting example

Repeat, until get a Repeat, until get a good resultgood result

RANSAC line fitting example

Repeat, until get a Repeat, until get a good resultgood result

RANSAC line fitting example

Repeat, until get a Repeat, until get a good resultgood result

How do we minimize a posterior energy?

So before we mentioned Bayesian estimation.So before we mentioned Bayesian estimation.

We explain first why just the RANSAC error We explain first why just the RANSAC error might not be so good.might not be so good.

Then how we can maximize an arbitrary Then how we can maximize an arbitrary posterior likelihood.posterior likelihood.

Cost function

RANSAC can be vulnerable to the correct RANSAC can be vulnerable to the correct choice of the threshold:choice of the threshold:– Too large all hypotheses are ranked equally.Too large all hypotheses are ranked equally.– Too small leads to an unstable fit.Too small leads to an unstable fit.

The interesting thing is that the same strategy The interesting thing is that the same strategy can be followed with any modification of the can be followed with any modification of the cost function.cost function.

Problem with RANSAC;threshold too high

Problem with RANSAC;threshold too high

This solution…This solution…

Problem with RANSAC;threshold too high

Is as good as this Is as good as this solutionsolution

Problem with RANSAC;threshold too low-no support

Problem 1; cost function

Examples of other cost functionsExamples of other cost functions– Least Median Squares; i.e. take the sample that Least Median Squares; i.e. take the sample that

minimized the median of the residuals.minimized the median of the residuals.– MLESAC/MLESAC use the posterior or likelihood MLESAC/MLESAC use the posterior or likelihood

of the data.of the data.– MINPRAN (Stewart), makes assumptions about MINPRAN (Stewart), makes assumptions about

randomness of datarandomness of data

LMS

Repeat M times:Repeat M times:– Sample minimal number of matches to estimate Sample minimal number of matches to estimate

two view relation.two view relation.

– Calculate error of all data.Calculate error of all data.

– Choose relation to minimize median of errors.Choose relation to minimize median of errors.

Pros and Cons LMS

PROPRO– Do not need any threshold for inliers.Do not need any threshold for inliers.

CONCON– Cannot work for more than 50% outliers.Cannot work for more than 50% outliers.– Problems if a lot of data belongs to a sub manifold Problems if a lot of data belongs to a sub manifold

(e.g. dominate plane in the image)(e.g. dominate plane in the image)

Con: LMS, subspace problem

Median error is same Median error is same for two solutions.for two solutions.

Con: LMS, subspace problem

No good solution if the No good solution if the number of outliers >50%number of outliers >50%

Pros LMS

One major advantage of LMS is that it can One major advantage of LMS is that it can yield a robust estimate of the variance of the yield a robust estimate of the variance of the errors.errors.

But care should be taken to use the right But care should be taken to use the right formula, as this depends on the distribution of formula, as this depends on the distribution of the errors, and degrees of freedom in the the errors, and degrees of freedom in the errors (codimension).errors (codimension).

Choosing the right model

We shall return to this laterWe shall return to this later

Robust Maximum Likelihood Estimation

Random Sampling can optimize any function:Random Sampling can optimize any function:

outlier

inlier ρ with ρ

222

222

i tet

teeed iR

Better, robust cost function, MLESACBetter, robust cost function, MLESAC

Mixture (Maxture) of Gaussian/Uniform?

Red-mixture, green-uniform, blue-Gaussian.Red-mixture, green-uniform, blue-Gaussian.

MAPSAC/MLESAC

This solution…This solution…

MLESAC/MLESAC

Is better than this Is better than this solutionsolution

MLESAC

Add in prior to get to MAP solutionAdd in prior to get to MAP solution

Interesting thing is that with MLESAC one could Interesting thing is that with MLESAC one could sample less than the minimal number of points to sample less than the minimal number of points to make an estimate (using prior as extra information).make an estimate (using prior as extra information).

Any posterior can be optimized; random sampling Any posterior can be optimized; random sampling good for matching AND FUNCTION OPTIMIZATION! good for matching AND FUNCTION OPTIMIZATION! e.g. MLESAC is a cheap way to optimize objective e.g. MLESAC is a cheap way to optimize objective functions regardless of outliers or not.functions regardless of outliers or not.

MLESAC

Once the benefits of MLESAC are seen there Once the benefits of MLESAC are seen there is no reason to continue to use RANSAC; is no reason to continue to use RANSAC; – in many situations the improvement in the solution in many situations the improvement in the solution

can be markedcan be marked– Especially if want to use prior information (e.g. the Especially if want to use prior information (e.g. the

F matrix changing smoothly over time).F matrix changing smoothly over time).– Gives an optimized solution Gives an optimized solution

AT NO EXTRA COST!AT NO EXTRA COST!

A RANSAC system for SAM

Structure and Motion Recovery

1. Finding Features 1. Finding Features 2. Matching Features2. Matching Features 3. Extracting Epipolar3. Extracting Epipolar GeometryGeometry

7. VRML Models7. VRML Models4. Extract edges4. Extract edges5. Match edges5. Match edges

6. Estimate Depth Map 6. Estimate Depth Map (dynamic programming)(dynamic programming)

Guide matches with Geometry

Plane

C

C’

m

m’

11

'

'

987

654

321

y

x

y

x

hhhhhhhhh

Hxx '0'Fxxt

0

1

'

'

1

987

654

321

y

x

yx

fffffffff

Concatenated Image space

2 Views- consider 4D space of image 2 Views- consider 4D space of image coordinates (x,y,x’,y’).coordinates (x,y,x’,y’).

Fundamental matrix is a 3D manifold in Fundamental matrix is a 3D manifold in this space.this space.

Homography is a 2D manifold in this Homography is a 2D manifold in this space.space.

Estimation of Motion model like fitting a manifold to space of 4D image

points in two images:

•Problem compounded in higher dimensionsProblem compounded in higher dimensions

Images of the same scene from different viewpoints

Feature Detectors need to consistently locate the position within the image of a landmark on the

3D object.

Stage 1 Corner Detection

Typical Features Detected

Stage 2 Feature Matching

Images of the same scene from different viewpoints

Initial Feature correspondence via Cross Correlation.

Stage 2 Feature Matching

Initial Feature correspondence via Cross Correlation

Many outliers.

Stage 3 Estimation of Epipolar Geometry

Images of the same scene from different viewpoints

Corresponding features must lie on correspondingepipolar lines.

All epipolar lines intersect at a common point.

Example: robust computationInterest points(500/image)

Putative correspondences (268)

Outliers (117)

Inliers (151)

Final inliers (262)

Automatic camera recovery:

Video

Video

How many samples?

Choose Choose NN so that, with probability so that, with probability pp, at least one random , at least one random sample is free from outliers. e.g. sample is free from outliers. e.g. pp=0.99=0.99

sepN 11log/1log

peNs 111

proportion of outliersproportion of outliers ee

ss 5%5% 10%10% 20%20% 25%25% 30%30% 40%40% 50%50%

22 22 33 55 66 77 1111 171733 33 44 77 99 1111 1919 3535

44 33 55 99 1313 1717 3434 7272

55 44 66 1212 1717 2626 5757 146146

66 44 77 1616 2424 3737 9797 29329377 44 88 2020 3333 5454 163163 588588

88 55 99 2626 4444 7878 272272 11771177

Adaptively determining the number of samples

ee is often unknown a priori, so pick worst case, e.g. 50%, and is often unknown a priori, so pick worst case, e.g. 50%, and

adapt if more inliers are found, e.g. 80% would yield adapt if more inliers are found, e.g. 80% would yield ee=0.2 =0.2

– NN=∞, =∞, sample_count sample_count =0=0

– While While N N >>sample_countsample_count repeat repeat Choose a sample and count the number of inliersChoose a sample and count the number of inliers Set e=1-(number of inliers)/(total number of points)Set e=1-(number of inliers)/(total number of points) Recompute Recompute NN from from ee Increment the Increment the sample_countsample_count by 1 by 1

– TerminateTerminate sepN 11log/1log

Number of Samples II

Make take many Make take many more samples than more samples than one would think due one would think due to degenerate point to degenerate point sets.sets.

Number of Samples II

These two These two points are points are inliers.inliers.

Number of Samples II

And yet the And yet the estimate estimate yielded is poor.yielded is poor.

Work of Tordof and Murray

Try and sample more frequently from strong Try and sample more frequently from strong matches.matches.

Model Selection!!

Problem 2, what model to fit?

There are many cases when we do not know There are many cases when we do not know the relation between the images, there may a the relation between the images, there may a choice of many.choice of many.

In this case a Bayesian solution might be to In this case a Bayesian solution might be to evaluate the likelihood of each.evaluate the likelihood of each.

Problem Degeneracy F or H?:Torr, Fitzgibbon & Zisserman 97, 99

AA

Two problem cases Two problem cases for SAM (structure for SAM (structure and motion) and motion) recovery.recovery.

A Camera RotationA Camera Rotation

B Planar object.B Planar object.

BB

Problem:Problem: Model Selection to determine whether Model Selection to determine whether FF or or HH

Problem Degeneracy F or H?:Torr, Fitzgibbon & Zisserman 97, 99

AA

Two problem cases Two problem cases for SAM (structure for SAM (structure and motion) and motion) recovery.recovery.

A Camera RotationA Camera Rotation

B Planar object.B Planar object.

BB

Solution:Solution: was to use was to use GRIC GRIC criteria, to be explained later.criteria, to be explained later.

When Homography describes scene:

A: Camera rotates, no new structure A: Camera rotates, no new structure information.information.

B: 2 views have a plane in common; can B: 2 views have a plane in common; can not put structure into the same projective not put structure into the same projective frame (3 degrees of freedom).frame (3 degrees of freedom).

This is an example of fitting manifolds of varying dimension:

2 Views----Consider Image coordinates 4D 2 Views----Consider Image coordinates 4D spacespace

Dimension 3 Dimension 3 Dimension 2Dimension 2

BilinearBilinear FF Matrix Matrix Homography Homography HH

LinearLinear Affine Affine FF Matrix Matrix Affinity Affinity AA

(non generic: quadratic transformations, (non generic: quadratic transformations, dimension 2.)dimension 2.)

Three views: same dimension for manifolds.Three views: same dimension for manifolds.

Concatenated or Joint Image Space

xxyy

X’X’

Y’Y’

Image coordinates in higher dimensional spaceImage coordinates in higher dimensional space

(x,y,x’,y’,…)(x,y,x’,y’,…)

Generic ProblemDetermine the degree and dimension ofDetermine the degree and dimension ofa manifold in a manifold in dd dimensional space. dimensional space.

Examples

•Problem compounded in higher dimensionsProblem compounded in higher dimensions

Robust Model Selection

•Curve Dim 2, degree 2Curve Dim 2, degree 2

•Line Dim 1, degree 1Line Dim 1, degree 1

•Point Dim 0, degree 1Point Dim 0, degree 1

•Outliers make a hard problem very hard!Outliers make a hard problem very hard!

Bayesian Model Comparison

A Horrible integral:

Can we simplify it?Can we simplify it?

Integrate out the Integrate out the ββ and and γγ..

Solve Via Sampling

Can use RANSAC to generate an importance Can use RANSAC to generate an importance sampling function and use sampling to sampling function and use sampling to evaluate the integral evaluate the integral – (Torr & Davidson ECCV 2002/PAMI 2002)(Torr & Davidson ECCV 2002/PAMI 2002)

GRIC approximation, Torr IJCV 2002

Quick hack: some combination of likelihood Quick hack: some combination of likelihood of model and a penalty term for complexity…of model and a penalty term for complexity…

Helps to penalize latent and model Helps to penalize latent and model parameters separately.parameters separately.

Summary of Idea

Can base approximation around MAP Can base approximation around MAP solution,solution,

Thus need to find MAP solution for each Thus need to find MAP solution for each model.model.

Sampling Problem

If lots of points lie on low D space data driven If lots of points lie on low D space data driven estimates can be quite poor.estimates can be quite poor.

E.g. many points lie near a single point, then hard to E.g. many points lie near a single point, then hard to get a line that passes through all the inliers.get a line that passes through all the inliers.

This gets worse in higher dimensionsThis gets worse in higher dimensions

Example for motion estimation, many points lie on a Example for motion estimation, many points lie on a plane.plane.

Sampling Problem

Sampling Problem

Sampling Problem

Solution

Derive a sampling strategy that takes into Derive a sampling strategy that takes into account the multiple models.account the multiple models.

Estimate lower dimensional ones first, e.g. Estimate lower dimensional ones first, e.g. planesplanes

Sample points on and off the planes to Sample points on and off the planes to estimate the higher dimensional manifolds estimate the higher dimensional manifolds e.g. F etc.e.g. F etc.

Model Selection and Multiple Motions

Lorry Sequence

Camera zooms out, lorry translates leftCamera zooms out, lorry translates left

Lorry Match Segmentation

Conclusion

Minimize the correct error from the outset.Minimize the correct error from the outset.

Be wary of the model selection problem.Be wary of the model selection problem.

END

Matlab Code and paper available:Matlab Code and paper available: Matching, est F, sfm, segmentation.Matching, est F, sfm, segmentation.

http://cms.brookes.ac.uk/staff/PhilipTorr/http://cms.brookes.ac.uk/staff/PhilipTorr/

philtorr@[email protected]@[email protected]

Bib

P.H.S. Torr, and D.W. Murray. P.H.S. Torr, and D.W. Murray. The Development and Comparison of Robust Methods for Estimating the FundaThe Development and Comparison of Robust Methods for Estimating the Fundamental Matrixmental Matrix. In . In International Journal of Computer Vision,International Journal of Computer Vision, pages 271—300, v 24, n 3, 1997. pages 271—300, v 24, n 3, 1997.

P.H.S. Torr and A. Zisserman. P.H.S. Torr and A. Zisserman. MLESAC: A New Robust Estimator with Application to Estimating Image GeomeMLESAC: A New Robust Estimator with Application to Estimating Image Geometrytry. In . In Journal of Computer Vision and Image UnderstandingJournal of Computer Vision and Image Understanding, pages 138—156, , pages 138—156, 78(1), 2000.78(1), 2000.

P.H.S. Torr. P.H.S. Torr. Geometric Motion Segmentation and Model SelectionGeometric Motion Segmentation and Model Selection. In . In Philosophical Transactions of the Royal Society APhilosophical Transactions of the Royal Society A , pages 1321—1340, 1998., pages 1321—1340, 1998.

P.H.S. Torr, A. Fitzgibbon and A. Zisserman. P.H.S. Torr, A. Fitzgibbon and A. Zisserman. The Problem of Degeneracy in Structure and Motion Recovery from UncalibrateThe Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequencesd Image Sequences. In . In International Journal of Computer VisionInternational Journal of Computer Vision, 32(1), pages 27—45, 1999., 32(1), pages 27—45, 1999.

P.H.S. Torr and C. Davidson. P.H.S. Torr and C. Davidson. IMPSAC: A synthesis of importance sampling and random sample consensusIMPSAC: A synthesis of importance sampling and random sample consensus. . In In IEEE Trans Pattern Analysis and Machine Intelligence, IEEE Trans Pattern Analysis and Machine Intelligence, 25(3), pages 354-365, 25(3), pages 354-365, 2003.2003.

P.H.S. Torr. P.H.S. Torr. Bayesian Model Estimation and Selection for Epipolar Geometry and Generic MBayesian Model Estimation and Selection for Epipolar Geometry and Generic Manifold Fittinganifold Fitting. In . In International Journal of Computer VisionInternational Journal of Computer Vision, 50(1), pages 27—45, 2002., 50(1), pages 27—45, 2002.

Bib

D. Myatt, P.H.S. Torr, S. Nasuto, and R. Craddock. D. Myatt, P.H.S. Torr, S. Nasuto, and R. Craddock. NAPSAC: High Noise, High Dimensional Robust EstimationNAPSAC: High Noise, High Dimensional Robust Estimation, In , In Proceedings British Proceedings British Machine Vision ConferenceMachine Vision Conference, pages 458-467,, pages 458-467, 2002. 2002. (oral)(oral). .

P.H.S. Torr and D.W. Murray. P.H.S. Torr and D.W. Murray. Outlier detection and motion segmentationOutlier detection and motion segmentation. In . In SPIE sensor SPIE sensor fusion conference VIfusion conference VI, pages 432—443, Sept. 1993. , pages 432—443, Sept. 1993. (oral)(oral). .

P.H.S. Torr, A. Zisserman and S. Maybank. P.H.S. Torr, A. Zisserman and S. Maybank. Robust Detection of Degeneracy.Robust Detection of Degeneracy. In In The Fifth The Fifth International Conference on Computer VisionInternational Conference on Computer Vision, pages 1037—1044, 1995. , pages 1037—1044, 1995. (oral)(oral)..

P.H.S. Torr, A. Zisserman, and D.W. Murray. P.H.S. Torr, A. Zisserman, and D.W. Murray. Motion Clustering using the Trilinear ConstraintMotion Clustering using the Trilinear Constraint. In . In Europe-China workshop on Geometric Europe-China workshop on Geometric Modelling and Invariants for Computer VisionModelling and Invariants for Computer Vision, pages 118—125, 1995. , pages 118—125, 1995. (oral)(oral)..

P.A. Beardsley, P.H.S. Torr and A. Zisserman. P.A. Beardsley, P.H.S. Torr and A. Zisserman. 3D Model Acquisition from Extended Image Sequences3D Model Acquisition from Extended Image Sequences. In . In The Fourth European The Fourth European Conference on Computer VisionConference on Computer Vision, pages 683—695, 1996. , pages 683—695, 1996. (oral)(oral). .

P.H.S. Torr and A. Zisserman. P.H.S. Torr and A. Zisserman. Feature Based Methods for Structure and Motion EstimationFeature Based Methods for Structure and Motion Estimation. In . In International Workshop on Vision Algorithms, International Workshop on Vision Algorithms, pages 278-295, 1999.pages 278-295, 1999.(oral-panel (oral-panel discussion)discussion). .

F. Schaffalitzky, A. Zisserman, R. Hartley and P.H.S. Torr. F. Schaffalitzky, A. Zisserman, R. Hartley and P.H.S. Torr. A Six Point Solution for Structure and MotionA Six Point Solution for Structure and Motion. In . In The Sixth European Conference on The Sixth European Conference on Computer VisionComputer Vision, pages 632—648, 2000. , pages 632—648, 2000. (poster)(poster). .

Related Bib

P.H.S. Torr and A. Fitzgibbon. P.H.S. Torr and A. Fitzgibbon. Invariant Fitting of Two View Geometry or “In Defiance of Invariant Fitting of Two View Geometry or “In Defiance of the eight point algorithm”the eight point algorithm”, In , In IEEE Trans Pattern Analysis and Machine IEEE Trans Pattern Analysis and Machine Intelligence, Intelligence, 26(5), pages 648-651, 2004. 26(5), pages 648-651, 2004.

P.H.S. Torr, A. Zisserman. P.H.S. Torr, A. Zisserman. Robust Parameterization and Computation of the Trifocal Robust Parameterization and Computation of the Trifocal TensorTensor. . Image and Vision Computing,Image and Vision Computing, pages 591—607, v 15, pages 591—607, v 15, 1997. 1997.

P.H.S. Torr, A. Zisserman. P.H.S. Torr, A. Zisserman. Performance Performance CharacterisationCharacterisation of Fundamental Matrix Estimation Under Image Degradat of Fundamental Matrix Estimation Under Image Degradationion. In . In Machine Vision and Applications, Machine Vision and Applications, pages 321—333, v pages 321—333, v 9, 1997.9, 1997.