MLESAC: A new robust estimator with application to estimating image geometry P. H. S. Torr and A. Zisserman Microsoft Research Ltd, St George House, 1 Guildhall St, Cambridge CB2 3NH, UK [email protected]and Robotics Research Group, Department of Engineering Science Oxford University, OX1 3PJ., UK [email protected]Received March 31, 1995; accepted July 15, 1996 A new method is presented for robustly estimating multiple view relations from point correspondences. The method comprises two parts, the first is a new robust estimator which is a generalization of the estimator. It adopts the same sampling strategy as to generate putative solutions, but chooses the solution to maximize the likelihood rather than just the number of in- liers. The second part to the algorithm is a general purpose method for automatically parametrizing these relations, using the output of . A difficulty with multi view image relations is that there are often non-linear constraints between the param- eters, making optimization a difficult task. The parametrization method overcomes the difficulty of non-linear constraints and conducts a constrained optimization. The method is general and its use is illustrated for the estimation of fundamental matri- ces, image-image homographies and quadratic transformations. Results are given for both synthetic and real images. It is demonstrated that the method gives results equal or superior to previous approaches. 1. INTRODUCTION This paper describes a new robust estimator which can be used in a wide variety of estimation tasks. In particular is well suited to estimating complex surfaces or more general manifolds from point data. It is applied here to the estimation of several of the multiple view relations that exist between images related by rigid motions. These are relations between corresponding image points in two or more views and include for example, epipolar geometry, projectivities etc. These image relations are used for several purposes: (a) matching, (b) recovery of structure [1, 8, 11, 27, 40] (if this is possible), (c) motion segmentation [31, 36], (d) motion model selection [14, 37, 35]. The paper is organized as follows: In Section 2 the matrix representation of the two view relations are given, including the constraints that the matrix elements must satisfy. For example, there is a cubic polynomial constraint on the matrix elements for the fundamental 1
19
Embed
MLESAC: A new robust estimator with application to estimating …vgg/publications/2000/Torr... · 2011. 8. 18. · MLESAC: A new robust estimator with application to estimating image
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MLESAC: A new robust estimator with applicationto estimating image geometry
for bothsyntheticandreal images.It is demonstratedthat themethodgivesresults
equalor superiorto previousapproaches.
1. INTRODUCTION
This paperdescribesa new robust estimator����������� which canbe usedin a widevarietyof estimationtasks.In particular ����������� is well suitedto estimatingcomplexsurfacesor moregeneralmanifoldsfrom point data.It is appliedhereto theestimationofseveralof themultiple view relationsthatexist betweenimagesrelatedby rigid motions.Thesearerelationsbetweencorrespondingimagepointsin two or moreviewsandincludefor example,epipolargeometry, projectivities etc. Theseimagerelationsare usedforseveral purposes: (a) matching,(b) recovery of structure[1, 8, 11, 27, 40] (if this ispossible),(c) motionsegmentation[31, 36], (d) motionmodelselection[14, 37, 35].
Thepaperis organizedasfollows: In Section2 thematrixrepresentationof thetwo viewrelationsaregiven, including the constraintsthat the matrix elementsmustsatisfy. Forexample,thereis acubicpolynomialconstraintonthematrixelementsfor thefundamental
1
2 TORRAND ZISSERMAN
matrix. It will beseenthatany parametrizationmustenforcethis constraintto accuratelycapturethetwo view geometry.
Due to the frequentoccurrenceof mismatches,a RANSAC [4] like robust estimatoris usedto estimatethe two view relation. The RANSAC algorithmis a hypothesisandverify algorithm. It proceedsby repeatedlygeneratingsolutionsestimatedfrom minimalsetof correspondencesgatheredfrom the data,and then testseachsolution for supportfrom thecompletesetof putative correspondences.��� �!����� is describedin Section4.In RANSAC the support is the numberof correspondenceswith error below a giventhreshold. We proposea new estimatorthat takes as supportthe log likelihood of thesolution (taking into accountthe distribution of outliers) and usesrandomsamplingtomaximizethis. Thislog likelihoodfor eachrelationis derivedin Section3. Thenew robustrandomsamplingmethod(dubbedMLESAC—MaximumLikelihoodEstimationSAmpleConsensus)is adumbratedin Section5.
Having obtaineda robust estimateusingMLESAC, the minimum point setbasiscanbe usedto parametrizethe constraintasdescribedin Section6. The MLE error is thenminimizedusingthisparametrizationandasuitablenon-linearminimizer. Theoptimizationis constrainedbecausethematrix elementsof many of thetwo view relationsmustsatisfycertainconstraints.Notethatrelationscomputedfrom thisminimalsetalwayssatisfytheseconstraints.Thusthenew contribution is threefold: (a) to improveRANSAC by useof abettercostfunction; (b) to develop this costfunction in termsof the likelihoodof inliersandoutliers(thusmakingit robust);and(c) to obtainaconsistentparametrizationin termsof a minimalpoint basis.
Notation. The imageof a 3D scenepoint " is # in thefirst view and # $ in thesecond,where # and # $ are homogeneousthreevectors, #&%('*)�+-,.+0/21-3 . The correspondence#�4(#65 will alsobedenotedas #87�4(#�9 . Throughout,underlininga symbol ) indicatestheperfectornoise-freequantity, distinguishingit from ):%;) <!= ) , whichis themeasuredvaluecorruptedby noise.
Theepipolarconstraintis representedby theFundamentalmatrix [3, 10]. This relationappliesfor generalmotionandstructurewith uncalibratedcameras.Considerthemovementof a setof point imageprojectionsfrom an objectwhich undergoesa rotationandnon-zerotranslationbetweenviews. After the motion, the setof homogeneousimagepoints> #6? @A+CBD%E/A+0FGF0FIHJ+ asviewedin thefirst imageis transformedto theset
where# %Q'*) +-, +0/21-3 isahomogeneousimagecoordinateandM is theFundamentalMatrix.
MLESAC: A NEW ROBUST ESTIMATOR 3
Shouldall the observedpointslie on a plane,or the camerarotateaboutits optic axisandnot translate,thenall thecorrespondenceslie on aprojectivity:# 5 %PRS# F (2)
Shouldall thepointsbeconsistentwith two (or more) M then# $ 3 M 7 # %;O and # $ 3 M 9 # %PO (3)
thus # $ % M 7 # T M 9 # (4)
hencethey conform to a quadratictransformation. The quadratictransformationis ageneralizationof the homography. It is causedby a combinationof a cameramotionandscenestructure,asall the scenepointsandthe cameraoptic centreslie on a criticalsurface[19], which is a ruled quadricsurface. Although the existenceof the criticalsurfaceis well known, little researchhasbeenput into effectively estimatingquadratictransformations.
2.1. Degreesof Freedomwithin Two View ParametrizationsThe fundamentalmatrix has9 elements,but only 7 degreesof freedom. Thus if the
fundamentalmatrix is parametrizedby the elementsof the UVTWU matrix M it is overparametrized.This is becausethe matrix elementsarenot independent,beingrelatedbya cubicpolynomialin thematrix elements,suchthat XY0Z2[ M]\ %^O . If this constraintis notimposedthenthe epipolarlines do not all intersectin a singleepipole[16]. Henceit isessentialthatthis constraintis imposed.
The projectivity has9 elementsand8 degreesof freedomastheseelementsareonlydefinedup to a scale. The quadratictransformationhas 18 elementsand 14 degreesof freedom[18]. Here if the constraintsbetweenthe parametersare not enforcedtheestimationprocessbecomesvery unstable,and good resultscannotbe obtained[18],whereasourmethodhasbeenableto accuratelyestimatetheconstraint.
2.2. Concatenatedor Joint Image Space
Eachpair of correspondingpoints # , # $ definesa singlepoint in a measurementspace_!`, formedby consideringthecoordinatesin eachimage. This spaceis the ‘joint image
space’[38] or the ‘concatenatedimagespace’[24]. It might be consideredsomewhateldritchto join thecoordinatesof thetwo imagesinto thesamespace,but thismakessenseif we assumethat the dataareperturbedby the samenoisemodel(discussedin the nextsubsection)in eachimage,implying thatthesamedistancemeasurefor minimizationmaybeusedin eachimage.Theimagecorrespondences
> # ? @�4 > # $? @a+-BD%Q/b+GF0F0F-HJ+ inducedbyarigid motionhaveanassociatedalgebraicvariety c in
_ `. Fundamentalmatricesdefine
athreedimensionalvarietyin_d`
, whereasprojectivitiesandquadratictransformationsareonly two dimensional.
Givena setof correspondencesthe(unbiased)minimumvariancesolutionfor M is thatwhichminimizesthesumof squaresof distancesorthogonalto thevarietyfrom eachpoint'*)�+-,.+-) $ +C, $ 1 in _ ` [12, 14, 15, 21, 23, 26, 35]. Thisisdirectlyequivalentto thereprojectionerrorof thebackprojected3D projectivepoint.
4 TORRAND ZISSERMAN
x
x x’
x’H-1
H
zz’
z’ = H z
H x’ H x-1
FIGURE 1
In previouswork suchas[17] thetransfererrorhasoftenbeenusedastheerrorfunctione.g.for fitting e this is fag�hGiJj elknm i6oqpsrWfag�hGi6oIj e i�p (5)
wheref�tqu
is the Euclideanimagedistancebetweenthe points. The transferdistanceisdifferentfrom theorthogonaldistanceasshown in Figure1. This is discussedfurther inrelationto themaximumlikelihoodsolutionderivedin Section3.
3. MAXIMUM LIKELIHOOD ESTIMATION IN THE PRESENCEOFOUTLIERS
Within this sectionthemaximumlikelihoodformulationis givenfor computingany ofthe multiple view relations. In the following we make the assumption,without loss ofgenerality, thatthenoisein thetwo imagesis Gaussianoneachimagecoordinatewith zeromeananduniformstandarddeviation v . Thusgivena truecorrespondencetheprobabilitydensityfunctionof thenoiseperturbeddataisw�x tqylz {|u�} ~��� m��L�L� � � �� ��� v�� ��� k����P�q����� �0��� � � k�� ��I� �-� ��� � � k.� ��C� ���G� � g¡ � � j (6)
where¢ is thenumberof correspondencesand{
is theappropriate2 view relation,e.g.thefundamentalmatrixor projectivity, and
i m¤£ g thetaskbecomesthatof finding themaximumlikelihoodestimate,²i m¤£ g of the trueposition
i m¤£ g , suchthat²i m¤£ g satisfiesthe relationandminimizes � ¯C� m¤£ g h ²° ¯ � § ° ¯ � p g r³h ²± ¯� § ± ¯� p g . The MLE
MLESAC: A NEW ROBUST ESTIMATOR 5
error ´ ? for the B th point is then´ 9? %¶µ·C¸ 7�¹ 9�º.») · ?]¼ ) · ?¾½ 9 < º.», ·?�¼ , ·?0½ 9 (8)
Thus ¿ ? ¸ 7�ÀLÀLÀ Á ´29? providesthe error function for the point data,and  for which ¿ ? ´Ã9?is a minimumis themaximumlikelihoodestimateof therelation(fundamentalmatrix, orprojectivity). Hartley andSturm[12] show how ´ , »# and »# $ maybefoundasthesolutionof a degree6 polynomial.A computationallyefficientfirst orderapproximationto theseisgivenin Torr et al. [32, 34, 35].
The above derivationassumesthat the errorsareGaussian,often however featuresaremismatchedandtheerror on Ä is not Gaussian.Thusthe error is modeledasa mixturemodelof Gaussiananduniformdistribution:-Å�Æ 'K´Ç1�%ÉÈÊ /Ë ÌÇÍ6Î 9 Y¾ÏÐÑ' ¼ ´29ÌbÎ 9 1Ñ<Ò'I/ ¼ Ê61 /ÓDÔ (9)
whereÊ is themixingparameterandÓ is justaconstant(thediameterof thesearchwindow),Îis thestandarddeviation of theerroron eachcoordinate.To correctlydetermineÊ andÓ entailssomeknowledgeof the outlier distribution; hereit is assumedthat the outlier
Givena suitableinitial estimatethereareseveralwaysto estimatethe parametersof themixture model, most prominentbeing the ��� algorithm [2, 20], but gradientdescentmethodscouldalsobeused. Becauseof thepresenceof outliersin thedatathe standardmethodof leastsquaresestimationis oftennotsuitableasaninitial estimate,andit is betterto usearobustestimatesuchas �ü���!����� which is describedin thenext section.
4. �ü���!�����The aim is to be ableto computeall theserelationsfrom imagecorrespondencesover
two views. This computationrequiresinitial matchingof points(corners)over the imagepairs. Cornersare detectedto sub-pixel accuracy using the Harris cornerdetector[9].Given a cornerat position 'K)Ñ+C,1 in the first image,the searchfor a matchconsidersallcornerswithin aregioncentredon 'K)Ñ+C,1 in thesecondimagewith athresholdonmaximumdisparity. Thestrengthof candidatematchesis measuredby sumof squareddifferencesinintensity. The thresholdfor matchacceptanceis deliberatelyconservative at this stagetominimiseincorrectmatches.Becausethematchingprocessis only basedonproximity andsimilarity, mismatcheswill oftenoccur. Thesearesufficientto renderstandardleastsquaresestimatorsuseless.Consequentlyrobust methodsmustbe adopted,which canprovide agoodestimateof thesolutionevenif someof thedataaremismatches(outliers).
Thesolutionfor M canbeobtainedfrom thetwo dimensionalnullspaceof
þ. Let � 7 and� 9 beobtainedfrom thetwo right handsingularvectorsof
þwith singularvaluesof zero,
thusthey form anorthogonalbasisfor thenull space.Let 7 and 9 bethe U TsU matricescorrespondingto � 7 and � 9 . ThenthethreefundamentalmatricesM� , �Ñ%^/b+ Ì +CU consistentwith
þcanbeobtainedfrom M %� � 7 <V'I/ ¼ D1� 9 , subjectto ascalingandtheconstraintXY0ZÃ[ M]\ % O (which givesa cubicin from which 1 or 3 realsolutionsareobtained).The
supportfor this fundamentalmatrix is determinedby the numberof correspondencesintheinitial matchsetwith error ´ (givenin (8)) below a threshold� . Theerrorusedis thenegativelog likelihoodwhichis derivedin thelastsection.If therearethreesolutions,theneachis testedfor support. This is repeatedfor many randomsets,andthe fundamentalmatrix with thelargestsupportis accepted.Theoutputis a setof cornercorrespondencesconsistentwith thefundamentalmatrix,anda setof mismatches(outliers).
For projectivitieseachcorrespondenceprovidestwo constraintson theparameters:� 3 7�� %PO and � 39�� %ÒO (12)
where � 7 %�� ) ,³/ OQOQO ¼ )ç)�5 ¼ ,�)Û5 ¼ )Û5��� 9 % � OQOQOQ) , / ¼ )ç, 5 ¼ ,a, 5 ¼ , 5��and � is thecorrespondingvectorof theelementsof R . Thusfour pointsmaybe used
to find an exact solution. ��� �!����� proceedsin muchthe samemanner, with minimalsetsof four correspondencesbeingrandomlyselected,andeachsetgeneratinga putativeprojectivity. Thesupportfor eachsetis measuredby calculatingthenegativelog likelihoodfor all the points in the initial matchset, and countingthe numberof correspondenceswith errorbelow a certainthresholddeterminedby considerationof the inlier andoutlierdistributions.
To estimateaquadratictransformationfrom sevencorrespondencesthemethodusedforgeneratingfundamentalmatricesis modified. A critical surfaceis a ruledquadricpassingthroughbothcameracentres.Sevencorrespondencesdefinea quadricthroughthecameracentres. If it is ruled thentherewill be threereal fundamentalmatricesM 7 , M 9 and M��formedfrom thedesignmatrix
þgivenin (11) of thesevenpoints. Thesematricescanbe
usedto generatethecritical surface.In thiscase,any two of thefundamentalmatricesmay
MLESAC: A NEW ROBUST ESTIMATOR 7
becombinedto givethequadratictransformationby usingEquation(4) (it doesnotmatterwhich two asany pair givesthesameresultasany otherpair). If only onesolutionis realthenanothersamplecanbetaken.
Howmanysamplesshouldbeused?. Ideallyeverypossiblesubsampleof thedatawouldbe considered,but this is usuallycomputationallyinfeasible,so an importantquestionishow many subsamplesof thedatasetarerequiredfor statisticalsignificance.FischlerandBolles[4] andRousseeuwandLeroy [22] proposedslightly differentmeansof calculation,but eachpropositiongivesbroadlysimilar numbers.Herewe follow thelatter’s approach.Thenumber� of samplesis chosensufficiently high to give a probability � in excessof�! #"
thata goodsubsampleis selected.Theexpressionfor this probability � is�Ò%Q/ ¼ '-/ ¼ '-/ ¼%$ 1'&b1�(�+ (13)
where $ is thefractionof contaminateddata,and ) thenumberof featuresin eachsample.Generallyit is better to take more samplesthan are neededas somesamplesmight bedegenerate.It canbeseenfrom this that, far from beingcomputationallyprohibitive, therobustalgorithmmayrequirefewer repetitionsthanthereareoutliers,asit is not directlylinkedto thenumberbut only theproportionof outliers. It canalsobeseenthatthesmallerthedatasetneededto instantiateamodel,thefewersamplesarerequiredfor agivenlevelofconfidence.If thefractionof datathatis contaminatedis unknown,asis usual,aneducatedworstcaseestimateof the level of contaminationmustbemadein orderto determinethenumberof samplesto be taken, this can be updatedas larger consistentsetsare foundallowing the algorithmto “jump out” of �ü��� ���s� e.g.if the worst guessis
O " andasetwith *AO " inliers is discovered,then $ couldbereducedfrom
O " toÌ O " . Generally,
assumingno morethan O " outliersthen500randomsamplesis morethansufficient.
5. THE ROBUST ESTIMATOR: �W���ü�����The �ü��� �.��� algorithmhasprovenvery successfulfor robustestimation,but having
Oneof theproblemswith ��� �!����� is thatif thethreshold� for consideringinliers issettoohigh thentherobustestimatecanbeverypoor. Considerationof �ü���!����� showsthatin effect it findstheminimumof a costfunctiondefinedas, % µ ?.- � ´ 9? � (14)
where- '�1 is
- 'q´ 9 1�%0/ O ´29213�é9constant ´29243�é9�F (15)
In otherwordsinliers scorenothingandeachoutlier scoresa constantpenalty. Thusthehigher ��9 is themoresolutionswith equalvaluesof
,tendingto poorestimatione.g.if �
weresufficiently largethenall solutionswouldhavethesamecostasall thematcheswouldbe inliers. In Torr andZisserman[34] it wasshown thatat no extra costthis undesirablesituationcanberemedied.Ratherthanminimizing
This is a simple, redescendingM-estimator[13]. It can be seenthat outliers are stillgiven a fixed penaltybut now inliers are scoredon how well they fit the data. We set�|%®/AF �!5 Î so thatGaussianinliers areonly incorrectlyrejectedfive percentof the time.Theimplementationof this new method(dubbed� ����� m-estimatorsampleconsensus)yields a modestto hefty benefit to all robust estimationswith absolutelyno additionalcomputationalburden. Oncethis is understoodthere is no reasonto use �ü� �!����� inpreferenceto this method.Similar schemesfor robustestimationusingrandomsamplingandM-estimatorswerealsoproposedin [29] and[25].
Thedefinitionof themaximumlikelihooderrorallowsusto suggesta further improve-mentover � ����� . As theaimis to minimisethenegativelog likelihoodof themixture ¼�+thenit makessenseto usethis asthescorefor eachof therandomsamples.Theproblemis thatthemixing parameterÊ is notdirectlyobserved. But givenany putativesolutionfortheparametersof themodelit is possibleto recover Ê thatprovidestheminimum ¼�+ , asthis is aonedimensionalsearchit provideslittle computationaloverhead
To estimateÊ , usingExpectationMaximization( � � ), asetof indicatorvariablesneedsto be introduced: 6b? , B]% /�F0F0F-H , where 6b?�%®/ if the B th correspondenceis an inlier, and6 ? % O if the B th correspondenceis an outlier. The ��� algorithmproceedsas followstreatingthe 6b? asmissingdata[5]: (1) generatea guessfor Ê , (2) estimatetheexpectationof the 6b? from the currentestimateof Ê , (3) make a new estimateof Ê from the currentestimateof 6 ? andgoto step(2). Thisprocedureis repeateduntil convergenceandtypicallyrequiresonly two or threeiterations.
In moredetailfor stage(1) theinitial estimateof Ê is 79 . For stage(2)denotetheexpectedvalueof 6b? by 72? thenit followsthat
and) < is thelikelihoodof a datumgiventhatit is anoutlier:)�<�%|'I/ ¼ Ên1 /Ó F (20)
For stage(3) ÊS% /H µ ? 7 ? F (21)
This methodis dubbed�W���ü����� (maximumlikelihoodconsensus).For realsystemsitis sometimeshelpful to put a prior on Ê , the expectedproportionof inliers, this depends
on theapplicationandis not pursuedfurtherhere.Thetwo algorithmsaresummarizedinTable1. Theoutputof MLESAC (aswith RANSAC) is aninitial estimateof therelation,togetherwith a likelihoodthateachcorrespondencesis consistentwith the relation. Thenext stepis to improvetheestimateof therelationusinga gradientdescentmethod.
6. NON-LINEAR MINIMIZA TION
Themaximizationof thelikelihoodis a constrainedoptimisationbecausea solutionforM , f or R is soughtthatenforcestherelationsbetweentheelementsof theconstraint.Ifa parametrizationenforcestheseconstraintsit will be termedconsistent. In thefollowingweintroduceaconsistentparametrizationanddescribevariationswhichresultin aminimalparametrization.A minimal parametrizationhasthe samenumberof parametersas thenumberof independentelements(degreesof freedom)of the constraint.Theadvantagesanddisadvantagesof suchminimalparametrizationswill bediscussed.
Thekey ideais to usethepointbasisprovidedby therobustestimatorastheparametriza-tion. For thesimplestcase,theprojectivity,afourpointbasisisprovided.By fixing )�+-, andvarying )Û5q+-,5 for eachcorrespondence,elementsof theprojectivity maybeparametrizedintermsof the4 correspondencesanda standardgradientdescentalgorithm[7] canbecon-ductedwith )Û5�+-,5 asparameters.Notethisparametrizationhasexactly8 DOF(2 variablesfor eachof the 4 correspondences).Anotherapproachis to alter all the 16 coordinates,the non-linearminimization conductedin this higher dimensionalparameterspacewilldiscardextraneousparametersautomatically. This approachhasthe disadvantagethat itrequiresan increasednumberof function evaluationsas therearemoreparametersthandegreesof freedom.Similarly, 7 pointsmaybeusedto encodethefundamentalmatrixandthe parametersso encodedareguaranteedto be consistent,i.e. their elementssatisfythenecessaryconstraints(Sometimesthe7 pointsmayprovide threesolutions,in which casetheonewith lowesterror is used).This methodof parametrizationin termof pointswasfirst proposedin Torr andZisserman[33].
A numberof variationson thefree/fixedpartitionwill now bediscussed,aswell ascon-straintsonthedirectionof movementduringtheminimisation.In all casestheparametriza-
10 TORRAND ZISSERMAN
tion is consistent,but maynot beminimal. Althougha non-minimalparametrizationoverparametrizesthe imageconstraint,the main detrimentaleffectsis likely to be the costofthenumericalsolutionandpoorconvergenceproperties.Theformeris oneof themeasuresusedto comparetheparametrizationsin Section7.
First theparametrizationsfor M aredescribed.Giventheminimalnumberof correspon-dencesthatcanencodeoneof the imagerelationsthreecoordinatescanbefixedandonevariede.g.we couldencodeM by sevencorrespondences'*)�?¡+-,b?�1�4 'K) $? +-, $? 1�B�% /�FGF0Fhg ,by fixing the )�+-,.+-) $ coordinatesof thesecorrespondencesthespaceof M is parametrizedby theseven , $ coordinates.This is referredto asparametrizationP1. TheparametrizationP1 for R and f fixes )Ñ+C, for theminimalbasissetandvaries) $ , , $ . Thisparametrizationis bothminimalandconsistent,but thedisadvantagefor M of this is thatshouldtheepipolarlinesin image2beparallelto the , axisthenthemovementof thesepointswill notchangeM .In orderto overcomethis disadvantagemethodP2 movescoordinatesin
_ `in a direction
orthogonalto theconstraintsurface(variety)definedby theimagerelation. Thedirectionof motion is illustratedin Figure2 for a two dimensionalcase. Hereandfor M thereisoneorthogonaldirectionto themanifold. In thecasesof R and f , methodP2moveseachcoordinatein )�+-,.+-) $ +-, $ in two directionsorthogonalto the manifold. Perturbingeachpoint in this spacethenhastwo degreesof freedom,so the parametrizationhas8 dof intotal (for R ), i.e. it is minimal. In fact,bothP1andP2 areminimal andconsistenthaving7 DOFfor M , 8 DOFfor R and14DOFfor f . A third methodP3 is now definedthatusesall thecoordinatesof thecorrespondencesencodingtheconstraintasparameters.Givinga 28 DOF parametrizationfor M and f , and16 for R . NotethatP3 is over parametrizedhaving moreparametersthandegreesof freedom.
By way of comparisonmethodP4 is the linearmethodfor eachconstraintandis usedasabenchmark.Furthermoreeachconstraintis estimatedusingstandardparametrizationsasfollows. Theprojectivity andquadratictransformationsareestimatedby fixing oneoftheelements(the largest)of thematrix. For theprojectivity this is minimal whereasit isnot for thequadratictransform.Thenon-linearparametrizationfixing thelargestelementis dubbedP5. P6 is Luong’sparametrizationfor thefundamentalmatrix. This is a 7 DOFparametrizationin termsof the epipolesandepipolarhomographydesignedby Luongetal [16], this is bothminimal andconsistent.
After applyingMLESAC, the non-linearminimizationis conductedusingthe methoddescribedin Gill andMurray [6], which is a modificationof the Gauss-Newton method.All the pointsareincludedin the minimization,but the effect of outliersareremovedasthe robust function placesa ceiling on the valueof their errors,(thus they do not affectthe Jacobianof the parameters),unlessthe parametersmove during the iteratedsearchto a value where that correspondencemight be reclassifiedas an inlier. This schemeallows outliersto bere-classedasinliers duringtheminimizationitself without incurringadditionalcomputationalcomplexity. This hasthe advantageof reducingthe numberoffalseclassifications,which might ariseby classifyingthe correspondencesat too early astage.
An advantageof themethodof Gill andMurrayis thatis doesnotrequirethecalculationof any secondorderderivativesorHessians.Furthermoreif thedataisoverparametrizedthealgorithmhasaneffectivestrategy for discardingredundantcombinationsof thevariables,andchoosingefficientsubsetsof directionto searchin parameterspace.Thismakesit idealfor comparingminimizationsconductedwith differentamountsof overparametrization.
MLESAC: A NEW ROBUST ESTIMATOR 11
Control Points
(a) (b)Direction of Perturbation Direction of Perturbation
(c) (d)
FIGURE 2
As amatchmaybeincorrect,it is desirablethat,if in thecourseof theestimationprocesswe discover that the corneris mismatched,we areable to alter this match. In order toachieve this we storefor eachfeaturenot only its match,but all its candidatematchesthat have a similarity scoreover a userdefinedthreshold. After eachestimationof therelation,in theiterativeprocessesdescribedabove,featuresthatareflaggedasoutliersarere-matchedto theirmostlikely candidatethatminimizesthenegative log likelihood.
Convergenceproblemsmight ariseif eitherthe chosenbasissetis exactly degenerate,or the dataasa whole aredegenerate.In the first casethe imagerelation  cannotbeuniquelyestimatedfrom thebasisset.To avoid this problemtherankof thedesignmatrix
þmatrix given by (11) canbe examined. If the null spaceis greaterthan2 (1 for R ),
which it surelywill begivendegeneratedata,thenthatparticularbasiscanbediscarded.Providedthe basispointsdo not becomeexactlydegeneratethenany basissetis suitablefor parametrizing .
In thesecondcase,shouldthedataasawholebedegeneratethenthealgorithmwill failto convergeto a suitableresult,the discussionof degeneracy is beyond the scopeof thispaperandis consideredfurtherin Torr et al. [35].
7. RESULTS
We have rigorouslytestedthevariousparametrizationson realandsyntheticdata.Twomeasuresarecompared:Thefirstassessestheaccuracyof thesolution.Thesecondmeasureis thenumberof costfunctionevaluationsmadei.e. thenumberof times i is evaluated.Inthecaseof syntheticdatathefirst measureisÎ & % @B8µ ? ·kj 9 º�»# · ? +C# · ?¤½H CEmln (22)
for thesetof inliers,where »# · ? is thepointclosestto thenoisefreedatum# · ? whichsatisfiestheimagerelation,# · ? is the B thpointin the o image,and j '�1 is theEuclideanimagedistance
12 TORRAND ZISSERMAN
betweenthepoints. This providesa measureof how far theestimatedrelationis from thetruedatai.e.we testthefit of ourcomputedrelationfrom thenoisydataagainsttheknowngroundtruth. In thecaseof realdatatheaccuracy is assessedfrom thestandarddeviationof theinliers Îqp % à6µ ? ´29?H ù ln F (23)
Firstexperimentsweremadeonsyntheticdatarandomlygeneratedin threespace;100setsof 100 3D pointsweregenerated.The pointsweregeneratedin the field of view 10-20focal lengthsfrom thecamera.Theimagedatawasperturbedby Gaussiannoise,standarddeviation /AF O , andthenquantizedto thenearest0.1pixel. Wethenintroducedmismatchedfeaturesto makeagivenpercentageof thetotal,between/ÃO and
O percent.With syntheticdatatheestimatecanbecomparedwith thegroundtruthasfollows: Thestandarddeviationof theerrorof theactualnoisefreeprojectionsof thesyntheticcorrespondencesto thefittedrelationis measured.Thisgivesagoodmeasureof thevalidity of eachmethodin termsofthegroundtruth.
A comparisonwasmadebetweentherobustestimatorslookingat thestandarddeviationof thegroundtrutherrorbeforeapplyingthegradientdescentstage,for variouspercentagesof outliers. The resultswerefoundto bedramaticallyimproved: a reductionof variancefrom 1.43 to 0.64 when estimatinga projectivity, suggestingthat MLESAC shouldbeadoptedinsteadof RANSAC. After the non-linearstagethe standarddeviation of thegroundtruth error
Î & dropsto 0.22. Figure 3 shows that estimatederror on syntheticdata(conformingto randomfundamentalmatrices)for four randomsamplingstylerobustestimators:RANSAC [4], LMS [22, 39], MSAC andMLESAC. It canbeseenthatMSACandMLESAC outperformthe other two estimators,providing a
¼ /GO " improvement.This is becausethe first two have a moreaccurateassessmentof fit, whereasLMS usesonly the median,andRANSAC countsonly the numberof inliers. For this exampletheperformanceof MSAC andMLESAC areveryclose,MLESAC givesslightly betterresultsbut at theexpenseof morecomputation(theestimationof themixing parameterÊ for eachputative solution). Thus the choiceof MLESAC or MSAC for a particularapplicationsdependson whetherspeedor accuracy is moreimportant.
Theinitial estimateof theseven(for afundamentalmatrixor critical surface)or four (foranimage-imagehomography/projectivity) pointbasisprovidedby stage2 is quiteclosetothe true solutionandconsequentlystage3 typically avoids local minima. In generalthenon-linearminimisationrequiresfar morefunctionevaluationsthantherandomsamplingstage. However, the numberrequiredvarieswith parametrization,and is an additionalmeasure(overvariance)on which to assesstheparametrization.
Fundamentalmatrix-syntheticdata.. The five parametrizationsfor M werecomparedalongwith thelinearmethodfor fitting theFundamentalmatrix. Theresultsaresummarisedin Table2. Luong’s methodP6 produceda standarddeviation of 0.32with anaverageof238functionevaluationsin thenon-linearstage.The28 DOFparametrizationP3 in termsof pointsdid significantlyworsein thetrials with anestimatedstandarddeviation of 0.53andan averageof 2787functionevaluations,whereasthe7 DOF P2 parametrizationdidsignificantly better with an estimatedstandarddeviation of 0.22 at an averageof 119function evaluations. It remainsyet to discover why the “orthogonal” parametrizationprovided the best results,but in generalit seemsthat movementperpendicularto the
MLESAC: A NEW ROBUST ESTIMATOR 13
10 20 30 40 50 60 700
0.5
1
1.5
2
2.5
3
3.5
4Comparison of Robust Estimators
Outlier Percentage
Err
or
Sta
nd
ard
De
via
tio
n
FIGURE 3
TABLE 2The DOF, average number of evaluations of the total cost function in the gradient
descentalgorithm,and thestandard deviation rZs (22) for theperfectsyntheticpointdatafor thefundamentalmatrix.
Method DOF Evaluations tvuP1 Vary w $ 7 120 0.34P2 OrthogonalPerturbation 7 119 0.22P3 Vary x , w , x $ , w $ 28 2787 0.53P4 Linear - - 0.85P5 Fix largestelementof y 7 260 0.54P6 Luong 7 238 0.32
Fundamentalmatrix-ChapelSequence. Figures4(a)-(b)show twoviewsof anoutdoorchapel,thecameramovesaroundthechapelrotatingto keepit in view. Thematchesfromthe initial crosscorrelationare shown in (c) and it can be seenthat they containa fairnumberof outliers. Thebasisprovidedby MLESAC is givenin (d). As theminimizationprogressesthebasispointsmoveonly afew hundredthsof apixeleach,but thesolutionwasmuchimproved,final inliersandoutliersareshown in (e)and(f); thestandarddeviationoftheinlying data
Î phavedecreasedfrom 0.67to 0.23.
Projectivity-syntheticdata.. Whenfitting theprojectivity theseveraldifferentparametriza-tions P1-P3,P5 wereused. In this case,the resultsof all parametrizationswerealmostidentical,with standarddeviationsof theerrorwithin OçF OJz pixelsof eachother. Thenum-berof functionevaluationsrequiredin thenon-linearminimizationwasonaverage124forthe 8 DOF orthogonalparametrization,comparedwith 457 for the 16 DOF. Hencethe 8
14 TORRAND ZISSERMAN
(a) (b) (c)
(d) (e) (f)
FIGURE 4
TABLE 3The DOF, average number of evaluations of the total cost function in the gradient
descentalgorithm,and thestandard deviation {Z| (22) for theperfectsyntheticpoint datafor thehomographymatrix.
DOF hasthe slight advantageof beingsomewhat faster. The lack of differencebetweenthe parametrizationsmight be explainedby the lack of complex constraintsbetweentheelementsof thehomographymatrix,which is definedup to a scalingby 9 elements.
Projectivity-CupData.. Figure 5 (a) shows the first and (b) the secondimageof acup viewed from a cameraundergoing cyclotorsionaboutits optic axis combinedwithan imagezoom. The matchesaregiven in (c), basis(d), inliers (e) outliers (f) for thisscenewhenfitting a projectivity. It canbeseenthatoutliersto thecyclotorsionareclearlyidentified. Thenonlinearstepdoesnot produceany new inliers astheMLESAC stephassuccessfullyeliminatedall mismatches,theerroron theinliers is reducedby �a� whentheimagecoordinatesin oneimagearefixedandthosein theothervaried.
MLESAC: A NEW ROBUST ESTIMATOR 15
(a) (b) (c)
(d) (e) (f)
FIGURE 5
TABLE 4The DOF, average number of evaluations of the total cost function in the gradient
descentalgorithm,and thestandard deviation �Z� (22) for theperfectsyntheticpoint datafor thequadratic transformation.
Quadratic Transformation-Modelhousedata.. Figure6 (a)(b) showsascenein whichacamerarotatesandtranslateswhilst fixatingonamodelhouse.Thestandarddeviationoftheinliersimprovesfrom 0.39aftertheestimationby MLESAC to 0.35afterthenon-linearminimization.Theimportantthingaboutthisimageis thatstructurerecoveryfor thisimagepairprovedhighlyunstable.Thereasonfor thisinstabilityisnotimmediatelyapparentuntilthe goodfit of the quadratictransformationis witnessed,indicatingthat structurecannot
16 TORRAND ZISSERMAN
be well recoveredfrom this scene. In fact the detectedcornersapproximatelylie nearaquadricsurfacewhich alsopassesthroughthecameracentres.This is shown in Figure7.
(a) (b) (c)
(d) (e) (f)
FIGURE 6
8. CONCLUSION
Within thispaperanimprovementover ���V������� : �A���]����� hasbeenshown to givebetterestimateson our testdata. A generalmethodfor constrainedparameterestimationhasbeendemonstrated,andit hasbeenshowntoprovideequalorsuperiorresultstoexistingmethods.In factfew suchgeneralpurposemethodsexist to estimateandparametrizecom-plex quantitiessuchascritical surfaces.Thegeneralmethod(of minimalparametrizationintermsof basispointsfoundfrom ����������� ) couldbeusedfor otherestimationproblemsin vision, for instanceestimatingthe Quadrifocaltensorbetweenfour images,complexpolynomialcurvesetc.The generalmethodologycould be usedoutsideof vision in anyproblemwhereminimal parametrizationsarenot immediatelyobvious,andthe relationsmaybedeterminedfrom someminimal numberof points.
Why doesthepointparametrizationwork sowell? Onereasonis thattheminimalpointset initially selectedby �A���]����� is known to provide a good estimateof the imagerelation(becausethereis a lot of supportfor this solution). Hencethe initial estimateofthepointbasisprovidedby ����������� is quitecloseto thetruesolutionandconsequentlythenon-linearminimisationtypically avoids local minima. Secondlytheparametrizationis consistentwhich meansthatduringthegradientdescentphaseonly imagerelationsthatmight actuallyarisearesearchedfor.
MLESAC: A NEW ROBUST ESTIMATOR 17
FIGURE 7
It hasbeenobservedthatthe �A�� �¡�¢�£ methodof robustfitting is goodfor initializingtheparameterestimationwhenthedataarecorruptedby outliers. In thiscasetherearejusttwo classto which a datummight belong,inliersor outliers.The ���� ]¡�¢�£ methodmaybe generalizedto the casewhenthe datahasarisenfrom a moregeneralmixture modelinvolving several classes,suchas in clusteringproblems. Preliminarywork to illustratethis hasbeenconducted[28].
AcknowledgementsThank you to the reviewers for helpful suggestions,and RichardHartley for helpful
discussions.
REFERENCES
1. P. Beardsley, P. H. S.Torr, andA. Zisserman.3Dmodelaquisitionfromextendedimagesequences.In B. Bux-ton andCipolla R., editors,Proc. 4th EuropeanConferenceon ComputerVision, LNCS1065,Cambridge,pages683–695.Springer–Verlag,1996.
2. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incompletedatavia the emalgorithm.J. R.Statist.Soc., 39B:1–38,1977.
3. O.D. Faugeras.Whatcanbeseenin threedimensionswith anuncalibratedstereorig? In G. Sandini,editor,Proc.2ndEuropeanConferenceon ComputerVision, LNCS588,SantaMargheritaLigure, pages563–578.Springer–Verlag,1992.®In this casethename ¯�°_±³²X´Zµ is still appropriateastheaim is still to maximizethe likelihood,now it is
theposteriorlikelihood
18 TORRAND ZISSERMAN
4. M. FischlerandR.Bolles.Randomsampleconsensus:aparadigmfor modelfitting with applicationto imageanalysisandautomatedcartography. Commun.Assoc.Comp.Mach., vol. 24:381–95,1981.
5. A. Gelman,J.Carlin,H. Stern,andD. Rubin. BayesianDataAnalysis. ChapmanandHall, 1995.
6. P. E. Gill andW. Murray. Algorithmsfor thesolutionof thenonlinearleast-squaresproblem. SIAMJ NumAnal, 15(5):977–992,1978.
8. C. Harris. TheDROID 3D vision system.TechnicalReport72/88/N488U,Plessey Research,Roke Manor,1988.
9. C.HarrisandM. Stephens.A combinedcornerandedgedetector. In Proc.Alvey Conf., pages189–192,1987.
10. R. I. Hartley. Estimationof relative camerapositionsfor uncalibratedcameras. In Proc. 2nd EuropeanConferenceon ComputerVision, LNCS588, SantaMargherita Ligure, pages579–587.Springer-Verlag,1992.
11. R. I. Hartley. Euclideanreconstructionfrom uncalibratedviews. In Proc. 2nd European-USWorkshoponInvariance, Azores, pages187–202,1993.
12. R. I. Hartley andP. Sturm. Triangulation.In DARPA Image UnderstandingWorkshop,Monterey, CA, pages957–966,1994.
13. P. J.Huber. Projectionpursuit.Annalsof Statistics, 13:433–475,1985.
14. K. Kanatani. StatisticalOptimizationfor GeometricComputation:TheoryandPractice. Elsevier Science,Amsterdam,1996.
15. M. KendallandA. Stuart.TheAdvancedTheoryof Statistics. CharlesGriffin andCompany, London,1983.
16. Q. T. Luong, R. Deriche,O. D. Faugeras,andT. Papadopoulo.On determiningthe fundamentalmatrix:analysisof differentmethodsandexperimentalresults. TechnicalReport1894,INRIA (SophiaAntipolis),1993.
17. Q. T. Luong and O. D. Faugeras.Determiningthe fundamentalmatrix with planes: Instability and newalgorithms.CVPR, 4:489–494,1993.
19. S.J.Maybank.Propertiesof essentialmatrices.Int. J. of Imaging SystemsandTechnology, 2:380–384,1990.
20. G.I. McLachlanandK. Basford.Mixture models: inferenceandapplicationsto clustering. MarcelDekker.New York, 1988.
21. V. Pratt.Direct leastsquaresfitting of algebraicsurfaces.ComputerGraphics, 21(4):145–152,1987.
22. P. J.Rousseeuw. RobustRegressionandOutlier Detection. Wiley, New York, 1987.
23. P.D. Sampson. Fitting conic sectionsto ‘very scattered’data: An iterative refinementof the Booksteinalgorithm.ComputerVision,Graphics,andImage Processing, 18:97–108,1982.
24. L. S.Shapiro.AffineAnalysisof Image Sequences. PhDthesis,OxfordUniversity, 1993.
25. C. V. Stewart. Bias in robust estimationcausedby discontinuitiesandmultiple structures.IEEE Trans.onPatternAnalysisandMachineIntelligence, vol.PAMI-19,no.8:818–833,1997.
29. P. H. S. Torr, P. A. Beardsley, andD. W. Murray. Robust vision. In J. Illingworth, editor, Proc. 5th BritishMachineVisionConference, York, pages145–155.BMVA Press,1994.
30. P. H. S.Torr andD. W. Murray. Outlier detectionandmotionsegmentation.In P. S.Schenker, editor, SensorFusionVI, pages432–443.SPIEvolume2059,1993.Boston.
31. P. H. S. Torr andD. W. Murray. Stochasticmotionclustering.In J.-O.Eklundh,editor, Proc.3rd EuropeanConferenceonComputerVision,LNCS800/801,Stockholm, pages328–338.Springer–Verlag,1994.
32. P. H. S. Torr andD. W. Murray. The developmentandcomparisonof robust methodsfor estimatingthefundamentalmatrix. Int Journal of ComputerVision, 24(3):271–300,1997.
MLESAC: A NEW ROBUST ESTIMATOR 19
33. P. H. S. Torr andA Zisserman.Robust parameterizationandcomputationof the trifocal tensor. Image andVisionComputing, 15:591–607,1997.
34. P. H. S. Torr and A. Zisserman. Robust computationand parametrizationof multiple view relations. InU Desai,editor, ICCV6, pages727–732.NarosaPublishingHouse,1998.
35. P. H. S.Torr, A Zisserman,andS.Maybank.Robustdetectionof degenerateconfigurationsfor thefundamentalmatrix. CVIU, 71(3):312–333,1998.
36. P. H. S. Torr, A. Zisserman,andD. W. Murray. Motion clusteringusingthe trilinear constraintover threeviews. In R. Mohr andC. Wu,editors,Europe-ChinaWorkshoponGeometricalModellingandInvariantsforComputerVision, pages118–125.Springer–Verlag,1995.
37. P.H.S.Torr. An assessmentof informationcriteria for motion modelselection. In CVPR97, pages47–53,1997.
38. W. Triggs. Thegeometryof projective reconstructioni: Matchingconstraintsandthe joint image. In Proc.5th Int’l Conf. on ComputerVision,Boston, pages338–343,1995.
39. Z. Zhang,R. Deriche,O. Faugeras,andQ. T. Luong. A robust techniquefor matchingtwo uncalibratedimagesthroughtherecovery of theunknown epipolargeometry. AI Journal, vol.78:87–119,1994.
40. Z. ZhangandO. Faugeras.3D DynamicSceneAnalysis. Springer-Verlag,1992.