Point Correspondence Performance Evaluation - haralick.orgharalick.org/conferences/point_correspondence_metrics.pdf · Point Correspondence Performance Evaluation Robert M. Haralick
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.
......Point Correspondence Performance Evaluation
Robert M. Haralick
Computer Science, Graduate CenterCity University of New York
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Types of Performance Characterization
White BoxEvaluate Each Component: Component Transfer FunctionNeeds Appropriate Random Perturbation Models for
Algorithm InputsAlgorithm Outputs
Black BoxEmpirical EvaluationNo Knowledge of Component Transfer Functions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Performance Characterization
An Algorithm hasInputs a and their data typesOutputs b and their data typesA Relationship Between Input and Output
Output b as a function of input aGiven a, b maximizes F (a, b)
Random Perturbation Model for Input aRandom Perturbation Model for Output bGiven Random Perturbation Distribution Acting on aDetermine Random Perturbation Distribution Acting on bDetermine Robustness
Do large perturbations on a small fraction of the input datacause a small perturbation on the output data?
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Performance Characterization
Performance characterization has to do with establishing thecorrespondence of the random variations and imperfectionswhich the algorithm produces on the output data caused by therandom variations and the imperfections on the input data.
(Haralick, 1994)
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. System Performance Characterization
A system performance characterization has a scoring functionthat evaluates the goodness of the output. The systemperformance characterization gives the distribution of thescoring function value as a function of the parametersdescribing the input perturbation and the tuning parameters.
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Experimental Protocols
Population of ideal inputsSimple Random SamplingStratified Sampling
Parameters of random perturbation distribution affectinginputsTuning Parameter SettingsScoring FunctionFix Tuning Parameters
Estimate Scoring Function Distribution as a function ofPerturbation Parameters
Fix inputEstimate Scoring Function Distribution as a function ofTuning Parameters
.. Finding Matching PointsFinding Matching points is often posed as an optimizationproblem and uses sensor projection geometry constraints
Determine a Window Size
Maximize Normalized Cross-correlation
Minimize Normalized Distance
Minimum Description Length
Swarming
Simulated Annealing
Gradient Descent
Expectation Maximization
Mutual Information
Total Least Squares
Random Walks with Restart
SoftPosit
Energy Minimization
Dynamic Programming
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Performance Evaluation
Optimization and triangulation do not give a performanceevaluation.Performance Evaluation includes:
Estimating the Covariance of the position of each 3D pointThe rule for deciding whether or not to accept thecorrespondences associated with an estimated 3D pointThe resulting False Alarm - Misdetect Rate
.. Variety of Related Point Correspondence Problems
Simultaneous Pose and CorrespondenceSensors calibratedStructure from MotionRigidity CheckingWide Baseline Stereo CorrespondenceSelf-Consistency
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Calibrated Sensors
Ground Truth Point CorrespondencesPerspective Geometry
Standard Photogrammetric ProcedureProjective Bundle AdjustmentInterior OrientationExterior Orientation
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. The Multi-Image Point Correspondence Problem
There are N > 1 calibrated sensorsTrue but unknown sensor parameters θ1, . . . , θN
3D Point q whose position is not known2D Corresponding points x1, . . . , xN , the sensor projectionsof q to the N sensorsξ1, . . . , ξN random perturbations of 2D sensor projectionpointsEstimated sensor parameters θ̂1, . . . , θ̂N
Model:xn = Pn(q, θn)x̂n = Pn(q, θn) + ξn, n = 1, . . . ,Nξn has N(0,Σξn)
θ̂n has multivariate uniformθ̂1, . . . , θ̂N , ξ1, . . . , ξN are independent
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Bayesian Setting
In the Bayesian setting, true sensor parameters are consideredas random variables with independent a priori densities
p(q) is the prior for 3D point q, taking into account all the 3D points that havealready been triangulated and pn(θ̂n) is a multivariate uniform.
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Robustification
Objective Function
log p(q)− 12
N∑n=1
(x̂n − Pn(q, θ̂n))′Σxn (q, θ̂n)
−1(x̂n − Pn(q, θ̂n)) +N∑
n=1
log pn(θ̂n)
The perturbations are smallThe optimization is only correct if in fact each x̂n doescorrespond to the 3D point qBut sometimes the correspondence is not correctRobustify the objective function
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Covariance Propagation
The optimization provides an estimate q̂ of qThe covariance Σq needs to be estimatedThe consistency of q̂ with respect to x̂1, . . . , x̂N has to bechecked
(Haralick, 1994)
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Covariance Propagation
X = (x1, . . . , xN)
X̂ = (x̂1, . . . , x̂N)
F (q,X ) = 0Minimize F (q̂, X̂ )
G(q,X ) = ∂F∂q
Σq =
(∂G∂q
)−1 ∂G∂X
ΣX (∂G∂X
)′(∂G∂q
)′−1
∂G∂q (q̂, X̂ )
∂G∂X (q̂, X̂ )
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Covariance Propagation
x̃n reprojection of estimated 3D point
x̃n = Pn(q̂, θ̂n)
Σx̃n(q̂, θ̂) =∂Pn
∂θΣθ̂n
(∂Pn
∂θ
)′+
∂Pn
∂qΣq̂θ̂
(∂Pn
∂θ
)′
+∂Pn
∂θΣθ̂q̂
(∂Pn
∂q
)′+
∂Pn
∂qΣq̂
(∂Pn
∂q
)′
∂Pn∂q (q̂, θ̂)∂Pn∂θ (q̂, θ̂)
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Self Consistency Check
Empirically measure the predictive power of a score withrespect to a given algorithm, population of scenes and imagingconditions.Decide x̂n is a corresponding point if
(x̃n − x̂n)′(Σx̃n +Σξn)
−1(x̃n − x̂n) < τn
Acceptance rate: the fraction of corresponding point sets thatare close enough to the sensor projection of their estimated 3Dpoints.
If x̂n is decided as not a corresponding point, look near x̃ for acorresponding point.(Leclerc and Luong, 2003)
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Scene and Image Populations
Textureless RegionsTextured RegionsUnoccluded CornersAmount of NoiseDepth Discontinuities
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Performance Characterization
Specify a populationLabel the true corresponding points: Ground TruthUse automatic procedure for finding corresponding pointsEstimate the 3D pointsDetermine the projection of the 3D pointsConsider only accepted pointsMatching Error compared to Ground Truth
Cumulative Distribution of distance to true positionThreshold defining when distance is close enoughMisdetect and False alarm rate
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Misdetect False Alarm Rate
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Tuning Parameters
Every algorithm has tuning parameters.How the Tuning Parameters are set influencesperformanceSome Tuning Parameters are set internal to programSome Tuning Parameters can be user setDefault Settings vs Tuned SettingsWhat is the sensitivity of the result to the settings of thetuning parameters?
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Performance Surface vs Tuning Parameters
Ruggedness of SurfaceSmoothness of SurfaceNumber of Local OptimaRatio of Local Optima values to Global Optima
Crossley, Nisbet, and Amos (2013)
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Performance Surface vs Tuning Parameters
Set a threshold θ of minimum acceptable performanceDetermine a hyperbox having the property
The fraction f of tuning parameter values in hyperbox yieldperformance > θ
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Estimating Performance Hyperbox Boundaries
N ExperimentsChoose tuning parameter M-tuples at randomEvaluate PerformanceDetermine Hyperbox Boundaries
Tuning Parameters (α1n, . . . , αMn), n = 1, . . . ,NGoodness Function ΨAcceptable Set A(θ) = {n | Ψ(α1n, . . . , αMn) > θ}bm min = minn∈A αmnbm max = maxn∈A αmnH = ×M
m=1[bm min, bm max ]
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. Goodness Fraction
N ExperimentsChoose Tuning Parameter M-tuples at random in H
Find largest Hypercube HC ∈ H such that(γ1, . . . , γM) ∈ HC implies Ψ(γ1, . . . , γM) < θ
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. References
M. Unger and A. Stojanovic, “A new Evaluation Criterion For PointCorrespondences in Stereo Images”, Analysis, Retrieval and Delivery ofMultimedia Content, Lecture Notes in Electrical Engineering, Volume 158, 2013,pp. 183-202.J. Yu, L. McMillan and P. Sturm, “Multi-Perspective Modelling, Rendering andImaging,” Computer Graphics Forum, Volume 29, 2010, pp. 227-246.E. Juhàsz, A. Tanàcs, Z. Kato, “Evaluation of Point Matching Methods forWide-baseline Stereo Correspondence on Mobile Platforms,” 8th InternationalSymposium on Image and Signal Processing and Analysis. September, 2013,Trieste, ItalyY. Leclerc and Q. Luong, “Self-Consistency and MDL: A Paradigm for evaluatingPoint-Correspondence Algorithms and Its Application to Detecting Changes inSurface Elevation,” International Journal of Computer Vision, Volume 51, 2003,pp. 63-83.R. Haralick “Propagating Covariance In Computer Vision,” InternationalConference on Pattern Recognition,Vol 1, Jerusalem, Israel, 1994, pp. 493-498.A. Bedekar and R. Haralick, “A Bayesian Method for Triangulation and ItsApplication to Finding Corresponding Points,” International Conference on ImageProcessing, 1995, pp. 362-365.S. Yi, R. Haralick, L. Shapiro, “Error Propagation in Machine Vision,” MachineVision and Applications, Volume 7, pp 93-114.R. Haralick, “Performance Characterization Protocol In Computer Vision”,DARPA Image Understanding Workshop, Monterey CA 1994.
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. ReferencesR. Szeliski, Computer Vision: Algorithms and Applications, Springer-Verlag,London, 2011.y. Xiuxiao M. Yang, “A Novel Method for Multi-image Matching SynthesizingImage and Object-Space Information”, Geo-spatial Information Science, Volume12, 2009, pp. 157-164.P. Tipwai and S. Madarasmi, “A Coarse-and-Fine Bayesian Belief Propagationfor Correspondence Problems in Computer Vision, MICAI 2007, LNAI 4827, pp.683-693.S. Anvar, W. Yau, E Teoh, “Finding the Correspondence Poings in Images ofMulti-Views,” 8th International Conference on Signal Image Technology andInternet Based Systems, 2012, pp. 275-280.A. Usumezbas and B. Kimia, “Generating Dense Point CorrespondenceGround-Truth Across Multiple Views”, 2nd Joint 3DIM/3DPVT Converence: 3Dimaging Modeling, Processing, Visualization and Transmission, 2012, pp.214-221.F. Dellaert, S. Seitz, C. Thorpe, S. Thrun, “EM MCMC, and Chain Flipping forStructure from Motion with Unknown Correspondence”, Machine Learning,Volume 50, 2003, pp. 45-71.C. Bodensteiner, W. Huebner, K. Juengling, J. Mueller, and M. Arens, “LocalMulti-modal Image matching Based on Self-Similarity”, 17th InternationalConference on Image Processing, 2010, pp. 937-940.A. Ericsson and J. Karlsson, “Measures for Benchmarking of AutomaticCorrespondence Algorithms”, Mathematical Imaging Vision, Volume 28, 2007,pp. 225-241.
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. ReferencesM. Irani, “Multi-Frame Correspondence Estimation Using Subspace Constraints”,International Journal of Computer Vision, Volume 48, 2002, pp. 173-194.S. Chaudhuri and S. Chatterjee, “Performance Analysis of Total Least SquaresMethods in Three-Dimensional Motion Estimation”, IEEE Transactions onRobotics and Automation, Volume 7, 1991, pp. 707-714.T. Kim, K. Lee, and S. Lee, “A Probabilistic Model For Correspondence ProblemsUsing Random Walks With Restart”, ACCV, 2009, pp. 16-425.X. Li and Z. Hu, “Rejecting Mismatches by Correspondence Function”,International Journal Computer Vision, Volume 89, 2010, pp. 1-17.R. Sàra, “Robust Correspondence Recognition For Computer Vision”,Computational Statistics of 17th ERS-IASC Symposium, 2006.P. David, D. Dementron, R. Duraiswami, and H. Samet, “SoftPosit: SimultaneousPose and Correspondence Determination”, International Journal Of ComputerVision, Volume 59, 2004, pp. 259-284.D. Scharstein and R. Szeliski, “A Taxonomy and Evaluation of Dense Two-FrameStereo Correspondence Algorithms”, International Journal of Computer Vision,Volume 47, 2002, pp. 7-42.M. Crossley, A. Nisbet, and M. Amos, “Quantifyhing the Impact of ParameterTuning on Nature-Inspired Algorithms”, Bioinspired Learning and Optimization,2013, pp. 925-932.M. Greiffenhagen, D. Comaniciu, H. Niemann, V. Ramesh, “ Design, Analysisand Engineering of Video Monitoring Systems: An Approach and Case Study”,Proceedings of the IEEE, Volume 89, 2001, pp. 1498-1517.R. Haralick, “Propagating Covariance In Computer Vision”, International Journalof Pattern Recognition and Artificial Intelligence, Volume 10, 1996, pp. 561-572.
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
.. References
R. Haralick, “Propagating Covariance In Computer Vision”, International Journalof Pattern Recognition and Artificial Intelligence, Volume 10, 1996, pp. 561-572.