Page 1
Image Similarity and the Earth Mover’s Distance
Empirical Evaluation of Dissimilarity Measures for Color and TextureY. Rubner, J. Puzicha, C. Tomasi and T.M. Buhmann
The Earth Mover’s Distance as a Metric for Image Retrieval
Y. Rubner, C. Tomasi and J.J. Guibas
The Earth Mover’s Distance is the Mallows Distance: Some Insights from Statistics
E. Levina and P.J. Bickel
Learning-Based Methods in Vision - Spring 2007
Frederik Heger(with graphics from last year’s slides)
1 February 2007
Page 2
2 LBMV Spring 2007 - Frederik Heger [email protected]
How Similar Are They?Images from Caltech 256
Page 3
3 LBMV Spring 2007 - Frederik Heger [email protected]
Similarity is Important for …
• Image classification• Is there a penguin in this picture?• This is a picture of a penguin.
• Image retrieval• Find pictures with a penguin in them.• Image as search query
• Find more images like this one.
• Image segmentation• Something that looked like this was called penguin before.
Page 4
4 LBMV Spring 2007 - Frederik Heger [email protected]
Space Shuttle Cargo Bay
Image Representations: Histograms
Normal histogram Cumulative histogram
•Generalize to arbitrary dimensions•Represent distribution of features
• Color, texture, depth, …
Images from Dave Kauchak
Page 5
5 LBMV Spring 2007 - Frederik Heger [email protected]
Image Representations: Histograms
Joint histogram• Requires lots of data• Loss of resolution to
avoid empty bins
Images from Dave Kauchak
Marginal histogram• Requires independent features• More data/bin than
joint histogram
Page 6
6 LBMV Spring 2007 - Frederik Heger [email protected]
Space Shuttle Cargo Bay
Image Representations: Histograms
Adaptive binning• Better data/bin distribution, fewer empty bins• Can adapt available resolution to relative feature importance
Images from Dave Kauchak
Page 7
7 LBMV Spring 2007 - Frederik Heger [email protected]
EASE Truss Assembly
Space Shuttle Cargo Bay
Image Representations: Histograms
Clusters / Signatures• “super-adaptive” binning• Does not require discretization along any fixed axis
Images from Dave Kauchak
Page 8
8 LBMV Spring 2007 - Frederik Heger [email protected]
Distance Metrics
-
-
-
= Euclidian distance of 5 units
= Grayvalue distance of 50 values
= ?
x
y
x
y
Page 9
9 LBMV Spring 2007 - Frederik Heger [email protected]
Issue: How to Compare Histograms?
Bin-by-bin comparisonSensitive to bin size. Could use wider bins …
… but at a loss of resolution
Cross-bin comparisonHow much cross-bin influence is necessary/sufficient?
Page 10
10 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Heuristic Histogram Distance:
Minkowski-form distance (Lp)
Special Cases:
L1 Mahattan distance
L2 Euclidian Distance
L Maximum value distance
Page 11
11 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Heuristic Histogram Distance:Weighted-Mean-Variance (WMV)
Info:• Per-feature similarity measure• Based on Gabor filter image representation• Shown to outperform several parametric models
for texture-based image retrieval
Page 12
12 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Nonparametric Test Statistic:Kolmogorov-Smirnov distance (KS)
Info:• Defined for only one dimension• Maximum discrepancy between cumulative
distributions• Invariant to arbitrary monotonic feature
transformations
Page 13
13 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Nonparametric Test Statistic:Cramer/von Mises type statistic (CvM)
Info:• Squared Euclidian distance between distributions• Defined for single dimension
Page 14
14 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Nonparametric Test Statistic:
2
Info:• Very commonly used
Page 15
15 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Information-theory Divergence:Kullback-Leibler divergence (KL)
Info:• Code one histogram using the other as true
distribution• How inefficient would it be?• Also widely used.
Page 16
16 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Information-theory Divergence:Jeffrey-divergence (JD)
Info:• Similar to KL divergence• But symmetric and numerically stable
Page 17
17 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Ground Distance Measure:Quadratic Form (QF)
Info:• Heuristic approach• Matrix A incorporates cross-bin information
Page 18
18 LBMV Spring 2007 - Frederik Heger [email protected]
Overview: Similarity Measures
Ground Distance MeasureEarth Mover’s Distance (EMD)
Info:• Based on solution of linear optimization problem
(transportation problem)• Minimal cost to transform one distribution to the
other• Total cost = sum of costs for individual features
Page 19
19 LBMV Spring 2007 - Frederik Heger [email protected]
Summary: Similarity Measures
Page 20
20 LBMV Spring 2007 - Frederik Heger [email protected]
Earth Mover’s Distance
≠
Page 21
21 LBMV Spring 2007 - Frederik Heger [email protected]
Earth Mover’s Distance
≠
Page 22
22 LBMV Spring 2007 - Frederik Heger [email protected]
Earth Mover’s Distance
=
Page 23
23 LBMV Spring 2007 - Frederik Heger [email protected]
Earth Mover’s Distance
=
(amount moved) * (distance moved)
Page 24
24 LBMV Spring 2007 - Frederik Heger [email protected]
How EMD Works
All movements
(distance moved) * (amount moved)
(distance moved) * (amount moved)
* (amount moved)
n clusters
Q
P
m clusters
Page 25
25 LBMV Spring 2007 - Frederik Heger [email protected]
How EMD Works
Move earth only from P to Q
P’
Q’n clusters
Q
P
m clusters
Page 26
26 LBMV Spring 2007 - Frederik Heger [email protected]
How EMD Works
n clusters
Q
P
m clusters
P cannot send more earth than there is
P’
Q’
Page 27
27 LBMV Spring 2007 - Frederik Heger [email protected]
How EMD Works
n clusters
Q
P
m clusters
Q cannot receive more earth than it can hold
P’
Q’
Page 28
28 LBMV Spring 2007 - Frederik Heger [email protected]
How EMD Works
n clusters
Q
P
m clusters
As much earth as possiblemust be moved
P’
Q’
Page 29
29 LBMV Spring 2007 - Frederik Heger [email protected]
Color-based Image Retrieval
Jeffrey divergence
Quadratic form distance
Earth Mover Distance
χ2 statistics
L1 distance
Page 30
30 LBMV Spring 2007 - Frederik Heger [email protected]
Red Car Retrievals (Color-based)
Page 31
31 LBMV Spring 2007 - Frederik Heger [email protected]
Zebra Retrieval (Texture-based)
Page 32
32 LBMV Spring 2007 - Frederik Heger [email protected]
EMD with Position Encoding
without position
with position
Page 33
33 LBMV Spring 2007 - Frederik Heger [email protected]
Issues with EMD
• High computational complexity• Prohibitive for texture segmentation
• Features ordering needs to be known • Open eyes / closed eyes example
• Distance can be set by very few features.• E.g. with partial match of uneven distribution weight
EMD = 0, no matter how many features follow
Page 34
34 LBMV Spring 2007 - Frederik Heger [email protected]
Help From Statisticians
• For even-mass distributions, EMD is equivalent to Mallows distance• (for uneven mass distributions,
the two distances behave differently)• Trick to compute Mallows distance
• 1-D marginals give better classification results than joint distributions (experimental results)
• Get marginals from empirical distribution by sorting feature vectors
Page 35
35 LBMV Spring 2007 - Frederik Heger [email protected]
EMD Summary / Conclusions
• Ground distance metric for image similarity
• Uses signatures for best adaptive binning and to lessen impact of prohibitive complexity
• Can deal with partial matches
• Good performance for color/texture classification
• Statistical grounding
Page 36
36 LBMV Spring 2007 - Frederik Heger [email protected]
Last Slide
Comments?
Questions?