IntroductionRelated WorkMethodology
Results and Discussion
Multiple Reflection Symmetry Detection viaLinear-Directional Kernel Density Estimation
M. Elawady1, O. Alata1, C. Ducottet1, C. Barat1, P. Colantoni2
1Universite de Lyon, CNRS, UMR 5516, Laboratoire Hubert Curien,Universite de Saint-Etienne, Jean-Monnet, F-42000 Saint-Etienne, France
2Universite Jean Monnet, CIEREC EA n0 3068, Saint-Etienne, France
17th International Conference on Computer Analysis of Images and Patterns
UMR • CNRS • 5516 • SAINT-ETIENNE
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 1 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 2 / 33
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 3 / 33
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Bilateral Symmetry
1Image from book: The Photographer’s Eye by Michael Freeman
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 4 / 33
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Bilateral Symmetry in Computer Vision
Medial Image Segmentation [1]
Aerial-based vehicle detection [2]
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 5 / 33
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Detection of Global Symmetries
Axis Legend: Strong, Weak
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 6 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 7 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
Baseline (Loy 2006) and its Successor (Mo 2011)
The general scheme (Loy and Eklundh 2006 [3]) consists of:
Disadvantages:
Depending mainly on the properties of hand-crafted features (i.e. SIFT).
For example: (smooth objects with noisy background)little feature points =⇒ lost symmetry.
(Mo and Draper 2011 [4]) proposed refinements in the general scheme.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 8 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
First Work (Cic 2014)
Instead of SIFT, the general idea (Cicconet et al. 2014 [5]) is extracting aregular set of wavelet segments with local edge amplitude and orientation.
Disadvantages:
Lacking neighborhood’s information inside the feature representation.
Depending on the scale parameter of the edge detector.
For example: (high texture objects with noisy background)inferior symmetrical info =⇒ incorrect symmetry.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 9 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
State-of-Art (Ela 2016)
(Single Symmetry) Investigating Cicconet’s edge features [5] within Loy’sscheme [3] by adding neighboring-pixel information.
(1) Mul�scale Edge Segment Extrac�on
(2) Triangula�on based on Local Symmetry Weights:
• Geometry Edge Orienta�ons (Cic)• Local Texture Histogram (Loy)
(3) Vo�ng Space for Peak Detec�on with Handling Orienta�on Discon�nuity.
θ
ρ0
π
Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 10 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 11 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Proposed Idea
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 12 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Symmetry Detection Algorithm: Main Steps
Input Image
d) Symmetry Selection
a) Feature ExtractionScale 1 Scale S
Mag
nit
ud
eO
rien
tati
on
His
togr
am
Point 1 Point 2 Point P
c) Kernel Density Estimator
b) Feature TriangulationEdge Magnitude
Symmetry Coefficient
Neighborhood Texture
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 13 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Multiscale Edge Segment Extraction I
Input Image
Scale 1
Scale 2
Scale S
Orientation 1
Orientation O/2
Orientation O
Amplitude Map
Orientation Map
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 14 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Multiscale Edge Segment Extraction II
A feature point pi and its local edge characteristics (J i , φi ) are extractedwithin each cell using a Morlet wavelet ψk,σ of constant scale σ andvarying orientation {φo , o = 1 . . .O}.
Amplitude Map Orientation Map
MaxAmplitude
CorrespondingOrientation
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 15 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Multiscale Edge Segment Extraction III
Neighboring textural histogram hi of size B:
hi (b) =∑
r∈D(pi )
J rδφb−φr , φb ∈ {bπ
B, b = 0 . . .B − 1, 8 ≤ B ≤ 32} (1)
where hi is l1 normalized and circular shifted respect to the maximummagnitude J i among the neighborhood window D(pi ).
0 36 72 108 1440
0.5
1
1.5
2
2.5
3
3.5
4
4.5#106
Magnitude Histogram
108 144 0 36 720
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hi)
0 36 72 108 1440
500
1000
1500
2000
2500
3000
Frequency Histogram
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 16 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Symmetry Triangulation I
(Textural Information) Symmetry degree of the two regions around i andj can be measured by comparing their corresponding local orientationhistograms hi and hj (reverse histogram of hj). Texture-based symmetrymeasure is given by:
d(i , j) =B∑
b=1
min(hi (b), hj(b)) (2)
108 144 0 36 720
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hp)
72 36 0 144 1080
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hq*)
1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Intersection
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 17 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Symmetry Triangulation II
(Edge Information) Semi-dense edge magnitude m(i , j) and mirrorsymmetry coefficient c(i , j) {similar to cosine distance} are defined as [5]:
m(i , j) = J iJ j (3)
c(i , j) = |τ iS(T⊥ij )τ j | (4)
where τ i = [cos(φi ), sin(φi )], and S(.) is a reflection matrix.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 18 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Symmetry Triangulation III
The candidate axis is parametrized by angle θn, and displacement ρn andhas pairwise symmetry weight ωn = ωi,j (i 6= j and σi = σj) is defined as:
ωn = ω(pi , pj) = m(i , j) c(i , j) d(i , j) (5)
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 19 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Kernel Density Estimation I
(Linear: Displacement) Linear kernel density estimator fl(.) is defined as
fl(x ; g) =1
Ng
N∑n=1
G(x − ρn
g) | G(u) =
1
(2π)12
e−12|u|2 (6)
where G(.) is a Gaussian kernel with bandwidth parameter g .
(Directional: Angle) Directional kernel density estimator fd(.) is defined as:
fd(y ; k) =C(k)
N
N∑n=1
L(yTµn; k) | L(x ; k) = ekx , C(k) =1
2πS(0, k)(7)
where L(.) is a von-Mises Fisher kernel with concentration parameter k, and
normalization constant C(k). S(.) is the modified Bessel function of the first
kind.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 20 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Kernel Density Estimation II
Thanks for the linear-directional density estimator fl,d(.) [6]. We define
the extended weighted version fl,d(.) as:
fl,d(x , y ; g , k) =C (k)
Ng
N∑n=1
ωnG (x − ρn
g)L(yTµn; k) (8)
y = [cos(θ), sin(θ)], µn = [cos(θn), sin(θn)]
assuming that linear and directional data are independent resulting dotproduct between accompanying kernels.
B
A2 A1 A3B
B
A2 A1 A3
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 21 / 33
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Kernel Density Estimation III
B
A2 A1 A3
Input Image
A2
A1
A3
Linear KDE fl (.)
[A1,A2,A3]B B
Directional KDE fd (.)
Edge Magnitude mn(.) Symmetry Coefficient cn(.) Neighborhood Texture dn(.)
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 22 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 23 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Evaluation Details
Datasets:PSUm dataset: Liu’s vision group proposed a symmetry groundtruth[7, 8] for Flickr images (# images = 142, # symmetries = 479) inECCV2010, CVPR2011 and CVPR2013.
NYm dataset: Cicconet et al. [9] presented a new symmetrydatabase (# images = 63, # symmetries = 188) in 2016.
Evaluation Metrics:True Positive [8]:
ang(SC ,GT ) < 10◦ (9)
dist(CenSC ,CenGT ) < 20%×min(LenSC , LenGT ) (10)
Precision, Recall, and Maximum F1 Score
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 24 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Quantitative Results I
(Precision-Recall Curves)
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1Loy2006 (0.29211)Cicconet2014 (0.15883)Elawady2016 (0.27744)Our2017 (0.32828)
PSUm (2010-2013)
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1Loy2006 (0.33657)Cicconet2014 (0.2365)Elawady2016 (0.38788)Our2017 (0.43373)
NYm (2016)
X-axis: Recall, Y-axis: Precision
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 25 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Quantitative Results II
(Statistical Comparison)
Max F1 Score and its equivalent Precision and Recall rates
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 26 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Qualitative Results I - PSUmColumns: (1) GT, (2) Our, (3) Ela2016 [10], and (4) Loy2006 [3]
Top 5 detections: red, yellow, green, blue, and magenta.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 27 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Qualitative Results II - NYmColumns: (1) GT, (2) Our, (3) Ela2016 [10], and (4) Loy2006 [3]
Top 5 detections: red, yellow, green, blue, and magenta.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 28 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Conclusion
Summary:1 A weighted joint density estimator is proposed to handle both orientation and
displacement information.
2 A reliable detection framework is developed for global multiple symmetries.
Future work:1 The proposed detection can be improved using a continuous maximal-seeking
technique to avoid over-extended axes.
2 Entropy-based balance measure can be introduced to describe the existence anddegree of global axes inside an image.
3 Possibility of integration within retrieval systems for artistic photographs andpaintings in museums
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 29 / 33
IntroductionRelated WorkMethodology
Results and Discussion
References I
[1] F. Abdolali, R. A. Zoroofi, Y. Otake, and Y. Sato, “Automatic segmentation ofmaxillofacial cysts in cone beam ct images,” Computers in biology and medicine,vol. 72, pp. 108–119, 2016.
[2] S. Ram and J. J. Rodriguez, “Vehicle detection in aerial images using multiscalestructure enhancement and symmetry,” in Image Processing (ICIP), 2016 IEEEInternational Conference on, pp. 3817–3821, IEEE, 2016.
[3] G. Loy and J.-O. Eklundh, “Detecting symmetry and symmetric constellations offeatures,” in Computer Vision–ECCV 2006, pp. 508–521, Springer, 2006.
[4] Q. Mo and B. Draper, “Detecting bilateral symmetry with feature mirroring,” inCVPR 2011 Workshop on Symmetry Detection from Real World Images, 2011.
[5] M. Cicconet, D. Geiger, K. C. Gunsalus, and M. Werman, “Mirror symmetryhistograms for capturing geometric properties in images,” in Computer Visionand Pattern Recognition (CVPR), 2014 IEEE Conference on, pp. 2981–2986,IEEE, 2014.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 30 / 33
IntroductionRelated WorkMethodology
Results and Discussion
References II
[6] E. Garcıa-Portugues, R. M. Crujeiras, and W. Gonzalez-Manteiga, “Kerneldensity estimation for directional–linear data,” Journal of Multivariate Analysis,vol. 121, pp. 152–175, 2013.
[7] I. Rauschert, K. Brocklehurst, S. Kashyap, J. Liu, and Y. Liu, “First symmetrydetection competition: Summary and results,” tech. rep., Technical ReportCSE11-012, Department of Computer Science and Engineering, ThePennsylvania State University, 2011.
[8] J. Liu, G. Slota, G. Zheng, Z. Wu, M. Park, S. Lee, I. Rauschert, and Y. Liu,“Symmetry detection from realworld images competition 2013: Summary andresults,” in Computer Vision and Pattern Recognition Workshops (CVPRW),2013 IEEE Conference on, pp. 200–205, IEEE, 2013.
[9] M. Cicconet, V. Birodkar, M. Lund, M. Werman, and D. Geiger, “A convolutionalapproach to reflection symmetry,” Pattern Recognition Letters, 2017.
[10] M. Elawady, C. Barat, C. Ducottet, and P. Colantoni, “Global bilateral symmetrydetection using multiscale mirror histograms,” in International Conference onAdvanced Concepts for Intelligent Vision Systems, pp. 14–24, Springer, 2016.
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 31 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Questions?
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 32 / 33
IntroductionRelated WorkMethodology
Results and Discussion
Appendix - Why Feature Normalization?
The feature points are normalized with keeping aspect ratio as following:
pi =pi − cW ,H
max(W ,H)(11)
where cW ,H represents the original image center (W2
, H2
).
Without Normalization With Normalization
Voting: unified space and independent parameters
M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 33 / 33