-
1Learning-Based Contour Detection & Contour-Based Object
DetectionLearning-based Contour Detection & Contour-based
Object Detection
Iasonas Kokkinos
21 January, 2011Visual Geometry Group, Oxford
Galen GroupINRIA-Saclay
Department of Applied MathematicsEcole Centrale de Paris
-
2Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Appearance descriptors and boundary detection
Coarse-to-fine inference (parsing)
Model learning
-
3Learning-Based Contour Detection & Contour-Based Object
Detection
Image ContoursObject/Surface Boundaries (edges)
-
4Learning-Based Contour Detection & Contour-Based Object
Detection
Image ContoursSymmetry axes (ridges/valleys)
-
5Learning-Based Contour Detection & Contour-Based Object
Detection
A biref anlaogy wtih txet
Waht mttares is waht hppaens on wrod bandouries
Mocpera iwht htsi
(compare with this)
Concrete evidence that our visual system employs boundary
detection
Contour-based approaches: shape matching, segmentation,
recognition,..
-
6Learning-Based Contour Detection & Contour-Based Object
Detection
How can we detect boundaries?Filtering approaches
Canny (1984), Morrone and Owens (1987), Perona and Malik
(1991),..
Scale-Space approaches
Tony Lindeberg `Edge Detection and Ridge Detection with
Automatic Scale Selection.’,
IJCV, 30(2), 117-156, (1998)
Witkin, A. P. "Scale-space filtering", IJCAI (1983)
Variational approaches
V. Caselles, R. Kimmel, G. Sapiro: Geodesic Active Contours.
IJCV22(1): 61-79 (1997)
K. Siddiqi, Y. Lauzière, A. Tannenbaum, S. Zucker: Area and
length minimizing flows
for shape segmentation. IEEE TIP 7(3): 433-443 (1998)
Gestalt-based approaches
Agnès Desolneux, Lionel Moisan, Jean-Michel Morel:
Meaningful
Alignments. International Journal of Computer Vision 40(1): 7-23
(2000)
M. Kass, A. Witkin and D. Terzopoulos, `Snakes: Active Contour
Models’, ICCV (1987)
-
7Learning-Based Contour Detection & Contour-Based Object
Detection
Learning-based approachesBoundary or non-boundary?
Use human-annotated segmentations
D. Martin, C. Fowlkes, J. Malik. "Learning to Detect Natural
Image Boundaries Using Local Brightness, Color and Texture
Cues", IEEE PAMI, 2004
S. Konishi, A.Yuille, J. Coughlan, S.C. Zhu, “Statistical Edge
Detection: Learning and Evaluating Edge Cues”, IEEE PAMI,
2003
Use human-annotated segmentations
-
8Learning-Based Contour Detection & Contour-Based Object
Detection
Progress during the last 40 years
Canny+ Hysteresis
Berkeley PB, ‘04
Berkeley gPb, ‘08
Humans
Prewitt, 1965
-
9Learning-Based Contour Detection & Contour-Based Object
Detection
θr
(x,y)
A closer look into gPb: featuresLocal features (Pb, 2004) Global
features (gPb, 2008)
N-Cuts eigenvectors
In specific:
-
10Learning-Based Contour Detection & Contour-Based Object
Detection
A closer look into gPb: classifierLogistic regression
-
11Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Appearance descriptors and boundary detection
Coarse-to-fine inference (parsing)
Model learning
-
12Learning-Based Contour Detection & Contour-Based Object
Detection
Wanted: `simple’ that `works well’ on
Learning
Given: Training set of feature-label pairs
`simple’: quantified by VC dimension, curvature,…
`works well’: quantified by loss criterion
-
13Learning-Based Contour Detection & Contour-Based Object
Detection
Logistic regression
Linear function:
Log-likelihood of training pair:
Loss function:
Optimization: Newton-Raphson (IRLS)
-
14Learning-Based Contour Detection & Contour-Based Object
Detection
At each round, add optimal pair
Anyboost
Additive form:
See training cost as function of
Steepest descent direction:
Find `closest’ to
Adaboost: exponential loss
sign weight
-
15Learning-Based Contour Detection & Contour-Based Object
Detection
Side-by-side
AnyboostLogistic regression
� Additive� Linear
� Summands: features � Summands: weak learners� Summands:
features
� fixed
� Summands: weak learners
� added `on the fly’
� Cost: minus label log likelihood � Cost: exponential loss
(Adaboost)
� : Coordinate descent� : Newton-Raphson
Connections: M. Collins, R. Schapire, Y. Singer `Logistic
Regression, AdaBoost and Bregman Distances’ COLT (2000)
-
16Learning-Based Contour Detection & Contour-Based Object
Detection
A compact combination
� Additive
� (linear part)
Goal: quick classification, using small (e.g. ) feature set.
� Remaining summands: weak learners (nonlinearities)
� Cost?
� : Newton-Raphson, at each iteration
� Slower, but off-line
-
17Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Boosting, Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Appearance descriptors and boundary detection
Coarse-to-fine inference (parsing)
Model learning
-
18Learning-Based Contour Detection & Contour-Based Object
Detection
Classifier
Loss
Cost function for training
Training set
additiveadditive
- but also potentially non-convex (local optimality)
- potentially better suited for the problem
non-additive: F-measure, Area Under Curve (AUC),…
M. Ranjbar, G. Mori and Y. Wang `Optimizing Complex Loss
Functions in Structured Prediction’ ECCV, 2010
T. Joachims, `A Support Vector Method for Multivariate
Performance Measures’, ICML, 2005
M. Jansche, `Maximum Expected F-Measure Training Of Logistic
Regression Models’, EMNLP, 2005
-
19Learning-Based Contour Detection & Contour-Based Object
Detection
F-measure
no reward for true negative decisions
Predicted label
Goal: deal with unbalanced datasets (many negative)
F-measure: geometric mean of precision and recall
false alarmstrue positives misses
precision recall
-
20Learning-Based Contour Detection & Contour-Based Object
Detection
F-measure approximation
predicted label
differentiable approximation
approximate F-measure
M. Jansche, ‘Maximum Expected F-Measure Training Of Logistic
Regression Models’, EMNLP, 2005
-
21Learning-Based Contour Detection & Contour-Based Object
Detection
function of responses
Anyboost
F-measure optimization via Anyboost
Previous iteration
Loss
Anyboost
Newton-Raphson for coefficients: Jansche’s paper
-
22Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Boosting, Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Appearance descriptors and boundary detection
Coarse-to-fine inference (parsing)
Model learning
-
23Learning-Based Contour Detection & Contour-Based Object
Detection
mom’s keychain
Sneaking into the fun room
dad’s keychaingrandma’s keychain
We know that dad cannot enter the fun room, either
Which key should we try?
Slide Credit: B. Babenko/T. Dietterich
-
24Learning-Based Contour Detection & Contour-Based Object
Detection
Multiple Instance Learning
Typical Learning Multiple Instance Learning
Slide Credit: K. Grauman
Typical Learning Multiple Instance Learning
Positive bag: at least one instance should be positiveNegative
bag: no instance should be positive
-
25Learning-Based Contour Detection & Contour-Based Object
Detection
Problem I: inconsistent orientation information
MIL and boundary detection
Problem II: inconsistent location information
rForm bag of image locations/orientations that can`support’
human boundary
Given orientation, location support:
Overall support for boundary at
-
26Learning-Based Contour Detection & Contour-Based Object
Detection
Anyboost for F-measure boosting (previous section)
function of
Loss
-
27Learning-Based Contour Detection & Contour-Based Object
Detection
function of
Loss
Anyboost for MIL & F-measure boosting
-
28Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Boosting, Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Appearance descriptors and boundary detection
Coarse-to-fine inference (parsing)
Model learning
-
29Learning-Based Contour Detection & Contour-Based Object
Detection
Weak learner selection
In all cases:
-
30Learning-Based Contour Detection & Contour-Based Object
Detection
Gains so far
0.7
0.8
0.9
1Effect of Training
Pre
cisi
on
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.2
0.3
0.4
0.5
0.6
Recall
Pre
cisi
on
Global PB, F = 0.697MIL + Full training set, F = 0.704MIL + Full
training set + Boosting, F = 0.711
-
31Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Boosting, Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Appearance descriptors and boundary detection
Coarse-to-fine inference (parsing)
Model learning
-
32Learning-Based Contour Detection & Contour-Based Object
Detection
Appearance Descriptors
Dense descriptors (DAISY-like)
Multi-scale Gaussian & Gabors, Infinite Impulse Response
implementations
Goal: capture context for boundary detection
-
33Learning-Based Contour Detection & Contour-Based Object
Detection
Discriminative dimensionality reduction
Squeeze discriminative information out of high-dimensional
descriptor
LDA: only 1-D (2 class separation)
Large Margin Nearest Neighbors, Neighborhood Component Analysis,
...
iterative, work with
-
34Learning-Based Contour Detection & Contour-Based Object
Detection
SAVEPCA
PCA vs SAVE (for SIFT features)
34
Projections from SAVE
Sca
le 1
Sca
le 2
Sca
le 3
Projections from PCA
-
35Learning-Based Contour Detection & Contour-Based Object
Detection
Dense descriptor projections
-
36Learning-Based Contour Detection & Contour-Based Object
Detection
Overall gains
0.7
0.8
0.9
1Effect of features
Pre
cisi
on
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.2
0.3
0.4
0.5
0.6
Recall
Pre
cisi
on
Global PB, F = 0.697MIL + Full training set + Boosting, F =
0.711DoG + LoG + Gabor, F = 0.719Context + DoG + LoG + Gabor, F =
0.726
-
37Learning-Based Contour Detection & Contour-Based Object
Detection
PComparisons with gPb
Glo
bal
Pb
P>.
5
-
38Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Boosting, Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Coarse-to-fine inference (parsing)
Model learning
Appearance descriptors and boundary detection
-
39Learning-Based Contour Detection & Contour-Based Object
Detection
Where can contours be useful?
Recognition?
But contours are highly redundant(only junctions/corners/endings
matter)
Attneave 1967
Contours carry most of the image information
But corners/blobs/junctions are hard to group post-hoc
-
40Learning-Based Contour Detection & Contour-Based Object
Detection
Parts
Object
Hierarchical Compositional Models
Contours
Tokens
Iasonas Kokkinos and Alan Yuille,Inference and Learning with
Hierarchical Shape Mode lsInt.l Journal of Computer Vision (IJCV),
to appear
-
41Learning-Based Contour Detection & Contour-Based Object
Detection
View production rules as composition rules
Build a parse tree for the object
Image ObjectParse Tree
Compositional Object Detection
-
42Learning-Based Contour Detection & Contour-Based Object
Detection
Composition of the `back’ structure
Problem: Too many options!(Combinatorial explosion)
-
43Learning-Based Contour Detection & Contour-Based Object
Detection
• A* Search
Exit
Cost so far
Cost to go
Heuristic cost
A* for object parsing
• How can we extend A* to parsing?– `The Generalized A*
Architecture’, P. Felzenszwalb and D. McAllester, JAIR, 2007
• How can we apply A* parsing to object detection?– ‘HOP:
Hierarchical Object Parsing’, I. Kokkinos and A. Yuille, CVPR
2009
43
Entry
-
44Learning-Based Contour Detection & Contour-Based Object
Detection
Heuristics to Fine Level
Bottom-Up
Top-Down
Coarse-level parsing
-
45Learning-Based Contour Detection & Contour-Based Object
Detection
Top-Down Guidance: Heuristic, Coarse Level
Fine-level parsing
Bottom-Up Composition, Fine level
-
46Learning-Based Contour Detection & Contour-Based Object
Detection
• A* Parsing
Coarse Level
Front Part Middle Part Back Part Object Goal
A* vs Knuth’s Lightest Derivation (DP)
• KLD Parsing (only fine level)
Fine Level
-
47Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Boosting, Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Coarse-to-fine inference(parsing)
Model learning
Appearance descriptors and boundary detection
-
48Learning-Based Contour Detection & Contour-Based Object
Detection
• Input: a set of unregistered images containing object
• Output: a hierarchical model and parsing cost criterion
Learning problem
• Learning pipeline– Contours– Parts– Cost
-
49Learning-Based Contour Detection & Contour-Based Object
Detection
X S(X)
Deformable model
• Active Appearance Models
• Edges/ridges: throw away appearance variation
-
50Learning-Based Contour Detection & Contour-Based Object
Detection
sT
M: UpdateE: Deform
Edges & RidgesInput Images
AAM Learning:
Learning deformable models
S
T
AAM Fit
I. Kokkinos and A. Yuille, Unsupervised Learning of Object
Deformation Models, ICCV 2007
-
51Learning-Based Contour Detection & Contour-Based Object
Detection
Recovering object contours
-
52Learning-Based Contour Detection & Contour-Based Object
Detection
Recovering object contours- ETHZ Shapes
-
53Learning-Based Contour Detection & Contour-Based Object
Detection
Recovering object parts
Perceptual grouping-based graph
Affinity propagation results
-
54Learning-Based Contour Detection & Contour-Based Object
Detection
Recovering object parts – ETHZ Shapes
-
55Learning-Based Contour Detection & Contour-Based Object
Detection
• Goal: learn cost that leads to accurate detection
Parts
Object
Discriminative cost training
– But, no manual annotations to train with– Sole information:
Class labels
Parts
Contours
Tokens
-
56Learning-Based Contour Detection & Contour-Based Object
Detection
Parses as hidden dataMIL-based formulationPositive bag Negative
bag
-
57Learning-Based Contour Detection & Contour-Based Object
DetectionImprovements in parsing
Round 2 Round 6
Improvement of cost function: better parsing
P. Gehler and O. Chapelle, Deterministic Annealing for Multiple
Instance Learning, AISTATS, 2007
-
58Learning-Based Contour Detection & Contour-Based Object
Detection
Improvement of cost function: better localization
Round 2 Round 6
-
59Learning-Based Contour Detection & Contour-Based Object
Detection
Parsing and localization results
-
60Learning-Based Contour Detection & Contour-Based Object
Detection
Benchmark results
-
61Learning-Based Contour Detection & Contour-Based Object
Detection
Failure casesFront-end failures
Missing appearance information/poor shape model
-
62Learning-Based Contour Detection & Contour-Based Object
Detection
Talk outlineBoundary Detection (35’)
Logistic regression and Anyboost
F-measure Boosting
MIL and boundary detection
Monte Carlo approximations for large-scale datasets
Object Detection (15’)
Monte Carlo approximations for large-scale datasets
Appearance descriptors and boundary detection
Coarse-to-fine inference (parsing)
Model learning
Conclusions (1’)
-
63Learning-Based Contour Detection & Contour-Based Object
Detection
Conclusion
It is not the same, indeed
Results: it is not too different
Future work: make it closer
combine contours and appearance descriptors
structured statistical models for shape
integrate segmentation (symmetry)
-
64Learning-Based Contour Detection & Contour-Based Object
Detection
Thank you
Acknowledgements
M.Bronstein: slide template