Mar 23, 2016
Relative Attributes
Mingxia [email protected]
1. Introduction 2. Relative Attributes[1]
3. Discussion 4. Our Intent Work
Outline
[1] Devi Parikh and Kristen Grauman.Relative Attributes. ICCV 2011 Best Paper.
Traditional Recognition
Dog Chimpanzee Tiger ???Tiger
3
Attributes-based Recognition
FurryWhite
BlackBig
StripedYellow
StripedBlackWhite
Big
Attributes provide a mode of
communication between humans and
machines!
[Lampert 2009][Farhadi 2009][Kumar 2009][Berg 2010][Parikh 2010]…
Zero-shot learningDescribing objectsFace verificationAttribute discoveryNameable attributes…
4
Dog Chimpanzee Tiger
Binary or Relative Attributes?
“Smiling”=1 “Smiling”=0
???
6
Smiling less than Gao, more than Ge
Binary or Relative Attributes?
Donkey
Mule
Horse
Binary or Relative Attributes?
Attributes for Mule
8[Oliva 2001] [Ferrari 2007] [Lampert 2009] [Farhadi 2009] [Kumar 2009] [Wang 2009] [Wang 2010] [Berg 2010] [Branson 2010] [Parikh 2010] [ICCV 2011] …
Is furryHas four-legs
Has tail
Tail longer than donkeys’
Legs shorter than horses’
Binary
9
Is furryHas four-legs
Has tail
Tail longer than donkeys’
Legs shorter than horses’
Relative
10
Tail longer than donkeys’
Legs shorter than horses’
Is furryHas four-legs
Has tail
Relative Attributes
11
Enhanced human-machine communication
More informative
Natural for humans
1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent Work
Outline
Contents
Relative attributes◦ Allow relating images and categories to each other◦ Learn ranking function for each attribute
Novel applications◦ Zero-shot learning from attribute comparisons◦ Automatically generating relative image
descriptions
13
1) Relative Attributes Annotation
14
coast (C), forest (F), highway (H), inside-city (I), mountain (M), open-country(O), street (S) and tall-building (T)8 categories, 6 attributes
Attri
bute
2) Learning Relative Attributes
For each attribute , supervision is“open”
15
2) Learning Relative Attributes
Learn a scoring function
16
Learned parameters
Image features
that best satisfies constraints:
2) Learning Relative Attributes
17
Ranking SVM
Based on [Joachims 2002]
Image Relative Attribute Score
12
34
56
Rank Margin
2) Learning Relative Attributes
Wm
N : total categories ◦ N = S + U
S : ‘seen’ categories ◦ training images are provided relative attribute
relation such as “lions are larger than dogs, as large as tigers, but less large than elephants”
U : ‘unseen’ categories ◦ training images are not provided
3) Relative Zero-shot Learning
How to relate Seen categories and Unseen categories through relative attributes?
3) Relative Zero-shot Learning
For attribute m
If attribute m is not used to describe to be the mean of all training image
3) Relative Zero-shot Learning
-
Step1: Compute the attribute rank score:
Step2: Compute the mean and covariance of according to relative attribute description.
Step 3: Assign class label for using the Maximum Likelihood Method:
3) Relative Zero-shot Learning
3) Relative Zero-shot Learning
Need not use all attributes, or all seen categories.
23
Age:ScarlettCliveHugh Jared Miley
Smiling:
JaredMileyScarlett Clive Hugh
3) Relative Zero-shot Learning
Clive
Infer image category using max-likelihood
Can predict new classes based on their relationships to existing classes – without training images
24
Age:ScarlettCliveHugh
Jared Miley
HughCliveScarlettSmiling:
JaredMileySm
iling
Age
Miley
S
J H
4) Automatic Relative Image Description
Density
Conventional binary description: not dense
Dense: Not dense:
Novel image
25
more dense than less dense than
DensityNovel image
26
4) Automatic Relative Image Description
Experiments
Datasets
28
Outdoor Scene Recognition (OSR)[Oliva 2001]
8 classes 2700 images 6 attributes: open, natural, etc.
Public Figures Face (PubFig)[Kumar 2009]
8 classes 800 images11 attributes: white, chubby, etc.
Ranker vs. Classifier
++
+
–
– –Percentage correctly ordered
pairs Classifier Ranker
Outdoor scenes 80% 89%
Celebrity faces 67% 82%
29
Zero-shot learning◦ Binary attributes:
Direct Attribute Prediction [Lampert 2009]
◦ Relative attributes via classifier scores
Automatic image-description◦ Binary attributes
30
Baselines
Relative Zero-shot Learning
An attribute is more discriminative when used relatively
Binary attributes
Rel. att. (classifier)
31
Rel. att.(ranker)
Relative zero-shot learning
ProposedBinary attributes
Classifier score
32
Relative (ours):
More natural than insidecity Less natural than highway
More open than street Less open than coast
Has more perspective than highway Has less perspective than insidecity
Binary (existing):
Not natural
Not open
Has perspective
33
Automatic Relative Image Description
Relative (ours):
More natural than tallbuilding Less natural than forest
More open than tallbuilding Less open than coast
Has more perspective than tallbuilding
Binary (existing):
Not natural
Not open
Has perspective
34
Automatic Relative Image Description
Relative (ours):
More Young than CliveOwenLess Young than ScarlettJohansson
More BushyEyebrows than ZacEfron Less BushyEyebrows than AlexRodriguez
More RoundFace than CliveOwenLess RoundFace than ZacEfron
Binary (existing):
Not Young
BushyEyebrows
RoundFace
Automatic Relative Image Description
35
(Viggo)
1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent work
Outline
Relative attributes representation
Attribution relation learning
Discussion
1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent work
Outline
Learning attributes’ relation automatically
Learning attributes’ relation with noise
Attribute selection
Instance selection for attributes
Our Intent Work
1. Introduction 2. Relative Attributes 3. Discussion 4. Our Intent work
Outline
Thank You