Semi-Automatic Semi-Automatic Image Annotation Image Annotation Liu Wenyin, Susan Dumais, Yanfeng Sun, HongJiang Zhang, Mary Czerwinski and Brent Field Microsoft Research
Jan 20, 2016
Semi-Automatic Semi-Automatic Image AnnotationImage Annotation
Liu Wenyin, Susan Dumais, Yanfeng Sun, HongJiang Zhang, Mary Czerwinski
and Brent FieldMicrosoft Research
Outline
Introduction: What, Why, and How
Our Approach: Semi-Automatic Processes and Algorithms
Automated Performance Evaluation
Usability Studies
Concluding Remarks
What it is and Why
Image Annotation is a process of labeling images with keywords to describe semantic content
For image indexing and retrieval in image databases
Annotated images can be found more easily using keyword-based search
Image Annotation Approaches
Totally Manual Labeling (Gong et al., 1994) Enter keywords when image is loaded/registered/browsed Accurate but labor-intensive, tedious, and subjective
Direct Manipulation Annotation (Shneiderman and Kang 2000) Drag and drop keywords (from a predefined list ) onto image Still manual, also limited to predefined keywords (can’t be many)
Automatic Approaches: Efficient but less reliable and not always applicable compared to human annotation---how to grab this when no text context? By Image Understanding/Recognition (Ono et al. 1996) By Associating with environmental text (Shen et al. 2000; Srihari et al.
2000; Lieberman 2000)
Our Proposed Approach
Semi-Automatic Approach User provides initial query and relevance feed back. Feedback used to “semi-automatically” annotate ima
ges Trade-off between manual and automatic Achieve both accuracy and efficiency Increase productivity
Employ Content-Based Image Retrieval (CBIR), text matching, and Relevance Feedback (RF)
CBIR and RF Process and Framework
Image Retrieval and Relevance Feedback System (IRRFS)
Image Browser Query
Interface
Image Retrieval and Relevance
Feedback Algorithm Module
Feedback
Interface
UI
User
Algorithms for Matching
Visual Similarity Measurement Features: color histogram/moments/coherence, Tam
ura coarseness, pyramid wavelet texture, etc Distance model: Euclidean distance
Semantic (Keywords) Similarity Measurement Features: keyword vectors, TF*IDF Metrics: dot product and cosine normalization
Overall similarity: weighted average of the above two
Algorithms to Refine Search
Image Relevance Feedback Algorithms There are many algorithms can be used Cox et al. (1996) Rui and Huang (2000) Vasconcelos and Lippman (1999)
Lu et al. 2000 is employed in MiAlbum for text and images Modified Rocchio’s Formula Uses both semantics (keywords) and image-based
features during relevance feedback
Semi-Automatic Annotation During Relevance Feedback
In each keyword-query search cycleWhen positive and negative examples provided, Increase the weight of the keyword for all positive
examples Decrease the weight of the keyword for all negative
examples Relevance feedback algorithm refines and puts more
relevant images in top ranks for further selection as positive examples
Repeat the feedback process
Possible Future Automatic Annotation
When a new image is added…
Find top N similar images using image metrics
Most frequent keywords among annotations of these top N similar images are potential annotations, and could be automatically added with low weight or presented to user as potential annotations
TBD--Need to be confirmed in further RF process
Automated Performance Evaluation
Test Ground Truth Database 12,200 images in 122 categories from Corel DB Category name is ground truth annotation
Automatic Experimental Process Use category name as query feature for image retrieval Among first 100 retrieved images, those belonging to this
category are used as positive feedback examples others as negative
Performance Metrics Retrieval accuracy and annotation coverage
Image retrieval accuracy and annotation coverage
0
10
20
30
40
50
60
70
80
90
1001 3 5 7 9 11 13
15
17
19
# of Feedback Iterations
Imag
e R
etri
eval
Acc
ura
cy/
An
no
tati
on
Co
ver
age
(%)
10% initialannotation
0% initialannotation
Usability Studies
Objectives 2 studies examined overall usability of MiAlbum The usability of the semi-automatic annotation strategy
Tasks Import pictures, annotate pictures, find pictures, and use
relevance feedback
Questionnaires including but not limited to Overall ease of entering annotations for images Impact of annotation on ease of searching for images Satisfaction of search refinement & relevance feedback
Questionnaire Results
Overall ease of entering annotations: 5.6/7.0Ease to search annotated photos: 6.3/7.0Intuitiveness of refining search: 4.1/7.0Other Comments Positive on “semi-automatic”: (1) When using the up
and down hands the software automatically annotated the photos chosen. (2) The ability to rate pictures on like/dislike and have the software go from there.
Negative: difficulties in understanding the feedback process and how the matching algorithm operated.
Concluding Remarks
A Semi-automatic Annotation Strategy Employing Available image retrieval algorithms and Relevance feedback
Automatic Performance Evaluation Efficient compared to manual annotation? More accurate than automatic annotation
Usability Studies Preliminary usability results are promising Need to improve the discoverability of the feedback process
and the underlying matching algorithm