Watch, Listen and Learn Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney
Post on 26-Jan-2016
25 Views
Preview:
DESCRIPTION
Transcript
Watch, Listen and Learn Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney
-Pratiksha Shah
What’s the problem
• We want to automatically annotate and index images (considering their ever growing number)
• Visual cues alone can be ambiguous (depend on lighting, and variety exhibited even by objects of the same kind)
Previous methods
• Previous work was focused on learning the association between visual and textual information
• Many researchers have worked on activity recognition in videos using only visual cues
• Some used co-training but in a different flavour (co-SVM, 2 visual views, 2 text views)
How we propose to solve it
• The two major factors are the features and approach.
• We want to use : {Visual information + Linguistic information + Unlabeled multi-modal data }
for the learning process using a co-training approach.• Visual and linguistic information are used as
separate cues and we expect they will complement each other during training
What is co-training • Training using 2 different (conditionally independent and
sufficient) views • First learn a separate classifier for each view • Most confident predictions of these classifiers are then
used to iteratively construct labeled training data• Change made: an unlabeled example is only labeled if a
pre-specified confidence threshold for that view is exceeded
• Used for classifying : Webpages (based on content and hyperlink views) Email ( based on header and body) object detection models
Co-training Algorithm: Algorithm: A classifier for each view using the labeled data with just the features for that
view.
– Loop following steps until there are no more unused unlabeled instances:1. Compute predictions and confidences of both classifiers for all of the
unlabeled instances.2. For each view, choose the m unlabeled instances for which its classifier hasthe highest confidence. For each such instance, if the confidence value is lessthan the threshold for this view, then ignore the instance and stop labelinginstances with this view, else label the instance and add it to the supervisedtraining set.3. Retrain the classifiers for both views using the augmented labeled data.
Text Feature
• Pre process text by removing stop-words • Stem the remaining words using Porter
stemming• Frequency of the resulting word stems
comprises of the final textual features. (“bag of words” representation)
Captioned images
Features used :• We want to capture overall texture and color
distributions in local regions• Texture – Gabor filter with 3 scales and 4
orientations • Color – Mean, Standard deviation and
skewness of per-channel RGB and lab color pixel values
Method (for captioned images)
• Divide each image into a 4 by 6 sized cells• Compute texture feature using Gabor filter for each • The resulting feature vector for each region is then
clustered using k-means • Each region is then assigned to one of the k-clusters
based on its closeness to cluster centroids • Final “bag of visual words” represents every image
with a vector of k values, each denoting number of regions of the image close to that value.
The University of Texas at Austin 10
Image FeaturesDivide images into 46 grid
Capture texture and colordistributions of each cell
into 30-dim vector
Cluster the vectors using k-Meansto quantize the features into a dictionary of visual words
Represent each image as histogram of visual words
[Fei-Fei et al. ‘05, Bekkerman & Jeon ‘07]
…QuickTime™ and a
TIFF (Uncompressed) decompressorare needed to see this picture.
N 30
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Example dataset
Results for captioned images
• Compare co-training to supervised SVM• Compare co-training to Semi-supervised EM• Compare co-training to Transductive SVM
Commented videos
• Features used:• we use features that describe both salient
spatial changes and interesting movements.• Maximize a normalized temporal laplacian
operation over spatial and temporal scale• HOG – 3x3x2 spatial temporal blocks,4-bin
HOG descriptor for every block => 72 element descriptor
Method (for commented videos)• Use spatio-temporal descriptor motion descriptor (Laptev)• To detect events, use significant local changes in image
values in both space and time.• Estimate the spatio-temporal extent of the detected events
by maximizing a normalized spatiotemporal Laplacian operator over both spatial and temporal scales
• A HOG(histogram of oriented gradients) is calculated at each interest point. The patch is partitioned into a grid with 3x3x2 spatio-temporal blocks, and four-bin HOG descriptors are then computed for all blocks and concatenated into a 72-element descriptor
• These descriptors are clustered to form a vocabulary
The University of Texas at Austin 15
Video FeaturesDetect Interest Points
Harris-Forstener Corner Detector for both spatial and temporal space
Describe Interest PointsHistogram of Oriented Gradients (HoG)
Create Spatio-Temporal VocabularyQuantize interest points to create 200
visual words dictionary
Represent each video as histogram of visual words
[Laptev, IJCV ‘05]
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture. …QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.N 72
Example dataset
Results for commented video• Compare co-training with supervised SVM for commented video dataset • Compare co-training with supervised SVM when commentary is not available
during testing
What does the future look like?
• Larger dataset + more categories for testing • Labeled data versus associated text • Already the results show that co-training gives
better results than existing semi-supervised and supervised methods.
Questions
?
top related