CS335 Principles of Multimedia Systems Content Based Media Retrieval Hao Jiang Computer Science Department Boston College Dec. 4, 2007.

CS335 Principles of Multimedia Systems

Content Based Media Retrieval

Hao Jiang

Computer Science Department

Boston College

Dec. 4, 2007


Introduction

With the increase of multimedia content on the web, we need methods to find the image, audio or video.

Text only schemes have limitations– A lot of manual work.– Inaccurate description.

Content based media retrieval studies how to– find media data using classification and recognition methods

based on “features” in audio, image and videos.– For example, we can use an exemplar (like a sample

picture) to find other similar ones.

Applications: – Digital library, personal image album and audio folders,

satellite image processing and medical applications.


An Image Retrieval Example (Viper)

The query input.


An Image Retrieval Example (Viper)

The query output.


User feedback.


Refined results. Better?


Another query for paintings.


Painting Search Result

The shortlist returned from the search.


Content Based Media Retrieval The input

– A text description.– One or more exemplar images, audio or video clips.– A sketch, e.g., a dark background with an orange disk in the

the middle used to search for sunset scenes.– Or the combination or them.

The output– A shortlist of images, or audio video clips.– The instances of the event you want to find in videos or

audios.– Can be structured into a web page or other documents.– Usually allows user feedback to improve the result.

The basic task in content based media retrieval is comparing and searching multimedia data.


How Do We Evaluate the Performance?

Precision and Recall– Precision = (# of relevant items) / (# of items retrieved)– Recall = (# of relevant items) / (Total # of related items

in the dataset)

The procedure of drawing a Recall-Precision Curve:– Compute the relevance score for each item in the database.– Sort the list.– Assume the sorted list is like

r r r n n r r r n n …

and we have total 6 relevant items in the database


The Recall-Precision Curve

1/6 2/6 3/6 4/6 5/6 1

1

Precision

Recall

Short list is like: r r r n n r r r n n …

Q: Why do not we just use a single value instead of a curve?


The “Best” Recall-Precision Curve

1

Precision

Recall1/(# of relevant items)

(# of relevant items)/(# of total items)

1


Image Retrieval Methods

To find images in a database, we have to compare images quantitatively based on “features”.

We can compare the images as a whole using features like:– Color, textures and their spatial layouts.

We can also segment images into regions and use similar features in object detection.

In some recent systems, people use salient features such as SIFT (Scale Invariant Transform) like features, learning and pattern recognition methods.


Color Histogram Methods

Color only schemes tend to find many unrelated images.http://amazon.ece.utexas.edu/~qasim/qdialog_IMGDATA2_v1_Birds_Swans.html


Improve Color Histogram Methods

If we can separate the foreground with background the result will be improved.

Foreground

Background


Improve Color Histogram Methods

Their spatial relations also help to find the right object.

ColorBlob 2

ColorBlob 1


Finding Shapes

Finding similar shapes is a very useful tool in managing large number of images.

Chamfer matching is a standard method to compare the similarity of shapes.

General Hough Transform can also be used to find shapes in images.


Shape Context

Shape context is another widely used feature in shape retrieval.

Cij is the distance of shape contexts hi and hj


Improve Matching Efficiency

Fast pruning in matching– Reprehensive shape contexts

– Shapemes

Greg Mori, Serge Belongie, and Jitendra Malik, Shape Contexts Enable Efficient Retrieval of Similar Shapes, CVPR, 2001


Example Results

Reprehensive shape contexts in shape matching


Current Trends and Challenges

We now show a more “recent” work

L. Fei-Fei, R. Fergus, and P. Perona. A Bayesian approach tounsupervised One-Shot learning of Object categories. ICCV 2003.

The goal is to detect whether an objectappears in an image.


SIFT features are used.The good features areIn fact learned fromSmall set of training images.


Motor bikeResults.


Competitions about Object Recognition

http://www.pascal-network.org/challenges/VOC/voc2007/

http://www.pascal-network.org/challenges/VOC/voc2007/layoutexamples/images/01_parts.jpg

http://www.pascal-network.org/challenges/VOC/voc2007/examples/bus_07.jpg

http://www.pascal-network.org/challenges/VOC/voc2007/examples/cat_06.jpg


Retrieve Other Multimedia Data

Audio retrieval– Find a audio clip in a large database.

Video retrieval– Find a specific video clip.– Find a video short that has specific person or action.– Browsing video …


Data Structures in Media Retrieval

In multimedia data retrieval we often need to find the “nearest Neighbor” in the database from the exemplar.

We can abstract each media object as a feature vector. Our goal is to organize the database so that we can locate the most similar vector as quickly as possible.

Q: Think of some data structures that help to improve the searching.


K-d Tree

A 2D k-d tree

a

b

c

d e

f

a

b

c d f

http://upload.wikimedia.org/wikipedia/en/b/bf/Kdtree_2d.svg

http://upload.wikimedia.org/wikipedia/en/f/f1/Kd_tree.png


Summary

Content based multimedia retrieval is still not mature. Many problems still need to be solved.

There is no single method that solves all the problems.

We need better object detection and classification schemes.

Other related problems like multimedia data mining are also attracting more and more interest.

CS335 Principles of Multimedia Systems Content Based Media Retrieval Hao Jiang Computer Science Department Boston College Dec. 4, 2007.

Documents

r r r n n r r r n n

multimedia data

database slide

recent systems

increase of multimedia

recall precision

total items

relevant items total