Content-based Image Retrieval Hai Le Supervisor: Sid Ray.

Content-based Image Retrieval

Hai Le

Supervisor: Sid Ray

Content-based Image RetrievalContent-based Image Retrieval

► Introduction

► CBIR fundamentals

► Overview of the system

► Relevance feedback

► Results and Discussion

► Conclusion

Content-based Image RetrievalIntroduction

► Search and retrieval of image databases

► Retrieval by query, such as an example image

► Application areas include, weather forecasting, iiiiiiiimedical research, fabric design, WWW search iiiiiiiijust to name a few..

► CBIR systems which are commercially iiiiiiiiavailable include IBM’s QBIC, Blobworld, iiiiiiiiVisualSeek, Virage etc.

Content-based Image RetrievalCBIR Fundamentals

Input query and image database

Extract Image features

Index images by sorting the images according to a distance measurement

Determine feature weighting

Present retrieved images to user

User Feedback?N Y







User Feedback?N Y


► User is searching for a specific target item from iiiiiiiiia known database

► User is searching for a class of similar items

► Query by example image, drawing etc.

► Exploiting multiple queries to refine searches

► Use of segmentation or object selection to iiiiiiiiisearch for image specifics







User Feedback?N Y


► Low level image features used as classifiers

► Most commonly used features include:

► Colour

► Texture

► Shape

► Transform domain features► DCT

► Wavelet filters







User Feedback?N Y


► Features are represented as a numerical value

► Similarities between images are determined by iiiiiiiiidifferences between values

► Typically, queries are viewed as points in a iiiiiiiiimultidimensional feature space, and the iiiiiiiii‘distance’ between points is determined

► Common distance functions include► Euclidean

► City block

► Images are indexed in terms of their distance iiiiiiiiifrom the query point







User Feedback?N Y


► Top N most similar images are retrieved

► Searches can be refined by user feedback

► Data which is fed back is usually used to iiiiiiiidetermine significant features

► A feature’s influence on a query can be iiiiiiiiincreased/decreased by applying a weighting iiiiiiiifacture during distance calculations

► Images are re-indexed

Content-based Image RetrievalOverview of the System

► Query by example image

► 3 selectable features for querying, with 189 iiiiiiiisub features

► Selection between 4 weighting schemes

► Incorporates user interaction, retrieved images iiiiiiiiare selected as either relevant or non-relevant


► Features used in the system:

► Colour histogram

► Colour moments

► Edge histogram


► Colour histogram using linear quantised HSV iiiiiiiiicolour space

► 18 hues (20 degrees separation), 3 iiiiiiiiisaturations, 3 values, total of 162 colours

► Each bin is proportional ie.

► Colour moments calculated by taking the iiiiiiiiiaverage, standard deviation and cube root of iiiiiiiiithe third moment, of each of the HSV channels


► Edge histogram is derived by the number of iiiiiiiiedge pixels present in the image and the iiiiiiiidirection of the edge pixels

► Sobel edge detector is applied on the image, if iiiiiiiithe edge strength surpasses a threshold the iiiiiiiidirection is recorded and quantised

► A histogram is generated from all the iiiiiiiidirectionality values

► Each bin represents a sub feature


► Euclidean metric used to measure distance

► Features which constitute large values iiiiiiiiovershadow those with small values

► Gaussian normalisation is applied to give equal iiiiiiiiemphasis on all features

i

ii HHD 2)21(

3' f

f


► Most similar images are those closest to the iiiiiiiiquery image in the feature space

► Images are indexed by their distance values

► The top N images are retrieved and displayed

Content-based Image RetrievalRelevance Feedback

► An image object can be modeled as:

► D is the image data, F = { fi }, features, R = { rij } iiiiiiiia set of representation or subfeatures, iiiiiiiieg. Histogram bins

► An objects similarity to the query is calculated iiiiiiiiby rij and its corresponding weight wij and a iiiiiiiisimilarity measure S

► Weighted Euclidean:

),,( RFDOO

221 )( ijijij rrwS


► Different queries are more reliant on certain iiiiiiiiifeatures than others

► Varying wij’s changes emphasis of features iiiiiiiiiduring distance calculation

► Term weighting using relevance feedback iiiiiiiiiinclude:

► Density estimation

► Support vector machine learning

► Self-organising maps


► System training data uses binary feedback, iiiiiiiiieither positive example or negative examples

► Number of retrieved images is 4 plus original iiiiiiiiiquery image

► Of the 4 examples, images marked as relevant iiiiiiiiiare placed in a positive subset, unselected iiiiiiiiiimages are placed in a negative subset

► Unretrieved images are left indifferent


► Weighting scheme devised by Rui et. al

► Form an M x N matrix, M = number of positive iiiiiiiiimages, N = number of features

► For each column of the matrix calculate the iiiiiiiistandard deviation

► The weight wij is calculated by taking:

ijijw

1


► Using both positive and negative images

► If means of rij positive and rij negative are iiiiiiiiisimilar, feature is insignificant

► Using variance:

ijp

ijnijpijw

||

2

2)(

ijp

ijnijpijw


► Positive images are similar because of a iiiiiiiiispecific characteristic, negative images are iiiiiiiiidifferent because of a number of iiiiiiiiicharacteristics

► Using negative standard deviation

ijp

ijnijnijpijw

||

Content-based Image RetrievalResults and Discussion

► Image database of 200, divided into individual iiiiiiiigroups ranging from 4-10 images per group

► group of 10 images selected for test case

► each image selected as query, precision and iiiiiiiiirecall calculated for each of the 4 term iiiiiiiiweighting schemes after 1 iteration

► Precision and recall averaged over the 10 iiiiiiiiimages




► Employing negative examples, showed iiiiiiiiiimprovement

► Type 3 showed slowest degradation

► Type 4 showed significant improvement

Content-based Image RetrievalConclusion

► CBIR system developed incorporating► variety of features

► functional interface

► ‘user friendly’ feedback mechanism

► Improved term weighting scheme over used iiiiiiiiin the MARS CBIR system

► Maintains same simplistic interface as MARS

► Future work► image classification

► segmentation/local features

► better feature representation eg. Gabor filtering

1

Content-based Image Retrieval Hai Le Supervisor: Sid Ray.

Documents

image specifics

user user feedback

similar images

database user

n y slide

query point slide

used features

user feedback data