MM2004_Hoi

11

A Novel Log-based Relevance Feedback Technique A Novel Log-based Relevance Feedback Technique in Content-based Image Retrievalin Content-based Image Retrieval

Steven Steven Chu-Hong Hoi & Michael R. LyuChu-Hong Hoi & Michael R. Lyu

Department of CSEDepartment of CSE

The Chinese University of Hong KongThe Chinese University of Hong Kong

Shatin, Hong Kong SARShatin, Hong Kong SAR

{chhoi, lyu}@cse.cuhk.edu.hk{chhoi, lyu}@cse.cuhk.edu.hk

ACM Multimedia 200412th Annual Conference, October 10 -16, 2004

New York City, Columbia University

22Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

OutlineOutline

Introduction & MotivationIntroduction & Motivation

Log-based Relevance FeedbackLog-based Relevance Feedback

Soft Label Support Vector MachineSoft Label Support Vector Machine

Experimental ResultsExperimental Results

Conclusions and Future WorkConclusions and Future Work

33Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

IntroductionIntroduction

Content-based Image Retrieval (CBIR)Content-based Image Retrieval (CBIR)– Attract much interest, studied for many yearsAttract much interest, studied for many years– AAn important componentn important component in multimedia retrieval in multimedia retrieval– Query based on low-level visual content: color, texture,Query based on low-level visual content: color, texture,

shape, etc. shape, etc.

QBE

Challenge: the semantic gap between low-level features and high-level conceptsChallenge: the semantic gap between low-level features and high-level concepts

44Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

IntroductionIntroduction

Relevance Feedback (RF) in CBIRRelevance Feedback (RF) in CBIR– A powerful technique, attack the semantic gap problemA powerful technique, attack the semantic gap problem– Using interactive mechanisms, soliciting users’ interactions, Using interactive mechanisms, soliciting users’ interactions,

learning users’ high-level conceptslearning users’ high-level concepts– Boosting retrieval performance effectivelyBoosting retrieval performance effectively– Many popular techniques: MARS, QEX, MindReader, OptiMany popular techniques: MARS, QEX, MindReader, Opti

mizing learning, SVM (active), Boosting, etc. mizing learning, SVM (active), Boosting, etc.

ProblemsProblems– Regular relevance feedback techniques: a lot of Regular relevance feedback techniques: a lot of timestimes of feed of feed

backback which will which will cost much time cost much time and and make users boringmake users boring

55Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

MotivationMotivation

Can users’ feedback logs information be used to Can users’ feedback logs information be used to improve the regular relevance feedback? improve the regular relevance feedback?

Relevance Feedback

Users’ Feedback Logs

Problem

?

66Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

LRF: Log-based Relevance FeedbackLRF: Log-based Relevance Feedback

Problem FormulationProblem Formulation– Construct a Relevance Matrix: Construct a Relevance Matrix: RMRM

– Each log session: (Each log session: (N N = + ) = + ) NN images are marked: relevant & irrelevant instances images are marked: relevant & irrelevant instances

– Values: relevant (+1), irrelevant (-1), unknown (0)Values: relevant (+1), irrelevant (-1), unknown (0)

Log Sessions

Image samples in the image database

1 -1 1 -1 -1 0 1 -1 -1 11

-1 1 -1 -1 -1 -1 -1 1 -1-10

77Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Log-based Relevance Feedback (cont’d)Log-based Relevance Feedback (cont’d)

Relationship MeasurementRelationship Measurement– For each given session For each given session k k , if the image , if the image i i is marked as is marked as

‘‘relevantrelevant’ (positive) and the image ‘’ (positive) and the image ‘jj’ is marked as ’ is marked as ‘‘irrelevantirrelevant’ (negative), then the elements are represented as’ (negative), then the elements are represented as

RM (RM (k, ik, i) = 1 ) = 1 andand RM ( RM (k, jk, j) = -1) = -1– For every two images: For every two images: ii and and jj, their relationship can be , their relationship can be

measured by a modified correlation function:measured by a modified correlation function:

88Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

LRF AlgorithmLRF Algorithm

Collection of training SamplesCollection of training Samples– Regular relevance feedbackRegular relevance feedback

Learn only with a limited number of training samplesLearn only with a limited number of training samples

Cannot achieve good performance without enough training samplesCannot achieve good performance without enough training samples

– Idea: finding more samples based on Idea: finding more samples based on NN initial samples initial samples

– For an initial positive sample For an initial positive sample ii, the relevance degrees between , the relevance degrees between every image sample every image sample jj of the database are computed by a soft label of the database are computed by a soft label function:function:

– By ranking the soft label values, we can collect a number of samples By ranking the soft label values, we can collect a number of samples with larger soft label values corresponding to the sample with larger soft label values corresponding to the sample i.i.

99Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

LRF Algorithm (cont’d)LRF Algorithm (cont’d)

The learning issue of the algorithmThe learning issue of the algorithm– Based on the initial marked samples and the log Based on the initial marked samples and the log

information, we can collect a large number of positive and information, we can collect a large number of positive and negative training samples associated with soft labels which negative training samples associated with soft labels which represent their confidence degrees.represent their confidence degrees.

– Problem: how to develop the algorithm to learn the data associated with soft labels ?

Proposed Solution: Soft Label Learning – Soft Label Support Vector Machine (SLSVM)

1010Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

SLSVM: Soft Label Support Vector MachineSLSVM: Soft Label Support Vector Machine

Problem FormulationProblem Formulation

SVM

1

11

1

11

0.5

1111Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

SLSVM (cont’d)SLSVM (cont’d)

1212Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

SLSVM (cont’d)SLSVM (cont’d)

1313Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental ResultsExperimental Results

DatasetsDatasets– Images selected from COREL image CDsImages selected from COREL image CDs– 20-Category: 2000 image instances20-Category: 2000 image instances– 50-Category: 5000 image instances50-Category: 5000 image instances– Each category contains a specific semantic meaningEach category contains a specific semantic meaning

1414Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental Results (cont’d)Experimental Results (cont’d)

Image RepresentationImage Representation– Color Moment Color Moment

9-dimension9-dimension

– Edge Direction Histogram Edge Direction Histogram 18-dimension18-dimension

Canny detector, 18 bins of 20 degreesCanny detector, 18 bins of 20 degrees

– Wavelet-based texture Wavelet-based texture 9-dimension9-dimension

Daubechies-4 wavelet, 3-level DWTDaubechies-4 wavelet, 3-level DWT

9 subimages are selected to generate the feature9 subimages are selected to generate the feature

1515Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental Results (cont’d)Experimental Results (cont’d)

Log FormatLog Format– Define a Log Session (LS) as a basic log unit, that Define a Log Session (LS) as a basic log unit, that

corresponds to a relevance feedback roundcorresponds to a relevance feedback round– Each log session contains 20 images marked by usersEach log session contains 20 images marked by users

Log CollectionLog Collection– Collect logs from 10 usersCollect logs from 10 users– Non-noisy logs: 100 LSNon-noisy logs: 100 LS– Noisy logs: Noisy logs:

20-Category: 103 LS, 7.2% noise20-Category: 103 LS, 7.2% noise

50-Category: 138 LS, 8.1% noise50-Category: 138 LS, 8.1% noise

1616Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental Results (cont’d)Experimental Results (cont’d)

Compared SchemesCompared Schemes– EU (Euclidean distance - baseline)EU (Euclidean distance - baseline)– RF_QEX (QEX: query expansion)RF_QEX (QEX: query expansion)

Multiple instance sampling, pick Multiple instance sampling, pick NN nearest samples recursively nearest samples recursively

– RF_SVMRF_SVMRegular relevance feedback by SVMRegular relevance feedback by SVM

– LRF_QEXLRF_QEXSimilar to RF_QEX, but we pick the samples weighted by soft Similar to RF_QEX, but we pick the samples weighted by soft labels in our framework (the larger the label, the smaller the labels in our framework (the larger the label, the smaller the distance)distance)

– LRF_SLSVMLRF_SLSVM

1717Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental Results (cont’d)Experimental Results (cont’d)

SettingsSettings– Same Kernels: e.g. RBF kernelSame Kernels: e.g. RBF kernel– Evaluation metric: Evaluation metric:

Average Precision = # of relevance / # of returned Average Precision = # of relevance / # of returned – Automatic evaluation: Automatic evaluation:

Taking average precision over 200 query executionsTaking average precision over 200 query executions

1818Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental Results (cont’d)Experimental Results (cont’d)

Performance ComparisonPerformance Comparison

1919Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental Results (cont’d)Experimental Results (cont’d)

Performance ComparisonPerformance Comparison

2020Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Experimental Results (cont’d)Experimental Results (cont’d)

Performance ComparisonPerformance Comparison

2121Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

ConclusionsConclusions

In this paper we proposed a new scheme to study In this paper we proposed a new scheme to study users’ feedback logs for improving the performance users’ feedback logs for improving the performance of regular relevance feedback in CBIR.of regular relevance feedback in CBIR.We introduce the soft label learning concept and We introduce the soft label learning concept and developed a modified SVM technique, i.e. Soft Label developed a modified SVM technique, i.e. Soft Label SVM, to construct the algorithm for log-based SVM, to construct the algorithm for log-based relevance feedback.relevance feedback.We evaluate our proposed method compared with We evaluate our proposed method compared with traditional techniques and demonstrate promising traditional techniques and demonstrate promising results.results.

2222Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Limitations & Future WorkLimitations & Future Work

The proposed LRF with SLSVM algorithm still suffers The proposed LRF with SLSVM algorithm still suffers performance drop when many noisy logs are appeared.performance drop when many noisy logs are appeared.Much noise may be involved when the scale of the image Much noise may be involved when the scale of the image database is increased. database is increased. When the number of log sessions is large, the dimension of the When the number of log sessions is large, the dimension of the relevance matrix may be a problem.relevance matrix may be a problem.Training time of SLSVM need be considered for large scale Training time of SLSVM need be considered for large scale datasets.datasets.

Open questions:Open questions:Can we work out more effective Soft Label Learning techniques Can we work out more effective Soft Label Learning techniques in the future?in the future?Can we include some noise filtering techniques into our Can we include some noise filtering techniques into our framework?framework?

2323Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

Thank You!Thank You!

Q & AQ & A

2424Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

References (part)References (part)

[He & King 2003] X. He, O. King, W.-Y. Ma, M. Li, and H. J. Zhang. Learning a semantic space from user’s relevance feedback for image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 13(1):39–48, Jan. 2003.

[Huang & Zhou 2001] T. S. Huang and X. S. Zhou. Image retrieval by relevance feedback: from heuristic weight adjustment to optimal learning methods. In Proceedings of IEEE International Conference on Image Processing (ICIP’01), Thessaloniki, Greece, Oct. 2001.

[Hong & Huang 2000] P. Hong, Q. Tian, and T. Huang. Incorporate support vector machines to content-based image retrieval with relevant feedback. In Proc. IEEE International Conference on Image Processing (ICIP’00), Vancouver, BC, Canada, 2000.

[Rui & Huang 1999] Y. Rui and T. S. Huang. A novel relevance feedback technique in image retrieval. In Proc. ACM Multimedia (MM’99), pages 67–70, Orlando, Florida, USA, 1999.on Image Processing (ICIP’00), Vancouver, BC, Canada,

[Tong & Change 2001] S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proceedings of the ninth ACM international conference on Multimedia, pages 107–118. ACM Press, 2001.

2525Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

AppendixAppendix

Kernel ComparisonKernel Comparison

2626Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalHoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image RetrievalACM Multimedia 2004

CBIRCBIR