1 Content Based Image Retrieval Using MPEG-7 Dominant Color Descriptor Student: Mr. Ka-Man Wong Supervisor: Dr. Lai-Man Po MPhil Examination Department.

1

Content Based Image Retrieval Using MPEG-7 Dominant Color Descriptor

Student : Mr. Ka-Man WongSupervisor : Dr. Lai-Man Po

MPhil ExaminationDepartment of Electronic Engineering

City University of Hong KongAugust 2004

2

Outlines of this presentation Objectives MPEG-7 visual descriptors A new similarity measure for dominant color descri

ptor Merged Palette Histogram Similarity Measure

A new relevance feedback for dominant color descriptor

Merged Palette Histogram Relevance Feedback MIRROR – A CBIR system using MPEG-7 visual

descriptors Conclusions

3

Objective of this research study To investigate Content Based Image Retrieval

(CBIR) based on color features To develop efficient techniques for MPEG-7

Dominant Color Descriptor (DCD) Merged Palette Histogram Similarity Measure Merged Palette Histogram Relevance Feedback

Apply proposed methods into a real system

4

MPEG-7 visual descriptors Color

Color structure, scalable color, dominant color, color layout

Texture Homogeneous texture, edge histogram, texture

browsing Shape

Contour shape, region shape, 3D shape Motion (for video contents)

Motion activity, camera motion, motion trajectory, parametric motion

They describe image/video contents in different aspects

5

MPEG-7 color descriptors Dominant color descriptor (DCD)

A compact color descriptor generated by color quantization

Color structure descriptor (CSD) Color histogram generated by structure block scanning a

pproach Scalable color descriptor (SCD)

Color histogram in a quantized HSV space with Haar transform.

Color layout descriptor (CLD) A compact color-spatial descriptor generated by dividing

the image by a 8x8 gird with DCT transform.

6

Relevance feedback Color might perform well, but it might not

match user’s expectation

Effectiveness could be further improved by involving users in the searching

7

MPEG-7 color descriptors Two major problems are found in DCD make it

unable to perform well Problems of its original similarity measure method Cannot use relevance feedback easily

We will focus on DCD in this research study New methods are developed to utilize DCD

Similarity measure Relevance feedback

8

Merged palette histogram similarity measure for dominant color descriptor

Dominant Color Descriptor Shortcomings of the existing similarity function Proposed Merged Palette Histogram Similarity Measure

9

Dominant Color Descriptor Feature representation

The dominant colors Percentage of area of the dominant color Maximum of 8 colors

(1) ),...,2,1(},,{ NipcF ii

Dominant Color Descriptor

percentage

color

10

Dominant Color Descriptor Feature extraction

GLA color quantization Each color have at least Td distance away in a perceptuall

y uniform CIELuv

Dominant Color Descriptor

Original Image

percentage

color

CQ

Color Quantized Image

11

Dominant Color Descriptor Similarity measure

A modified Quadratic Histogram Distance Measure (QHDM)

1 2 1 22 2 2

1 2 1 2 1 ,2 1 21 1 1 1

( , ) 2 (2)N N N N

i j i j i ji j i j

D F F p p a p p

Percentage p

color

Percentage q

color

12

Dominant Color Descriptor Since each DCD may have different set of

colors, QHDM is used to account for identical colors and similar colors.

1 2 1 22 2 2

1 2 1 2 1 ,2 1 21 1 1 1

( , ) 2 (2)N N N N

i j i j i ji j i j

D F F p p a p p

Percentage p

color

Percentage q

color

13

Shortcomings of the QHDM similarity function Limitations of QHDM

Distance upper bound is not fixed Completely different image cannot be identified by its upper

bound The similarity coefficient does not well model color simila

rity It does not balance between color distance and area of matc

hing The new Merged Palette Histogram Similarity Meas

ure method Can compare identical colors as well as similar colors Use area of matching for similarity measure

14

Proposed Merged Palette Histogram Similarity Measure MPHSM Process - 1

Find the closest pair of colors using Euclidian distance in CIELuv color space

2 2 21 2 1 2 1 2 1 2( , ) ( ) ( ) ( ) (3)d C C l l u u v v

15

Proposed Merged Palette Histogram Similarity Measure MPHSM process - 2

If the distance smaller than a threshold Td, merge them to form a new common palette color

Step 1 – 2 iterates until the minimum distance larger than Td

1 1 2 2( , )

1 2

(4)i i j jm i j

i j

p c p cc

p p

16

Proposed Merged Palette Histogram Similarity Measure MPHSM process - 3

A new common palette is then generated Form new descriptors based on the common palette

Dominant Color Descriptor

Common Palette

Merged Palette Histogram

17

Proposed Merged Palette Histogram Similarity Measure MPHSM process - 4

Histogram intersection is used to measure the similarity

Count the non-overlapping area as the distance

1 2 1 2 1 21 1

1( , ) 1 min( , ) (5)

2

m mN N

m m mi mi mi mii i

D F F p p p p

18

Experimental results MPHSM improves DCD for both datasets

While using Corel_1k dataset MPHSM outperforms QHDM significantly

*ANMRR (smaller means better)ANMRR (MPEG-7 CCD) ANMRR (Corel_1k)

DCD-MPHSM 0.2604 0.3946

DCD-QHDM 0.2834 0.5648

19

Experimental results Visual results - Query #32 from MPEG-7 CCD

Query image

QHDM results, ANMRR=0.4 MPHSM result, ANMRR=0.0111

20

Experimental results Visual results – Query #15 from Corel_1k

Query image

QHDM result, ANMRR=0.6464 MPHSM result, ANMRR=0.4819

21

Conclusions on Merged Palette Histogram Similarity Measure MPHSM generates a common palette Can match similar colors Uses area of matching as the similarity Boosts DCD in terms of ANMRR Gives better visual results

22

Merged palette histogram for dominant color descriptor relevance feedback

Feature weighting relevance feedback technique and its limitations Proposed Merged Palette Histogram Relevance Feedback Experimental results

23Feature weighting relevance feedback technique and its limitations Feature weighting relevance feedback

technique Assumes a fixed feature space (histograms) Taking liner combinations on matching histogram

bins. Simple approach: Histogram averaging

1' (7)

k

jij

i

pH

k

+( ) / 2 =

24Feature weighting relevance feedback technique and its limitations But DCDs of images might have different set of

colors, similar images might not have any exactly matched colors.

Two problems

H1 2

1

H2 2

1

H’4

1

25

Limitation of feature weighting relevance feedback technique Problems

The number of colors in updated query may greatly exceed the limit of the number of colors defined by MPEG-7 as the number of selected images increase.

Similar colors are separated. By definition of DCD, similar colors should be grouped together.

H1 2

1

H2 2

1

H’4

1

26

Limitation of feature weighting relevance feedback technique The Merged Palette Histogram Relevance

Feedback The updated query contains common colors among

selected images Represent the selected images efficiently

27

Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback

(MPH-RF) process - initialize Obtain all DCD from selected images

28

Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback

(MPH-RF) process - 1 Link all DCD together

+ + =

6 colors 8 colors 6 colors 20 colors

29

Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback

(MPH-RF) process - 2 Palette Merging

Find the closest pair of colors based on Euclidian distance in CIELuv

If minimum distance smaller than Td merge the color pair and sum up the percentages of merged colors

Iterate until minimum distance > Td1 1 2 2

1 1 2 2

1 1 2 2

(8)i i j jm

i j

m

w p c w p cc

w p w p

p w p w p

20 colors 9 colors

30

Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback

(MPH-RF) process - 3 Approximation

Cut the least significant colors if number of colors >8

9 colors 8 colors

31

Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback

(MPH-RF) process - 4 Re-normalization

Adjust the histogram sum into 1 An updated query is generated

Approximated MPH Updated QueryHistogram Sum =1

32

Experimental results MPH-RF gives improvement on all combinations of

similarity measures and datasets. Combination of MPHSM and MPH-RF gives a significant

improvement Three iterations of relevance feedback give a significant

result

*ANMRR – smaller means betterMPEG-7 CCD Corel_1k

Initial After 3 RFRF

ImprovementInitial After 3 RF

RF Improvement

DCD-MPHSM

0.2604 0.1752 0.0852 0.3946 0.3298 0.0648

DCD-QHDM

0.2834 0.2117 0.0717 0.5468 0.4900 0.0568

33

Experimental results Visual results – Query #50 from MPEG-7 CCD, MPHSM

Query image

Ground truth images

Initial retrieval, 4 of 8 ground truths hit, NMRR=0.5

First RF retrieval, 6 of 8 ground truths hit, NMRR=0.2782

Second RF retrieval, 7 of 8 ground truths hit, NMRR=0.1541

34

Experimental results Visual results – Query #13 from Corel_1k,

MPHSM

Query image

Ground truth images

Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043

First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688

Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688

35

Conclusions on Merged Palette Histogram Relevance Feedback MPH-RF generates a new DCD query using

palette merging technique Represents the selected relevant images

naturally and effectively MPH-RF boosts all situations of DCD searching

36

MIRROR – A CBIR system using MPEG-7 visual descriptors

MPEG-7 Image Retrieval Refinement based On Relevance feedback Systems structure Demo

37

MIRROR – A CBIR system using MPEG-7 visual descriptors System structure

ImageDB

SimilarityMeasure

RelevanceFeedback

user initial input user feedback

MPEG-7data

Feature Extraction

reference image relevant image(s)

SimilaritySorting

Output Images

user feedback

-7

)

38

MIRROR – A CBIR system using MPEG-7 visual descriptors Demo

Demo 1: Similarity Measure

Demo 2: Relevance Feedback

http://www.ee.cityu.edu.hk/~mirror/

39

Conclusions of this research work By utilizing MPHSM and MPH-RF DCD, DCD

becomes compact as well as accurate Similarity measure

Merged Palette Histogram Similarity Measure Relevance Feedback

Merged Palette Histogram Relevance Feedback

Proposed methods are implemented into a real system.

CBIR functions Evaluation tools

40

Q & A