PIBE: A Personalizable Image Browsing Enginecvdb04.irisa.fr › Talks › 6_p21_Bartolini.pdf · Principles of PIBE nThree main ingredients behind the BT: n image descriptors (e.g.

PIBE:A Personalizable Image

Browsing Engine

Ilaria Bartolini, Paolo Ciaccia, and Marco PatellaIEIIT-BO/CNR – DEIS

University of Bologna, Italy

CVDB - Paris, June 13, 2004

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 2

Outline

n Why Image Browsing?n Existing Browsing Systems n PIBE (Personalizable Image Browsing Engine)

n Functionalitiesn Customizable Hierarchical Browsing Structuren Visual Examples

n Experimental Evaluationn Conclusions & Future Work

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 3

Motivation

n Exploration of large image databases (DB’s) is a complex and tedious task

n Difficulties in characterizing the image contentand in defining suitable comparison criteria

n Effective interaction with large DB’s has to include not just querying but also browsingfacilities:n to determine a good starting point for searchingn to get an overall view of the DB contents

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 4

Existing Browsing Systems

n Several browsing systems existn Information considered to compute the similarity between

images: n automatically extracted low-level features (e.g. color,

texture, etc.)n manually extracted semantic conceptsn both

n Typically, they provide an hierarchical view of the image DB

n Problems that limit their applicability to large image DB’s:

n based on static browsing structure n providing customization facilities that need to reorganize a

large part of the DBn no persistent updating of the browsing structure

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 5

PIBE (Personalizable Image Browsing Engine)

n A novel adaptive image browsing enginen customizable hierarchical structure called Browsing

Tree (BT)n graphical personalization actions to modify the BTn “local” reorganization of the DB

n specific similarity criteria for each portion (sub-tree) of the BT

n user customizations persist across different sessions

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 6

PIBE Architecture

n PIBE builds off linethe BT from the features

FeatureOrganization

FeatureExtraction

ImageDB

FeatureDB BT

Off-line

Browsing Engine

Browsing ProcessorVisualizer

GUI

On-line

n At run timethe user interacts with the system by means of a GUI

n Browsing Engine:n Browsing Processor

manages requests to modify the BT

n Visualizer displays the BT’s content

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 7

Principles of PIBE

n Three main ingredients behind the BT:n image descriptors (e.g. color histograms)

n representable as points in a N-dimensional spacen (dis)similarity functions (e.g. weighted Euclidean)

n instance of a parametrized class of functions to support personalization actions

n clustering algorithms (e.g. k-means)n BT is a hierarchical structure derived from a hierarchical

clustering algorithm or, alternatively, by recursivelyapplying a partitioning algorithm

n PIBE is parametric with respect to the choices of above points. The combination of choices should guarantee:n scalability with respect to the cardinality of the image DBn suitable cardinality of each cluster

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 8

Browsing Tree

n PIBE uses:n 32-D HSV color histograms (p)n weighted Euclidean distances

n k-means algorithm applied to the whole image DB and, recursively, to each of the derived k clusters producing a default BT (wi = 1)

n each node of the BT corresponds to a cluster Cj of images and maintains then centroid c(Cj) of Cjn representative image p(Cj) of Cj defined as

n local weight vector wj computed as

2/12

32

1

)();,(

−= ∑=

iii

i qpwd wqp

2,, /1 ijijw σ∝

{ }jjjj CCcdCp ∈= pwp ););(,(minarg)(

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 9

Browsing Tree Example (k=3)

p(C1) p(C2) p(C3)

p(C4) p(C5) p(C6)

p(C13) p(C14) p(C15) p(C19) p(C20) p(C21)

p, s q r

C0

C1

C4 C6

C13 C14 C19

v, u

C5

n = number of images to be clusteredO(kn logkn)

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 10

Browsing

n Visualizer and Browsing Processor implement the navigation mechanismn images displayed in a 2-D screen (spatial

visualization technique) using relative distancesn to display images p and q the weighted Euclidean

distance corresponding to the most specific cluster containing both images is used

n Two modalities:n vertical: the user selects a representative image on

the display and zooms in the cluster contentn horizontal: the user explores regions of the space

where no representative image is present

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 11

X

Browsing Examples

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 12

X

Browsing Examples

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 13

Personalization Facilities

n PIBE provides graphical facilities allowing the user to move a “source” cluster representative image p(Cs) on a “target” one p(Ct)

n Asymmetric operations n Ct is the “leader”

n Two different actionsn “moving image”

n p(Cs) is considered as single imagen “moving cluster”

n p(Cs) represents the whole cluster Cs

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 14

Updating the BT

n Browsing Processor is in charge of the BT updates (we consider the more demanding case of moving cluster action)n wt is recomputed (Ct = Ct ∪ Cs) n Cs is deleted from the BTn k-means algorithm is recursively applied on the updated

Ct = Ct ∪ Cs using wt (O(k |Ct| logk|Ct|))n representative images of clusters that contained Cs need to be

updatedn if Ct and Cs have the same parent Cp,

p(Cp) does not changen if Ct and Cs have different parents Cpt and Cps

(but the same grand-parent Cgp), only p(Cpt) and p(Cps) have to be updated

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 15

Experimental Evaluation

n 2,000 color images of IMSI-PHOTOS data setn Goal: “given a set G of target images

(in our experiments, 51 “fishes”), evaluate how PIBE is able to reflect user’s needs in localizing them”

n Scenarios involved:n Default: this strategy uses the default BTn PIBE: a customized version of the BT is considered, following

two ways to insert the target images of G into a single target cluster Ctn Top-down (Ct is an internal node that contains a large

number of target images) n Bottom-up (Ct is the leaf node containing the highest

number of images in G)

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 16

Effectiveness

n Computed as precision (P) of the target cluster Ct with respect to the set G of target images

n P is the the number of target images in Ct over the cardinality of Ct

63.7551802

77.4248621

67.6523340

P (%)

n. relevant images

n. images

n. actions

Top-down strategy (k=11)

98.08515230

100343420

100222210

100220

P (%)

n. relevant images

n. images

n. actions

Bottom-up strategy (k=11)

P = | Ct ∩ G|/| Ct |

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 17

Efficiency

n Computed as time (T) of a browsing session, i.e. number of mouse clicks needed to reach all target images by means of the PIBE GUI

n We computed the saved-time (ST) as

ST = (1 - TPIBE/TDefault)*100

where TPIBE is the time needed using a customized BT, whereas TDefault is that associated to the default BT

63.642055Bottom-up

47.272955Top-Down

ST (%)TPIBETDefaultstrategy

(k=11)

Bartolini, Ciaccia, and Patella CVDB – June 13, 2004 18

Conclusions & Future Work

n PIBE as a new solution to image browsing:n personalization graphical actionsn hierarchical structure (Browsing Tree) n persistent and local customizations

n We are investigating several issues:n variable fan-out for Browsing Tree nodesn “split” actionsn integration of semantic concepts extracted

by textual descriptors (using lexical ontologies, e.g. WordNet)