Cluster based landmark and event detection for tagged photo collections

Cluster-based Landmark and Event Detection on Tagged Photo Collections

Symeon Papadopoulos, Christos Zigkolis,

Yiannis Kompatsiaris, Athena Vakali

user generated content creates new opportunities

real-world depicted in users’ online collections

need new tools for content organization

potential for many insights into what people see, do and like

image clustering

event

landmark

clusters landmarks + events

the framework

+ +

photos tags geo

1 2

event

landmark

3

landmark

4

overview

step 1: create photo similarity graph

1 2

event

landmark

3

landmark

4

tag similarity

visual similarity casa mila, la pedrera

co-occurrence

latent semantic indexing

SIFT

SURF

step 2: use graph to cluster the photos

1 2

event

landmark

3

landmark

4

v

neighborhood of node v + node itself = structure of node v

v v

N(v) v Γ(v)

the concept of node structure

u v

Γ(v) Γ(u) ∩

Γ(v) Γ(u)

structural similarity between nodes v and u

the concept of structural similarity (1)

C

B A

high structural similarity

low structural similarity

photo cluster 1

photo cluster 2

the concept of structural similarity (2)

O (km m)

average node degree

# edges

graph-based clustering

k-means clustering O (I C n D)

hierarchical agglomerative clustering

O (n2 log n)

# iterations

# clusters

# nodes

# dimensions

complexity

step 3: detect landmarks & events

event

landmark landmark

1 2

3 4

#users / #photos

duration

[1 day, 2 users / 10 photos]

[2 years, 50 users / 120 photos]

Quack et al., CIVR 2008

baseline features

Event Tags

Landmark Tags additional features

step 4: post-process landmark clusters

event

landmark landmark

1 2

3 4

cluster merging based on proximity

low frequency tags

helado tropical

park güell jaume oller

barcelona spain cielos

park

field

sclupture el beso

generic tags

CLUSTER TAGS

cluster tag filtering

results

207,750 photos

7,768 users

33,959 unique tags

compare graph-based vs. k-means clustering

user study geospatial coherence

high geospatial coherence

low geospatial coherence

user study

precision recall κ-statistic

graph-based

k-means

VISUAL

1.000

0.806

0.110

0.324

1.000

0.226

precision recall κ-statistic

graph-based

k-means

TAG

0.950

0.848

0.182

0.307

0.820

0.564

geospatial coherence

radius std. deviation

graph-based

k-means

VISUAL

357 m

2.4 km

1.18 km

1.73 km

graph-based

k-means

TAG

456 m

767 m

1.15 km

1.76 km

357 m 1.18 km

456 m 1.15 km

classification performance

16% - 23% improvement thanks to tag features

sagrada familia, cathedral, catholic 15.2m

la pedrera, casa mila 31.8m

parc guell 9.6m

boqueria, market, mercado, ramblas 82.1m

camp nou, fc barcelona, nou camp 18.7m

landmark localization accuracy

music, concert, gigs, dj 43.1%

conference, presentation 6.5%

local traditional, parades 4.6%

racing, motorbikes, f1 3.3%

event category composition

clusttour

www.clusttour.gr

twitter.com/clusttour facebook.com/clusttour