Contextual Wisdom Social Relations and Correlations for Multimedia Event Annotation Amit Zunjarwad, Hari Sundaram and Lexing Xie.

Contextual Wisdom

Social Relations and Correlations for Multimedia Event Annotation

Amit Zunjarwad, Hari Sundaram and Lexing Xie

I don’t want to spend time annotating :( help!

April 18, 2023@NUS 2

Talk Outline

Observations

Events

Generalization:Sum of Partial Observations

Similarity, Co-Occurrence and Trust

@I2R April 18, 2023 3

Experiments:compare against SVM

Conclusions

An Annotation Puzzle

@NUS April 18, 2023 4

April 18, 2023 5@NUS

Observing Flickr Data

The pool statistics reveal a power law distribution• Less than 11% of the tags have more than 10 photos• There are not enough instances to learn most of the

concepts! The global flickr pool is interesting:

April 18, 2023 6@NUS

Learnability

April 18, 2023 7@NUS

Learnability

The pool statistics reveal a power law distribution• Less than 11% of the photos have more than 10

instances• There are not enough instances to learn most of the

concepts! The global flickr pool is interesting: • Most of the tags have over 100 instances• Photos reveal very high visual diversity

The Power law is a fundamental property of online networks – cannot be wished away.

April 18, 2023 8@NUS

Learnability

Singapore People Walking Orchard rd. After MRT Experimenting Walking Day Outdoor..

April 18, 2023 9@NUS

Scalability

The assumption of consensual semantics Search for “yamagata”

April 18, 2023 10@NUS

The Role of context

April 18, 2023@NUS 11

What if the answer didn’t completely lie in the pixels?

EventsWhat are they?

April 18, 2023@NUS 12

An event refers to a real-world occurrence, spread over space and time.

Observations form event meta data [Westermann / Jain 2007]• Images / text / sounds describe events

April 18, 2023 13@NUS

Defining Events

when

where

who

what

author

image

Event context refers to the set of attributes that help in understanding the semantics • Images / Who / Where / When / What / Why / How

Context is always application dependent • Ubiquitous computing community – location, identity

and time are main considerations

April 18, 2023 14@NUS

Context

[Mani and Sundaram 2007]

Event archival – events involve people, places and artifacts

Exploit different forms of knowledge: • (Global) Similarity – media, events, people. • (Personal) Co-occurrence – what are the joint statistics

of occurrence?• (Social) Trust – determining whom to trust for effective

annotation?

April 18, 2023 15@NUS

Four Problems

SimilarityGlobal, Systemic knowledge

April 18, 2023@NUS 16

A bottom up approach • Edge, color and texture histograms for images. • Rely on ConceptNet for text tags

Why ConceptNet and not WordNet?• Expands on pure lexical terms, to compound terms –

“buy food”• Expands on number of relations – from three to twenty• Contains practical knowledge – we can infer that a

student is near a library.

April 18, 2023 17@NUS

Event similarity

ConceptNet provides three functions:• GetContext(node): the neighborhood of the concept “book” includes “knowledge”, “library”

• GetAnalogousConcepts(node): concepts that share incoming relations; analogous concepts for the concept “people” are “human”, “person”, “man”

• FindPathsBetweenNodes(node1,node2) – returns a set of paths.

Our similarity measure is built using these functions.

April 18, 2023 18@NUS

A base similarity measure

The similarity between two concepts (e,f) is defined as follows:

We current use a uniform weighting on all three as the composite measure

April 18, 2023 19@NUS

Concept similarity

fe

fCeC

eA fA

context

analogous

path based1

1 1( , )

N

pi i

s e fN h

The distance between two concept sets is a modified Haussdorf similarity.

April 18, 2023 20@NUS

Computing similarity between sets

A

B

| |

1

1( , | ) max{ ( , )}

| |

A

H k ii

k

S A B m m a bA

Similarity between facets are computed using a weighted sum of frequency and the concept similarity measure:

Time distance is based on text tags, not actual time data – allows for temporal descriptions as “summer”, “holidays” etc.

Only frequency is used for “who” facet.April 18, 2023 21@NUS

Facet similarity (4w)

1 21 2 1 2

2

1 | |( , ) , | ,

2 | | H

L Ls L L S L L CS

L

Color, texture and edges are computed• 166 bin HSV color histogram• 71 bin edge histogram• 3 texture features

Euclidean distance on the composite feature vector.

The distance between two events is then a weighted sum of distances across all event facets.

April 18, 2023 22@NUS

Image facet similarity

April 18, 2023 23@NUS

The global similarity matrix Ms

Co-occurrencePersonal, statistical knowledge

April 18, 2023@NUS 24

The concept co-occurrences are just frequency counts.

(i= fun , j = new york) then the index (i,j) contains the number of occurrences of this tuple.

Notes:• Each concept is given a globally unique index• Co-occurrence matrixes are locally compact

Each user k, has a co-occurrence matrix Mck

associated with the user.

April 18, 2023 25@NUS

Statistics are computed per person

TrustPeople we like

April 18, 2023@NUS 26

Narrow understanding of “trust” a priori value is important Computing trust:• Compute event-event similarity

Trust propagation• Biased PageRank algorithm

• Trust vectors are row normalized

April 18, 2023 27@NUS

Activity based trust

activity matrix apriori

(1 ) k t = A t + p

The recommendation algorithm

April 18, 2023@NUS 28

The framework is event centric We know:

How to combine the three?

April 18, 2023 29@NUS

A review of what we know

similarity co-occurrence trust vectors

global personal social

1. Compute the social network trust vector (t) for the current user.

2. Compute the trusted, global co-occurrence matrix, for all tuples.

3. Iterate:

April 18, 2023 30@NUS

details

1

( , ) ( ) ( , ),N

k kc c

i

a b t i a b

M M

who where whatwhen image event

query

,

,c

s

y M x q

x M y q

Experiments

April 18, 2023@NUS 31

Developed and event based archival system

8 graduate students 58 events, 250 images,

over two weeks SVM – baseline

comparison Two cases• Uniform trust (global)• Personal trust

April 18, 2023 32@NUS

Details

Training is difficult – very small pool.• Modified bagging strategy • Train five symmetric classifiers• Pick one which maximizes the F-score

April 18, 2023 33@NUS

SVM training

Global Case:• 31 classifiers (who:8, when: 6, where: 10, what: 7)• Minimum number of images: 10 • Tested on 50 images (why?)

April 18, 2023 34@NUS

Uniform trust

Facets SVM CM (uniform)

H M X U H M

Who 13 23 5 9 22 28

When 11 20 6 13 24 26

Where 12 19 3 16 23 27

What 13 21 8 8 31 19

Event 10 12 22 6 22 28

H Hits

M Misses

X Unknown

U Undecidable

Trained classifiers per person• Very small pool• Min images – 5• 28 classifiers (who:9, when: 4, where: 6, what: 9)

April 18, 2023 35@NUS

Personal Network

Facets SVM CM (network)

H M X U H M

Who 45 81 62 62 183 67

When 51 96 73 30 167 83

Where 62 76 59 53 179 71

What 72 89 23 66 204 46

Events 0 0 250 0 153 97

H Hits

M Misses

X Unknown

U Undecidable

April 18, 2023 36@NUS

Positive examplesSVM ‘sky diving’

Social Network based method ‘fun’

The Sum of Partial ObservationsBeyond web 2.0 hype

April 18, 2023@NUS 37

Which media object summarizes “my trip to Singapore?”

April 18, 2023 38@NUS

Experiential fragments

April 18, 2023 39@NUS

A reconsideration of a traditional idea

@NUS

The Creation of participatory knowledge

April 18, 2023 40

Conclusions

April 18, 2023@NUS 41

An event based annotation system• Media are event meta-data• Issues: learnability, scalability, context

Employ three kinds of knowledge• Global – conceptnet, image similarity• Personal – statistical co-occurrence • Social – trust

Recommendations• Employ iterative schemes (HITS / PageRank)

Results:• Outperform SVM in small pools

April 18, 2023 42@NUS

summary

Power law tag distribution• Data pool will remain small for most tags• Fundamental issue

Participatory knowledge is powerful – trust within context is important issue.

Future work: • Careful math analysis of coupling equations• Event structure / relationships need to be incorporated • Multi-source (email / Calendar / IM / blogs)

integration.

April 18, 2023 43@NUS

Conclusions

Thanks!Esp. Dick Bulterman, Mohan

April 18, 2023@NUS 44