Multimedia Analytics: Synergy Between Human and Machine by ...videos.rennes.inria.fr/Workshop-Multimedia... · I Multimedia data increasingly important I Valuable sources of knowledge,

MULTIMEDIA ANALYTICS: SYNERGYBETWEEN HUMAN AND MACHINE BY

VISUALIZATION

Marcel Worring, Jan Zahalka, Stevan Rudinac

Intelligent Systems Lab Amsterdam Amsterdam Data ScienceUniversity of AmsterdamAmsterdam Data Science

INTRODUCTION

I Multimedia data increasingly important

I Valuable sources of knowledge, for example:

I Forensics: analyze multimedia data for evidence of ISISinvolvement

I Travel industry: analyze social media data to map trendingplaces of interest. . .

INTRODUCTION

I Multimedia data increasingly importantI Valuable sources of knowledge, for example:

I Forensics: analyze multimedia data for evidence of ISISinvolvement

I Travel industry: analyze social media data to map trendingplaces of interest. . .

MULTIMEDIA AS A KNOWLEDGE SOURCE

I Night Watch by Rembrandt. How to describe it?g

I Art? Painting? People? Military unit? Amsterdam?g

I Art? Painting? People? Military unit? Amsterdam? . . .Content, technical parameters, geo location, . . .

I Description depends on context provided by the analystAnalyst needs to interact with the system

Comments

Metadata. . .

I Multimedia items contain multiple types of dataIntegrating them improves the information gain

I What if we have millions of images, tags, metadata. . . ?Intelligent navigation capabilities required from the system

MULTIMEDIA ANALYTICS

I How do we move towards interactive, intelligent, andintegrated multimedia systems?

I Possible answer: multimedia analytics

MultimediaAnalysis

MultimediaAnalytics

InfoVis Visual Analytics

MULTIMEDIA ANALYTICS

I How do we move towards interactive, intelligent, andintegrated multimedia systems?

I Possible answer: multimedia analytics

MultimediaAnalysis

MultimediaAnalytics

InfoVis Visual Analytics

RELATED WORK

I Extensive survey work involving ∼ 800 references

I Covered relevant work from last 10 years:

I Multimedia analyticsI Multimedia visualizationI Information visualizationI Visual analyticsI Automated multimedia analysis

I Multimedia Analytics Article Library (MAAL):

I staff.fnwi.uva.nl/j.zahalka/maal.htmlI 374 catalogued references

RELATED WORK

I Extensive survey work involving ∼ 800 referencesI Covered relevant work from last 10 years:

I Multimedia Analytics Article Library (MAAL):

I staff.fnwi.uva.nl/j.zahalka/maal.htmlI 374 catalogued references

RELATED WORK

I Extensive survey work involving ∼ 800 referencesI Covered relevant work from last 10 years:

I Multimedia Analytics Article Library (MAAL):I staff.fnwi.uva.nl/j.zahalka/maal.htmlI 374 catalogued references

PIPELINE

interactive

model update

navigation

directions

Visualization

Knowledge

Category 1people

61 items. . .

Category 2nature

93 items. . .

DataMM collection

Images

Annotations

Metadata

I Multimedia instantiation of the visual analytics process (Keim et al., Visualanalytics: Scope and challenges, 2008)

TASK MODEL

Exploration

Search

Categorization

I Exploration: uncovering the overall structureI Search: finding particular items

I Exploration-search axis: E-S ratio changes dynamicallyI Mental model attributes: semantic→ categorical

TASK MODEL

Exploration

Search

Categorization

I Exploration: uncovering the overall structureI Search: finding particular itemsI Exploration-search axis: E-S ratio changes dynamically

I Mental model attributes: semantic→ categorical

TASK MODEL

Exploration

Search

Categorization

TASK MODEL

Exploration

Search

Categorization

TASK MODEL

Exploration

Search

Categorization

I Exploration: uncovering the overall structureI Search: finding particular itemsI Exploration-search axis: E-S ratio changes dynamicallyI Mental model attributes: semantic→ categorical

TASK MODEL

Exploration

Search

Categorization

I Exploration: uncovering the overall structureI Search: finding particular itemsI Exploration-search axis: E-S ratio changes dynamicallyI Mental model attributes: semantic→ categorical

CATEGORIZATION

I Categorization — assigning individual multimedia itemsinto categories defined by the analyst

CHALLENGE: THE GAPS

Complex and abstract semantics

Recognized instantly

Put in context

Limited semantics

Takes time, computationally costly

No context

semantic gap

New categories on the fly

Non-exclusive categories

Dynamic category semantics

Static no. of classes

Exclusive classes

Static class semantics

pragmatic gap

I Multimedia analysis capabilities very different for humansand machines

I Semantic gap [Smeulders et al. 2000] — richness ofsemantics

I Pragmatic gap (our work) — flexibility of the model

CHALLENGE: THE GAPS

Put in context

Limited semantics

No context

semantic gap

Exclusive classes

pragmatic gap

CHALLENGE: THE GAPS

Put in context

Limited semantics

No context

semantic gap

Exclusive classes

pragmatic gap

SIMILARITY BROWSER

FORK BROWSER

PHOTO CUBE

MULTIMEDIA PIVOT TABLES

STATE OF THE ART

Limited Intermediate Advanced

Limited

Inter-mediate

AdvancedGoal

I-SI NewdlesVisitInformedia

Canopy

Similaritybrowser

INA browser

MediaTable

semantic gap

pragmatic gap

I Systems advance w.r.t. gapsI Algorithms and techniques allow realization of our model

STATE OF THE ART

Limited

Inter-mediate

AdvancedGoal

Canopy

Similaritybrowser

INA browser

MediaTable

semantic gap

pragmatic gap

I Systems advance w.r.t. gaps

I Algorithms and techniques allow realization of our model

STATE OF THE ART

Limited

Inter-mediate

AdvancedGoal

Canopy

Similaritybrowser

INA browser

MediaTable

semantic gap

pragmatic gap

I Systems advance w.r.t. gapsI Algorithms and techniques allow realization of our model

INSTANTIATING THE MODEL

NEW YORKER MELANGE

I Interactive New York venue recommender

I “Explore the city through the eyes of social media usersthat share interests with you.”

I newyorkermelange.com

I ACM Multimedia Grand Challenge 2014 1st Prize

NEW YORKER MELANGE

I Interactive New York venue recommenderI “Explore the city through the eyes of social media users

that share interests with you.”

I newyorkermelange.com

NEW YORKER MELANGE

that share interests with you.”I newyorkermelange.com

NEW YORKER MELANGE

that share interests with you.”I newyorkermelange.com

NEW YORKER MELANGE: INGREDIENTS

Visual & textfeatures for

venues & users

Grid, map

Interesting venuesto visit

indicate

relevant

users & venues

suggest

relevant

users & venues

Exploration SearchNY Melange

NEW YORKER MELANGE: INGREDIENTS

Visual & textfeatures for

venues & users

Grid, map

Interesting venuesto visit

indicate

relevant

users & venues

suggest

relevant

users & venues

Exploration SearchNY Melange

NEW YORKER MELANGE

DATASET

New York venuesVenue images

Images, metadata

Q(venue name,geo)

DATASET

New York venuesVenue images

Images, metadata

Q(venue name,geo)

DATASET

I >1M New York venue images with metadata

I Real dataset with a purposeI Query strategy designed to reduce noise

I Exploitable size-noise tradeoff

I Each image has a venue category label→ ready forclassification

DATASET

I >1M New York venue images with metadataI Real dataset with a purpose

I Query strategy designed to reduce noise

DATASET

I >1M New York venue images with metadataI Real dataset with a purposeI Query strategy designed to reduce noise

DATASET

I >1M New York venue images with metadataI Real dataset with a purposeI Query strategy designed to reduce noise

I Exploitable size-noise tradeoffI Each image has a venue category label→ ready for

classification

VENUE/USER TOPICS

Dataset

Images

Annotations

Foursquare

Flickr

Picasa

Features

visual

concepts

latent

topics

ConvNet

Clustering

Venuetopics

Visual

Usertopics

Visual

VENUE/USER TOPICS

Dataset

Images

Annotations

Foursquare

Flickr

Picasa

Features

visual

concepts

latent

topics

ConvNet

Clustering

Venuetopics

Visual

Usertopics

Visual

VENUE/USER TOPICS

Dataset

Images

Annotations

Foursquare

Flickr

Picasa

Features

visual

concepts

latent

topics

ConvNet

Clustering

Venuetopics

Visual

Usertopics

Visual

USER PREFERENCE LEARNING

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

Initial

interface

Negatives

Positives

+relevant

venues

topics

(random sample)

Linear

ranking

selection

topics

interface

+relevant

+non-relevant

EVALUATION: SCHEME

I Real user data

I 25% of the visited venues withheld, rest used to seed thesystem

I 10 interaction roundsI Measure: average recall of the withheld venuesI Only exact withheld venues count as match

EVALUATION: SCHEME

I Real user dataI 25% of the visited venues withheld, rest used to seed the

system

I 10 interaction roundsI Measure: average recall of the withheld venuesI Only exact withheld venues count as match

EVALUATION: SCHEME

systemI 10 interaction rounds

I Measure: average recall of the withheld venuesI Only exact withheld venues count as match

EVALUATION: SCHEME

systemI 10 interaction roundsI Measure: average recall of the withheld venues

I Only exact withheld venues count as match

EVALUATION: SCHEME

systemI 10 interaction roundsI Measure: average recall of the withheld venuesI Only exact withheld venues count as match

EVALUATION: RESULTS

1 2 3 4 5 6 7 8 9 100

Interaction Round

AverageRecall

Baseline

NYM-VNYM-T

NYM-VT

FUTURE OF MELANGE: SOFTWARE

I Consolidated Melange deployable everywhere

I AmsterdamI Hong KongI BeijingI Washington, D. C.I PragueI Rennes

I Consolidated Melange deployable everywhereI Amsterdam

I Hong KongI BeijingI Washington, D. C.I PragueI Rennes

I Consolidated Melange deployable everywhereI AmsterdamI Hong Kong

I BeijingI Washington, D. C.I PragueI Rennes

I Consolidated Melange deployable everywhereI AmsterdamI Hong KongI Beijing

I Washington, D. C.I PragueI Rennes

I Consolidated Melange deployable everywhereI AmsterdamI Hong KongI BeijingI Washington, D. C.

I PragueI Rennes

I Consolidated Melange deployable everywhereI AmsterdamI Hong KongI BeijingI Washington, D. C.I Prague

I Rennes. . .

I Consolidated Melange deployable everywhereI AmsterdamI Hong KongI BeijingI Washington, D. C.I PragueI Rennes

CONCLUSION

I A model of multimedia analytics integration, tasks andchallenges

I Based on extensive survey work

I Multimedia Analytics Article Library:staff.fnwi.uva.nl/j.zahalka/maal.html

I Current state-of-the-art techniques allow realization

I Ample research opportunities in closing the gaps

I Model already successfuly instantiated

I New Yorker Melange: newyorkermelange.com

ImagesText

Metadata

CONCLUSION

I Based on extensive survey workI Multimedia Analytics Article Library:staff.fnwi.uva.nl/j.zahalka/maal.html

I Current state-of-the-art techniques allow realization

I Ample research opportunities in closing the gaps

ImagesText

Metadata

CONCLUSION

I Current state-of-the-art techniques allow realizationI Ample research opportunities in closing the gaps

ImagesText

Metadata

CONCLUSION

I Current state-of-the-art techniques allow realizationI Ample research opportunities in closing the gaps

I Model already successfuly instantiatedI New Yorker Melange: newyorkermelange.com

ImagesText

Metadata

Multimedia Analytics: Synergy Between Human and Machine by ...videos.rennes.inria.fr/Workshop-Multimedia... · I Multimedia data increasingly important I Valuable sources of knowledge,

Documents

SYNERGY - Euro-Pharmat · stent SYNERGY sont décrites dans...

Parameterized veriﬁcation of networks of many identical...

RENESAS SYNERGY PLATFORM · Synergy Software Package SSP...

Synergy Basic

Synergy Brochure

Synergy Comprehensive Exercise Rehab System › shop ›...

Synergy Spanish Action Guide - Amazon S3 · Synergy Spanish...

Floating Solar Prospects in Central and West Asia · Ciel.....

Synergy revision

[Mairal,Bach,Ponce,Sapiro&ZissermanCVPR’08]videos.rennes.i...

Synergy Mill Manual - Weber Systems: Synergy CAD CAM

Bimanual Haptic Interaction with Virtual Environments...

Olga UGOlga UGM -Synergy PresentationM -Synergy Presentation

HPE Synergy infographic - HPE Zone HPE Synergy... · Title:...

Synergy centres

Structural Monitoring on Germanys Offshore Research...