Semantics at the multimedia fragment level SSSW 2013

Semantics at the multimedia fragment level or how enabling

the remixing of online media Raphaël Troncy <raphael.troncy@eurecom.fr>

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 2

Once upon a time …

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 3

… leading to sharing Media Fragments

Publishing status message containing a Media Fragment URI Use a ‘#’ ! Highlight a

video sequence

Highlight a region to pay attention to

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 4

What are Media Fragments?

t 0 20 35 temporal media fragment

spatial media fragment

track media fragment

11/07/2013 - - 5 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Media Fragments (temporal)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 6

Fragment beginning Fragment end Playback progress

Original resource length

Media Fragments (spatial) + Demo

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 7

semi-opaque overlay

highlighted fragment

Media Fragments URIs

Bookmark / Share parts (fragments) of audio/video content

Annotate media fragments

Search for media fragments

Mash-ups

Conserve bandwidth

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 8

http://www.w3.org/TR/media-frags-reqs/ http://www.w3.org/TR/media-frags/

Video annotation

11/07/2013 - - 9 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Video interactivity

Cubism Expressionism

Fauvism

FACETS / PROPERTIES OF CONCEPT

CONCEPT IN PLAYER

CONTENT ENRICHMENT

11/07/2013 - - 10 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Video Accessibility

What is required to make video accessible on the Web?

Technologies: Annotating: automatic (speech transcription) and manual (social

collaborative annotation tool) Addressing: pointing to, retrieving, transmitting only parts of media Rendering: video visualization for the impaired, Braille output

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 11

Benchmarking: Sphinx, HTK, Julius

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 12

Semantic indexing at the fragment level

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 13

Benchmarking: Sphinx, HTK, Julius

NER on subtitle blocks

Interlinking with the Linked Data Cloud to enable semantic search

What is a Named Entity recognition task?

A task that aims to locate and classify the name of a person or an organization, a location, a brand, a product, a numeric expression including time, date, money and percent in a textual document

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 14

NER Tools and Web APIs

Standalone software GATE Stanford CoreNLP Temis

Web APIs

http://nerd.eurecom.fr/

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 15

Compare performances of NER and NEL tools Understand strengths and weaknesses of different Web APIs Adapt NER processing to different context

(Learn how to) Combine NER (/ NEL) tools

Participate in various benchmarks

NERD: Named Entity Recognition and Disambiguation

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 16

What is NERD? REST API2 ontology1

UI3

1 http://nerd.eurecom.fr/ontology 2 http://nerd.eurecom.fr/api/application.wadl

3 http://nerd.eurecom.fr

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 17

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 18/15

Alchemy API

DBpedia Spotlight

Evri Extractiv Lupedia Open Calais

Saplo Wikimeta Yahoo! Zemanta

Language EN,FR, GR,IT, PT,RU, SP,SW

EN GR* PT* SP*

EN,IT

EN EN,FR, IT

EN,FR SP

EN, SW

EN,FR SP

EN EN

Granularity OEN OEN OED OEN OEN OEN OED OEN OEN OED

Entity position

N/A char offset

N/A word offset

range of chars

char offset

N/A POS offset

range of

chars

N/A

Classification schema

Alchemy DBpedia FreeBase Scema.or

g

Evri DBpedia DBpedia LinkedM

DB

Open Calais

N/A ESTER

Yahoo FreeBase

Number of classes

324 320 5 34 319 95 5 7 13 81

Response Format

JSON MicroF XML RDF

HTML JSON RDF XML

HTML

JSON

RDF

HTML JSON RDF XML

HTML JSON RDFa XML

JSON MicroFormat

JSON JSON XML

JSON XML

XML JSON RDF

Quota (calls/day)

30000 unl 3000

3000 unl 50000 1333 unl 5000 10000

Factual comparison of 10 Web NER tools

Aligned the taxonomies used by the extractors

NERD Ontology

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 19

NERD type Occurrence

Person 10

Organization 10

Country 6

Company 6

Location 6

Continent 5

City 5

RadioStation 5

Album 5

Product 5

... ...

Building the NERD Ontology

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 20

NERD REST API

GET, POST, PUT,

DELETE

/document /user /annotation/{extractor} /extraction /evaluation ...

JSON

“entities” : [{ “entity”: “Tim Berners-Lee” , “type”: “Person” , “uri”: "http://dbpedia.org/resource/Tim_berners_lee", “nerdType”: "http://nerd.eurecom.fr/ontology#Person", “startChar”: 30, “endChar”: 45, “confidence”: 1, “relevance”: 0.5 }]

Rizzo G., Troncy R. (2012), NERD: A Framework for Unifying Named Entity Recognition and Disambiguation Web Extraction Tools. In: European chapter of the Association for Computational Linguistics (EACL'12), Avignon, France.

RDF

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 21

NERD meets NIF

Model documents through a set of strings deferencable on the Web

: offset_23107_ 23110 a str:String ; str:referenceContext :offset_0_26546 .

: offset_23107_ 23110 sso:oen dbpedia:W3C.

dbpedia:W3C rdf:type nerd:Organization .

Map string to entity

Classification

Rizzo G, Troncy R., Hellmann S. and Bruemmer M. (2012), NERD meets NIF: Lifting NLP Extraction Results to the Linked Data Cloud. In: (LDOW'12) Linked Data on the Web (WWW'12), Lyon, France.

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 22

NERD User Dashboard

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 23

NERD User Interface

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 24

History of NER benchmarks CoNLL 2003 and CoNLL 2005

schema (4 types): person, organization, location and miscellaneous

ACE 2004, ACE 2005 and ACE 2007 schema (7 types): person, organization, location, facility, weapon,

vehicle and geo-political entity entity recognition, co-ref, find relationships among entities extracted

TAC 2009 (Knowledge Base Track) schema (3 types): person, organization and location create a knowledge base from the named entities extracted

ETAPE 2012 (Named Entity Task) schema: Quaero (7 main types, 32 sub-types)

MSM 2013: tweet corpus ! schema (4 types): person, organization, location, miscellaneous

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 25

ETAPE 2012 challenge

genre train dev test sources

TV news 7h 40m 1h 40m 1h 40m BFM Story, Top QUestions (LCP)

TV debates 10h 30m 5h 10m 5h 10m Pile et Face, Ca vous regarde, Entre les lignes (LCP)

TV amusements - 1h 05m 1h 05m La place du village (TV8)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 26

Train Dev Eval Item length 26h 10h 55m 10h 55m Nb files 44 15 15 Nb words 290517 91656 115511 Nb Named Entities 46763 14398 13055 Nb unique categories 33 33 33

NERD @ ETAPE (naïve combined strategy)

(eA1,tA1,URIA1,siA1,eiA1) ... ... ...

`

(eA2,tA2,URIA2,siA2,eiA2) (eA3,tA3,URIA3,siA3,eiA3)

(eN2,tN2,URIN2,siN2,eiN2) (eN1,tN1,URIN1,siN1,eiN1)

extraction

cleaning

fusion When at least 2 extractors classify the same entity with a different type then we apply a preferred selection order

(empirically defined): Wikimeta, AlchemyAPI, OpenCalais, Lupedia

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 27

Participation at ETAPE (combined+ strategy)

(eA1,tA1,URIA1,siA1,eA1)

`

(eA2,tA2,URIA2,siA2,eiA2)

(eN2,tN2,URIN2,sN2,eN2) (eN1,tN1,URIN1,sN1,eN1)

...

ETAPE Train & Dev

Learned model

Created static rules

fusion Conflicts handled by

priority selection: own, Wikimeta,AlchemyAPI,OpenCalais,Lupedia

POS tagger

Apply rules

(e1,t1,URI1,si1,ei1)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 28

NERD Global results

SLR Precision Recall F-measure %correct

combined 86.85% 35.31% 17.69% 23.44% 17.69%

combined+ 188.81% 15.13% 28.40% 19.45% 28.40%

Combined+ : Eval corpus differs substantially from the Train & Dev corpora. The static rules do not fit well the Eval corpora and they introduce classification noise.

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 29

Per-extractor results SLR Precision Recall F-measure %correct

alchemyapi 37.71% 47.95% 5.45% 9.68% 5.45%

lupedia 39.49% 22.87% 1.56% 2.91% 1.56%

opencalais 37.47% 41.69% 3.53% 6.49% 3.53%

wikimeta 36.67% 19.40% 4.25% 6.95% 4.25%

combined (nerd)

86.85% 35.31% 17.69% 23.44% 17.69%

combined+ (nerd+)

188.81% 15.13% 28.40% 19.45% 28.40%

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 30

- 31 11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Learning How to Combine NER Extractors

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 32

NERD on CoNLL 2003 (NER task)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 33

NERD on MSM 2013 (NER task)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 34

NERD on MSM 2013 (NEL task)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 35

Media Fragment Enricher: http://mfe.synote.org/mfe/

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 36

Linking pieces of knowledge

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 37

Linking pieces of knowledge

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 38

Named Entities for Video Classification

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 39

Workflow

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 40

Media Fragment Enricher Services

Media Fragment Enricher UI

Metadata & timed-text

NERD Client RDFizator Triple Store

Categori-zation

Video and metadata preview

Video replay with subtitles and aligned NEs

1: Video URL

2: Metadata

3: meta-data 4:NERDify

5:Timed Text 6: NEs with time

alignment (json)

7: RDFize (ttl)

8: Generate Category

9: SPARQL query

Channel signature based on NE distribution

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 41

Media Collector

Composition of media item extractors (12 SNs) Rely on search APIs + a fix 30s timeout window to provide results Fallback on screen scraping when necessary (Twitter ecosystem)

Implemented as a NodeJS server

Serialize results in a common schema (JSON)

Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 11/07/2013 - - 42

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 43

Deep link Permalink

Clean text for NLP processing

Aggregate view of ALL social interactions

12 Social Networks

Media Finder (www2013)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 44

Media Finder (zooming on media items)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 45

Media Finder (timeline view)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 46

Media Finder Architecture

Media items harvesting using the Media Server http://eventmedia.eurecom.fr/media-

server/search/{combined}/{term} https://github.com/vuknje/media-server (@tomayac fork)

Image near de-duplication DCT signature on image and video frame,

Hamming distance between image pairs

Clustering and disambiguation Named Entity Extraction using NERD Topic Generation using LDA

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 47

Media Finder (named entities clustering)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 48

Media Finder (zooming in a cluster)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 49

Media Finder: http://mediafinder.eurecom.fr/

Live Topic Generation from Event Streams WWW 2013 Demo Session http://www.youtube.com/watch?v=8iRiwz7cDYY

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 50

Tracking an event: Italian Election

Repeated queries over a period of time We have tracked and analyzed media posts tagged as

elezioni2013 from 2013-02-26 to 2013-03-03 Cron job: every 30 minutes over the 6 days Slice the data in 24 hours slots

Research questions: Can we re-create the news headlines?

Storyboarding: http://mediafinder.eurecom.fr/story/elezioni2013

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 51

Tracking an event: Italian Election

Dataset: ~16501 microposts containing (duplicate) media items ~21087 Named Entities extracted

Clustering NER and LDA Generate Bag of Entities (BOE) disambiguated with a

DBpedia URI

Examples: Monti, Bersani, Italia, Berlusconi, Grillo, Stelle

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 52

Tracking an event: Italian Election

Tracking and Analyzing The 2013 Italian Election ESWC 2013 Demo Session http://www.youtube.com/watch?v=jIMdnwMoWnk

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 53

Multimedia and Semantic Web

Different Ecosystems: Local identifiers Specific metadata formats

Huge amount of Multimedia Content

Low number of links between content

11/07/2013 - - 54 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Multimedia and Semantic Web

Universal Identifiers: URI’s

Common description formats

Easy interlinking between content

11/07/2013 - - 55 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Media Fragments

nerd:Location Cafe Rick

Nerd:Person H. Bogart

Nerd:Person I. Bergman

nerd:Location Casablanca

Media Fragment URI 1.0 Chapters Scenes Shots etc…

http://data.linkedtv.eu/media/e2899e7f#t=14,15

LinkedTV Ontology

11/07/2013 - - 56 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Hypervideo

nerd:Location Cafe Rick

Nerd:Person H. Bogart

Nerd:Person I. Bergman

nerd:Location Casablanca

Nerd:Person E. Tierney

nerd:Location China

11/07/2013 - - 57 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Web + TV experience

http://www.youtube.com/watch?v=4mSC685AG7k

11/07/2013 - - 58 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013

Research Vision (context)

Knowledge Graphs everywhere Google Knowledge Graph, Microsoft Entity Graph,

Yahoo! Web of Things, Wikidata Open Data, Structured Data, Linked Data

The rise of social media Events happen all the time and are the topic of social network

conversations, also in form of event-related multimedia data Videos and photos are (re-)shared on multiple social networks Events can be

planned or unplanned

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 59

(Read the background story http://www.washingtonpost.com/about-those-2005-and-2013-photos-of-the-crowds-in-st-peters-square)

Research Vision (opportunity)

Video is a first class citizen on the Web Annotations: Ontology and API for Media Resources Access: Media Fragments URI NERD platform for extracting key information from

learning resources including videos

The Linked Media vision Extracting semantic knowledge from social media Collect, enrich and visualize media memes shared by

the crowd Generate visual stories about what is happening in the

world (summarization)

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 60

Winter School: http://winterschool.mediamixer.eu/

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 61

Credits

Giuseppe Rizzo, Vuk Milicic, José Luis Redondo Garcia (EURECOM)

Thomas Steiner (Google Inc.)

Marieke van Erp (Free University of Amsterdam)

Yunjia Li (University of Southampton)

… and many other students

11/07/2013 - Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013 - 62

http://www.slideshare.net/troncy

11/07/2013 - - 63 Semantics at the multimedia fragment level - SSSW, Cercedilla, July 2013