Presentation on MediaEval Search & Linking task 2013

Set-‐up of the Search & Hyperlinking Task

in the Series

Evalua'on of Video Hyperlinking

Roeland Ordelman -‐ Audiovisual Content Access

University of Twente, The Netherlands Netherlands Ins@tute for Sound and Vision

Cross-‐Media Interac@on

Video hyperlinking

•  Topic with increasing aEen@on •  Relate video (fragments) to other mul@media sources •  Various perspec@ves and approaches, e.g.,: –  Recommenda@on/Detail-‐on-‐demand –  Seman@c/Event Web –  Storytelling/narra@ves –  Summariza@on/Clustering

•  Link up communi@es by means of special sessions, workshops and benchmarks –  Mul@media Modeling (MMM 2014) – Dublin. Special session on

Mul@media Hyperlinking and Retrieval – CfP due 01 August 2013 hEp://mmm2014.org/special-‐sessions/

7/2/13 DGA workshop -‐ July 2013, Paris

MediaEval

•  a mul$media benchmarking ini@a@ve running several tasks

•  evaluates new algorithms for mul@media access and retrieval linked to a use case

•  emphasis on the "mul$" in mul@media: audio, speech, visual content, tags, users, context.

•  new evalua@on paradigms involving users

h3p://www.mul:mediaeval.org


Video linking scenario

•  Audiovisual archive explora$on scenario •  Video-‐to-‐video linking •  Start with archive search:

assuming an “informa$on need” •  Poten@ally search may go into:

serendipitous or exploratory type of behavior •  Create meaningful cross-‐sec:ons through archive •  Audiovisual archive interlinked on the fragment level •  Automated “Storytelling” type of approaches


Terminology •  Video (e.g, 2 hours)

•  Interes@ng segment (e.g. 10 min)

•  Anchor: segment for which a user requests a link (e.g., 1 min) “I want to know more about this”

•  Hyperlink

•  Target: relevant segment for given anchor (e.g., 5 min)


ME 2013 Search & Hyperlinking •  ME 2012 S&HL “brave new” task: –  Search: retrieve known-‐item video segment given a natural language descrip@on

–  Linking: link known-‐item video segment to similar segments in the collec@on (blib.tv)

•  ME 2013 S&HL “regular” task –  Search: retrieve known-‐item video segment given a natural language descrip@on

–  Linking: link user-‐defined anchor within the known-‐item to relevant target video segments in the collec@on (bbc archive)


ME 2013 Search & Hyperlinking


Anchor Jump-‐in point

Task set-‐up

1.  Organizers: provide anchors to par@cipants

2.  Par@cipants: find relevant targets

3.  Organizers: evaluate relevance targets

4.  Organizers+Par@cipants: discuss at workshop 7/2/13 DGA workshop -‐ July 2013, Paris

1. Provide anchors

•  Requirements: – Large video data set – Real ‘users’ selec@ng anchors

•  Data: – 270h of BBC archival data

•  One complete week •  Excluded: material not owned by BBC (e.g., Dr. Who)

•  Users: …


Researchers & Educators AXES Research system

Journalists Research

Academic researchers & students

Inves:gate

Academic educators

Educate

Home users AXES Home system

Home users Entertainment, Edutainment

Main group User Target

Media Professionals AXES Pro system

Broadcast Professionals

Reuse

Media Archivists Annotate

User requirements study


!

!!

Users

•  Recruitment Agency •  In an@cipa@on on a rather complex task: public users with computer & online experience

•  30 Public Users – Na@onality: UK – Age group: 16-‐30 – Computer familiarity: high – Online ac@vity: high


ANCHOR SELECTION BBC User Trials


Search for an interes@ng video


potentially interesting videoSEARCH

BBC ARCHIVE


Formulate Informa@on

need

Text search Visual search

AXES PRO system

Select an interes@ng clip


BBC ARCHIVE

potentially interesting video

interesting clip

SEARCH

Select an anchor


BBC ARCHIVE


interesting clip

interesting segment for linking

SEARCH


Describe target

Describe anchor

Select anchor using sliders

Trial evalua@on

•  Most par@cipants understood the task •  Not difficult •  Technology would be a very nice-‐to-‐have men@oning the “serendipitous” type of browsing behavior

•  Par@cipants want music as link anchors (which we did not an@cipate on!)


Classifica@on of selected clips


0

5

10

15

20

25

Classifica@on of anchors


0

10

20

30

40

50

60

70

80

90

Whole Scene Speech Other Object, moving

Object, sta:c

First FA cup Final at Wembley

Mayor Bloomberg of New York talking about Ken Livingstone

2. Automa@c target selec@on •  ~ 15 Par@cipants

•  Data –  50 NL queries, 50 anchors –  270h of BBC archival data (used for anchor selec@on) –  + One month before ~1000h

•  Metadata –  Archival metadata –  Sub@tles –  Speech transcripts (provided by LIMSI & LIUM) –  Shot segmenta@on –  (Self-‐derived features)



BBC ARCHIVE


interesting clip


related video clip

video hyperlink

SEARCH

LINK

3. Evaluate target relevance


BBC ARCHIVE


interesting clip


related video clip

video hyperlink

SEARCH

LINK

RelevanceAssessment

Relevance assessment •  Default procedure:

–  Par@cipants of 1st user trials –  Evaluate for their own anchors the automa@cally generated targets from

ME par@cipants •  Risks:

–  No shows –  Par@cipants can only evaluate top-‐N (N-‐max=5?) targets for a given

anchor •  Fall-‐back op@on:

–  Use Amazon Mechanical Turk approach –  (same as we did in ME-‐2012 task) –  Drawback

•  relevance assessment not based on personal trail “informa@on need” >> “clip selec@on” >> “anchor selec@on” >> “target evalua@on”


Assessment interface


Original anchor & target

descrip@on

ANCHOR SUGGESTED TARGETS

Relevance assessment & comments

Evalua@on metrics •  MAP evalua@on for both searching and linking •  Issues: – MAP does not provide means to evaluate segmenta@on (e.g., correct video but jump-‐in point is incorrect)

–  limit relevant results to a window of N (N=60 seconds) before and awer the actual beginning of the relevant segment

•  Addi@onal metric: – Mean Generalized Average Precision (mGAP)

•  awards runs that are closer to the jump-‐in point of the relevant content


4. Discuss at workshop



18-‐19 October 2013 Barcelona, Spain


Work funded by:

Task Organizers: Robin Aly Maria Eskevitch Gareth Jones Roeland Ordelman


Presentation on MediaEval Search & Linking task 2013

Technology

paris bbc archive

paris anchor jumpinpoint

onscenario video

me2013shlregulartask

noshows par

cipants data

cipants risks

gate academic educators