Search and Anchoring Video Archives SAVA Overview Maria Eskevich, Robin Aly , David N. Racca Roeland Ordelman, Shu Chen, Gareth J.F. Jones
Search and Anchoring Video Archives
SAVA Overview
Maria Eskevich, Robin Aly, David N. Racca
Roeland Ordelman, Shu Chen, Gareth J.F. Jones
Outline
• Task definiGon • Dataset (Videos + user input) • Ground truth creaGon • EvaluaGon procedure • Results
5/13/13 LIME workshop -‐ WWW2013
Terminology • Video (e.g, 2 hours)
• Search result (e.g. 10 min)
• Anchor: segment for which a user requests a link (e.g., 1 min) “I want to know more about this”
• Hyperlink
• Target: relevant segment for given anchor (e.g., 5 min)
7/2/13 DGA workshop -‐ July 2013, Paris
Use Case
7/2/13 DGA workshop -‐ July 2013, Paris
Video 1
Video 2 Video 3
Text query: Speech cue: “hunger around the globe”Visual cue: “hungry people slim bodies”
Search results:Video Start End Jump-InVideo1 13:30 15:00 13:30Video10 15:10 17:00 15:10Video12 29:50 31:00 29:50
Target Target
Result 1
Anchor Anchor Anchor Anchor
HyperlinkHyperlink
Search Task DefiniGon
Video 1
Text query: Speech cue: “hunger around the globe”Visual cue: “hungry people slim bodies”
Search results:Video Start End Jump-InVideo1 13:30 15:00 13:30Video10 15:10 17:00 15:10Video12 29:50 31:00 29:50
Result 1
User -‐ Input ParGcipant Submission
Anchoring Task DefiniGon
Video 1Anchor? Anchor? Anchor? Anchor?
Input ParGcipant Submission
Video Start End Video
Task history
• ME 2011 Rich Speech Retrieval (predecessor) • ME 2012 S&HL “brave new” task: – Search & Linking (blip.tv)
• ME 2013 S&HL “regular” task – Search: (known-‐item) Linking: (bbc collec=on)
• ME 2014 S&HL “regular” task – Search: (mulG relevant) Linking: (mul= relevant)
• ME 2015 Search & Anchoring + Linking@TRECVid – Search: mulG relevant – Anchoring: "brave new task"
7/2/13 DGA workshop -‐ July 2013, Paris
Dataset: Video collecGon
• Test collecGon Search: – copyright cleared broadcasts from the period of 12.05.2008 – 31.07.2008
– 2686 hours – ~200 videos rebroadcast or audio-‐visual signal was out of sync.
• Anchoring test collecGon – 33 videos for anchoring for anchors of 2013 and 2014 ediGon
5/13/13 LIME workshop -‐ WWW2013
Dataset: Query GeneraGon
• Users – BBC employees – BriGsh Film InsGtute – Journalists, + prospecGve Students
• InstrucGons: – Personal – Teleconference session
• SubjecGve impression: task difficult but doable.
5/13/13 LIME workshop -‐ WWW2013
GeneraGon of Info'Need
7/2/13 DGA workshop -‐ July 2013, Paris
Formulate InformaGon
need
Text search
Visual search
Dataset: Search Queries
• 42 queries for search task, e.g. <top> <itemId>item_35</itemId> <queryText>michael jackson quincy jones</queryText> <visualCues>singer, dancing, michael jackson</visualCues> </top>
5/13/13 LIME workshop -‐ WWW2013
Ground truth creaGon
• Search sub-‐task: – MTurk – top-‐10 of all runs – Showing descripGon for desired segments – Binary relevance judgment with explanaGon
• Anchoring sub-‐task: – Combine top-‐25 segments to connected parts – MTurk for these segments – Binary relevance judgments
5/13/13 LIME workshop -‐ WWW2013
Search Assessment
Search result DescripGon informaGon
need
Relevance judgment
Judgment details &
VerificaGon
Anchoring Assessment
Context
Submired anchor
Relevance judgment
Judgment details & verificaGon
Context
EvaluaGon: Search Task
• Depending on jump-‐in point • Measures: – MAP, P_10 adapted binary measures (overlap important decision)
• Maisp
5/13/13
EvaluaGon: Anchoring Task
• Measures: P@10 , Recall • Segments overlapping with relevant segments are considered relevant
• Recall: How many of the known-‐relevant segments were found
Conclusions
• Task defined by users • Search task: maisp measure • First steps anchoring task • Few runs prevent strong conclusions