Me13sh task overview

Search and Hyperlinking Overview

Robin Aly, Maria Eskevich, Roeland Ordelman, Gareth J.F. Jones

LIME workshop - WWW2013

Outline

• The search and hyperlinking task• Dataset (Videos + user input)• Ground truth creation• Evaluation procedure• Results

5/13/13

DGA workshop - July 2013, Paris

ME 2013 Search & Hyperlinking

• ME 2012 S&HL “brave new” task: – Search: retrieve known-item video segment given a

natural language description – Linking: link known-item video segment to similar

segments in the collection (blib.tv)• ME 2013 S&HL “regular” task– Search: retrieve known video segment (known-

item) given a textual query and visual cues– Linking: link user-defined anchor within the known-

item to relevant target video segments7/2/13

DGA workshop - July 2013, Paris

Terminology• Video (e.g, 2 hours)

• Interesting segment (e.g. 10 min)

• Anchor: segment for which a user requests a link (e.g., 1 min)

“I want to know more about this”• Hyperlink

• Target: relevant segment for given anchor (e.g., 5 min)7/2/13

DGA workshop - July 2013, Paris

ME 2013 Search & Hyperlinking

7/2/13

AnchorJump-in point

LIME workshop - WWW2013

Dataset: Video collection

• copyright cleared broadcasts from the period of 01.04.2008 – 11.05.2008

• 1667 hours, 2323 videos• ~200 videos were rebroadcast

5/13/13

LIME workshop - WWW2013

Dataset: Query

• 28 Users- Policemen, Hair dresser, Bouncer, Sales manger, Student, Self-employed

• Two hour session:– Browse the archive– Define known-item– Define anchor

5/13/13

DGA workshop - July 2013, Paris7/2/13

Formulate Information

need

Text search Visual search

AXES PROsystem

DGA workshop - July 2013, Paris7/2/13

Describe target

Describe anchor

Select anchorusing sliders

LIME workshop - WWW2013

Dataset: outcome (1/2)

• 50 known-items<top> <itemId>item_9</itemId> <queryText>animal park, kenya marathon , wildlife reserve</queryText> <visualQueues>animals, kenya wildlife reserve,marathon</visualQueues></top>

5/13/13

LIME workshop - WWW2013

Dataset: outcome (2/2)

• 98 anchors<anchor> <anchorId>anchor_3</anchorId> <startTime>5.29</startTime> <endTime>6.03</endTime> <item> <fileName>v20080506_093000_...</fileName> </item</anchor>

5/13/13

LIME workshop - WWW2013

Ground truth creation

• Search sub-task: done (known-items)• Judgments linking sub-task:– top-10 of one run per participant (7) by users

locally at BBC– top-10 of most runs using Amazon's mechanical

turk– ~70% Agreement– Similar measurements

5/13/13

LIME workshop - WWW2013

Unifying Qrels

5/13/13

Judgment 1

Judgment 2

Result

Case1: Merge relevant Case2: Prioritize relevant over non-relevant

Ground truth statistics

Evaluation: search sub-task

• Depending on jump-in point• Measures: MRR, MGAP, MSAP• Rebroadcasts not considered

5/13/13

Evaluation: linking sub-task

• Measures: P@5,10,20 and map• Resolving conflicts

Judgedsegment

Submittedsegment

Result:

RESULTS

Per Task

Results: search sub-task

Results: hyperlinking sub-task

Conclusions

• Task defined by real users• Innovative elements perceived as interesting• Crowdsourcing valid evaluation strategy• Experimental setup works• Provides a promising starting point for further

research

The Search and Hyperlinking task was funded by

We are grateful to Jana Eggink and Andy O'Dwyer

from the BBC for preparing the collection and hosting the user trials.

... and of course Martha for advise & crowdsourcing access.