Linking inside a video collection

LIME workshop - WWW2013

Linking inside a video collection- what and how to measure

First User Trials @ BBC

5/13/13

Robin Aly University of Twente, The NetherlandsMaria Eskevich Dublin City University, IrelandGareth Jones Dublin City University, IrelandRoeland Ordelman Netherlands Institute for Sound and Vision

University of Twente, The NetherlandsShu Chen Dublin City University, Ireland


VIDEO HYPERLINKING:AN AUDIOVISUAL ARCHIVE PERSPECTIVE

5/13/13


CAN VIDEO HYPERLINKING PLAY A ROLE IN AUDIOVISUAL ARCHIVE ACCESS?

Huge amounts of audiovisual contentDaily ingest of professional productionsLarge investments in digitization of older materials

5/13/13

Instead of …

Search for: “Things to do in London”

Relevant clip Alternative clips

We could do better …LINK ANCHORS orSEGMENTS

LINK TARGETS


Video linking scenario

• Archival viewing scenario (+1 week after broadcast)• Detail-on-demand perspective• Video-to-video linking• Start-off with archive search• Assuming an information need• Potentially search may go into more serendipitous or

exploratory types of behavior

• Create meaningful cross-sections through archive• Audiovisual archive interlinked on the fragment level• “Storytelling” type of approaches

5/13/13


VIDEO LINKING: USER PERSPECTIVE

Linked video is not yet analyzed on a fundamental level with users in real-life scenarios

5/13/13

Main research questions

• Do users understand the concept of video linking?• Do users think video linking is useful?• What elements in a video a user would see as a potential

video hyperlink – What are potential link anchors? – Can we classify these anchors?

• What users would expect as the endpoint (target) of such a hyperlink

• Understanding this is important to be able to build technology for automatic link generation in video (feature selection, avoid “link explosion”)


WHICH USERS TO ADDRESS?

5/13/13

Researchers & Educators

AXES Researchsystem

Journalists Research

Academic researchers & students

Investigate

Academic educators

Educate

Home users

AXES Home system

Home users Entertainment, Edutainment

Main group User Target

Media Professionals AXES Pro system

Broadcast Professionals

Reuse

Media Archivists Annotate

User group definitions


Experimental set-up

• Three parts:1. First user trials:– Explain concept of video hyperlinking– Let users define anchors in archival data

2. Find link targets in the collection3. Second user trials:– Users assess link targets

5/13/13


HOW TO GET THE LINK TARGETS?

5/13/13


Automatic link generation

• MediaEval benchmark evaluation– Searching & Linking task– Data set: BBC

• Task: find relevant links on the basis of – Anchors on their own (cond.1)– Anchors in context of the video segment they are

defined in (cond.2)• Systems use available archival metadata, subtitles,

speech transcripts, self-derived features

5/13/13


Data set• Anchor selection:

– 270h of BBC archival data• One complete week• Material not owned by BBC (e.g., Dr. Who) excluded

• For link target selection we could add:– One month before, one month after – Quasi-random selection of archival data

• All data have:– Archival metadata– EPG data– Subtitles– Speech transcripts – Shot segmentation

5/13/13


Participants

• Recruitment Agency• In anticipation on a rather complex task:

public users with computer & online experience

• 30 Public Users– Nationality: UK– Age group: 16-30– Computer familiarity: high– Online activity: high

5/13/13

LIME workshop - WWW2013 5/13/13


Student EnglishLearning Support

SupervisorSales Specialist

DJFinance Assistant

Student EconomicsAnalytical Chemist

CarerArrivals Coordinator

Bar StaffStudent English

Bar StaffPolice Officer

Hair and Beauty StylistAdministrator

Student Games Development

Operational Risk OfficerSales Manager

LabourerStudentLabourer

UnemployedSelf-employed interior

designerSpeech therapy student

Shop AssistantNightclub Promotions

Housing Officer Accountant

Photography studentRestaurant front of

house/waitress5/13/13


Trial set-up

• Introduction– Explaining the concept of video linking– Explaining the task

• Link anchor selection

• Interview (experimenter + 3 participants)

5/13/13



Information need

Text search Visual search





PRELIMINARY RESULTS

5/13/13


Overall

• Most participants understood the task• Not difficult• Technology would be a very nice-to-have

mentioning the “serendipitous” type of browsing behavior

• Data set too small for searching• Participants take music as link anchors (which

we did not anticipate!)

5/13/13


Manual assessment of user data

• Check validity of– selected clips and descriptions– selected anchors and description of anchor – description of expected link target– segmentation

• Classification of– Selected clips according to google-categories– Selected anchors:

• Whole scene• Speech• Object, moving• Object, static• Other (=Music)

5/13/13


Classification of selected clips

5/13/13

Documentar

y

Comedy

Sp

orts

Sp

ecial In

terestMusic

News

TV

Show

Healt

h And Fitness

Anim

ation

other

Child

ren and Fa

mily

Gay

and Le

sbian

Politi

cal

0

5

10

15

20

25


Classification of anchors

5/13/13Whole Scene Speech Other Object, moving Object, static

0

10

20

30

40

50

60

70

80

90


Anchor statistics

5/13/13


NEXT STEPS

5/13/13





Wrap up

• Description of experimental set-up to have better understanding of user behavior in video hyperlinking in archival, video2video scenario

• In spite of complexity, participants understood the task• First results indicate that

– Public users are focused on edutainment and entertainment– Prefer linking based on whole scenes– Are interested as well in music links

• Coming months: automatic link generation in MediaEval followed by link target assessment with same users at BBC

5/13/13

The authors:Robin Aly University of Twente, The NetherlandsRoeland Ordelman Netherlands Institute for Sound and Vision

University of Twente, The NetherlandsMaria Eskevich Dublin City University, IrelandGareth Jones Dublin City University, IrelandShu Chen Dublin City University, Ireland

Linking inside a video collection

Technology

video linking

video segment

video collection

user perspectivelinked

concept of video hyperlinking

potential link anchors

link anchors whichwe

apotential video hyperlink