Video Hyperlinking (LNK) Task at TRECVid 2016

VideoHyperlinking(LNK)TRECVid 2016

MariaEskevich,MarthaLarson(Radboud University)RobinAly,RoelandOrdelman (UniversityofTwente)

GarethJ.F.Jones(DublinCityUniversity)BenoitHuet (EURECOM)

Taskintuition

16Nov.2016 TRECVidVideoHyperlinkingOverview

Taskintuition

...thequeen...


Terminology• Video (e.g.,2hours)

• Videoclip (e.g.,10min)

• Anchor:segment(unconstraint)forwhichauserrequestsalink(e.g.,1min)“Iwanttoknowmoreaboutthis”• Hyperlink

• Target:relevantsegmentforgivenanchor(Nosegmentationenforced)


Collectiondetails

• Blip10000dataset:14,838semi-professionallycreatedvideos• ASRtranscripts:LIMSI(2012,2016)andLIUM(2012)• Shotsegmentation(2012)• Visualfeatures(AlexNet):1000visualconcepts


Anchorcreation


Observationfrom2015• Toomuchambiguityinanchors•Previouslycreatedanchorswerenotalwaysmultimodal


Strategyfor2016• Truemultimodalityofanchorsreduceambiguity•Weassumemorereliablerelevanceassessment

TV’16:focusonmultimodality

Lookforanchorscontainingacombinationofverbal-visualinformation:• Verballinguisticcues:̀ cansee',`seeinghere',`thislooks’,`lookslike',`showing',`wanttoshow’• Visualcues:actionsandobjectscrucialforthisvideoarenotexplicitlynamedormentioned


Anchorcreation:practicaldetails

• Devset:28anchorscollectedforSearchandHyperlinking2012MediaEvaltask• Testset:94anchorsdefinedby2humanannotators(mediaprofessionals)• AnchorVerificationstageonAmazonMTurk:• Feedbackcollectionfrom3workersforeachanchor(26and93respectivelyfromdevelopmentandtestsets)• ExamplesintheHITweretakenfrombothdevelopmentandtestsets(2+1)• 3anchorsfromthetestsetwerediscardedasconfirmednottobeproperlyverbal-visual• 90testsetanchorsverifiedfortaskevaluationusage


AnchorVerification:HITLayout


Anchorexample• Textualdescription:• “Wearelookingforvideosofpeopleexplainingwhereandhowtheylive.”


Relevanceassessment


TargetVetting:Forcedchoice

CaseID Choiceoftargetdescription Feedbackondecisionmakingprocess Relevancedecision

1 Correct Positive Relevant

2 Correct Negative ManualCheck

3 Other Positive Non-Relevant

4 Other Negative Non-Relevant

Participants• Numberof registrations:16

• INF BeijingUniversityofPostsandTelecommunicationUniversityAutonoma deMadridShandongUniversityXianJiaoTongUniversitySingapore

• VIREO CityUniversityofHongKong• IRISA CNRS,IRISA,INSA,Universite deRennes1• EURECOM EURECOM• FXPAL FXPALOALTOLABORATORY,INC• NII_Hitachi_UIT NationalInstituteof Informatics,Japan(NII);Hitachi,Ltd;UniversityofInformationTechnology, VNU-HCM,Vietnam(HCM-UIT)

• ORAND ORANDS.A.Chile• PKU_ICST PekingUniversity• VideoVision PESUniversity• EURECOM_POLITO Politecnico diTorinoEurecom• REGIMVID REGIM,ResearchGroupon IntelligentMachinesNationalSchoolofEngineers,

UniversityofSfax,Tunisia• Trimps TheThirdResearchInstituteoftheMinistryofPublicSecurity,P.R.China• HAWKEYE TsinghuaUniversity• TUZ TUBITAKUZAY• Waseda Waseda University• IIP_WHU Whuhan University

• Submittingparticipants:5groups(20officialruns)

Targetvetting:details

• Variabilityin20runssubmissions:• Top5ranks:7216diverseanchor-targetpairs- NEW!• Top10ranks:14271diverseanchor-targetpair

• Targetvetting:HITprocessing• 3judgmentsforeachanchor-targetpairfromthetop5(21648HITsubmissions)• Batchtargetvettingassessment:forcedlogic• Finalrelevancejudgmentassignmentforeachanchor-targetpairisbasedonmajoritydecisionover3crowdworkersdecisions

Evaluationmetrics


Officialmetrics

• Precisionatrank5• MAiSP

Assess-ments

RawResults

Res1 Res3 Res2

Effort

Reward

Results

TV'16:Teams(P@5)


TV'16:Teams(MAiSP)


TV’16:Textualfeatures(P@5)

TV’16:Visualfeaturesandsegmentation(P@5)

TV’16:Textualfeatures(MAiSP)

TV’16:Visualfeaturesandsegmentation(MAiSP)

Summaryandobservationssofar• Disruptivedataissuein2016:nodirectcomparisonwith2015• However:thetaskhappened!• Collectionwithstate-of-the-artfeatures• Improvedanchorscreationprocessthatleadstohigherrelevanceassessmentreliability

• Post-workshopanalysisisinprogress


Practicalplansfor2017• Continuationofthetask• Default:BlipTV collection

• 2016multimodalanchors• Potentialaddition:BBC2015collection

• Nomultimodaldevelopmentanchors• Whatistheaddedvalue?• Differentusescenario• Differentanchor/targetdensityinprofessionalcontent

• Taskfocus:• Multimodalityinanchors• Featuresofinterest?


Video Hyperlinking (LNK) Task at TRECVid 2016

Science