MediaEval 2014 Multimedia Benchmark Workshop

MediaEval 2014Multimedia Benchmark Workshop

16-17 October 2014Reial Acadèmia de Bones Lletres de Barcelona

Welcome to MediaEval

• We have arrived at the culmination of 2014.• We gather to:– Present our algorithms,– Discuss our solutions,– Lay pathways for future work.

Read more about the tasks in the program notes of the MediaEval 2014 Workshop.

The MediaEval Benchmark…

• …evaluates new algorithms for multimedia access and retrieval.

• …emphasizes the "multi" in multimedia: speech, audio, visual content, tags, users, context.

• …innovates new tasks and techniques focusing on the human and social aspects of multimedia content.

MediaEval History• MediaEval was founded as VideoCLEF in 2008.• Ran in 2008 and 2009 at the Cross Language

Evaluation Forum (CLEF) and became MediaEval in 2010.

• Ran in 2010 and 2011 as an independent benchmark under the sponsorship of the PetaMedia Network of Excellence.

• In 2012, ran as a completely “bottom-up benchmark” with no parent project.

• In 2014, workshop was not co-located with a major conference.

The 11 MediaEval 2014 Tasks

• (New!) C@MERATA: Question Answering on Classical Music Scores• (New!) Crowdsourcing: Crowdsorting Timed Comments about Music • Diverse images: Retrieving Diverse Social Images• Emotion in Music • Placing: Multimodal Location Estimation• QUESST: Query by Example Search on Speech (ex SWS) Task• Search and Hyperlinking • SED: Social Event Detection in Web Multimedia• (New!) SEM: Synchronization of multi-user Event Media • Visual Privacy • Violence: Violent Scenes Detection

2014 Survey Responses

2011 2012 2013 20140

20406080

100120140160180200

2014 Working Notes Papers

In 2014, we had 11 task overview papers (grey) and 85 accepted participant papers (black).

2014 Workshop Participants

Poster Session

• Each poster session has a focus task or tasks, marked in the program.

• Next is the “ThursAM” session, dedicated to Social Event Detection.

• QUESST people can start putting up their posters around the perimeter.

• During lunch we will have the “QUESST quest”: guided tour of QUESST posters.

Internet

• We follow the grand tradition of Santa Croce in Fossabanda.

• Wireless internet is provided solely for three purposes:

1. turning on your out-of-office auto-reply

2. the occasional tweet #mediaeval14

3. For Task organizersto submit MDMs

Technical Retreats

• Purpose– Detailed technical discussion of tasks– Making future plans– All are welcome, but mostly interesting for task

participants• Place– Your task organizers will tell you where the technical

retreats are.– Task organizers please talk to the Technical Retreat

chair, to know where your retreat is.

Workshop Keynote

• MediaEval originally arose in CLEF (a European information retrieval benchmark)

• We have a tradition of networking with other benchmarks.

• This includes a keynote speaker from another benchmark each year.

• This year: Sergio Escalera from Universitat de Barcelona “ChaLearn Challenge and Workshop.

Slides

• Please load your slides on this computer in the break just before your session.

• There is a folder for your task on the desktop.• If you do not want your slides online, please

delete them from the folder after the session.

Bibliography

• MediaEval Group• http://www.citeulike.org/groupfunc/16499/ho

me• Please post your publications that use

MediaEval data to the bibliography.• Important for understanding the contribution

of MediaEval tasks to the research community.

Resources

• Tasks publish their own data sets.• Please send me a link and I will add it to the

website: http://www.multimediaeval.org/datasets

• Many tasks submit papers to the MMSys Data Set trackhttp://www.mmsys.org/index.php/2015-mmsys-data-track(Dataset paper Submission Deadline: November 10, 2014)

Final Session

• MDMs are “MediaEval Distinctive Mentions”• Task organizers assign MDMs to point out an

interesting scientific contribution (that is not necessarily top ranked contribution).

• Task organizers also assign MDMs to praise or tease participants or each others.

• The final session of the workshop is devoted to MDMs and outlook on next year.

MediaEval Letters

• “MediaEval Letters” is a section in the working notes papers that will contain papers on topics other than tasks.

• Examples:– Discussions of reproducibility and reproduction– Documentation of best practices– Multi-year retrospectives on tasks

• Scheduled to be introduced in 2015.

Thank you’s

Workshop Organization

Local Chair Xavier Anguera (Telefonica Research, Spain)

Coordination ChairMohammad Soleymani (University of Geneva, Switzerland)

Poster Chairs Bogdan Ionescu (University Politehnica of Bucharest, Romania) Xavier Giró (Universitat Politecnica de Catalunya, Barcelona)

Logistics ChairSaskia Peters (Delft University of Technology, Netherlands)

Workshop Organization

Filmmaker Olivier Van Laere and company

Technical Retreat ChairMichele Trevisiol

Proceedings ChairMichael Riegler, Simula Research Lab AS, Norway

Documentation Chair Richard Sutcliffe, University of Essex, UK

Workshop Organization

Resources and Recognition committee (MediaEval Community Council)Mohammad Soleymani (University of Geneva, Switzerland)Guillaume Gravier (IRISA, France) Bogdan Ionescu (University Politehnica of Bucharest, Romania) Gareth Jones (Dublin City University, Ireland)Martha Larson (Delft University of Technology, Netherlands)

Workshop Assistants:Sandra Avila, Bogdan Boteanu, Shu Chen, Justin Chiu, Tom Collins, Taufik Edy

Sutanto, Maria Eskevich, Vaiva Imbrasaite, Irene Gris, Santosh Kesiraju, Daniel Manchon, Amaia Salvador, Emanuele Sansone, Patrick Schwab, Carles Ventura.

Statistics and Research Assistants:Bogdan Boteanu and Anca Radu (University Politehnica of Bucharest, Romania)

Thank you to Task Organizers

• Anna Aljanaki, Utrecht University, Netherlands• Ahmed Al-Obaidi, University of Reading, U• Robin Aly, University of Twente, Netherland• Xavier Anguera, Telefonica Research, Spain• Atta Badii, University of Reading, UK• Andi Buzo, University Politehnica of Bucharest, Romania• Liangliang Cao, IBM TJ Watson, USA• Shu Chen, Dublin City University, Ireland• Jaeyoung Choi, ICSI, USA

Thank you to Task Organizers

• Nicola Conci, University of Trento, Italy• Tim Crawford, Goldsmiths College, University of

London, UK• Francesco De Natale, University of Trento, Italy• Touradj Ebrahimi, EPFL, Switzerland• Volker Eiselein, Technische Universität Berlin, German• Maria Eskevich, Dublin City University, Irelan• Christian Fedorczak, Thales Security Solutions &

Services• Gerald Friedland, ICSI, US

Thank you to Task Organizers

• Chris Fox, University of Essex, UK• Eduard Hovy, Carnegie Mellon University, US• Bogdan Ionescu, University Politehnica of Bucharest,

Romania• Yu-Gang Jiang, Fudan University, Shanghai, China• Gareth Jones, Dublin City University, Ireland• Pavel Korshunov, EPFL, Switzerland• Martha Larson, Delft University of Technology,

Netherlands• Mihai Lupu, Vienna University of Technology, Austria

Thank you to Task Organizers

• Vu Lam Quang, University of Information Technology, VNU-HCMC, Vietnam

• David Racca, Dublin City University, Ireland• Giuseppe Rizzo, Univ. of Torino, Italy• Luis Javier Rodriguez-Fuentes, University of the Basque

Country, Spain• Deane L. Root, University of Pittsburgh, US• Markus Schedl, Johannes Kepler University, Linz, Austria• Mats Sjöberg, University of Helsinki, Finland• Mohammad Soleymani, University of Geneva, Switzerland

Thank you to Task Organizers

• Richard Sutcliffe, University of Essex, UK• Igor Szőke, Brno University of Technology ,

Czech Republic• Bart Thomee, Yahoo Labs, San Francisco, USA• Raphael Troncy, EURECOM, France• Karthik Yadati, Delft University of Technology,

Netherlands• Yi-Hsuan Yang, Academia Sinica, Taiwan

MediaEval Supporters

• ISCA SIG Speech and Language in Multimedia (contact at workshop: Martha or Gareth)

• IEEE STC Social Networking• IEEE SIG Audio and Speech Processing for

Multimedia (contact at workshop: Xavi Anguera)

Thank you to sponsors

• ELIAS ESF Research Networking Programme on Evaluating Information Access Systems.

• Friends of SIGIR.• Technicolor.• COMMIT/ Dutch national project.

Thank you

• Thank you to the:Reial Acadèmia de

Bones Lletres de Barcelona

• Thank you to UPF for the piano.

How to propose a 2015 task?

• Come to the meeting tomorrow 10-13 here.• Send in a task proposal:http://www.multimediaeval.org/mediaeval2015• Proposal deadline: 22 December 2014• Tasks are decided by amount of interest shown

on the survey.• Main tasks must have five participants who vow

to cross the finish line come hell or high water.