Welcome to CLEF 2009 Carol Peters ISTI-CNR, Pisa, Italy
May 26, 2015
Welcome to CLEF 2009
Carol PetersISTI-CNR, Pisa, Italy
Happy 10th Birthday
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF Objectives
Stimulate the development of multilingual IR systems (for European languages !)
Create a CLIR/MLIA community Construct publicly available test-suites
BY Conducting annual evaluation campaigns
Designing tracks/tasks to meet emerging needs and to stimulate research in the”right” direction
Objective: truly multilingual/multimedia systems
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Evolution of CLEFCLEF 2000 Tracks
• mono-, bi- & multilingual text doc retrieval (Ad Hoc)• mono- and cross-language information on structured scientific data (Domain-Specific)
CLEF 2001 New
• interactive cross-language retrieval (iCLEF)
CLEF 2002 New
•cross-language spoken document retrieval (CL-SR)
CLEF 2003 New
• multiple language question answering (QA@CLEF) • cross-language retrieval in image collections (ImageCLEF)
CLEF 2005 New
• multilingual retrieval of Web documents (WebCLEF)• cross-language geographical retrieval (GeoCLEF)
CLEF 2008New
• cross-language video retrieval (VideoCLEF)• multilingual information filtering (INFILE@CLEF)
CLEF 2009New
• intellectual property (CLEF-IP)• log file analysis (LogCLEF)• large-scale grid experiments (Grid@CLEF)
CLEF Tracks: 2000 - 2009
Multilingual textual document retrieval (Ad Hoc) Interactive cross-language retrieval (iCLEF) Multiple language question answering (QA@CLEF) Cross-language retrieval in image collections
(ImageCLEF) Multilingual information filtering (INFILE@CLEF) Cross-language video retrieval (VideoCLEF) Intellectual property (CLEF-IP) – New this year Log file analysis (LogCLEF) – New this year MorphoChallenge (in collaboration with Pascal NoE)
CLEF 2009 Tracks
Pilot: Grid Experiments (Grid@CLEF)
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF2009 Coordination
Adaptive Informatics Research Centre, Helsinki University of Technology, Finland
Berlin School of Library and Information Science, Humboldt-Universität zu Berlin
Business Information Systems, U. Applied Sciences Western Switzerland, Sierre, Switzerland
CEA LIST, France Centre for Evaluation of Human Language & Multimodal
Communication Technologies (CELCT), Italy Centruum vor Wiskunde en Informatica, Amsterdam, NL Computer Science Department, U. Basque Country, Spain Computer Vision and Multimedia Lab, U. Geneva, CH Data Base Research Group, U. Tehran, Iran Dept. of Computer Science and Information Systems, U.
Limerick, Ireland Dept. of Information Engineering, U. Padua, Italy Dept. of Information Science, U. Hildesheim, Germany Dept. of Information Studies, U. Sheffield, UK Dept. of Medical Informatics and Clinical Epidemiology,
Oregon Health and Science U., USA Dept. Medical Informatics, U. Hospitals and University of
Geneva, Switzerland Department of Medical Informatics, Aachen University of
Technology (RWTH), Germany
Evaluations and Language Resources Distribution Agency, Paris, France
GERiiCO, Université de Lille, France Idiap Research Institute, Switzerland Information Retrieval Facility (IRF), Austria Laboratoire d'Informatique pour la Mécanique et les
Sciences de l'Ingénieur (LIMSI), Orsay, France U. Nacional de Educación a Distancia, Madrid, Spain Linguateca, Sintef ICT, Norway Linguistic Modelling Lab., Bulgarian Acad Sci Matrixware Information Services, Austria Mediamatics, Delft University of Technology, The
Netherlands Mitre Corporation, USA NIST, USA NLE Lab., Universidad Politènica de Valencia, Spain Research Institute for Artificial Intelligence, Romanian
Academy, Romania Romanian Institute for Computer Science, Romania Royal Institute of Technology (KTH), Stockholm,
Sweden School of Computing, DCU, Ireland Swedish Institute of Computer Science, Sweden TALP Research Center, Universitat Politécnica de
Catalunya, Barcelona, Spain.
CLEF is coordinated by the Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche, PisaThe following Institutions are contributing to the organisation of the different tracks of the CLEF 2008 campaign:
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF Steering Committee
Maristella Agosti, U.Padove, Italy Martin Braschler, Zurich, Switzerland Amedeo Cappelli, ISTI-CNR & CELCT, Italy Hsin-Hsi Chen, National Taiwan U., Taipei, Taiwan Khalid Choukri, ELRA/ELDA, Paris, France Paul Clough, University of Sheffield, UK Thomas Deselaers, RWTH Aachen University,
Germany Giorgio Di Nunzio, U. Padova, Italy David A. Evans, Clairvoyance Corporation, USA Nicola Ferro, U. Padova, Italy Christian Fluhr, CEA-LIST, Fontenay-aux-Roses,
France Norbert Fuhr, University of Duisburg, Germany Frederic C. Gey, U.C. Berkeley, USA Julio Gonzalo, LSI-UNED, Madrid, Spain Donna Harman, NIST, USA Gareth Jones, Dublin City University, Ireland Franciska de Jong, University of Twente,
Netherlands Noriko Kando, NII, Tokyo, Japan Jussi Karlgren, SICS, Sweden Michael Kluck, German Institute for International and
Security Affairs, Berlin, Germany
Natalia Loukachevitch, Moscow State University, Russia Bernardo Magnini, ITC-irst, Trento, Italy Thomas Mandl, U. Hildesheim, Germany Paul McNamee, Johns Hopkins University, USA Henning Müller, University & University Hospitals of
Geneva, Switzerland Douglas W. Oard, University of Maryland, USA Anselmo Peňas, LSI-UNED, Madrid, Spain Vivien Petras, Humboldt University, Berlin, Germany Maarten de Rijke, University of Amsterdam,
Netherlands Diana Santos, Linguateca, Sintef, Oslo, Norway Jacques Savoy, University of Neuchatel, Switzerland Peter Schäuble, Eurospider Information Technologies,
Switzerland Richard Sutcliffe, University of Limerick, Ireland Hans Uszkoreit, German Research Center for Artificial
Intelligence (DFKI), Germany Felisa Verdejo, LSI-UNED, Madrid, Spain José Luis Vicedo, University of Alicante, Spain Ellen Voorhees, NIST, USA Christa Womser-Hacker, University of Hildesheim,
Germany
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF2009:Track Coordinators
Ad Hoc: Abolfazl AleAhmad, Hadi Amiri, Eneko Agirre, Giorgio Di Nunzio, Nicola Ferro, Nicolas Moreau, Arantxa Otegi, Vivien Petras
iCLEF: Paul Clough, Julio Gonzalo, Jussi Karlgren QA@CLEF: Iñaki Alegria, Davide Buscaldi, Luís Miguel Cabral, Pere R. Comas,
Corina Forascu, Pamela Forner, Olivier Galibert, Danilo Giampiccolo, Nicolas Moreau, Djamel Mostefa, Petya Osenova, Anselmo Peñas, Álvaro Rodrigo, Sophie Rosset, Paolo Rosso, Diana Santos, Richard Sutcliff and Jordi Turmo
ImageCLEF: Brian Bakke, Steven Bedrick, Barbara Caputo, Paul Clough, Peter Dunker, Thomas Deselaers, Thomas Deserno, Ivan Eggel, Mark Oliver Güld, William Hersh, Patric Jensfelt, Charles E. Kahn Jr., Jana Kludas, Jayashree Kalpathy–Cramer, Henning Müller, Stefanie Nowak, Monica Lestari Paramita, Andrzej Pronobis, Saïd Radhouani, Mark Sanderson, Tatiana Tommasi, Theodora Tsikrika and Petra Welter
VideoCLEF: Gareth J.F. Jones, Martha Larson and Eamonn Newman INFILE: Romaric Besançon, Stéphane Chaudiron, Khalid Choukri, Meriama Laïb,
Djamel Mostefa and Ismaïl Timimi CLEF-IP: Florina Piroi, Giovanna Roda, John Tait and Veronika Zenz LogCLEF: Maristella Agosti, Giorgio Di Nunzio, Christine Doran, Inderjeet Mani,
Thomas Mandl, Julia Maria Schulz and Alexander Yeh Grid@CLEF: Nicola Ferro and Donna Harman MorphoChallenge: Graeme W. Blackwood, William Byrne Mikko Kurimo, Ville T.
Turunen and Sami Virpioja
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF 2009: Participating Groups
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF: Trend in Participation
CLEF2009: Europe = 81(69); N. America = 18(12); Asia = 16(15), S. America = 1(3), Africa = 1(1)
CLEF 2000 - 2009 Participation
0
10
20
30
40
50
60
70
80
90
100
110
120
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Oceania
Africa
South Am erica
North Am erica
As ia
Europe
CLEF 2009 Workshop30 September – 2 October, Corfu,
Greece
CLEF2009:Participants per Track
Ad Hoc: 28(26) iCLEF: 6(6) QA@CLEF: 25(29) ImageCLEF: 62 (42) INFILE: 5(1)
VideoCLEF: 7(5) MorphoChallenge:
9(6)
CLEF-1P: 15 (new) LogCLEF: 6 (new) Grid@CLEF: 2 (pilot)
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Participation by Track
CLEF 2009 Tracks
AdHoc
iCLEF
QA@CLEF
ImageCLEF
VideoClef
INFILE
MorphoChal lenge
CLEF-IP
LogCLEF GridCLEF
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF 2000 – 2009Participation per Track
CLEF 2000 - 2009 Tracks
0
10
20
30
40
50
60
70
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009Years
Par
ticip
atin
g G
roup
s
AdHoc
DomSpec
iCLEF
CL-SR
QA@CLEF
ImageCLEF
WebClef
GeoClef
VideoClef
INFILE
MorphoChallenge
CLEF-IP
LogCLEF
GridCLEF
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF Achievements Creation of very active multidisciplinary international
research community Investigation of core issues in MLIA
development of multiple language processing tools creation of linguistic resources implementation of appropriate cross-language retrieval models
and algorithms for different tasks and languages; Creation of important reusable test collections and
resources in diverse media for a large number of European languages
Significant and quantifiable improvements in the performance of MLIA systems
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Test Collections2000 News documents in 4 languages GIRT German Social Science database
2009 A subset of CLEF multilingual corpus of news documents (Ad Hoc WSD-Robust) Hamshahri Persian newspaper corpus (Ad Hoc Persian task) TEL Library catalog records plus log files (Ad Hoc TEL task and LogCLEF) Flickr web-based image database (iCLEF) ResPubliQA document collection, (QAatCLEF: ResPubliQA) Transcripts of European parliamentary sessions & French news broadcasts (QAST) BELGAPICTURE image collection (ImageCLEFPhoto) Multilingual collections of Wikipedia documents and images (ImageCLEFwiki) Articles and images from Radiology and Radiography; IRMA collection for medical image annotation (ImageCLEFmed and medAnnotation) Dutch and English documentary television programs (VideoCLEF) Agence France Press (AFPnewswire stories in Arabic, French and English (INFILE) Patent documents from the European Patent Office (CLEF-IP)
Outline of Talk
Why, How, What
DIRECT@work in CLEF
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF Publications
9 CLEF Post-workshop Proceedings, Springer LNCS
Nearly 1000 Working Notes Reports on-line Google Scholar > 15,000 publications with
CLEF – cross-language information retrieval
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF 2008 Proceedings
Evaluating Systems for Multilingual and Multimodal Information Access 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers Series: Lecture Notes in Computer Science , Vol. 5706Peters, C.; Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, Th.; Peñas, A.; Petras, V (Eds.)
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Many, many thanks to Danilo Giampiccolo, CELCT
CLEF & TrebleCLEF
CLEF 2008 & 2009 is an activity of the TrebleCLEF Coordination Action under the Seventh Framework Programme of the European Commission.
TrebleCLEF organises a set of dissemination activities in the multilingual information access field.
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Treble-CLEF
The CLEF research results have led to development of a new generation of multilingual retrieval system prototypes
BUT lack of technology transfer
Treble-CLEF extends the CLEF activity by: continuing to promote MLIA R&D via evaluation campaigns; providing a consistent training activity: tutorials, workshops,
summer school; producing best practice guidelines for system implementation; providing resources to encourage the multilingual system
development.
www.trebleclef.eu
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Approach Evaluation
test collections and laboratory evaluation
user evaluation log analysis
Best Practices & Guidelines system-oriented aspects of
MLIA applications collaborative user studies user-oriented aspects of MLIA
interfaces
Dissemination and Training tutorials workshops summer school
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Treble-CLEF Events
Workshop on Novel Methodologies for Evaluation in Information Retrieval, ECIR’08, Glasgow, Scotland
Workshop on Best Practices for the Development of Multilingual Information Access Systems, Segovia, Spain, June 08
Workshop on Best Practices for System Developers: Bringing Multilingual Information Access to Operational Systems, Winterthur, Switzerland, October 2008
Workshop on Best Practices in Query Log Analysis, London, May 2009
Summer School on Multilingual information Access, Pisa, June 2009
MLIA Technology Day – Dissemination of Results of Best Practices Workshops, 8 December 2009
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
CLEF 2010
Can CLEF continue without external funding for the central coordination?
??????
Participate in the discussion on Friday afternoon
CLEF 2009 Workshop30 September – 2 October, Corfu, Greece
Before I Forget
1.Posters: Put up between 8.00 & 13.30 Thursday morning
Pool Room 2: ImageCLEF PostersPool Room 1: All other Posters & Demos
2.USB StickECDL Conference ProceedingsAll Workshop Proceedings
3.Social Dinner - TonightHotel restaurant – 8 pm (Badge)
CLEF 2002Rome
CLEF 2006Alicante
CLEF 2005Vienna
CLEF 2004Bath
CLEF 2003Trondheim
CLEF 2008Aarhus
CLEF 2001Darmstadt
CLEF 2007Budapest
CLEF 2000
Lisbon
CLEF2009Corfu
CLEF2010?????
Come to CLEF – and see Europe!
CLEF 2009
Thank you for your attention
and
ENJOY THE Workshop !
CLEF 2008 Workshop, Aarhus, Denmark 17-19 September 2008