Results of the fifth edition of the BioASQ Challenge A. Nentidis, K. Bougiatiotis, A. Krithara, G. Paliouras and I. Kakadiaris NCSR “Demokritos”, University of Houston 4th of August 2017 BioNLP Workshop, Vancouver G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
32
Embed
Results of the fifth edition of the BioASQ · PDF fileResults of the fifth edition of the BioASQ Challenge, ... Test 2 100 3.49 5.13 Test 3 100 4.03 5.47 ... Results of the fifth
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Results of the fifth edition of the BioASQChallenge
A. Nentidis, K. Bougiatiotis, A. Krithara, G. Paliouras and I.Kakadiaris
NCSR “Demokritos”, University of Houston
4th of August 2017
BioNLP Workshop, Vancouver
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
IntroductionWhat is BioASQ
A competition
I BioASQ is a series of challenges on biomedical semanticindexing and question answering (QA).
I Participants are required to semantically index content fromlarge-scale biomedical resources (e.g. MEDLINE) and/or
I to assemble data from multiple heterogeneous sources (e.g.scientific articles, knowledge bases, databases)
I to compose informative answers to biomedical naturallanguage questions.
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Presentation of the challengeTasks
Task A: Hierarchical text classification
I Organizers distribute new unclassified MEDLINE articles.I Participants have 21 hours to assign MeSH terms to the articles.I Evaluation based on annotations of MEDLINE curators.
Febr
uary
06
Febr
uary
13
Febr
uary
20
March
1
March
06
March
13
March
20
March
27
April 0
3
April 1
0
April 2
4
May01
May08
May15
May22
1st batch 3rd batch2nd batch End of Task5a
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Presentation of the challengeTasks
Task B: IR, QA, summarization
I Organizers distribute English biomedical questions.I Participants have 24 hours to provide: relevant articles,
snippets, concepts, triples, exact answers, ideal answers.I Evaluation: both automatic (GMAP, MRR, Rouge etc.) and
manual (by biomedical experts).
March
08
March
09
March
22
March
23
April 0
5
April 0
6
April 1
9
April 2
0May
3May
4
2nd batch 4th batch 5th batch3rd batch1st batch
Phase APhase B
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Presentation of the challengeNew task
Task C: Funding Information Extraction
I Organizers distribute PMC full-text articles.I Participants have 48 hours to extract: grant-IDs, funding
agencies, full grants (i.e. the combination of a grant-ID and thecorresponding funding agency).
I Evaluation based on annotations of MEDLINE curators.
April 1
1
April 1
8
Dry Run Test Batch
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Presentation of the challengeBioASQ ecosystem
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Presentation of the challengeBioASQ ecosystem
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Presentation of the challengePer task
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Task 5AHierarchical text classification
I Training data
version 2015 version 2016 version 2017Articles 11,804,715 12,208,342 12,834,585Total labels 27,097 27,301 27,773Labels per article 12.61 12.62 12.66Size in GB 19 19.4 20.5
Total 40,119 (34,272) 33,162 (30,934) 42,435 ( 21,323)
The numbers in parentheses are the annotated articles for each test dataset.
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Task 5ASystem approaches
I Feature Extraction: Representing each abstractI tf-idf of words and bi-wordsI doc2vec embeddings of paragraphs
I Concept Matching: Finding relevant MeSH labelsI k-NN between article-vector representationsI Linear SVM binary classifiers for each MESH labelI Recurrent Neural Networks for sequence-to-sequence predictionI UIMA-ConceptMapper and MeSHLabeler tools for boosting NER
and Entity-to-MeSH matchingI Latend Dirichlet Allocation and Labeled LDA utilizing topics found
in abstractsI Ensemble methodologies and stacking
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Task 5AEvaluation Measures
Flat measures
I Accuracy (Acc.)I Example Based Precision (EBP)I Example Based Recall (EBR)I Example Based F-Measure (EBF)I Macro Precision/Recall/F-Measure
(MaP, MaR,MaF)I Micro Precision/Recall/F-Measure
(MiP,MIR,MiF)
Hierarchical measures
I Hierarchical Precision (HiP)I Hierarchical Recall (HiR)I Hierarchical F-Measure (HiF)I Lowest Common Ancestor Precision
(LCA-P)I Lowest Common Ancestor Recall (LCA-R)I Lowest Common Ancestor F-measure
(LCA-F)
A. Kosmopoulos, I. Partalas, E. Gaussier, G. Paliouras and I. Androutsopoulos: EvaluationMeasures for Hierarchical Classification: a unified view and novel approaches. Data Mining andKnowledge Discovery, 29:820-865, 2015.
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Task 5A resultsEvaluation
I Systems ranked using MiF (flat) and LCA-F (hierarchical).I Results, in all batches and for both measures :
1. Fudan2. AUTH-Atypon
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017
Task 5A results
G. Paliouras. Results of the fifth edition of the BioASQ Challenge, 4th of August 2017