AQA: a multilingual Anaphora annotation scheme for Question Answering E. Boldrini, P. Martínez Barco, B. Navarro Colorado, M. Puchol Blasco, C. Vargas.

AQA: a multilingual Anaphora annotation scheme for

Question Answering

E. Boldrini, P. Martínez Barco, B. Navarro Colorado, M. Puchol Blasco, C. Vargas Sierra

[eboldrini/patricio/borja/marcel/]@dlsi.ua.es chelo.vargas@ua.es

CBA 2008 Corpus-Based Approaches to Coreference Resolution in Romance Languages

Outline• Introduction

• Corpus

• Principles

• Previous work

• Problematic cases

• Evaluation

• Conclusion

Introductioninteraction• AQA: multilingual annotation scheme for anaphora

resolution that can be applied in machine learning for the improvement of QA systems

• To understand and annotate the way anaphora is used in each language

• To be able to detect the antecedent of each the anaphora and find the correct answer

• INTERACTION between the user and the system

Introduction - Corpus - Principles - Previous work - Problematic cases - Evaluation - Conclusion

Introductionlanguages

• Languages: Italian, Spanish, English

• Advantages: participate successfully in competitions in which the question is formulated in a language and the system shows the answer in another language

• Disadvantages: languages with different characteristics

Introductionlanguages

• Languages: Italian, Spanish, English

• Advantages: can participate successfully in competitions in which the question is formulated in a language and the system shows you the answer in another language

• Disadvantages: languages with different characteristics

<t><q id="q065">¿Qué medio de transporte se utilizó en la Expedición Kon-tiki?</q><q id="q066">¿Cuántas personas <link rel="dir" status="ok" type="pron" ref="" ant="a" refq="q065">la</link> tripulaban?</q></t>

<t><q id="q265">Quale mezzo di trasporto venne usato nella spedizione Kon-Tiki?</q><q id="q266">Quanti membri d'equipaggio aveva <link rel="dir" status="ok" type="elips" ref="" ant="a" refq="q265">0</link>?</q></t>

<t><q id="q465">What transport was used in the Kon-Tiki Expedition?</q><q id="q466">How many people crewed <link rel="dir" status=”no" type="pron" ref="" ant=”q" refq="q465">it</link>?</q></t>

Corpus

• Corpus for CLEF 2008 in English, Italian and Spanish

• 200 questions per language

• Topic-related questions

• Categories of questions: factoid, definition, and list

Principlesannotated elements

• Each group has a topic

<t><q id="q429">Between what days was <de id="n16">the battle of Brunete</de>?</q><q id="q430">Where was the article of <de id="n17">Gerda Taro</de> about <link rel="dir" status="ok" type="dd" ref="n16" ant="q" refq="q429">this battle</link> published?</q><subt><q id="q431">Which hospital were <link rel="dir" status="ok" type="pron" ref="n17" ant="q" refq="q430">she</link> moved to after her accident?</q></subt></t>

• If there is a subtopic, we mark it

Principlesannotated elments

• Each question (question/answer pair) has a number

• Each anaphora has a number, the same of its antecedent

• We indicate if the antecedent is in the question or in the answer

<t><q id="q482">Which city is the headquarters of the China's Eastern Fleet?</q><q id="q483">How far from China's capital city is <link rel="dir" status="ok" ant="a" refq="q482" type="pron" ref="">it</link>?</q><q id="q484">What was <link rel="indir" status="ok" ant="a" refq="q482" type="dd" ref="">its population</link> in 2002?</q></t>

• We indicate the number of the question or the answer where the antecedent is situated

• We select the type of anaphora

<t><q id="q453">In which country is <de id="n28">the Colditz Castle</de>?</q><q id="q454">Exactly in which state is <link rel="dir" status="ok" type="pron" ref="n28" ant="q" refq="q453">it</link>?</q><q id="q455">Who was the first who escaped from <link rel="dir" status="ok" type="adv" ref="n28" ant="q" refq="q453">there</link> ?</q>

<t><q id="q412">Who published the Evangelium Vitae <de id="n6">encyclical</de>?</q><q id="q413">How many <link rel="dir" status="ok" ant="q" refq="q412" type="elips" ref="n6">0</link> did <link rel="dir" status="ok" ant="a" refq="q412" type="pron" ref="">he</link> publish?</q></t>

• We select the type of relation

<t><q id="q416">Which islands are in <de id="n9">the Pelagie Islands</de>?</q><q id="q417">Which is <link rel="indir" status="ok" type="dd" ref="n9" ant="q" refq="q416">the biggest one</link>?

• We underline if the annotator has doubts or not

Previuos work• UCREL (Fligelstone, 1992; Garside et al., 1997): first scheme

for anaphora resolution

• MUC: inclusion of the coreference task in MUC-6 and MUC-7

• Last decade of 20th century: anaphora resolution project for French (Popescu, Belis and Robba, 1997).

• Martínez-Barco and Palomar (2001): An annotation scheme for dialogues applied to anaphora resolution algorithm.

• MATE/GNOME (Poesio, 2004): meta-model

Previuos workwhat we added

• MATE/GNOME (Poesio, 2004): meta-model

• Element link in the text with the information about the anaphora

• Identification of the question/answer pair

• Topic/subtopic

• Antecedent in the question or in the answer

• Status of the annotation

• Applied to three languages

• Applied to collections of questions

Problematic cases

• World knowledge

• An antecedent contains another one

• Collective nouns

• Two antecedents, but separated

• Doubtful position of the antecedent

• An anaphora inside a discourse entity

Problematic cases• World knowledge

<t><q id="q404">Which was <de id="n2">the "gordo" in the 1995 Christmas</de>?</q><q id="q405">Which was <link rel="indir" status="no" type="dd" ref="n2" ant="q" refq="q404">the prize</link>?</q></t>

Problematic cases• World knowledge• World knowledge

• An antecedent contains another one<t><q id="q427">Who were <de id="n14">the founders of <de id="n15">Magnum Photos</de> </de>?</q><q id="q428">In what year did <link rel="dir" status="ok" ant="q" refq="q427" type="pron" ref="n14">they</link> found <link rel="dir" status="ok" type="pron" ref="n15" ant="q" refq="q427">it</link>?</q></t>

• World knowledge

<t><q id="q432">What is <de id="n18">the starring cast</de> of the film Beetlejuice?</q><q id="q433">Who of <link rel="dir" status="ok" type="pron" ref="n18" ant="q" refq="q432">them</link> is the main character?</q></t>

• World knowledge

<t><q id="q465">What transport was used in the Kon-Tiki Expedition?</q><q id="q466">How many people crewed <link rel="dir" status=”no" type="pron" ref="" ant=”q" refq="q465">it</link>?</q></t>

• World knowledge

• An anaphora inside a discourse entity• An anaphora inside a discourse entity

• World knowledge

<t><q id="q434">What is <de id="n19">a censer</de> ?</q><q id="q435">What name is given to <de id="n20"> <link rel="dir" status="no" type="pron" ref="n19" ant="q" refq="q434">the one</link> of the Cathedral of Santiago de Compostela </de>?</q><q id="q436">How much does <link rel="dir" status="ok" type="pron" ref="n20" ant="q" refq="q434">it</link> weight?</q></t>

Evaluation

• Annotation

• 2 annotators

• Blind annotation

• Evaluation

• Each language independently

• Global results

Evaluationsubdivision

• Topic boundary

• Anaphora detection

• Anaphora attibutes

• Antecedent recognition

Evaluationtopic boundary

• Class N: new topic

• Class S: same topic

SPANISH ITALIAN ENGLISH

A1\A2 S N A1\A2 S N A1\A2 S N

S 62 0 S 62 0 S 61 0

N 0 138 N 0 138 N 1 138

Kappa 1 Kappa 1 Kappa 0,988

Evaluationanaphora detection

Anaphors detected by A1 70 69 67

Anaphors detected by A2 70 69 68

Anaphors detection agreement 70 69 67

Different anaphora boundary 1 1 0

Evaluationanaphora attributes (antecedent)

A1\A2 Q A A1\A2 Q A A1\A2 Q A

Q 64 0 Q 62 0 Q 61 0

A 0 6 A 0 7 A 0 6

Kappa 1 Kappa 1 Kappa 1

Evaluationanaphora attributes (type)

A1 A2 A1 A2 A1 A2

Elips 33 33 32 32 3 3

Pron 13 15 13 13 42 42

Adv 1 1 2 2 1 1

Sup 1 0 0 0 0 0

DD 22 21 22 22 21 21

Kappa 0,955 1 1

Evaluationanaphora attributes (relation)

• Dir: direct relation

• Indir: bridging relation

A1\A2 DIR INDIR A1\A2 DIR INDIR A1\A2 DIR INDIR

DIR 52 0 Q 51 0 Q 52 0

INDIR 4 14 INDIR 1 17 INDIR 2 13

Kappa 0,838 Kappa 0,961 Kappa 0,909

Evaluationantecedent recognition

Total antecedents into the answer (agreement) 6 7 6

Total antecedents into the question (agreement) 64 62 61

Anaphors pointing to the same question (refq) (agreement)

64 62 61

Antecedents with different boundary (disagreement)

Evaluationglobal results

• Total agreement results

• Spanish: 60/70 = 0,857

• Italian: 60/69 = 0,869

• English: 59/67 = 0,880

• Average: 0,868

Conclusion

• Multilingual annotation scheme for anaphora resoultion

• For the improvement of QA system: the system can detect the antecedent of each anaphora and extract the correct answer

• For a true interaction between the system and the user

• Simple but complete

• Positive results of the evaluation

Future work

• Integration of other languages

• Application of the annotation scheme to other corpora

Evaluationmeasure used

• Kappa

AQA: a multilingual Anaphora annotation scheme for Question Answering E. Boldrini, P. Martínez Barco, B. Navarro Colorado, M. Puchol Blasco, C. Vargas.

corpus corpus

corpusbased approaches

romance languages slide

system introduction

list introduction corpus

coreference resolution

conclusion slide

anaphora resolution

Documents

AUTORA DEL TRABAJO: ALBA PUCHOL VILLORO

Autovetor e Autovalor Boldrini

104998670 Curriculum Vitae Luis Puchol

Anaphora Resolution

Anaphora in English vs. Anaphora in Arabic.

Hablar en Publico - Luis Puchol

TEXT PROCESSING 1 Anaphora resolution Introduction to...

anaphora and deixis

Anaphora Resolution 3

Anaphora and Discourse - Fachbereich...

Boldrini - A'lgebra Linear.pdf

Questões Boldrini

Anaphora and Discourse

El Libro de Luis Puchol

Cap05 TransfLinear Boldrini livro do boldrini

Pronominal Anaphora resolution