1 Textual Entailment as a Framework for Applied Semantics Ido Dagan Bar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor, Roy Bar Haim, Maayan Geffet, Moshe Koppel, Efrat Marmorshtein, Bar Ilan University Shachar Mirkin Hebrew University, Israel Hristo Tanev, Bernardo Magnini, Alberto Lavelli, Lorenza Romano ITC-irst, Italy Bonaventura Coppola, Milen Kouylekov University of Trento and ITC-irst, Italy Danilo Giampiccolo, CELCT, ItalyDan Roth, UIUC
67
Embed
1 Textual Entailment as a Framework for Applied Semantics Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor, Roy Bar.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Textual Entailment as a Framework
for Applied Semantics
Ido Dagan Bar-Ilan University, Israel
Joint works with:Oren Glickman, Idan Szpektor, Roy Bar Haim, Maayan Geffet, Moshe Koppel, Efrat Marmorshtein, Bar Ilan UniversityShachar Mirkin Hebrew University, IsraelHristo Tanev, Bernardo Magnini, Alberto Lavelli, Lorenza Romano ITC-irst, ItalyBonaventura Coppola, Milen Kouylekov
University of Trento and ITC-irst, ItalyDanilo Giampiccolo, CELCT, Italy Dan Roth, UIUC
2
Applied Semantics forText Understanding/Reading
• Understanding text meaning refers to the semantic level of language
• An applied computational framework for semantics is needed
• Such common framework is still missing
3
Desiderata for Modeling Framework
• A framework for a target level of language processing should provide:
1) Generic module for applications2) Unified paradigm for investigating language
phenomena3) Unified knowledge representation
• Most semantics research is scattered – WSD, NER, SRL, lexical semantics relations…
(e.g. vs. syntax)– Dominating approach - interpretation
4
Outline
• The textual entailment task – what and why?• Evaluation – PASCAL RTE Challenges• Modeling approach:
– Knowledge acquisition
– Inference (briefly)
– Application example
• An alternative framework for investigating semantics
5
Natural Language and Meaning
Meaning
Language
Ambiguity
Variability
6
Variability of Semantic Expression
Model variability as relations between text expressions:• Equivalence: expr1 expr2 (paraphrasing)• Entailment: expr1 expr2 – the general case
– Incorporates inference as well
Dow ends up
Dow climbs 255
The Dow Jones Industrial Average closed up 255
Stock market hits a record high
Dow gains 255 points
7
Typical Application Inference
Overture’s acquisition by Yahoo
Yahoo bought Overture
Question Expected answer formWho bought Overture? >> X bought Overture
• Similar for IE: X buy Y
• Similar for “semantic” IR: t: Overture was bought …
• Summarization (multi-document) – identify redundant info
• MT evaluation (and recent ideas for MT)
• Educational applications
text hypothesized answer
entails
8
KRAQ'05 Workshop - KNOWLEDGE and REASONING for ANSWERING QUESTIONS
(IJCAI-05)
CFP:– Reasoning aspects:
* information fusion, * search criteria expansion models * summarization and intensional answers, * reasoning under uncertainty or with incomplete
knowledge,– Knowledge representation and integration:
* levels of knowledge involved (e.g. ontologies, domain knowledge),
* knowledge extraction models and techniques to optimize response accuracy
… but similar needs for other applications – can entailment provide a common empirical task?
9
Classical Entailment Definition
• Chierchia & McConnell-Ginet (2001):A text t entails a hypothesis h if h is true in every circumstance (possible world) in which t is true
• Strict entailment - doesn't account for some uncertainty allowed in applications
10
“Almost certain” Entailments
t: The technological triumph known as GPS … was incubated in the mind of Ivan Getting.
h: Ivan Getting invented the GPS.
11
Applied Textual Entailment• Directional relation between two text
fragments: Text (t) and Hypothesis (h):
t entails h (th) if, typically, a human reading t would infer that h is most likely true
• Operational (applied) definition:– Human gold standard - as in NLP applications– Assuming common background knowledge – which
is indeed expected from applications!
12
Probabilistic InterpretationDefinition: • t probabilistically entails h if:
– P(h is true | t) > P(h is true)• t increases the likelihood of h being true • ≡ Positive PMI – t provides information on h’s truth
• P(h is true | t ): entailment confidence– The relevant entailment score for applications– In practice: “most likely” entailment expected
13
The Role of Knowledge
• For textual entailment to hold we require:– text AND knowledge h
but – knowledge should not entail h alone
• Systems are not supposed to validate h’s truth without utilizing t
• Features model similarity and mismatch• Classifier determines relative weights of information sources• Train on development set and auxiliary t-h corpora
t,hSimilarity Features:
Lexical, n-gram,syntacticsemantic, global
Feature vector
Classifier
YES
NO
20
Results
First Author (Group)AccuracyAverage Precision
Hickl (LCC)75.4%80.8%
Tatu (LCC)73.8%71.3%
Zanzotto (Milan & Rome)63.9%64.4%
Adams (Dallas)62.6%62.8%
Bos (Rome & Leeds)61.6%66.9%
11 groups58.1%-60.5%
7 groups52.9%-55.6%
Average: 60%Median: 59%
21
Analysis• For the first time: deeper methods (semantic/
• COLING-04, ACL-05Lexical entailment via distributional similarity– Individual features characterize semantic propertiesObtain characteristic features via bootstrappingTest characteristic feature inclusion (vs. overlap)
• COLING-ACL-06Integrate pattern-based extraction– NP such as NP1, NP2, …– Complementary information to distributional evidence– Integration using ML with minimal supervision (10 words)
32
Acquisition Example
• Does not overlap traditional ontological relations
•Top-ranked entailments for “company”:
firm, bank, group, subsidiary, unit, business, supplier, carrier, agency, airline, division, giant,
• The essence of our proposal: – Formulate various semantic problems as entailment tasks
– Base applied inference on entailment “engines” and KBs
• Interpretations and mapping methods may compete• Open question: which inference
– can be represented at language level?
– requires logical or specialized representation and inference? (temporal, spatial, mathematical, …)
65
Meeting the knowledge challenge – by a coordinated effort?
• A vast amount of “entailment rules” needed• Speculation: is it possible to have a public effort
for knowledge acquisition?– Simple, uniform representations
– Assuming mostly automatic acquisition (millions of rules?)
– Human Genome Project analogy
• Preliminary: RTE-3 Resources Pool at ACLWiki
66
Textual Entailment ≈ Human Reading Comprehension
• From a children’s English learning book(Sela and Greenberg):
Reference Text: “…The Bermuda Triangle lies in the Atlantic Ocean, off the coast of Florida. …”
Hypothesis (True/False?): The Bermuda Triangle is near the United States
???
67
Optimistic Conclusions: Textual Entailment…
is a promising framework for applied semantics:– Defines new semantic problems to work on– May be modeled probabilistically– Appealing potential for knowledge acquisition