Document-level Sentiment Inference with Social, Faction, and Discourse Context
Eunsol Choi, Hannah Rashkin, Luke Zettlemoyer, Yejin Choi
ACL 2016
1
2
Targeted Sentiment InferenceExtract sentiment relation between real-world entities in news articles.
Russia Belarus
positive? negative? neutral?
Opinion Holder Opinion Target
(Stoyanov and Claire 11,Yang and Cardie 14, Deng and Wiebe 14)
Russia criticizes Belarus
This work: Document-level Sentiment Inference
3
NegativePositive
Factual Tie
Russia criticizes Belarus
Input: News article Output: Sentiment Graph
4
• Russia criticizes Belarus for … • The speaker of the Russian parliament
Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim
Belarus
Challenge 1: Read the entire story
Russia
positive? negative? neutral?
5
• Entities form factions inside the document.
• Subset of sentiment relations can only inferred via relationships among
entities.
Challenge 2: Inference across entities
Tim Cook
C.E.O
Apple
Katy PerryRihanna
Taylor Swift
Elie Goulding
Miley Cyrus
Challenge 3: Sentiment Beyond Sentence Boundaries
6
Russia heat, smog trigger health problems..……….………………………………… “We never care to work with a future perspective in mind,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”
Alexei Skripkov Russia
This work: Document-level Sentiment Inference
7
NegativePositive
Factual Tie
Russia criticizes Belarus
Input: News article Output: Sentiment Graph
8
Evidence: Explicit Sentiment Textual Cues
…….….Belarus for permitting Georgian President Mikheil Saakhashvili to appear on its television.…….……. Saakhashvili announced Thursday that he did not understand Russia’s claims..…….…….……s with Ge
did not understand
permit
criticize
NegativePositive
9
NegativePositiveFactionEntities in the same faction shares opinions
Homophily (Lazarsfeld and Merton, 1954)
Evidence:Sentiment Inference Through Factions
president
10
NegativePositiveFactionEnemy of an enemy is a friend.
Social Balance Theory (Heider, 1946)
Evidence:Sentiment Inference Through Relations
This work: Document-level Sentiment Inference
11
NegativePositive
Factual Tie
Russia criticizes Belarus
Input: News article Output: Sentiment Graph
Related Work
Opinion argument
Opinion attribute
Opinion expression
Chavez faced America 's hostility
SubjectiveSourceTarget
IS-FROM
IS-ABOUT
Opinion Frame
Trigger: “hostility” Polarity: negativeIntensity: highSource: “America”Target: “Chavez”
12
Corpus-level Approach (KBP)2013/4 Sentiment Task
Conan O’Brien is positive towards
David Litterman, South Korea, Pokemon
Query Entity Attitude
Answer
Fine-grained Approach (MPQA)(Deng and Wiebe 15)
Related Work
Opinion argument
Opinion attribute
Opinion expression
Chavez faced America 's hostility
SubjectiveSourceTarget
IS-FROM
IS-ABOUT
Opinion Frame
Trigger: “hostility” Polarity: negativeIntensity: highSource: “America”Target: “Chavez”
13
Corpus-level Approach (KBP)2013/4 Sentiment Task
Conan O’Brien is positive towards
David Litterman, South Korea, Pokemon
Query Entity Attitude
Answer
Fine-grained Approach (MPQA)(Deng and Wiebe 15)
This Work:
• Document-level approach• Considers all named entities• Focuses on entity-entity interactions
Dataset
14
• Document-level sentiment data collection
• Dataset Statistics
Document Annotations
15
Evaluation Dataset Collection
Evaluation Dataset Collection
Mark the relationship between the pair as one of below.
16
Positive Not Negative
Unbiased Not Positive
Negative
Evaluation Dataset Collection
Mark the relationship between the pair as one of below.
17
Positive Not Negative
Unbiased Not Positive
Negative
When sentiment is not explicitly stated but can be inferred, mark it as Inferred.
Document Output
18
Evaluation Dataset Collection
Document Output
19
Evaluation Dataset Collection
•Documents drawn from previous datasets (KBP, MPQA).
•All named entity pairs beyond sentence boundaries.
•Considers the first 15 sentences of document.
•Encouraged to capture implicit sentiment.
Dataset Statistics
20
KBP
MPQA
# of Labeled Entity Pairs Per Doc
0 30 60 90 120
11.6
5
91
44.7
9.5
11.6
Positive Unbiased Negative
Same polarity
Annotator Agreement (Cohen’s Kappa) 0.71
# Docs Avg. # of entities
MPQA 54 10.6
KBP 154 7.9
Total 15,185 entity pair labels
Data Characteristics
• High overlap (about 90%) with KBP and MPQA
• Denser (10x more) entity pair annotations than KBP/MPQA
• Cover all named entity pairs
• Capture inferred sentiment
21
Inferred Sentiment
Positive Negative
Inferred Ratio 70% 58%
The FINRA announced the fine, saying Goldman lacked adequate procedures to ensure the required disclosure.
FINRA Goldman
22
Sentiment Beyond Sentence Boundaries
~25% of entity pairs with sentiment occurs from pairs not appear in the same sentence.
23
Russia heat, smog trigger health problems..……….………………………………… “We never work with a future perspective,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”
Alexei Skripkov Russia
Model
24
• Document-level ILP model framework
• Soft constraints capturing entity-entity interactions
• Individual entity pair model
Document level ILP model
25
Objective = + + +
NegativePositive
Factual Tie
Document level ILP model
NegativePositive
Factual Tie
26
Objective = + + +
Global soft constraints capturing entity-entity interactions
Document level ILP model
27
Objective = + + +
• Russia criticizes Belarus for … • The speaker of the Russian parliament
Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim
Belarus Russia
Opinion Holder Opinion Target
positive? negative? neutral?
Individual entity pair sentiment classifier model scores
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
28
Saakhashvili Georgia
president of
Objective = + + +
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
29
Saakhashvili Georgia
president of
Objective = + + +
Russia
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
30
Saakhashvili Georgia
president of
Objective = + + +
Russia
Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)
Inference from Faction Relation
31
Saakhashvili Georgia
president of
Objective = + + +
Russia
Inference from Faction Relation
32
Objective = + + +
Faction Detector
Heuristics to decide whether an entity belongs to the other.
• modifier, compound, possessive or appositive on dependency path
33
Saakhfshvilli
Georgiapresident of
Faction Detector
Taylor Swift
SelenaGomez
’s friend
34
Theresa May
Andrea Leadso
m
’s rival Gryzlov Russiaspeaker of
Good Examples Errors
On small annotated study, ~30% recall, ~60% precision
Inference from Sentiment Relation
35
Russia
Georgia
Belarus
Social Balance Theory(Heider, 1946): Models balance or imbalance of sentiment relation in triadic relations.
Objective = + + +
Inference from Sentiment Relation
36
Russia
Georgia
Belarus
Social Balance Theory(Heider, 1946): Models balance or imbalance of sentiment relation in triadic relations.
Objective = + + +
Russia
Georgia
Belarus
Inference from Sentiment Relation
37
Objective = + + +
38
Reciprocity (Gouldner, 1960): Opinions are often reciprocal.
Inference from Sentiment Relation
Russia Georgia Belarus
Objective = + + +
39
Inference from Sentiment Relation
Russia Georgia Belarus
Objective = + + +
Reciprocity (Gouldner, 1960): Opinions are often reciprocal.
40
Inference from Sentiment Relation
Objective = + + +
Reciprocity (Gouldner, 1960): Opinions are often reciprocal.
Data shows that these constraints are often satisfied.
5
20
35
50
65
80
Faction Balance Reciprocity
Random Data
41
• Incorporates the scores from a pairwise classifier model
42
Objective = + + +
• Factors from linear SVM classifier model
Individual Entity Pair Model
• Document Features
• Dependency Path Features
• Quotation Features
• Russia criticizes Belarus for … • The speaker of the Russian parliament
Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim
Russia Belarus
43
Individual Entity Pair Model
Document Features• Capture the entity salience.
• Do they appear in headline?• Do they occur together frequently?
44
The most frequently mentioned entity is 3.4 times more likely to have sentiment.
Dependency Path Features• The polarity of path by MPQA sentiment lexicon (Wilson et al 05)
• Path containing [dobj, rev_subj]
45
Skah Norway
Quotation Features
46
Russia heat, smog trigger health problems..……….………………………………… “We never care to work with a future perspective in mind,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”
Alexei Skripkov Russia
• The polarity of quotes by MPQA sentiment lexicon
Evaluation
47
Experimental Set-up
• Precision, Recall, and F1 score
• Positive and negative label sets
• Half as development set, half as a test set
48
Comparison Systems• Random (following the data distribution of the dev set)
• Sentence-level RNN classifier:
• Sentiment model on movie domain (Socher 2013)
• Sentiment labels from sentences of the pair co-occurring
• Pairwise classifier (without global inference)
49
Results (KBP)F1
Sco
re
0
10
20
30
40
Positive Negative
Random Sentence Pairwise Global
50
Ablation StudyF1
Sco
re
20
25
30
35
40
Base +Reciprocity +Balance Theory +Faction
Positive Negative
51
Contributions
• Document-level sentiment inference among pairs of entities
• Intuitions from social balance theory and reciprocity
• Models factual relationship affecting sentiment relation
52
Future work• Incorporating additional types of factual relationships
• Refine global constraints based on entity types
53
Data is available!
http://homes.cs.washington.edu/~eunsol/project_page/acl16/
Thanks!Questions?
54