Multilingual Relation Extraction using Compositional Universal Schema Haw-Shiuan Chang, Abdurrahman Munir, Ao Liu, Johnny Tian-Zheng Wei, Aaron Traylor, Ajay Nagesh, Nicholas Monath, Patrick Verga, Emma Strubell, Andrew McCallum These slides are mostly prepared by Patrick Verga (Team ID: UMass_IESL)
40
Embed
Multilingual Relation Extraction using Compositional ... Relation Extraction using Compositional Universal Schema ... relation extraction using compositional universal ... sults in
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Multilingual Relation Extraction using Compositional Universal Schema
Haw-Shiuan Chang, Abdurrahman Munir, Ao Liu, Johnny Tian-Zheng Wei, Aaron Traylor, Ajay Nagesh,
Nicholas Monath, Patrick Verga, Emma Strubell, Andrew McCallum
These slides are mostly prepared by Patrick Verga
(Team ID: UMass_IESL)
Wei Li studies at Xinghua U. Her 2008 publications include W. Li. “Scalable NLP” ACL, 2008.
Entity Extraction
(NER)Search
Text docs
Entity Mentions
Entities, Relations
Relation Mentions
Wei Li W. Li
Xinghua U.
Member( Wei Li,
Xinghua U.)
Wei Li W. Li
Xinghua U.
Text docs
Text docs
(PER)
(PER)
(ORG)
Slot Filling (SF)
2
Relation Extraction
Query Expansion
Wei Li, Member, ______?
…
Wei Li W. Li
Entity Aliases
Queries
Wei Li, Member,
Xinghua U. …
KB
Wei Li studies at Xinghua U. Her 2008 publications include W. Li. “Scalable NLP” ACL, 2008.
Entity Extraction
(NER)
Resolution (Coref)
Xinghua U.
Text docs
Entity Mentions
Entities, Relations
Relation Mentions
Wei Li W. Li
Xinghua U.
Member( Wei Li,
Xinghua U.)
Wei Li W. Li
Xinghua U.
Text docs
Text docs
(PER)
(PER)
(ORG)
Knowledge Base (KB) Construction
3
Relation Extraction
Wei Li, Member, ______?
…Queries
KB
Wei Li studies at Xinghua U. Her 2008 publications include W. Li. “Scalable NLP” ACL, 2008.
Entity Extraction
(NER)
Resolution (Coref)
answer
Text docs
Entity Mentions
Entities, Relations
Relation Mentions
Wei Li W. Li
Xinghua U.
Member( Wei Li,
Xinghua U.)
Wei Li W. Li
Xinghua U.
Text docs
Text docs
(PER)
(PER)
(ORG)
Resources
4
Relation Extraction
RelationFactorySpanish?
Wei Li, Member, ______?
…Queries
KB
Wei Li studies at Xinghua U. Her 2008 publications include W. Li. “Scalable NLP” ACL, 2008.
Entity Extraction
Resolution (Coref)
query
answer
Text docs
Entity Mentions
Entities, Relations
Relation Mentions
Wei Li W. Li
Xinghua U.
Wei Li W. Li
Xinghua U.
Text docs
Text docs
(PER)
(PER)
(ORG)
Relation Extraction
5
Member( Wei Li,
Xinghua U.)
Relation Extraction
Compositional Universal Schema
Verga, P., Belanger, D., Strubell, E., Roth, B., and McCallum, A. (2016). Multilingual relation extraction using compositional universal schema. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
Relation Types
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Melinda, Bill
Steve, Microsoft
Bill, MicrosoftBill, Steve
Microsoft, SeattleMelinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
structured textual 50k columns
7
Universal Schema
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
for which similarity is difficult to be captured by an Ope-nIE approach because of their syntactically complex con-structions. This motivates the technique in Section 3.2,which uses a deep architecture applied to raw tokens, in-stead of rigid rules for normalizing text to obtain patterns.
Sentence (context tokens italicized) OpenIE patternKhan ’s younger sister, AnnapurnaDevi, who later married Shankar, de-veloped into an equally accomplishedmaster of the surbahar, but custom pre-vented her from performing in public.
arg1’s * sisterarg2
A professor emeritus at Yale, Mandel-brot was born in Poland but as a childmoved with his family to Paris wherehe was educated.
arg1 * moved with* family to arg2
Kissel was born in Provo, Utah, buther family also lived in Reno.
arg1 * lived inarg2
Table 1: Examples of sentences expressing relations.Context tokens (italicized) consist of the text occurringbetween entities (bold) in a sentence. OpenIE patterns areobtained by normalizing the context tokens using hand-coded rules. The top example expresses the per:siblingsrelation and the bottom two examples both express theper:cities of residence relation.
2.4 Universal SchemaWhen applying Universal Schema (Riedel et al., 2013)(USchema) to relation extraction, we combine the Ope-nIE and link-prediction perspectives. By jointly mod-eling both OpenIE patterns and the elements of a targetschema, the method captures broader relational structurethan multi-class classification approaches that just modelthe target schema. Furthermore, the method avoids thedistant supervision alignment difficulties of Section 2.2.
Riedel et al. (2013) augment a knowledge graph froma seed KB with additional edges corresponding to Ope-nIE patterns observed in the corpus. Even if the user doesnot seek to predict these new edges, a joint model over alledges can exploit regularities of the OpenIE edges to im-prove modeling of the labels from the target schema.
The data still consist of (s, r, o) triples, which can bepredicted using link-prediction techniques such as low-rank factorization. Riedel et al. (2013) explore a varietyof approximations to the 3-mode (s, r, o) tensor. Onesuch probabilistic model is:
P ((s, r, o)) = �
�u
>s,o
v
r
�, (1)
where �() is a sigmoid function, us,o
is an embeddingof the entity pair (s, o), and v
r
is an embedding of therelation r, which may be an OpenIE pattern or a rela-tion from the target schema. All of the exposition and re-sults in this paper use this factorization, though many ofthe techniques we present later could be applied easily to
the other factorizations described in Riedel et al. (2013).Note that learning unique embeddings for OpenIE rela-tions does not guarantee that similar patterns, such as thefinal two in Table 1, will be embedded similarly.
As with most of the techniques in Section 2.1, the dataonly consist of positive examples of edges. The absenceof an annotated edge does not imply that the edge is false.In fact, we seek to predict some of these missing edges astrue. Riedel et al. (2013) employ the Bayesian Person-alized Ranking (BPR) approach of Rendle et al. (2009),which does not explicitly model unobserved edges asnegative, but instead seeks to rank the probability of ob-served triples above unobserved triples.
Recently, Toutanova et al. (2015) extended USchemato not learn individual pattern embeddings v
r
, but insteadto embed text patterns using a deep architecture appliedto word tokens. This shares statistical strength betweenOpenIE patterns with similar words. We leverage this ap-proach in Section 3.2. Additional work has modeled theregularities of multi-hop paths through knowledge graphaugmented with text patterns (Lao et al., 2011; Lao et al.,2012; Gardner et al., 2014; Neelakantan et al., 2015).
2.5 Multilingual EmbeddingsMuch work has been done on multilingual word embed-dings. Most of this work uses aligned sentences fromthe Europarl dataset (Koehn, 2005) to align word embed-dings across languages (Gouws et al., 2015; Luong et al.,2015; Hermann and Blunsom, 2014). Others (Mikolovet al., 2013; Faruqui et al., 2014) align separate single-language embedding models using a word-level dictio-nary. Mikolov et al. (2013) use translation pairs to learna linear transform from one embedding space to another.
However, very little work exists on multilingual re-lation extraction. Faruqui and Kumar (2015) performmultilingual OpenIE relation extraction by projecting alllanguages to English using Google translate. However,as explained in Section 2.3 the OpenIE paradigm is notamenable to prediction within a fixed schema. Further,their approach does not generalize to low-resource lan-guages where translation is unavailable – while we usetranslation dictionaries to improve our results, our experi-ments demonstrate that our method is effective even with-out this resource.
3 Methods3.1 Universal Schema as Sentence ClassifierSimilar to many link prediction approaches, (Riedel et al.,2013) perform transductive learning, where a model islearned jointly over train and test data. Predictions aremade by using the model to identify edges that were un-observed in the test data but likely to be true. The ap-proach is vulnerable to the cold start problem in collab-
Relation Typesstructured textual 50k columns
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
8
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###~y
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
9
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
10
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
11
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
####
###
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
Pattern Encoder
####
###
####
###
####
###
####
###
####
###
####
###
####
###
12
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
LSTM
13
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
####
###
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
Pattern Encoder
####
###
####
###
####
###
####
###
####
###
####
###
####
###
LSTMMelinda,
Bill
Entity pair Relation
spouse
married to
#######
####
###
and
####
###
####
###
LSTM
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
JoeJane
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
14
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
####
###
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
Pattern Encoder
####
###
####
###
####
###
####
###
####
###
####
###
####
###
spousequery:
LSTM
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Pattern Encoder
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
15
JoeJane
LSTM
spousequery:
####
###
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
Pattern Encoder
Joe Jane
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
16
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
LSTM
####
###
spousequery:
####
###
Spanish Relation Extraction
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
mar
ried-
to
HQ-in
Steve, Microsoft
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Obama, MichelleObama, US
18
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
Bill, MicrosoftMelinda, Dallas
Bill, SeattleObama, Michelle
English
Spanish
Multilingual Relation Extraction
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
mar
ried-
to
HQ-in
Steve, Microsoft
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
rBill, Microsoft
Bill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
19
Multilingual Universal Schema
Relation Typesstructured English
textual
Spanish
textual
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
mar
ried-
to
HQ-in
Steve, Microsoft
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
rBill, Microsoft
Bill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
20
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
LSTM
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
mar
ried-
to
HQ-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
21
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
2121
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
##############
####
###
####
###
####
###
LSTM
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
mar
ried-
to
HQ-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
##############
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
22
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
22
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
22
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTM
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
mar
ried-
to
HQ-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTM
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
mar
ried-
to
HQ-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
23
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTMCE
O ch
airm
an
pres
iden
t lea
der-o
f he
ad-o
f m
arrie
d-to
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
Melinda, Bill
Entity pair Relation
Michelle, Obama
spouse
married to
esposa
####
##resident
####
####
####
esposaspouse
####
##
cat
####
##
married
####
##
resident
####
##
presidente
####
##
casado
####
##
gato
24
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTMCE
O ch
airm
an
pres
iden
t lea
der-o
f he
ad-o
f m
arrie
d-to
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
25
####
##
####
##
esposa
spouse
####
##
cat
####
##
married
####
##
resident
####
##
casado
####
##
gato
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTMCE
O ch
airm
an
pres
iden
t lea
der-o
f he
ad-o
f m
arrie
d-to
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
####
##
presidente
26##
#### ####
##
esposaspouse
####
##
cat
####
##
married
####
##
presidente
####
##
casado
####
##
gato
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTMCE
O ch
airm
an
pres
iden
t lea
der-o
f he
ad-o
f m
arrie
d-to
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
####
##
resident
####
####
####
esposaspouse
####
##
cat
####
##
married
####
##
casado
####
##
gato ####
##
spouse / esposa
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTMCE
O ch
airm
an
pres
iden
t lea
der-o
f he
ad-o
f m
arrie
d-to
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
####
##
resident
####
##
presidente
####
##
spouse / esposa
####
##
cat
####
##
married
####
##
casado
####
##
gato
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
##############
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
##############
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Michelle
Obama, US
vectorembeddindistributerepresentati
ofsemantilearnedparamete#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
#######
####
###
#######
####
###
LSTMCE
O ch
airm
an
pres
iden
t lea
der-o
f he
ad-o
f m
arrie
d-to
HQ
-in
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
espo
sa
pres
iden
te
vive e
n re
siden
te
fund
ador
####
##
resident
####
##
presidente
Search Engine Supervision
KB
Wei Li studies at Xinghua U. Her 2008 publications include W. Li. “Scalable NLP” ACL, 2008.
Entity Extraction
Resolution (Coref)
query
answer
Text docs
Entity Mentions
Entities, Relations
Relation Mentions
Wei Li W. Li
Xinghua U.
Member( Wei Li,
Xinghua U.)
Wei Li W. Li
Xinghua U.
Text docs
Text docs
(PER)
(PER)
(ORG)
Search Engine as a Resource
30
RelationFactory
Relation Extraction
Barack Obama Michelle Obama
Entity 1 Entity 2
Corpus
Barack Obama Michelle Obama…...
…...
Barack Obama Michelle Obama…...
Distant Supervision
per:spouse wife
and
president
lives with
understands
married to
Key phrases for per:spouse
High frequency signal
Low frequency signal
Global background noise
Entity background noise
High correlation noise
Not expressing a relation
Freebase selection bias
wife
and
president
lives with
married to
Only valid in some entity pairs
Other relation noiseunderstands They might have other relations
Limitations of Distant Supervision
Barack Obama Michelle ObamaEntity 1 Entity 2
wife
and
president
lives with
married to
Corpus
Barack Obama Michelle Obama?
Key phrases
understands
Entity 1 Entity 2
P( ___ = Michelle Obama)
Asking in another direction?
Barack Obama … wife … ______
Barack Obama … and … ______
P( ___ = Michelle Obama)
Hard to accurately estimate this when corpus is not large enough
Asking Google!
wife "his wife"
his married husband "wife of"
"married to" "her husband"
to her
wife "his wife" married husband
"married to" "wife of"
"is married" "is married to" “her husband”
“( wife”
Distant supervision (after applying several noises
removal techniques)
After applying search engine supervision
Noise Filtering Examples
Lower their influence on Universal Schema
Results
KB, XLING Hop0 Hop1 Total
Prec Recall F1 Prec Recall F1 F1
USchema 0.475 0.015 0.028 0.071 0.003 0.005 0.021
LSTM 0.421 0.038 0.069 0.106 0.018 0.030 0.057
The recall is low because we don’t include Chinese query and Chinese corpus
TAC 2016 Component Testing
SF, ENG Hop0 Hop1 Total
Prec Recall F1 Prec Recall F1 F1
DS 0.244 0.168 0.199 0.500 0.013 0.025 0.159
SES+DS 0.229 0.203 0.215 0.444 0.013 0.025 0.174
SF Hop0 (ENG: 350, SPA: 150) Hop1 (ENG: 185, SPA: 53) Total
Prec Recall F1 Prec Recall F1 F1
2015 system 0.302 0.154 0.204 0.081 0.088 0.084 0.155
2016 system 0.229 0.203 0.215 0.444 0.013 0.025 0.174
Spanish 0.1364 0.285 0.187 N/A N/A N/A 0.160
KB Hop0 Hop1 Total
Prec Recall F1 Prec Recall F1 F1
2015 system 0.390 0.128 0.192 0.253 0.078 0.119 0.168
2016 system 0.271 0.154 0.196 0.039 0.071 0.051 0.127
Spanish 0.280 0.121 0.169 0.060 0.048 0.053 0.124
TAC 2016 ResultsLDC Max All Micro Scores
2015 KB evaluation
2015 system 0.227 0.149 0.192 0.038 0.097 0.054 0.120