10/28/2014 EMNLP 2014 in Doha, Qatar Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures Kazuma Hashimoto (UT) Pontus Stenetorp (UT) Makoto Miwa (TTI) Yoshimasa Tsuruoka (UT) University of Tokyo (UT) Toyota Technological Institute (TTI)
119
Embed
Jointly Learning Word Representations and Composition Functions Using …hassy/publications/... · 2014-10-28 · Word Representations and Composition Functions Using Predicate-Argument
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
10/28/2014 EMNLP 2014 in Doha, Qatar
Jointly LearningWord Representations and Composition Functions
Using Predicate-Argument Structures
Kazuma Hashimoto (UT)
Pontus Stenetorp (UT)
Makoto Miwa (TTI)
Yoshimasa Tsuruoka (UT)
University of Tokyo (UT)Toyota Technological Institute (TTI)
• Neural networks + large unlabeled corpora
Neural Word Vector Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
• Neural networks + large unlabeled corpora
– Learn word (i.e. single token) representations
• e.g.) word2vec
(Mikolov+ 2013; Mnih and Kavukcuoglu 2013; inter alia)
Neural Word Vector Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
• Neural networks + large unlabeled corpora
– Learn word (i.e. single token) representations
• e.g.) word2vec
(Mikolov+ 2013; Mnih and Kavukcuoglu 2013; inter alia)
– Learn composed vector representations
• e.g.) compositional neural language models
for verb-object vectors (Tsubaki+ 2013)
Neural Word Vector Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
Relation to Previous Work
10/28/2014 EMNLP 2014 in Doha, Qatar
word2vec
Compositional
neural
language models
Our model
single token
representations ✓ ✓ ✓recursive structures
of syntactic relations x x ✓pre-training
✓ x ✓composition
x ✓ ✓
Relation to Previous Work
10/28/2014 EMNLP 2014 in Doha, Qatar
word2vec
Compositional
neural
language models
Our model
single token
representations ✓ ✓ ✓recursive structures
of syntactic relations x x ✓pre-training
✓ x ✓composition
x ✓ ✓
Relation to Previous Work
10/28/2014 EMNLP 2014 in Doha, Qatar
word2vec
Compositional
neural
language models
Our model
single token
representations ✓ ✓ ✓recursive structures
of syntactic relations x x ✓pre-training
✓ x ✓composition
x ✓ ✓
Relation to Previous Work
10/28/2014 EMNLP 2014 in Doha, Qatar
word2vec
Compositional
neural
language models
Our model
single token
representations ✓ ✓ ✓recursive structures
of syntactic relations x x ✓pre-training
✓ x ✓composition
x ✓ ✓
• Learning word and composed representations
A Joint Learning Model
10/28/2014 EMNLP 2014 in Doha, Qatar
• Learning word and composed representations
– using syntactic structures of unlabeled corpora
d vectors
A Joint Learning Model
10/28/2014 EMNLP 2014 in Doha, Qatar
• Learning word and composed representations
– using syntactic structures of unlabeled corpora
– without pre-trained word vectors
A Joint Learning Model
10/28/2014 EMNLP 2014 in Doha, Qatar
• Learning word and composed representations
– using syntactic structures of unlabeled corpora
– without pre-trained word vectors
A Joint Learning Model
10/28/2014 EMNLP 2014 in Doha, Qatar
storm
downpourpay
solve
overcome
• Learning word and composed representations
– using syntactic structures of unlabeled corpora
– without pre-trained word vectors
A Joint Learning Model
10/28/2014 EMNLP 2014 in Doha, Qatar
storm
downpour
heavy rainmake payment
paysolve problem
achieve objective
bridge gap
solve
overcome
• Learning word and composed representations
– using syntactic structures of unlabeled corpora
– without pre-trained word vectors
A Joint Learning Model
10/28/2014 EMNLP 2014 in Doha, Qatar
storm
downpour
heavy rainmake payment
paysolve problem
achieve objective
bridge gap
solve
overcome
State-of-the-art scores
for phrase similarity tasks with transitive verbs
1. Learning word representations
using predicate-argument structures
2. Jointly learning word representations and
composition functions
3. Evaluation on phrase similarity tasks
4. Conclusion
Overview
10/28/2014 EMNLP 2014 in Doha, Qatar
1. Learning word representations
using predicate-argument structures
2. Jointly learning word representations and
composition functions
3. Evaluation on phrase similarity tasks
4. Conclusion
Overview
10/28/2014 EMNLP 2014 in Doha, Qatar
• Standard dependency structures
– Relations between heads and modifiers
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
the heavy rain caused the car accidents
• Standard dependency structures
– Relations between heads and modifiers
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
the heavy rain caused the car accidents
nndet
det
amod
nsubj dobj
root
• Standard dependency structures
– Relations between heads and modifiers
• Predicate-Argument Structures (PASs)
– Relations between predicates and arguments
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
the heavy rain caused the car accidents
nndet
det
amod
nsubj dobj
root
the heavy rain caused the car accidents
• Each predicate in a sentence has
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
(Enju parser (Miyao and Tsujii 2008))
the heavy rain caused the car accidents
• Each predicate in a sentence has
– a specific category
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
(Enju parser (Miyao and Tsujii 2008))
the heavy rain caused the car accidents
• Each predicate in a sentence has
– a specific category
– zero or more arguments
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
(Enju parser (Miyao and Tsujii 2008))
the heavy rain caused the car accidents
• Each predicate in a sentence has
– a specific category
– zero or more arguments
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
(Enju parser (Miyao and Tsujii 2008))
the heavy rain caused the car accidentsadjective
argument 1
• Each predicate in a sentence has
– a specific category
– zero or more arguments
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
(Enju parser (Miyao and Tsujii 2008))
the heavy rain caused the car accidentsverbadjective
argument 1
argument 1 argument 2
• Each predicate in a sentence has
– a specific category
– zero or more arguments
Predicate-Argument Structures (PASs)
10/28/2014 EMNLP 2014 in Doha, Qatar
(Enju parser (Miyao and Tsujii 2008))
the heavy rain caused the car accidentsverbadjective noun
argument 1 argument 1
argument 1 argument 2
• Given a PAS, discriminating between
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
• Given a PAS, discriminating between
– a word in the specific PAS
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
• Given a PAS, discriminating between
– a word in the specific PAS and
– a word drawn from a noise distribution
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
• Given a PAS, discriminating between
– a word in the specific PAS and
– a word drawn from a noise distribution
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
rain cause accidentverb
argument 1 argument 2
• Given a PAS, discriminating between
– a word in the specific PAS and
– a word drawn from a noise distribution
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
a target word: cause
rain cause accidentverb
argument 1 argument 2
• Given a PAS, discriminating between
– a word in the specific PAS and
– a word drawn from a noise distribution
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
a target word: cause
a noise distribution
(scaled unigram distribution
in (Mikolov+, 2013))
rain cause accidentverb
argument 1 argument 2
• Given a PAS, discriminating between
– a word in the specific PAS and
– a word drawn from a noise distribution
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
a target word: causevs
a drawn word: eat
a noise distribution
(scaled unigram distribution
in (Mikolov+, 2013))
rain eat accidentverb
argument 1 argument 2
• Given a PAS, discriminating between
– a word in the specific PAS and
– a word drawn from a noise distribution
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
a target word: causevs
a drawn word: eat
a noise distribution
(scaled unigram distribution
in (Mikolov+, 2013))
rain eat accidentverb
argument 1 argument 2
context information
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
rain cause accidentverb
argument 1 argument 2
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident word vectors
rain cause accidentverb
argument 1 argument 2
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+argument
2
word vectors
rain cause accidentverb
argument 1 argument 2
𝑝 cause =
tanh(ℎ𝑎𝑟𝑔1𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(rain) +
ℎ𝑎𝑟𝑔2𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(accident))
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+argument
2
𝑝 cause =
tanh(ℎ𝑎𝑟𝑔1𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(rain) +
ℎ𝑎𝑟𝑔2𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(accident))
word vectors
rain cause accidentverb
argument 1 argument 2
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+argument
2
𝑝 cause =
tanh(ℎ𝑎𝑟𝑔1𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(rain) +
ℎ𝑎𝑟𝑔2𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(accident))
word vectors
rain cause accidentverb
argument 1 argument 2
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+argument
2
𝑝 cause =
tanh(ℎ𝑎𝑟𝑔1𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(rain) +
ℎ𝑎𝑟𝑔2𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(accident))
word vectors
rain cause accidentverb
argument 1 argument 2
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause
𝑠
argument 2
𝑝 cause =
tanh(ℎ𝑎𝑟𝑔1𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(rain) +
ℎ𝑎𝑟𝑔2𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(accident))
𝑠 = 𝑣 cause ∙ 𝑝(cause)
𝑠′ = 𝑣 eat ∙ 𝑝 cause
word vectors
rain cause accidentverb
argument 1 argument 2
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑝 cause =
tanh(ℎ𝑎𝑟𝑔1𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(rain) +
ℎ𝑎𝑟𝑔2𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(accident))
𝑠 = 𝑣 cause ∙ 𝑝(cause)
𝑠′ = 𝑣 eat ∙ 𝑝 cause
word vectors
rain cause accidentverb
argument 1 argument 2
A Word Prediction Model Using PASs
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝐜𝐨𝐬𝐭: 𝐦𝐚𝐱(𝟎, 𝟏 − 𝒔 + 𝒔′)
𝑝 cause =
tanh(ℎ𝑎𝑟𝑔1𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(rain) +
ℎ𝑎𝑟𝑔2𝑣𝑒𝑟𝑏_𝑎𝑟𝑔12
∗ 𝑣(accident))
𝑠 = 𝑣 cause ∙ 𝑝(cause)
𝑠′ = 𝑣 eat ∙ 𝑝 cause
word vectors
rain cause accidentverb
argument 1 argument 2
• Learning word representations based on
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Learning word representations based on
– specific PAS categories
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Learning word representations based on
– specific PAS categories
– selectional preferences
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Learning word representations based on
– specific PAS categories
– selectional preferences
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Learning word representations based on
– specific PAS categories
– selectional preferences
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Learning word representations based on
– specific PAS categories
– selectional preferences
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Learning word representations based on
– specific PAS categories
– selectional preferences
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
``rain’’ can be
• a subject of ``cause’’
(not ``eat’’)
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Learning word representations based on
– specific PAS categories
– selectional preferences
What We Expect from the Model
10/28/2014 EMNLP 2014 in Doha, Qatar
``rain’’ can be
• a subject of ``cause’’
(not ``eat’’)
• a cause of ``accident’’
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
Examples
10/28/2014 EMNLP 2014 in Doha, Qatar
eat at restaurantpreposition
argument 1 argument 2
heavy rainadjective
argument 1
Examples
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 eat 𝑣 a𝑡
argument 1
+predicate
eat at restaurantpreposition
argument 1 argument 2
heavy rainadjective
argument 1
Examples
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 eat 𝑣 a𝑡
argument 1
+
restaurant cupboard
𝑠
predicate
𝑠′
eat at restaurantpreposition
argument 1 argument 2
heavy rainadjective
argument 1
Examples
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 eat 𝑣 a𝑡
argument 1
+
restaurant cupboard
𝑠
predicate
𝑠′
𝑣 rain
argument 1
+
heavy delicious
𝑠 𝑠′
eat at restaurantpreposition
argument 1 argument 2
heavy rainadjective
argument 1
• Providing additional context information
Adding Bag-of-Words Contexts
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Providing additional context information
– Nouns and Verbs in the same sentences
Adding Bag-of-Words Contexts
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident
argument 1
+
cause eat
𝑠
argument 2
𝑠′
• Providing additional context information
– Nouns and Verbs in the same sentences
Adding Bag-of-Words Contexts
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident 𝑣 road 𝑣 injure
argument 1
+
cause eat
𝑠
argument 2
𝑠′
+
• Providing additional context information
– Nouns and Verbs in the same sentences
Adding Bag-of-Words Contexts
10/28/2014 EMNLP 2014 in Doha, Qatar
𝑣 rain 𝑣 accident 𝑣 road 𝑣 injure
argument 1
+
cause eat
𝑠
argument 2
𝑠′
+BoW
• Learning representations composed by
Beyond Single Word Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
• Learning representations composed by
– multiple words and
Beyond Single Word Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
• Learning representations composed by
– multiple words and
– specific relation categories
Beyond Single Word Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
• Learning representations composed by
– multiple words and
– specific relation categories
Beyond Single Word Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
storm
downpour
• Learning representations composed by
– multiple words and
– specific relation categories
Beyond Single Word Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
storm
downpour
heavy rainadjective
argument 1
• Learning representations composed by
– multiple words and
– specific relation categories
Beyond Single Word Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
storm
downpour
heavy rain
heavy rainadjective
argument 1
• Using connections on graphs of PASs
A Specific PAS as a Single Token
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 rain 𝑣 accident
rain cause accidentverb
argument 1 argument 2
• Using connections on graphs of PASs
A Specific PAS as a Single Token
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
rain cause accidentverb
argument 1 argument 2
heavyadjective
argument 1
carnoun
argument 1
𝑣 rain 𝑣 accident
• Using connections on graphs of PASs
A Specific PAS as a Single Token
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy__rain 𝑣 car__accident
rain cause accidentverb
argument 1 argument 2
heavyadjective
argument 1
carnoun
argument 1
parameterization
• Using connections on graphs of PASs
A Specific PAS as a Single Token
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
Same as Previously!
rain cause accidentverb
argument 1 argument 2
heavyadjective
argument 1
carnoun
argument 1
𝑣 heavy__rain 𝑣 car__accident
parameterization
• Similar tokens for each PAS representation
in terms of cosine similarity
Learned PAS Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
heavy_rain chief_executive world_war
rain
thunderstorm
downpour
blizzard
much_rain
general_manager
vice_president
executive_director
project_manager
managing_director
second_war
plane_crash
riot
last_war
great_war
• Similar tokens for each PAS representation
in terms of cosine similarity
Learned PAS Representations
10/28/2014 EMNLP 2014 in Doha, Qatar
make_payment solve_problem meeting_take_place
make_order
carry_survey
pay_tax
pay
impose_tax
achieve_objective
bridge_gap
improve_quality
deliver_information
encourage_development
hold_meeting
event_take_place
end_season
discussion_take_place
do_work
1. Learning word representations
using predicate-argument structures
2. Jointly learning word representations and
composition functions
3. Evaluation on phrase similarity tasks
4. Conclusion
Overview
10/28/2014 EMNLP 2014 in Doha, Qatar
Why Composition?
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy__rain 𝑣 car__accident
Why Composition?
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
fully parameterized
PAS representations𝑣 heavy__rain 𝑣 car__accident
Why Composition?
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
fully parameterized
PAS representations
• Very large number of combinations of words
𝑣 heavy__rain 𝑣 car__accident
Why Composition?
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
fully parameterized
PAS representations
• Very large number of combinations of words
Data sparseness
𝑣 heavy__rain 𝑣 car__accident
Why Composition?
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
fully parameterized
PAS representations
• Very large number of combinations of words
Data sparseness
• Ignoring information from individual words
𝑣 heavy__rain 𝑣 car__accident
Incorporating Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy rain 𝑣 car accident
Incorporating Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy rain 𝑣 car accident
𝑣 heavy 𝑣 rain 𝑣 car 𝑣 accident word vectors
Incorporating Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy rain 𝑣 car accident
𝑣 heavy 𝑣 rain 𝑣 car 𝑣 accident
𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏 𝒈𝒏𝒐𝒖𝒏_𝒂𝒓𝒈𝟏composition functions
word vectors
Incorporating Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy rain 𝑣 car accident
𝑣 heavy 𝑣 rain 𝑣 car 𝑣 accident
𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏 𝒈𝒏𝒐𝒖𝒏_𝒂𝒓𝒈𝟏composition functions
composed vectors
word vectors
Incorporating Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy rain 𝑣 car accident
𝑣 heavy 𝑣 rain 𝑣 car 𝑣 accident
𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏 𝒈𝒏𝒐𝒖𝒏_𝒂𝒓𝒈𝟏composition functions
composed vectors
Same as Previously!
word vectors
Incorporating Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
argument 1
+
cause eat
𝑠
argument 2
𝑠′
𝑣 heavy 𝑣 rain 𝑣 car 𝑣 accident
𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏 𝒈𝒏𝒐𝒖𝒏_𝒂𝒓𝒈𝟏composition functions
𝑣 heavy rain 𝑣 car accident
• Simple element-wise composition functions
with and without tanh
Composition Functions in this Work
10/28/2014 EMNLP 2014 in Doha, Qatar
• Simple element-wise composition functions
with and without tanh
– e.g.)
Composition Functions in this Work
10/28/2014 EMNLP 2014 in Doha, Qatar
Composition Function 𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏
𝑣 heavy rain = 𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏(𝑣 heavy , 𝑣 rain )
• Simple element-wise composition functions
with and without tanh
– e.g.)
Composition Functions in this Work
10/28/2014 EMNLP 2014 in Doha, Qatar
Composition Function 𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏
Add𝑙 𝑣 heavy + 𝑣 rain
Add𝑛𝑙 tanh(𝑣 heavy + 𝑣 rain )
𝑣 heavy rain = 𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏(𝑣 heavy , 𝑣 rain )
• Simple element-wise composition functions
with and without tanh
– e.g.)
Composition Functions in this Work
10/28/2014 EMNLP 2014 in Doha, Qatar
Composition Function 𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏
Add𝑙 𝑣 heavy + 𝑣 rain
Add𝑛𝑙 tanh(𝑣 heavy + 𝑣 rain )
WAdd𝑙 𝑚𝑝𝑟𝑒𝑑𝑎𝑑𝑗_𝑎𝑟𝑔1
∗ 𝑣 heavy +𝑚𝑎𝑟𝑔1𝑎𝑑𝑗_𝑎𝑟𝑔1
∗ 𝑣 rain
WAdd𝑛𝑙 tanh(𝑚𝑝𝑟𝑒𝑑𝑎𝑑𝑗_𝑎𝑟𝑔1
∗ 𝑣 heavy + 𝑚𝑎𝑟𝑔1𝑎𝑑𝑗_𝑎𝑟𝑔1
∗ 𝑣 rain )
𝑣 heavy rain = 𝒈𝒂𝒅𝒋_𝒂𝒓𝒈𝟏(𝑣 heavy , 𝑣 rain )
Learned Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
make payment solve problem run company
make repayment
make money
make indemnity
make saving
make sum
solve dilemma
solve task
solve difficulty
solve trouble
solve contradiction
run firm
run industry
run corporation
run enterprise
run club
• Similar composed representations in terms of
cosine similarity
Learned Composed Vectors
10/28/2014 EMNLP 2014 in Doha, Qatar
people kill animal animal kill people meeting take place