CS460/626 : Natural Language Processing/Speech, NLP and the Web (Lecture 23, 24– Parsing Algorithms; Parsing in case of Ambiguity; Probabilistic Parsing) Pushpak Bhattacharyya CSE Dept., IIT Bombay 8 th , 10 th March, 2011 (Lectures 21 and 22 were on Sentiment Analysis by Aditya Joshi)
CS460/626 : Natural Language Processing/Speech, NLP and the Web (Lecture 23, 24– Parsing Algorithms; Parsing in case of Ambiguity; Probabilistic Parsing). Pushpak Bhattacharyya CSE Dept., IIT Bombay 8 th , 10 th March , 2011 - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CS460/626 : Natural Language Processing/Speech, NLP and the Web
(Lecture 23, 24– Parsing Algorithms; Parsing in case of Ambiguity; Probabilistic Parsing)
Pushpak BhattacharyyaCSE Dept., IIT Bombay
8th, 10th March, 2011
(Lectures 21 and 22 were on Sentiment Analysis by Aditya Joshi)
A note on Language Modeling Example sentence
“ ^ The tortoise beat the hare in the race.”
Guided Guided by Guided byby frequency Language world
Knowledge KnowledgeN-gram (n=3) CFG Probabilistic
CFGDependencyGrammar
Prob. DG
^ the tortoise 5*10-3
S-> NP VP S->NP VP 1.0
Semantic Roles agt, obj, sen, etc.Semantic Rules are always between “Heads”
Semantic Roles with probabilitiesthe tortoise beat
3*10-2NP->DT N NP->DT N
0.5tortoise beat the 7*10-5
VP->V NP PP
VP->V NP PP 0.4
beat the hare5*10-6
PP-> P NP PP-> P NP 1.0
Parse Tree
S
NP VP
DT N V NP PP
The Tortoise beat DT N P NP
the hare in DT N
the race
UNL Expression
beat@past
tortoise@def hare@defrace@de
f
agt obj scn (scene)
Purpose of LM Prediction of next word (Speech
Processing) Language Identification (for same
script) Belongingness check (parsing) P(NP->DT N) means what is the
probability that the ‘YIELD’ of the non terminal NP is DT N
Need for Deep Parsing Sentences are linear structures But there is a hierarchy- a tree-
hidden behind the linear structure There are constituents and
branches
PPs are at the same level: flat with respect to the head word “book”
NP
PPAP
big
The
of poems
with the blue cover
[The big book of poems with theBlue cover] is on the table.
book
No distinction in terms of dominance or c-
command
PP
“Constituency test of Replacement” runs into problems
One-replacement: I bought the big [book of poems with
the blue cover] not the small [one] One-replacement targets book of
poems with the blue cover Another one-replacement:
I bought the big [book of poems] with the blue cover not the small [one] with the red cover
One-replacement targets book of poems
More deeply embedded structureNP
PP
AP
big
The
of poems
with the blue cover
N’1
Nbook
PP
N’2
N’3
Grammar and Parsing Algorithms
A simplified grammar S NP VP NP DT N | N VP V ADV | V
Example Sentence
People laugh1 2 3
Lexicon:People - N, V Laugh - N, V
These are positions
This indicate that both Noun and Verb is
possible for the word “People”
Top-Down Parsing State Backup State Action-----------------------------------------------------------------------------------------------------1. ((S) 1) - -
(Here P means preposition)NP2 attaches to NP1 ?or NP2 attaches to V ?
Parse Trees for a Structurally Ambiguous SentenceLet the grammar be – S NP VPNP DT N | DT N PPPP P NPVP V NP PP | V NPFor the sentence,“I saw a boy with a telescope”
Parse Tree - 1S
NP VP
N V NP
Det N PP
P NP
Det N
I saw
a boy
with
a telescope
Parse Tree -2S
NP VP
N V NP
Det N
PP
P NP
Det NI saw
a boy with
a telescope
Parsing Structural Ambiguity
Parsing for Structurally Ambiguous Sentences Sentence “I saw a boy with a telescope” Grammar:
S NP VPNP ART N | ART N PP | PRONVP V NP PP | V NP
ART a | an | theN boy | telescopePRON IV saw
Ambiguous Parses Two possible parses:
PP attached with Verb (i.e. I used a telescope to see)
( S ( NP ( PRON “I” ) ) ( VP ( V “saw” ) ( NP ( (ART “a”) ( N “boy”))( PP (P “with”) (NP ( ART “a” ) ( N
“telescope”))))) PP attached with Noun (i.e. boy had a
telescope) ( S ( NP ( PRON “I” ) ) ( VP ( V “saw” )
( NP ( (ART “a”) ( N “boy”) (PP (P “with”) (NP ( ART “a” ) ( N
“telescope”))))))
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
4 ( ( VP ) 2 ) − Consumed “I”
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
4 ( ( VP ) 2 ) − Consumed “I”
5 ( ( V NP PP ) 2 ) ( ( V NP ) 2 ) − Verb Attachment Rule used
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
4 ( ( VP ) 2 ) − Consumed “I”
5 ( ( V NP PP ) 2 ) ( ( V NP ) 2 ) − Verb Attachment Rule used
6 ( ( NP PP ) 3 ) − Consumed “saw”
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
4 ( ( VP ) 2 ) − Consumed “I”
5 ( ( V NP PP ) 2 ) ( ( V NP ) 2 ) − Verb Attachment Rule used
6 ( ( NP PP ) 3 ) − Consumed “saw”
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
4 ( ( VP ) 2 ) − Consumed “I”
5 ( ( V NP PP ) 2 ) ( ( V NP ) 2 ) − Verb Attachment Rule used
6 ( ( NP PP ) 3 ) − Consumed “saw”
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
8 ( ( N PP) 4 ) − Consumed “a”
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
4 ( ( VP ) 2 ) − Consumed “I”
5 ( ( V NP PP ) 2 ) ( ( V NP ) 2 ) − Verb Attachment Rule used
6 ( ( NP PP ) 3 ) − Consumed “saw”
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
8 ( ( N PP) 4 ) − Consumed “a”
9 ( ( PP ) 5 ) − Consumed “boy”
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
2 ( ( NP VP ) 1 ) − − Use NP ART N | ART N PP | PRON
3 ( ( ART N VP ) 1 ) (a) ( ( ART N PP VP ) 1 )
(b) ( ( PRON VP ) 1)
− ART does not match “I”,
backup state (b) used
3B
( ( PRON VP ) 1 ) − −
4 ( ( VP ) 2 ) − Consumed “I”
5 ( ( V NP PP ) 2 ) ( ( V NP ) 2 ) − Verb Attachment Rule used
6 ( ( NP PP ) 3 ) − Consumed “saw”
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
8 ( ( N PP) 4 ) − Consumed “a”
9 ( ( PP ) 5 ) − Consumed “boy”
10
( ( P NP ) ) − −
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
… … … … …
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
8 ( ( N PP) 4 ) − Consumed “a”
9 ( ( PP ) 5 ) − Consumed “boy”
10
( ( P NP ) 5 ) − −
11
( ( NP ) 6 ) − Consumed “with”
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
… … … … …
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
8 ( ( N PP) 4 ) − Consumed “a”
9 ( ( PP ) 5 ) − Consumed “boy”
10
( ( P NP ) 5 ) − −
11
( ( NP ) 6 ) − Consumed “with”
12
( ( ART N ) 6 ) (a) ( ( ART N PP ) 6 )(b) ( ( PRON ) 6)
−
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
… … … … …
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
8 ( ( N PP) 4 ) − Consumed “a”
9 ( ( PP ) 5 ) − Consumed “boy”
10
( ( P NP ) 5 ) − −
11
( ( NP ) 6 ) − Consumed “with”
12
( ( ART N ) 6 ) (a) ( ( ART N PP ) 6 )(b) ( ( PRON ) 6)
−
13
( ( N ) 7 ) − Consumed “a”
Top Down ParseState Backup State Action Comments
1 ( ( S ) 1 ) − − Use S NP VP
… … … … …
7 ( ( ART N PP ) 3 ) (a) ( ( ART N PP PP ) 3 )
(b) ( ( PRON PP ) 3 )
8 ( ( N PP) 4 ) − Consumed “a”
9 ( ( PP ) 5 ) − Consumed “boy”
10
( ( P NP ) 5 ) − −
11
( ( NP ) 6 ) − Consumed “with”
12
( ( ART N ) 6 ) (a) ( ( ART N PP ) 6 )(b) ( ( PRON ) 6)
−
13
( ( N ) 7 ) − Consumed “a”
14
( ( − ) 8 ) − Consume “telescope”
Finish Parsing
Top Down Parsing - Observations Top down parsing gave us the Verb
Attachment Parse Tree (i.e., I used a telescope)
To obtain the alternate parse tree, the backup state in step 5 will have to be invoked
Is there an efficient way to obtain all parses ?
I saw a boy with a telescope1 2 3 4 5 6 7 8
Colour Scheme : Blue for Normal Parse Green for Verb Attachment Parse Purple for Noun Attachment Parse Red for Invalid Parse
Bottom Up Parse
I saw a boy with a telescope1 2 3 4 5 6 7 8
Bottom Up Parse
NP12 PRON12S1?NP12VP2?
I saw a boy with a telescope1 2 3 4 5 6 7 8
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3?
I saw a boy with a telescope1 2 3 4 5 6 7 8
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
NP35ART34N45
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
NP35ART34N45
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5?
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?NP35ART34N45
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5?
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?NP35ART34N45 NP68ART67N7?
NP6?ART67N78PP8?
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5?
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?NP35ART34N45 NP68ART67N7?
NP6?ART67N78PP8?
NP68ART67N78
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5?
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?NP35ART34N45 NP68ART67N7?
NP6?ART67N78PP8?
NP68ART67N78
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5? PP58P56NP68
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?NP35ART34N45 NP68ART67N7?
NP6?ART67N78PP8?
NP68ART67N78
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5? PP58P56NP68
NP38ART34N45PP58
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?NP35ART34N45 NP68ART67N7?
NP6?ART67N78PP8?
NP68ART67N78
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5? PP58P56NP68
NP38ART34N45PP58
VP28V23NP38VP28V23NP35PP58
I saw a boy with a telescope1 2 3 4 5 6 7 8
NP3?ART34N45PP5?
Bottom Up Parse
NP12 PRON12S1?NP12VP2? VP2?V23NP3?PP??
VP2?V23NP3? NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?NP35ART34N45 NP68ART67N7?
NP6?ART67N78PP8?
NP68ART67N78
VP25V23NP35 S15NP12VP25
VP2?V23NP35PP5? PP58P56NP68
NP38ART34N45PP58
VP28V23NP38VP28V23NP35PP58
S18NP12VP28
Bottom Up Parsing - Observations Both Noun Attachment and Verb
Attachment Parses obtained by simply systematically applying the rules
Numbers in subscript help in verifying the parse and getting chunks from the parse
ExerciseFor the sentence,“The man saw the boy with a
telescope” & the grammar given previously, compare the performance of top-down, bottom-up & top-down chart parsing.
1( | ) = 1mP w t If t is a parse tree for the sentence w1m, this will be 1 !!
Assumptions of the PCFG model Place invariance :
P(NP DT NN) is same in locations 1 and 2 Context-free :
P(NP DT NN | anything outside “The child”) = P(NP DT NN)
Ancestor free : At 2,P(NP DT NN|its ancestor is VP)
= P(NP DT NN)
S
NP
The child
VP
NP
The toy
1
2
Probability of a parse tree Domination :We say Nj dominates from k
to l, symbolized as , if Wk,l is derived from Nj
P (tree |sentence) = P (tree | S1,l ) where S1,l means that the start symbol S dominates the word sequence
W1,l
P (t |s) approximately equals joint probability of constituent non-terminals dominating the sentence fragments (next slide)
Probability of a parse tree (cont.)S1,l
NP1,2 VP3,l
N2V3,3 PP4,l
P4,4 NP5,lw2
w4
DT1
w1 w3
w5 wl
P ( t|s ) = P (t | S1,l )= P ( NP1,2, DT1,1 , w1,
N2,2, w2,VP3,l, V3,3 , w3,
PP4,l, P4,4 , w4, NP5,l, w5…l | S1,l )
= P ( NP1,2 , VP3,l | S1,l) * P ( DT1,1 , N2,2 | NP1,2) * D(w1 | DT1,1) * P (w2 | N2,2) * P (V3,3, PP4,l | VP3,l) * P(w3 | V3,3) * P( P4,4, NP5,l | PP4,l ) * P(w4|P4,4) * P (w5…l | NP5,l)
(Using Chain Rule, Context Freeness and Ancestor Freeness )
HMM ↔ PCFG O observed sequence ↔ w1m
sentence X state sequence ↔ t parse
tree model ↔ G grammar
Three fundamental questions
HMM ↔ PCFG How likely is a certain observation given the
model? ↔ How likely is a sentence given the grammar?
How to choose a state sequence which best explains the observations? ↔ How to choose a parse which best supports the sentence?
arg max ( | , )X
P X O 1arg max ( | , )mt
P t w G↔
1( | )mP w G( | )P O ↔
HMM ↔ PCFG
How to choose the model parameters that best explain the observed data? ↔ How to choose rule probabilities which maximize the probabilities of the observed sentences?arg max ( | )P O
1arg max ( | )m
GP w G↔
Interesting Probabilities
The gunman sprayed the building with bullets 1 2 3 4 5 6 7
(4,5) NP
N1
NP
(4,5)NP
What is the probability of having a NP at this position such that it will derive “the building” ? -
What is the probability of starting from N1 and deriving “The gunman sprayed”, a NP and “with bullets” ? -
Inside Probabilities
Outside Probabilities
Interesting Probabilities Random variables to be considered
The non-terminal being expanded. E.g., NP
The word-span covered by the non-terminal. E.g., (4,5) refers to words “the building”
While calculating probabilities, consider: The rule to be used for expansion :
E.g., NP DT NN The probabilities associated with the RHS