1 Basic Parsing with Context-Free Grammars Slides adapted from Julia Hirschberg and Dan Jurafsky
Basic Parsing with Context-Free Grammars
Feb 08, 2016
Basic Parsing with Context-Free Grammars. Slides adapted from Julia Hirschberg and Dan Jurafsky. Homework – Getting Started. Data News articles in TDT4 Make sure you are looking in ENG sub-directory You need to represent each article in .arff form - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Basic Parsing with Context-Free Grammars
Slides adapted from Julia Hirschberg and Dan Jurafsky
2
Homework ndash Getting Started
Datandash News articles in TDT4ndash Make sure you are looking in ENG sub-directoryndash You need to represent each article in arff formndash You need to write a program that will extract ldquofeaturesrdquo from
each articlendash (Note that files with POS tags are now available in
Eng_POSTAGGED) The arff file contains independent variables as well
as the dependent variable
3
An example
Start with classifying into topic Suppose you want to start with just the
words Two approaches
1 Use your intuition to choose a few words that might disambiguate
2 Start with all words
4
What would your arff file look like
Words are the attributes What are the values
Binary present or not Frequency how many times it occurs TFIDF how many times it occurs in this document (TF
= term frequency) divided by how many times it occurs in all documents (DF = document frequency
5
news_2865input
ltDOCgtltSOURCE_TYPEgtNWIREltSOURCE_TYPEgtltSOURCE_LANGgtENGltSOURCE_LANGgtltSOURCE_ORGgtNYTltSOURCE_ORGgtltDOC_DATEgt20010101ltDOC_DATEgtltBROAD_TOPICgtBT_9ltBROAD_TOPICgtltNARROW_TOPICgtNT_34ltNARROW_TOPICgtltTEXTgtReversing a policy that has kept medical errors secret for more than two decades federal
officials say they will soon allow Medicare beneficiaries to obtain data about doctors who botched their care Tens of thousands of Medicare patients file complaints each year about the quality of care they receive from doctors and hospitals But in many cases patients get no useful information because doctors can block the release of assessments of their performance Under a new policy officials said doctors will no longer be able to veto disclosure of the findings of investigations Federal law has for many years allowed for review of care received by Medicare patients and the law says a peer review organization must inform the patient of the ``final disposition of the complaint in each case But the federal rules used to carry out the law say the peer review organization may disclose information about a doctor only ``with the consent of that practitioner The federal manual for peer review organizations includes similar language about disclosure Under the new plan investigators will have to tell patients whether their care met ``professionally recognized standards of health care and inform them of any action against the doctor or the hospital Patients could use such information in lawsuits and other actions against doctors and hospitals that provided substandard care The new policy came in response to a lawsuit against the government
6
All words for news_2865input
Class BT_9 dependent Reversing 1 independent A 100 Policy 20 That 50 Has 75 Kept 3 Commonwealth 0 (news_2816input) Independent 0 States 0 Preemptive 0 Refugees 0
7
Try it
Open your file Select attributes using Chi-square You can cut and paste resulting attributes to a file Classify
How does it work
Try n-grams POS or date next in same wayndash How many features would each give you
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
2
Homework ndash Getting Started
Datandash News articles in TDT4ndash Make sure you are looking in ENG sub-directoryndash You need to represent each article in arff formndash You need to write a program that will extract ldquofeaturesrdquo from
each articlendash (Note that files with POS tags are now available in
Eng_POSTAGGED) The arff file contains independent variables as well
as the dependent variable
3
An example
Start with classifying into topic Suppose you want to start with just the
words Two approaches
1 Use your intuition to choose a few words that might disambiguate
2 Start with all words
4
What would your arff file look like
Words are the attributes What are the values
Binary present or not Frequency how many times it occurs TFIDF how many times it occurs in this document (TF
= term frequency) divided by how many times it occurs in all documents (DF = document frequency
5
news_2865input
ltDOCgtltSOURCE_TYPEgtNWIREltSOURCE_TYPEgtltSOURCE_LANGgtENGltSOURCE_LANGgtltSOURCE_ORGgtNYTltSOURCE_ORGgtltDOC_DATEgt20010101ltDOC_DATEgtltBROAD_TOPICgtBT_9ltBROAD_TOPICgtltNARROW_TOPICgtNT_34ltNARROW_TOPICgtltTEXTgtReversing a policy that has kept medical errors secret for more than two decades federal
officials say they will soon allow Medicare beneficiaries to obtain data about doctors who botched their care Tens of thousands of Medicare patients file complaints each year about the quality of care they receive from doctors and hospitals But in many cases patients get no useful information because doctors can block the release of assessments of their performance Under a new policy officials said doctors will no longer be able to veto disclosure of the findings of investigations Federal law has for many years allowed for review of care received by Medicare patients and the law says a peer review organization must inform the patient of the ``final disposition of the complaint in each case But the federal rules used to carry out the law say the peer review organization may disclose information about a doctor only ``with the consent of that practitioner The federal manual for peer review organizations includes similar language about disclosure Under the new plan investigators will have to tell patients whether their care met ``professionally recognized standards of health care and inform them of any action against the doctor or the hospital Patients could use such information in lawsuits and other actions against doctors and hospitals that provided substandard care The new policy came in response to a lawsuit against the government
6
All words for news_2865input
Class BT_9 dependent Reversing 1 independent A 100 Policy 20 That 50 Has 75 Kept 3 Commonwealth 0 (news_2816input) Independent 0 States 0 Preemptive 0 Refugees 0
7
Try it
Open your file Select attributes using Chi-square You can cut and paste resulting attributes to a file Classify
How does it work
Try n-grams POS or date next in same wayndash How many features would each give you
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
3
An example
Start with classifying into topic Suppose you want to start with just the
words Two approaches
1 Use your intuition to choose a few words that might disambiguate
2 Start with all words
4
What would your arff file look like
Words are the attributes What are the values
Binary present or not Frequency how many times it occurs TFIDF how many times it occurs in this document (TF
= term frequency) divided by how many times it occurs in all documents (DF = document frequency
5
news_2865input
ltDOCgtltSOURCE_TYPEgtNWIREltSOURCE_TYPEgtltSOURCE_LANGgtENGltSOURCE_LANGgtltSOURCE_ORGgtNYTltSOURCE_ORGgtltDOC_DATEgt20010101ltDOC_DATEgtltBROAD_TOPICgtBT_9ltBROAD_TOPICgtltNARROW_TOPICgtNT_34ltNARROW_TOPICgtltTEXTgtReversing a policy that has kept medical errors secret for more than two decades federal
officials say they will soon allow Medicare beneficiaries to obtain data about doctors who botched their care Tens of thousands of Medicare patients file complaints each year about the quality of care they receive from doctors and hospitals But in many cases patients get no useful information because doctors can block the release of assessments of their performance Under a new policy officials said doctors will no longer be able to veto disclosure of the findings of investigations Federal law has for many years allowed for review of care received by Medicare patients and the law says a peer review organization must inform the patient of the ``final disposition of the complaint in each case But the federal rules used to carry out the law say the peer review organization may disclose information about a doctor only ``with the consent of that practitioner The federal manual for peer review organizations includes similar language about disclosure Under the new plan investigators will have to tell patients whether their care met ``professionally recognized standards of health care and inform them of any action against the doctor or the hospital Patients could use such information in lawsuits and other actions against doctors and hospitals that provided substandard care The new policy came in response to a lawsuit against the government
6
All words for news_2865input
Class BT_9 dependent Reversing 1 independent A 100 Policy 20 That 50 Has 75 Kept 3 Commonwealth 0 (news_2816input) Independent 0 States 0 Preemptive 0 Refugees 0
7
Try it
Open your file Select attributes using Chi-square You can cut and paste resulting attributes to a file Classify
How does it work
Try n-grams POS or date next in same wayndash How many features would each give you
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
4
What would your arff file look like
Words are the attributes What are the values
Binary present or not Frequency how many times it occurs TFIDF how many times it occurs in this document (TF
= term frequency) divided by how many times it occurs in all documents (DF = document frequency
5
news_2865input
ltDOCgtltSOURCE_TYPEgtNWIREltSOURCE_TYPEgtltSOURCE_LANGgtENGltSOURCE_LANGgtltSOURCE_ORGgtNYTltSOURCE_ORGgtltDOC_DATEgt20010101ltDOC_DATEgtltBROAD_TOPICgtBT_9ltBROAD_TOPICgtltNARROW_TOPICgtNT_34ltNARROW_TOPICgtltTEXTgtReversing a policy that has kept medical errors secret for more than two decades federal
officials say they will soon allow Medicare beneficiaries to obtain data about doctors who botched their care Tens of thousands of Medicare patients file complaints each year about the quality of care they receive from doctors and hospitals But in many cases patients get no useful information because doctors can block the release of assessments of their performance Under a new policy officials said doctors will no longer be able to veto disclosure of the findings of investigations Federal law has for many years allowed for review of care received by Medicare patients and the law says a peer review organization must inform the patient of the ``final disposition of the complaint in each case But the federal rules used to carry out the law say the peer review organization may disclose information about a doctor only ``with the consent of that practitioner The federal manual for peer review organizations includes similar language about disclosure Under the new plan investigators will have to tell patients whether their care met ``professionally recognized standards of health care and inform them of any action against the doctor or the hospital Patients could use such information in lawsuits and other actions against doctors and hospitals that provided substandard care The new policy came in response to a lawsuit against the government
6
All words for news_2865input
Class BT_9 dependent Reversing 1 independent A 100 Policy 20 That 50 Has 75 Kept 3 Commonwealth 0 (news_2816input) Independent 0 States 0 Preemptive 0 Refugees 0
7
Try it
Open your file Select attributes using Chi-square You can cut and paste resulting attributes to a file Classify
How does it work
Try n-grams POS or date next in same wayndash How many features would each give you
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
5
news_2865input
ltDOCgtltSOURCE_TYPEgtNWIREltSOURCE_TYPEgtltSOURCE_LANGgtENGltSOURCE_LANGgtltSOURCE_ORGgtNYTltSOURCE_ORGgtltDOC_DATEgt20010101ltDOC_DATEgtltBROAD_TOPICgtBT_9ltBROAD_TOPICgtltNARROW_TOPICgtNT_34ltNARROW_TOPICgtltTEXTgtReversing a policy that has kept medical errors secret for more than two decades federal
officials say they will soon allow Medicare beneficiaries to obtain data about doctors who botched their care Tens of thousands of Medicare patients file complaints each year about the quality of care they receive from doctors and hospitals But in many cases patients get no useful information because doctors can block the release of assessments of their performance Under a new policy officials said doctors will no longer be able to veto disclosure of the findings of investigations Federal law has for many years allowed for review of care received by Medicare patients and the law says a peer review organization must inform the patient of the ``final disposition of the complaint in each case But the federal rules used to carry out the law say the peer review organization may disclose information about a doctor only ``with the consent of that practitioner The federal manual for peer review organizations includes similar language about disclosure Under the new plan investigators will have to tell patients whether their care met ``professionally recognized standards of health care and inform them of any action against the doctor or the hospital Patients could use such information in lawsuits and other actions against doctors and hospitals that provided substandard care The new policy came in response to a lawsuit against the government
6
All words for news_2865input
Class BT_9 dependent Reversing 1 independent A 100 Policy 20 That 50 Has 75 Kept 3 Commonwealth 0 (news_2816input) Independent 0 States 0 Preemptive 0 Refugees 0
7
Try it
Open your file Select attributes using Chi-square You can cut and paste resulting attributes to a file Classify
How does it work
Try n-grams POS or date next in same wayndash How many features would each give you
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
6
All words for news_2865input
Class BT_9 dependent Reversing 1 independent A 100 Policy 20 That 50 Has 75 Kept 3 Commonwealth 0 (news_2816input) Independent 0 States 0 Preemptive 0 Refugees 0
7
Try it
Open your file Select attributes using Chi-square You can cut and paste resulting attributes to a file Classify
How does it work
Try n-grams POS or date next in same wayndash How many features would each give you
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
7
Try it
Open your file Select attributes using Chi-square You can cut and paste resulting attributes to a file Classify
How does it work
Try n-grams POS or date next in same wayndash How many features would each give you
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
8
CFG Example Many possible CFGs for English here is an example
(fragment)ndash S NP VPndash VP V NPndash NP DetP N | AdjP NPndash AdjP Adj | Adv AdjPndash N boy | girlndash V sees | likesndash Adj big | smallndash Adv very ndash DetP a | the
the very small boy likes a girl
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
9
Modify the grammar
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
10
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
12
Derivations of CFGs
String rewriting system we derive a string (=derived structure)
But derivation history represented by phrase-structure tree (=derivation structure)
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
13
Formal Definition of a CFG
G = (VTPS) V finite set of nonterminal symbols
T finite set of terminal symbols V and T are disjoint
P finite set of productions of the formA A V and (T V)
S V start symbol
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
14
Context
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)A -gt B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
15
Key Constituents (English)
Sentences Noun phrases Verb phrases Prepositional phrases
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
16
Sentence-Types Declaratives I do not
S -gt NP VP Imperatives Go around again
S -gt VP Yes-No Questions Do you like my hat
S -gt Aux NP VP WH Questions What are they going to do
S -gt WH Aux NP VP
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
18
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
19
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
20
NPs
NP -gt Pronounndash I came you saw it they conquered
NP -gt Proper-Nounndash New Jersey is west of New York Cityndash Lee Bollinger is the president of Columbia
NP -gt Det Nounndash The president
NP -gt Det Nominal Nominal -gt Noun Noun
ndash A morning flight to Denver
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
21
PPs
PP -gt Preposition NPndash Over the housendash Under the housendash To the treendash At playndash At a party on a boat at night
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
22
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
23
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
24
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
26
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
27
Recursion
Wersquoll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -gt NP PP [[The flight] [to Boston]]VP -gt VP PP [[departed Miami] [at noon]]
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
28
Recursion
Of course this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
29
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc
NP -gt NP PP
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
30
Implications of recursion and context-freeness
If you have a rule likendash VP -gt V NP
ndash It only cares that the thing after the verb is an NPIt doesnrsquot have to know about the internal affairs of that NP
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
31
The point
VP -gt V NP (I) hate
flights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
32
Grammar Equivalence
Can have different grammars that generate same set of strings (weak equivalence)
ndash Grammar 1 NP DetP N and DetP a | thendash Grammar 2 NP a N | NP the N
Can have different grammars that have same set of derivation trees (strong equivalence)
ndash With CFGs possible only with useless rulesndash Grammar 2 NP a N | NP the Nndash Grammar 3 NP a N | NP the N DetP many
Strong equivalence implies weak equivalence
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
33
Normal Forms ampc
There are weakly equivalent normal forms (Chomsky Normal Form Greibach Normal Form)
There are ways to eliminate useless productions and so on
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
34
Chomsky Normal Form
A CFG is in Chomsky Normal Form (CNF) if all productions are of one of two forms
A BC with A B C nonterminals A a with A a nonterminal and a a terminal
Every CFG has a weakly equivalent CFG in CNF
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
35
ldquoGenerative Grammarrdquo
Formal languages formal device to generate a set of strings (such as a CFG)
Linguistics (Chomskyan linguistics in particular) approach in which a linguistic theory enumerates all possible stringsstructures in a language (=competence)
Chomskyan theories do not really use formal devices ndash they use CFG + informally defined transformations
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
36
Nobody Uses Simple CFGs (Except Intro NLP Courses)
All major syntactic theories (Chomsky LFG HPSG TAG-based theories) represent both phrase structure and dependency in one way or another
All successful parsers currently use statistics about phrase structure and about dependency
Derive dependency through ldquohead percolationrdquo for each rule say which daughter is head
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
37
Penn Treebank (PTB) Syntactically annotated corpus of newspaper texts
(phrase structure) The newspaper texts are naturally occurring data but
the PTB is not PTB annotation represents a particular linguistic
theory (but a fairly ldquovanillardquo one) Particularities
ndash Very indirect representation of grammatical relations (need for head percolation tables)
ndash Completely flat structure in NP (brown bag lunch pink-and-yellow child seat )
ndash Has flat Ss flat VPs
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Example from PTB( (S (NP-SBJ It) (VP s (NP-PRD (NP (NP the latest investment craze)
(VP sweeping (NP Wall Street))) (NP (NP a rash) (PP of
(NP (NP new closed-end country funds) (NP (NP those
(ADJP publicly traded) portfolios) (SBAR (WHNP-37 that) (S (NP-SBJ T-37)
(VP invest (PP-CLR in
(NP (NP stocks) (PP of (NP a single foreign country)))))))))))
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
39
Types of syntactic constructions Is this the same construction
ndash An elf decided to clean the kitchenndash An elf seemed to clean the kitchen An elf cleaned the kitchen
Is this the same constructionndash An elf decided to be in the kitchenndash An elf seemed to be in the kitchenAn elf was in the kitchen
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
40
Types of syntactic constructions (ctd)
Is this the same constructionThere is an elf in the kitchenndash There decided to be an elf in the kitchenndash There seemed to be an elf in the kitchen
Is this the same constructionIt is rainingit rainsndash It decided to rainbe rainingndash It seemed to rainbe raining
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
41
Types of syntactic constructions (ctd)
Conclusion to seem whatever is embedded surface
subject can appear in upper clause to decide only full nouns that are referential
can appear in upper clause Two types of verbs
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Types of syntactic constructions Analysis
an elf
S
NP VP
V
to decide
S
NP VP
V
to be
PP
in thekitchen
S
VP
V
to seem
S
NP VP
V
to be
PP
in thekitchen
an elfan elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Types of syntactic constructions Analysis
an elf
S
NP VP
V
decided
S
NP
PRO
VP
V
to be
PP
in thekitchen
S
NPi VP
V
seemed
S
NP VP
V
to be
PP
in thekitchen
an elf
ti
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
46
Types of syntactic constructions Analysis
to seem lower surface subject raises to upper clause raising verb
seems (there to be an elf in the kitchen)there seems (t to be an elf in the kitchen)it seems (there is an elf in the kitchen)
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
47
Types of syntactic constructions Analysis (ctd)
to decide subject is in upper clause and co-refers with an empty subject in lower clause control verb
an elf decided (an elf to clean the kitchen)an elf decided (PRO to clean the kitchen)an elf decided (he cleansshould clean the kitchen)it decided (an elf cleansshould clean the kitchen)
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
48
Lessons Learned from the RaisingControl Issue
Use distribution of data to group phenomena into classes
Use different underlying structure as basis for explanations
Allow things to ldquomoverdquo around from underlying structure -gt transformational grammar
Check whether explanation you give makes predictions
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
The Big Picture
Empirical Matter
FormalismsbullData structuresbullFormalismsbullAlgorithmsbullDistributional Models
Maud expects there to be a riotTeri promised there to be a riotMaud expects the shit to hit the fanTeri promised the shit to hit the
or
Linguistic TheoryContent Relate morphology to semanticsbull Surface representation (eg ps)bull Deep representation (eg dep)bull Correspondence
uses
descriptivetheory is
about
explanatorytheory is about
predicts
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
50
Syntactic Parsing
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
51
Syntactic Parsing
Declarative formalisms like CFGs FSAs define the legal strings of a language -- but only tell you lsquothis is a legal string of the language Xrsquo
Parsing algorithms specify how to recognize the strings of a language and assign each string one (or more) syntactic analyses
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
52
Parsing as a Form of Search
Searching FSAsndash Finding the right path through the automatonndash Search space defined by structure of FSA
Searching CFGsndash Finding the right parse tree among all possible parse
treesndash Search space defined by the grammar
Constraints provided by the input sentence and the automaton or grammar
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
CFG for Fragment of EnglishS NP VP VP VS Aux NP VP PP -gt Prep NPNP Det Nom N old | dog | footsteps |
young
NP PropN V dog | include | preferNom -gt Adj Nom Aux doesNom N Nom Prep from | to | on | ofNom N PropN Bush | McCain |
ObamaNom Nom PP Det that | this | a| theVP V NP
TopD BotUp Eg LCrsquos
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Parse Tree for lsquoThe old dog the footsteps of the youngrsquo for Prior CFG
S
NP VP
NPV
DETNOM
N PP
DET NOM
N
The old dogthe
footstepsof the young
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
55
Top-Down Parser
Builds from the root S node to the leaves Expectation-based Common search strategy
ndash Top-down left-to-right backtrackingndash Try first rule with LHS = Sndash Next expand all constituents in these treesrulesndash Continue until leaves are POSndash Backtrack when candidate POS does not match input string
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
56
Rule Expansion
ldquoThe old dog the footsteps of the youngrdquo Where does backtracking happen What are the computational disadvantages What are the advantages
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
57
Bottom-Up Parsing
Parser begins with words of input and builds up trees applying grammar rules whose RHS matches
Det N V Det N Prep Det NThe old dog the footsteps of the young
Det Adj N Det N Prep Det NThe old dog the footsteps of the youngParse continues until an S root node reached or no further node expansion possible
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
58
Whatrsquos rightwrong withhellip
Top-Down parsers ndash they never explore illegal parses (eg which canrsquot form an S) -- but waste time on trees that can never match the input
Bottom-Up parsers ndash they never explore trees inconsistent with input -- but waste time exploring illegal parses (with no S root)
For both find a control strategy -- how explore search space efficiently
ndash Pursuing all parses in parallel or backtrack or hellipndash Which rule to apply nextndash Which node to expand next
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
59
Some Solutions
Dynamic Programming Approaches ndash Use a chart to represent partial results
CKY Parsing Algorithmndash Bottom-upndash Grammar must be in Normal Formndash The parse tree might not be consistent with linguistic theory
Early Parsing Algorithmndash Top-downndash Expectations about constituents are confirmed by inputndash A POS tag for a word that is not predicted is never added
Chart Parser
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
60
Earley Parsing
Allows arbitrary CFGs Fills a table in a single sweep over the input
wordsndash Table is length N+1 N is number of wordsndash Table entries represent
Completed constituents and their locations In-progress constituents Predicted constituents
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
61
States
The table-entries are called states and are represented with dotted-rulesS -gt VP A VP is predicted
NP -gt Det Nominal An NP is in progress
VP -gt V NP A VP has been found
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
62
StatesLocations
It would be nice to know where these things are in the input sohellipS -gt VP [00] A VP is predicted at the
start of the sentence
NP -gt Det Nominal [12] An NP is in progress the Det goes from 1 to 2
VP -gt V NP [03] A VP has been found starting at 0 and ending
at 3
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
63
Graphically
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
64
Earley
As with most dynamic programming approaches the answer is found by looking in the table in the right place
In this case there should be an S state in the final column that spans from 0 to n+1 and is complete
If thatrsquos the case yoursquore donendash S ndashgt α [0n+1]
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
65
Earley Algorithm
March through chart left-to-right At each step apply 1 of 3 operators
ndash Predictor Create new states representing top-down expectations
ndash Scanner Match word predictions (rule with word after dot) to words
ndash Completer When a state is complete see what rules were looking for
that completed constituent
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
66
Predictor
Given a statendash With a non-terminal to right of dotndash That is not a part-of-speech categoryndash Create a new state for each expansion of the non-terminalndash Place these new states into same chart entry as generated state
beginning and ending where generating state ends ndash So predictor looking at
S -gt VP [00] ndash results in
VP -gt Verb [00] VP -gt Verb NP [00]
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
67
Scanner
Given a statendash With a non-terminal to right of dotndash That is a part-of-speech categoryndash If the next word in the input matches this part-of-speechndash Create a new state with dot moved over the non-terminalndash So scanner looking at
VP -gt Verb NP [00]ndash If the next word ldquobookrdquo can be a verb add new state
VP -gt Verb NP [01]ndash Add this state to chart entry following current onendash Note Earley algorithm uses top-down input to disambiguate POS
Only POS predicted by some state can get added to chart
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
68
Completer
Applied to a state when its dot has reached right end of role Parser has discovered a category over some span of input Find and advance all previous states that were looking for this
categoryndash copy state move dot insert in current chart entry
Givenndash NP -gt Det Nominal [13]ndash VP -gt Verb NP [01]
Addndash VP -gt Verb NP [03]
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
69
Earley how do we know we are done
How do we know when we are done Find an S state in the final column that spans
from 0 to n+1 and is complete If thatrsquos the case yoursquore done
ndash S ndashgt α [0n+1]
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
70
Earley
More specificallyhellip1 Predict all the states you can upfront2 Read a word
1 Extend states based on matches2 Add new predictions3 Go to 2
3 Look at N+1 to see if you have a winner
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
71
Example
Book that flight We should findhellip an S from 0 to 3 that is a
completed statehellip
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
72
Example
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
73
Example
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
74
Example
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
75
Details
What kind of algorithms did we just describe ndash Not parsers ndash recognizers
The presence of an S state with the right attributes in the right place indicates a successful recognition
But no parse treehellip no parser Thatrsquos how we solve (not) an exponential problem in
polynomial time
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
76
Converting Earley from Recognizer to Parser
With the addition of a few pointers we have a parser
Augment the ldquoCompleterrdquo to point to where we came from
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Augmenting the chart with structural information
S8S9
S10
S11
S13S12
S8
S9
S8
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
78
Retrieving Parse Trees from Chart
All the possible parses for an input are in the table We just need to read off all the backpointers from every
complete S in the last column of the table Find all the S -gt X [0N+1] Follow the structural traces from the Completer Of course this wonrsquot be polynomial time since there could be
an exponential number of trees So we can at least represent ambiguity efficiently
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
79
Left Recursion vs Right Recursion
Depth-first search will never terminate if grammar is left recursive (eg NP --gt NP PP)
)(
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Solutionsndash Rewrite the grammar (automatically) to a weakly
equivalent one which is not left-recursiveeg The man on the hill with the telescopehellipNP NP PP (wanted Nom plus a sequence of PPs)NP Nom PPNP NomNom Det NhellipbecomeshellipNP Nom NPrsquoNom Det NNPrsquo PP NPrsquo (wanted a sequence of PPs)NPrsquo e Not so obvious what these rules meanhellip
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
81
ndash Harder to detect and eliminate non-immediate left recursion
ndash NP --gt Nom PPndash Nom --gt NP
ndash Fix depth of search explicitlyndash Rule ordering non-recursive rules first
NP --gt Det Nom NP --gt NP PP
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
82
Another Problem Structural ambiguity
Multiple legal structuresndash Attachment (eg I saw a man on a hill with a
telescope)ndash Coordination (eg younger cats and dogs)ndash NP bracketing (eg Spanish language teachers)
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
83
NP vs VP Attachment
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
84
Solution ndash Return all possible parses and disambiguate using
ldquoother methodsrdquo
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
85
Summing Up
Parsing is a search problem which may be implemented with many control strategiesndash Top-Down or Bottom-Up approaches each have
problems Combining the two solves some but not all issues
ndash Left recursionndash Syntactic ambiguity
Next time Making use of statistical information about syntactic constituentsndash Read Ch 14
- Slide 1
- Homework ndash Getting Started
- An example
- What would your arff file look like
- news_2865input
- All words for news_2865input
- Try it
- CFG Example
- Modify the grammar
- Slide 10
- Slide 11
- Derivations of CFGs
- Formal Definition of a CFG
- Context
- Key Constituents (English)
- Sentence-Types
- Slide 17
- Slide 18
- Slide 19
- NPs
- PPs
- Slide 22
- Slide 23
- Slide 24
- Slide 25
- Slide 26
- Recursion
- Slide 28
- Slide 29
- Implications of recursion and context-freeness
- The point
- Grammar Equivalence
- Normal Forms ampc
- Chomsky Normal Form
- ldquoGenerative Grammarrdquo
- Nobody Uses Simple CFGs (Except Intro NLP Courses)
- Penn Treebank (PTB)
- Example from PTB
- Types of syntactic constructions
- Types of syntactic constructions (ctd)
- Slide 41
- Types of syntactic constructions Analysis
- Slide 43
- Slide 44
- Slide 45
- Slide 46
- Types of syntactic constructions Analysis (ctd)
- Lessons Learned from the RaisingControl Issue
- The Big Picture
- Syntactic Parsing
- Slide 51
- Parsing as a Form of Search
- CFG for Fragment of English
- Top-Down Parser
- Rule Expansion
- Bottom-Up Parsing
- Whatrsquos rightwrong withhellip
- Some Solutions
- Earley Parsing
- States
- StatesLocations
- Graphically
- Earley
- Earley Algorithm
- Predictor
- Scanner
- Completer
- Earley how do we know we are done
- Slide 70
- Example
- Slide 72
- Slide 73
- Slide 74
- Details
- Converting Earley from Recognizer to Parser
- Augmenting the chart with structural information
- Retrieving Parse Trees from Chart
- Left Recursion vs Right Recursion
- Slide 80
- Slide 81
- Another Problem Structural ambiguity
- Slide 83
- Slide 84
- Summing Up
-
Related Documents
