CS3300 - Compiler Design Parsing V. Krishna Nandivada IIT Madras * Acknowledgement These slides borrow liberal portions of text verbatim from Antony L. Hosking @ Purdue, Jens Palsberg @ UCLA and the Dragon book. Copyright c 2014 by Antony L. Hosking. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from [email protected]. V.Krishna Nandivada (IIT Madras) CS3300 - Aug 2014 2 / 98 * The role of the parser code source tokens errors scanner parser IR A parser performs context-free syntax analysis guides context-sensitive analysis constructs an intermediate representation produces meaningful error messages attempts error correction For the next several classes, we will look at parser construction V.Krishna Nandivada (IIT Madras) CS3300 - Aug 2014 3 / 98 * Syntax analysis by using a CFG Context-free syntax is specified with a context-free grammar. Formally, a CFG G is a 4-tuple (V t , V n , S, P), where: V t is the set of terminal symbols in the grammar. For our purposes, V t is the set of tokens returned by the scanner. V n , the nonterminals, is a set of syntactic variables that denote sets of (sub)strings occurring in the language. These are used to impose a structure on the grammar. S is a distinguished nonterminal (S ∈ V n ) denoting the entire set of strings in L(G). This is sometimes called a goal symbol. P is a finite set of productions specifying how terminals and non-terminals can be combined to form strings in the language. Each production must have a single non-terminal on its left hand side. The set V = V t ∪ V n is called the vocabulary of G V.Krishna Nandivada (IIT Madras) CS3300 - Aug 2014 4 / 98
25
Embed
CS3300 - Compiler Design - Parsingkrishna/courses/2014/odd-cs3300/...CS3300 - Compiler Design Parsing V. Krishna Nandivada IIT Madras * Acknowledgement These slides borrow liberal
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CS3300 - Compiler DesignParsing
V. Krishna Nandivada
IIT Madras
*
Acknowledgement
These slides borrow liberal portions of text verbatim from Antony L.Hosking @ Purdue, Jens Palsberg @ UCLA and the Dragon book.
Context-free syntax is specified with a context-free grammar.Formally, a CFG G is a 4-tuple (Vt,Vn,S,P), where:
Vt is the set of terminal symbols in the grammar.For our purposes, Vt is the set of tokens returned by thescanner.
Vn, the nonterminals, is a set of syntactic variables thatdenote sets of (sub)strings occurring in the language.These are used to impose a structure on the grammar.
S is a distinguished nonterminal (S ∈ Vn) denoting the entireset of strings in L(G).This is sometimes called a goal symbol.
P is a finite set of productions specifying how terminals andnon-terminals can be combined to form strings in thelanguage.Each production must have a single non-terminal on itsleft hand side.
The set V = Vt∪Vn is called the vocabulary of GV.Krishna Nandivada (IIT Madras) CS3300 - Aug 2014 4 / 98
*
Notation and terminology
a,b,c, . . . ∈ Vt
A,B,C, . . . ∈ Vn
U,V,W, . . . ∈ V
α,β ,γ, . . . ∈ V∗u,v,w, . . . ∈ Vt∗
If A→ γ then αAβ ⇒ αγβ is a single-step derivation using A→ γ
Similarly,→∗ and⇒+ denote derivations of ≥ 0 and ≥ 1 steps
If S→∗ β then β is said to be a sentential form of G
L(G) = {w ∈ Vt∗ | S⇒+ w}, w ∈ L(G) is called a sentence of G
We have derived the sentence x + 2 ∗ y.We denote this 〈goal〉→∗ id + num ∗ id.Such a sequence of rewrites is a derivation or a parse.The process of discovering a derivation is called parsing.
These two derivations point out a problem with the grammar.It has no notion of precedence, or implied order of evaluation.To add precedence takes additional machinery:
Regular expressions are used to classify:identifiers, numbers, keywordsREs are more concise and simpler for tokens than a grammarmore efficient scanners can be built from REs (DFAs) thangrammars
Context-free grammars are used to count:brackets: (), begin. . .end, if. . .then. . .elseimparting structure: expressions
Syntactic analysis is complicated enough: grammar for C has around 200productions. Factoring out lexical analysis as a separate phase makescompiler more manageable.
Top-down parsersstart at the root of derivation tree and fill inpicks a production and tries to match the inputmay require backtrackingsome grammars are backtrack-free (predictive)
Bottom-up parsersstart at the leaves and fill instart in a state valid for legal first tokensas input is consumed, change state to encode possibilities(recognize valid prefixes)use a stack to store both state and sentential forms
A top-down parser starts with the root of the parse tree, labelled withthe start or goal symbol of the grammar.To build a parse, it repeats the following steps until the fringe of theparse tree matches the input string
1 At a node labelled A, select a production A→ α and construct theappropriate child for each symbol of α
2 When a terminal is added to the fringe that doesn’t match theinput string, backtrack
3 Find next node to be expanded (must have a label in Vn)
The key is selecting the right production in step 1.
If the parser makes a wrong step, the “derivation” process does notterminate.Why is it bad?
We saw that top-down parsers may need to backtrack when theyselect the wrong productionDo we need arbitrary lookahead to parse CFGs?
in general, yesuse the Earley or Cocke-Younger, Kasami algorithms
Fortunatelylarge subclasses of CFGs can be parsed with limited lookaheadmost programming language constructs can be expressed in agrammar that falls in these subclasses
Among the interesting subclasses are:LL(1): left to right scan, left-most derivation, 1-token lookahead;
andLR(1): left to right scan, reversed right-most derivation, 1-token
Basic idea:For any two productions A→ α | β , we would like a distinct way ofchoosing the correct production to expand.For some RHS α ∈ G, define FIRST(α) as the set of tokens thatappear first in some string derived from α.That is, for some w ∈ V∗t , w ∈ FIRST(α) iff. α ⇒∗ wγ.Key property:Whenever two productions A→ α and A→ β both appear in thegrammar, we would like
FIRST(α)∩ FIRST(β ) = φ
This would allow the parser to make a correct choice with alookahead of only one symbol!
Given a left-factored CFG, to eliminate left-recursion:1 Input: Grammar G with no cycles and no ε productions.2 Output: Equivalent grammat with no left-recursion. begin3 Arrange the non terminals in some order A1,A2, · · ·An;4 foreach i = 1 · · ·n do5 foreach j = 1 · · · i−1 do6 Say the ith production is: Ai→ Ajγ ;7 and Aj→ δ1|δ2| · · · |δk;8 Replace, the ith production by:9 Ai→ δ1γ|δ2γ| · · · δnγ;
Question:By left factoring and eliminating left-recursion, can wetransform an arbitrary context-free grammar to a form where itcan be predictively parsed with a single token lookahead?
Answer:Given a context-free grammar that doesn’t meet ourconditions, it is undecidable whether an equivalent grammarexists that does meet our conditions.
Many context-free languages do not have such a grammar:
{an0bn | n≥ 1}∪{an1b2n | n≥ 1}
Must look past an arbitrary number of a’s to discover the 0 or the 1 andso determine the derivation.
1 int A()2 begin3 foreach production of the form A→ X1X2X3 · · ·Xk do4 for i = 1 to k do5 if Xi is a non-terminal then6 if (Xi() 6= 0) then7 backtrack; break; // Try the next production
8 else if Xi matches the current input symbol a then9 advance the input to the next symbol;
10 else11 backtrack; break; // Try the next production
For a string of grammar symbols α, define FIRST(α) as:the set of terminals that begin strings derived from α:{a ∈ Vt | α ⇒∗ aβ}If α ⇒∗ ε then ε ∈ FIRST(α)
FIRST(α) contains the tokens valid in the initial position in α
To build FIRST(X):1 If X ∈ Vt then FIRST(X) is {X}2 If X→ ε then add ε to FIRST(X)3 If X→ Y1Y2 · · ·Yk:
1 Put FIRST(Y1)−{ε} in FIRST(X)2 ∀i : 1 < i≤ k, if ε ∈ FIRST(Y1)∩·· ·∩ FIRST(Yi−1)
(i.e., Y1 · · ·Yi−1⇒∗ ε)then put FIRST(Yi)−{ε} in FIRST(X)
3 If ε ∈ FIRST(Y1)∩·· ·∩ FIRST(Yk) then put ε in FIRST(X)Repeat until no more additions can be made.
Provable facts about LL(1) grammars:1 No left-recursive grammar is LL(1)2 No ambiguous grammar is LL(1)3 Some languages have no LL(1) grammar4 A ε–free grammar where each alternative expansion for A begins
with a distinct terminal is a simple LL(1) grammar.Example
S→ aS | a is not LL(1) because FIRST(aS) = FIRST(a) = {a}S→ aS′
Table driven Predictive parsingInput: A string w and a parsing table M for a grammar GOutput: If w is in L(G), a leftmost derivation of w; otherwise, indicate an error
1 push $ onto the stack; push S onto the stack;2 inp points to the input tape;3 X = stack.top();4 while X 6= $ do5 if X is inp then6 stack.pop(); inp++;
7 else if X is a terminal then8 error();
9 else if M[X,a] is an error entry then10 error();
11 else if M[X,a] = X→ Y1Y2 · · ·Yk then12 output the production X→ Y1Y2 · · ·Yk;13 stack.pop();14 push Yk,Yk−1, · · ·Y1 in that order;
Here is a typical example where a programming language fails tobe LL(1):stmt → asginment | call | otherassignment → id := expcall → id (exp-list)
This grammar is not in a form that can be left factored. We mustfirst replace assignment and call by the right-hand sides of theirdefining productions:statement → id := exp | id( exp-list ) | otheraWe left factor:statement → id stmt’ | otherstmt’ → := exp (exp-list)
See how the grammar obscures the language semantics.
An error is detected when the terminal on top of the stack doesnot match the next input symbol or M[A,a] = error.
Panic mode error recoverySkip input symbols till a “synchronizing” token appears.
Q: How to identify a synchronizing token?Some heuristics:
All symbols in FOLLOW(A) in the synchronizing set for thenon-terminal A.Semicolon after a Stmt production: assgignmentStmt;assignmentStmt;If a terminal on top of the stack cannot be matched? –
pop the terminal.issue a message that the terminal was inserted.
RecallFor a grammar G, with start symbol S, any string α such thatS⇒∗ α is called a sentential formIf α ∈ V∗t , then α is called a sentence in L(G)
Otherwise it is just a sentential form (not a sentence in L(G))A left-sentential form is a sentential form that occurs in the leftmostderivation of some sentence.A right-sentential form is a sentential form that occurs in the rightmostderivation of some sentence.
An unambiguous grammar will have a unique leftmost/rightmostderivation.
Reduction:At each reduction step, a specific substring matching the body ofa production is replaced by the non-terminal at the head of theproduction.
Key decisionsWhen to reduce?What production rule to apply?
Reduction Vs DerivationsRecall: In derivation: a non-terminal in a sentential form isreplaced by the body of one of its productions.A reduction is reverse of a step in derivation.
Bottom-up parsing is the process of “reducing” a string w to thestart symbol.Goal of bottum-up parsing: build derivation tree in reverse.
1 S →E2 E→E+T3 | E−T4 | T5 T→ T ∗F6 | T/F7 | F8 F→ num9 | id
Stack Input Action$ id − num ∗ id S$id − num ∗ id R9$〈factor〉 − num ∗ id R7$〈term〉 − num ∗ id R4$〈expr〉 − num ∗ id S$〈expr〉 − num ∗ id S$〈expr〉 − num ∗ id R8$〈expr〉 − 〈factor〉 ∗ id R7$〈expr〉 − 〈term〉 ∗ id S$〈expr〉 − 〈term〉 ∗ id S$〈expr〉 − 〈term〉 ∗ id R9$〈expr〉 − 〈term〉 ∗ 〈factor〉 R5$〈expr〉 − 〈term〉 R3$〈expr〉 R1$〈goal〉 A
Informally, we say that a grammar G is LR(k) if, given a rightmostderivation
S = γ0⇒ γ1⇒ γ2⇒ ·· · ⇒ γn = w,
we can, for each right-sentential form in the derivation:1 isolate the handle of each right-sentential form, and2 determine the production by which to reduce
by scanning γi from left to right, going at most k symbols beyond theright end of the handle of γi.
Formally, a grammar G is LR(k) iff.:1 S⇒∗rm αAw⇒rm αβw, and2 S⇒∗rm γBx⇒rm αβy, and3 FIRSTk(w) = FIRSTk(y)
⇒ αAy = γBxi.e., Assume sentential forms αβw and αβy, with common prefix αβ
and common k-symbol lookahead FIRSTk(y) = FIRSTk(w), such thatαβw reduces to αAw and αβy reduces to γBx.But, the common prefix means αβy also reduces to αAy, for the sameresult.Thus αAy = γBx.
LR(1) grammars are often used to construct parsers.We call these parsers LR(1) parsers.
virtually all context-free programming language constructs can beexpressed in an LR(1) formLR grammars are the most general grammars parsable by adeterministic, bottom-up parserefficient parsers can be implemented for LR(1) grammarsLR parsers detect an error as soon as possible in a left-to-rightscan of the inputLR grammars describe a proper superset of the languagesrecognized by predictive (i.e., LL) parsers
LL(k): recognize use of a production A→ β seeing first ksymbols derived from β
LR(k): recognize the handle β after seeing everythingderived from β plus k lookahead symbols
An LR(1) parser for either Algol or Pascal has several thousand states,while an SLR(1) or LALR(1) parser for the same language may haveseveral hundred states.
The table construction algorithms use sets of LR(k) items orconfigurations to represent the possible states in a parse.An LR(k) item is a pair [α,β ], where
α is a production from G with a • at some position in the RHS,marking how much of the RHS of a production has already beenseen
β is a lookahead string containing k symbols (terminals or $)Two cases of interest are k = 0 and k = 1:
LR(0) items play a key role in the SLR(1) table constructionalgorithm.
LR(1) items play a key role in the LR(1) and LALR(1) tableconstruction algorithms.
Given an item [A→ α •Bβ ], its closure contains the item and any otheritems that can generate legal substrings to follow α.Thus, if the parser has viable prefix α on its stack, the input shouldreduce to Bβ (or γ for some other item [B→•γ] in the closure).
Let I be a set of LR(0) items and X be a grammar symbol.Then, GOTO(I,X) is the closure of the set of all items
[A→ αX •β ] such that [A→ α •Xβ ] ∈ I
If I is the set of valid items for some viable prefix γ, then GOTO(I,X) isthe set of valid items for the viable prefix γX.GOTO(I,X) represents state after recognizing X in state I.
function GOTO(I,X)let J be the set of items [A→ αX •β ]
What’s the point of the lookahead symbols?carry along to choose correct reduction when there is a choicelookaheads are bookkeeping, unless item has • at right end:
in [A→ X •YZ,a], a has no direct usein [A→ XYZ•,a], a is useful
allows use of grammars that are not uniquely invertible†
The point: For [A→ α•,a] and [B→ α•,b], we can decide betweenreducing to A or B by looking at limited right context
Given an item [A→ α •Bβ ,a], its closure contains the item and anyother items that can generate legal substrings to follow α.Thus, if the parser has viable prefix α on its stack, the input shouldreduce to Bβ (or γ for some other item [B→•γ,b] in the closure).
function closure1(I)repeat
if [A→ α •Bβ ,a] ∈ Iadd [B→•γ,b] to I, where b ∈ FIRST(βa)
Let I be a set of LR(1) items and X be a grammar symbol.Then, GOTO(I,X) is the closure of the set of all items
[A→ αX •β ,a] such that [A→ α •Xβ ,a] ∈ I
If I is the set of valid items for some viable prefix γ, then GOTO(I,X) isthe set of valid items for the viable prefix γX.goto(I,X) represents state after recognizing X in state I.
function goto1(I,X)let J be the set of items [A→ αX •β ,a]
Build lookahead into the DFA to begin with1 construct the collection of sets of LR(1) items for G′2 state i of the LR(1) machine is constructed from Ii
1 [A→ α •aβ ,b] ∈ Ii and goto1(Ii,a) = Ij⇒ ACTION[i,a]← “shift j”
To construct LALR(1) parsing tables, we can insert a single step intothe LR(1) algorithm
(1.5) For each core present among the set of LR(1) items, findall sets having that core and replace these sets by theirunion.The goto function must be updated to reflect thereplacement sets.
The resulting algorithm has large space requirements, as we still arerequired to build the full set of LR(1) items.
The revised (and renumbered) algorithm1 construct the collection of sets of LR(1) items for G′2 for each core present among the set of LR(1) items, find all sets
having that core and replace these sets by their union (update thegoto1 function incrementally)
3 state i of the LALR(1) machine is constructed from Ii.1 [A→ α •aβ ,b] ∈ Ii and goto1(Ii,a) = Ij⇒ ACTION[i,a]← “shift j”
3 [S′→ S•,$] ∈ Ii ⇒ ACTION[i,$]← “accept”4 goto1(Ii,A) = Ij ⇒ GOTO[i,A]← j5 set undefined entries in ACTION and GOTO to “error”6 initial state of parser s0 is closure1([S′→•S,$])
Observe that we can:represent Ii by its basis or kernel:items that are either [S′→•S,$]or do not have • at the left of the RHScompute shift, reduce and goto actions for state derived from Ii
directly from its kernel
This leads to a method that avoids building the complete canonicalcollection of sets of LR(1) items
The error mechanismdesignated token errorvalid in any productionerror shows synchronization points
When an error is discoveredpops the stack until error is legalskips input tokens until it successfully shifts 3 (some default value)error productions can have actions
This mechanism is fairly general
Read the section on Error Recovery of the on-line CUP manual
Recursive descentA hand coded recursive descent parser directly encodes agrammar (typically an LL(1) grammar) into a series of mutuallyrecursive procedures. It has most of the linguistic limitations ofLL(1).LL(k)An LL(k) parser must be able to recognize the use of a productionafter seeing only the first k symbols of its right hand side.LR(k)An LR(k) parser must be able to recognize the occurrence of theright hand side of a production after having seen all that is derivedfrom that right hand side with k symbols of lookahead.