Combinator parsing

Combinator ParsingBy Swanand Pagnis

Higher-Order Functions for ParsingBy Graham Hutton

• Abstract & Introduction

• Build a parser, one fn at a time

• Moving beyond toy parsers

Abstract

In combinator parsing, the text of parsers resembles BNF notation. We present the basic method, and a number of extensions. We address the special problems presented by whitespace, and parsers with separate lexical and syntactic phases. In particular, a combining form for handling the “offside rule” is given. Other extensions to the basic method include an “into” combining form with many useful applications, and a simple means by which combinator parsers can produce more informative error messages.

• Combinators that resemble BNF notation

• Whitespace handling through "Offside Rule"

• "Into" combining form for advanced parsing

• Strategy for better error messages

Introduction

Primitive Parsers

• Take input

• Process one character

• Return results and unused input

Combinators

• Combine primitives

• Define building blocks

• Return results and unused input

Lexical analysis and syntax

• Combine the combinators

• Define lexical elements

• Return results and unused input

input: "from:swiggy to:me" output: [("f", "rom:swiggy to:me")]

input: "42 !=> ans" output: [("4", "2 !=> ans")]

rule: 'a' followed by 'b' input: "abcdef" output: [(('a','b'),"cdef")]

rule: 'a' followed by 'b' input: "abcdef" output: [(('a','b'),"cdef")]

Combinator

Language choice

Suggested: Lazy Functional Languages

Miranda: Author's choice

Haskell: An obvious choice. 🤓

Racket: Another obvious choice. 🤓

Ruby: 🍎 to 🍊 so $ for learning

OCaml: Functional, but not lazy.

Haskell %

Simple when stick to fundamental FP

• Higher order functions

• Immutability

• Recursive problem solving

• Algebraic types

Let's build a parser, one fn at a time

type Parser a b = [a] !-> [(b, [a])]

Types help with abstraction

• We'll be dealing with parsers and combinators

• Parsers are functions, they accept input and return results

• Combinators accept parsers and return parsers

A parser is a function that accepts an input and returns parsed results and the unused input for each result

Parser is a function type that accepts a list of type a and returns all possible results as a list of tuples of type (b, [a])

(Parser Char Number) input: "42 it is!" !-- a is a [Char] output: [(42, " it is!")] !-- b is a Number

type Parser a b = [a] !-> [(b, [a])]

Primitive Parsers

succeed !:: b !-> Parser a b succeed v inp = [(v, inp)]

Always succeeds Returns "v" for all inputs

failure !:: Parser a b failure inp = []

Always fails Returns "[]" for all inputs

satisfy !:: (a !-> Bool) !-> Parser a a satisfy p [] = failure [] satisfy p (x:xs) | p x = succeed x xs !-- if p(x) is true | otherwise = failure []

satisfy !:: (a !-> Bool) !-> Parser a a satisfy p [] = failure [] satisfy p (x:xs) | p x = succeed x xs !-- if p(x) is true | otherwise = failure []

Guard Clauses, if you want to Google

literal !:: Eq a !=> a !-> Parser a a literal x = satisfy (!== x)

match_3 = (literal '3') match_3 "345" !-- !=> [('3',"45")] match_3 "456" !-- !=> []

succeed failure satisfy literal

Combinators

match_3_or_4 = match_3 `alt` match_4 match_3_or_4 "345" !-- !=> [('3',"45")] match_3_or_4 "456" !-- !=> [('4',"56")]

alt !:: Parser a b !-> Parser a b !-> Parser a b (p1 `alt` p2) inp = p1 inp !++ p2 inp

(p1 `alt` p2) inp = p1 inp !++ p2 inpList concatenation

(match_3 `and_then` match_4) "345" # !=> [(('3','4'),"5")]

🐉

and_then !:: Parser a b !-> Parser a c !-> Parser a (b, c) (p1 `and_then` p2) inp = [ ((v1, v2), out2) | (v1, out1) !<- p1 inp, (v2, out2) !<- p2 out1 ]

and_then !:: Parser a b !-> Parser a c !-> Parser a (b, c) (p1 `and_then` p2) inp = [ ((v1, v2), out2) | (v1, out1) !<- p1 inp, (v2, out2) !<- p2 out1 ]

List comprehensions

(v11, out11) (v12, out12) (v13, out13)

…

(v21, out21) (v22, out22)

…

(v31, out31) (v32, out32)

…

(v31, out31)

p1

p2

((v11, v21), out21) ((v11, v22), out22)

…

(match_3 `and_then` match_4) "345" # !=> [(('3','4'),"5")]

Manipulating values

match_3 = (literal '3') match_3 "345" !-- !=> [('3',"45")] match_3 "456" !-- !=> []

(number "42") "42 !=> answer" # !=> [(42, " answer")]

(keyword "for") "for i in 1!..42" # !=> [(:for, " i in 1!..42")]

using !:: Parser a b !-> (b !-> c) !-> Parser a c (p `using` f) inp = [(f v, out) | (v, out) !<- p inp ]

((string "3") `using` float) "3" # !=> [(3.0, "")]

Levelling up

many !:: Parser a b !-> Parser a [b] many p = ((p `and_then` many p) `using` cons) `alt` (succeed [])

0 or many

(many (literal 'a')) "aab" !=> [("aa","b"),("a","ab"),("","aab")]

(many (literal 'a')) "xyz" !=> [("","xyz")]

some !:: Parser a b !-> Parser a [b] some p = ((p `and_then` many p) `using` cons)

1 or many

(some (literal 'a')) "aab" !=> [("aa","b"),("a","ab")]

(some (literal 'a')) "xyz" !=> []

positive_integer = some (satisfy Data.Char.isDigit)

negative_integer = ((literal '-') `and_then` positive_integer) `using` cons

positive_decimal = (positive_integer `and_then` (((literal '.') `and_then` positive_integer) `using` cons)) `using` join

negative_decimal = ((literal '-') `and_then` positive_decimal) `using` cons

number !:: Parser Char [Char] number = negative_decimal `alt` positive_decimal `alt` negative_integer `alt` positive_integer

word !:: Parser Char [Char] word = some (satisfy isLetter)

string !:: (Eq a) !=> [a] !-> Parser a [a] string [] = succeed [] string (x:xs) = (literal x `and_then` string xs) `using` cons

(string "begin") "begin end" # !=> [("begin"," end")]

xthen !:: Parser a b !-> Parser a c !-> Parser a c p1 `xthen` p2 = (p1 `and_then` p2) `using` snd

thenx !:: Parser a b !-> Parser a c !-> Parser a b p1 `thenx` p2 = (p1 `and_then` p2) `using` fst

ret !:: Parser a b !-> c !-> Parser a c p `ret` v = p `using` (const v)

succeed, failure, satisfy, literal, alt, and_then, using, string, many, some, string, word, number, xthen, thenx, ret

Expression Parser & Evaluator

data Expr = Const Double | Expr `Add` Expr | Expr `Sub` Expr | Expr `Mul` Expr | Expr `Div` Expr

(Const 3) `Mul` ((Const 6) `Add` (Const 1))) # !=> "3*(6+1)"

parse "3*(6+1)" # !=> (Const 3) `Mul` ((Const 6) `Add` (Const 1)))

(Const 3) Mul ((Const 6) `Add` (Const 1))) # !=> 21

BNF Notation

expn !::= expn + expn | expn − expn | expn ∗ expn | expn / expn | digit+ | (expn)

Improving a little:

expn !::= term + term | term − term | term term !::= factor ∗ factor | factor / factor | factor factor !::= digit+ | (expn)

Parsers that resemble BNF

addition = ((term `and_then` ((literal '+') `xthen` term)) `using` plus)

subtraction = ((term `and_then` ((literal '-') `xthen` term)) `using` minus)

multiplication = ((factor `and_then` ((literal '*') `xthen` factor)) `using` times)

division = ((factor `and_then` ((literal '/') `xthen` factor)) `using` divide)

parenthesised_expression = ((nibble (literal '(')) `xthen` ((nibble expn) `thenx`(nibble (literal ')'))))

value xs = Const (numval xs) plus (x,y) = x `Add` y minus (x,y) = x `Sub` y times (x,y) = x `Mul` y divide (x,y) = x `Div` y

expn = addition `alt` subtraction `alt` term

term = multiplication `alt` division `alt` factor

factor = (number `using` value) `alt` parenthesised_expn

expn "12*(5+(7-2))" # !=> [ (Const 12.0 `Mul` (Const 5.0 `Add` (Const 7.0 `Sub` Const 2.0)),""), … ]

value xs = Const (numval xs) plus (x,y) = x `Add` y minus (x,y) = x `Sub` y times (x,y) = x `Mul` y divide (x,y) = x `Div` y

value = numval plus (x,y) = x + y minus (x,y) = x - y times (x,y) = x * y divide (x,y) = x / y

expn "12*(5+(7-2))" # !=> [(120.0,""), (12.0,"*(5+(7-2))"), (1.0,"2*(5+(7-2))")]

expn "(12+1)*(5+(7-2))" # !=> [(130.0,""), (13.0,"*(5+(7-2))")]

Moving beyond toy parsers

Whitespace? 🤔 (

white = (literal " ") `alt` (literal "\t") `alt` (literal "\n")

white = many (any literal " \t\n")

/\s!*/

any p = foldr (alt.p) fail

any p [x1,x2,!!...,xn] = (p x1) `alt` (p x2) `alt` !!... `alt` (p xn)

white = many (any literal " \t\n")

nibble p = white `xthen` (p `thenx` white)

The parser (nibble p) has the same behaviour as parser p, except that it eats up any white-space in the input string before or afterwards

(nibble (literal 'a')) " a " # !=> [('a',""),('a'," "),('a'," ")]

symbol = nibble.string

symbol "$fold" " $fold " # !=> [("$fold", ""), ("$fold", " ")]

The Offside Rule

w = x + y where x = 10 y = 15 - 5 z = w * 2

w = x + y where x = 10 y = 15 - 5 z = w * 2

When obeying the offside rule, every token must lie either directly below, or to the right of its first token

i.e. A weak indentation policy

The Offside Combinator

type Pos a = (a, (Integer, Integer))

prelex "3 + \n 2 * (4 + 5)" # !=> [('3',(0,0)), ('+',(0,2)), ('2',(1,2)), ('*',(1,4)), … ]

satisfy !:: (a !-> Bool) !-> Parser a a satisfy p [] = failure [] satisfy p (x:xs) | p x = succeed x xs !-- if p(x) is true | otherwise = failure []

satisfy !:: (a !-> Bool) !-> Parser (Pos a) a satisfy p [] = failure [] satisfy p (x:xs) | p a = succeed a xs !-- if p(a) is true | otherwise = failure [] where (a, (r, c)) = x

satisfy !:: (a !-> Bool) !-> Parser (Pos a) a satisfy p [] = failure [] satisfy p (x:xs) | p a = succeed a xs !-- if p(a) is true | otherwise = failure [] where (a, (r, c)) = x

offside !:: Parser (Pos a) b !-> Parser (Pos a) b offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)] where inpON = takeWhile (onside (head inp)) inp inpOFF = drop (length inpON) inp onside (a, (r, c)) (b, (r', c')) = r' !>= r !&& c' !>= c

offside !:: Parser (Pos a) b !-> Parser (Pos a) b

offside !:: Parser (Pos a) b !-> Parser (Pos a) b offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]

(nibble (literal 'a')) " a " # !=> [('a',""),('a'," "),('a'," ")]

offside !:: Parser (Pos a) b !-> Parser (Pos a) b offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]

offside !:: Parser (Pos a) b !-> Parser (Pos a) b offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)] where inpON = takeWhile (onside (head inp)) inp

offside !:: Parser (Pos a) b !-> Parser (Pos a) b offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)] where inpON = takeWhile (onside (head inp)) inp inpOFF = drop (length inpON) inp

offside !:: Parser (Pos a) b !-> Parser (Pos a) b offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)] where inpON = takeWhile (onside (head inp)) inp inpOFF = drop (length inpON) inp onside (a, (r, c)) (b, (r', c')) = r' !>= r !&& c' !>= c

(3 + 2 * (4 + 5)) + (8 * 10)

(3 + 2 * (4 + 5)) + (8 * 10)

(offside expn) (prelex inp_1) # !=> [(21.0,[('+',(2,0)),('(',(2,2)),('8',(2,3)),('*',(2,5)),('1',(2,7)),('0',(2,8)),(')',(2,9))])]

(offside expn) (prelex inp_2) # !=> [(101.0,[])]

Quick recap before we 🛫

∅ !|> succeed, fail !|> satisfy, literal !|> alt, and_then, using !|> many, some !|> string, thenx, xthen, return !|> expression parser & evaluator !|> any, nibble, symbol !|> prelex, offside

Practical parsers

🎯Syntactical analysis 🎯Lexical analysis 🎯Parse trees

type Parser a b = [a] !-> [(b, [a])] type Pos a = (a, (Integer, Integer))

data Tag = Ident | Number | Symbol | Junk deriving (Show, Eq) type Token = (Tag, [Char])

(Symbol, "if") (Number, "123")

Parse the string with parser p, & apply token t to the result

(p `tok` t) inp = [ (((t, xs), (r, c)), out) | (xs, out) !<- p inp] where (x, (r,c)) = head inp

(p `tok` t) inp = [ ((<token>,<pos>),<unused input>) | (xs, out) !<- p inp] where (x, (r,c)) = head inp

(p `tok` t) inp = [ (((t, xs), (r, c)), out) | (xs, out) !<- p inp] where (x, (r,c)) = head inp

((string "where") `tok` Symbol) inp # !=> ((Symbol,"where"), (r, c))

many ((p1 `tok` t1) `alt` (p2 `tok` t2) `alt` !!... `alt` (pn `tok` tn))

[(p1, t1), (p2, t2), …, (pn, tn)]

lex = many.(foldr op failure) where (p, t) `op` xs = (p `tok` t) `alt` xs

🐉

lex = many.(foldr op failure) where (p, t) `op` xs = (p `tok` t) `alt` xs

# Rightmost computation cn = (pn `tok` tn) `alt` failure

# Followed by (pn-1 `tok` tn-1) `alt` cn

many ((p1 `tok` t1) `alt` (p2 `tok` t2) `alt` !!... `alt` (pn `tok` tn))

lexer = lex [ ((some (any_of literal " \n\t")), Junk), ((string "where"), Symbol), (word, Ident), (number, Number), ((any_of string ["(", ")", "="]), Symbol)]

lexer = lex [ ((some (any_of literal " \n\t")), Junk), ((string "where"), Symbol), (word, Ident), (number, Number), ((any_of string ["(", ")", "="]), Symbol)]

lexer = lex [ ((some (any_of literal " \n\t")), Junk), ((string "where"), Symbol), (word, Ident), (number, Number), ((any_of string ["(", ")", "="]), Symbol)]

head (lexer (prelex "where x = 10")) # !=> ([((Symbol,"where"),(0,0)), ((Ident,"x"),(0,6)), ((Symbol,"="),(0,8)), ((Number,"10"),(0,10)) ],[])

(head.lexer.prelex) "where x = 10" # !=> ([((Symbol,"where"),(0,0)), ((Ident,"x"),(0,6)), ((Symbol,"="),(0,8)), ((Number,"10"),(0,10)) ],[])

(head.lexer.prelex) "where x = 10" # !=> ([((Symbol,"where"),(0,0)), ((Ident,"x"),(0,6)), ((Symbol,"="),(0,8)), ((Number,"10"),(0,10)) ],[])

Function composition

length ((lexer.prelex) "where x = 10") # !=> 198

Conflicts? Ambiguity?

In this case, "where" is a source of conflict. It can be a symbol, or identifier.

lexer = lex [ {- 1 -} ((some (any_of literal " \n\t")), Junk), {- 2 -} ((string "where"), Symbol), {- 3 -} (word, Ident), {- 4 -} (number, Number), {- 5 -} ((any_of string ["(",")","="]), Symbol)]

Higher priority, higher precedence

Removing Junk

strip !:: [(Pos Token)] !-> [(Pos Token)] strip = filter ((!!= Junk).fst.fst)

((!!= Junk).fst.fst) ((Symbol,"where"),(0,0)) # !=> True ((!!= Junk).fst.fst) ((Junk,"where"),(0,0)) # !=> False

(fst.head.lexer.prelex) "where x = 10" # !=> [((Symbol,"where"),(0,0)), ((Junk," "),(0,5)), ((Ident,"x"),(0,6)), ((Junk," "),(0,7)), ((Symbol,"="),(0,8)), ((Junk," "),(0,9)), ((Number,"10"),(0,10))]

(strip.fst.head.lexer.prelex) "where x = 10" # !=> [((Symbol,"where"),(0,0)), ((Ident,"x"),(0,6)), ((Symbol,"="),(0,8)), ((Number,"10"),(0,10))]

Syntax Analysis

characters !|> lexical analysis !|> tokens

tokens !|> syntax analysis !|> parse trees

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

Script

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

Definition

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

Body

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

Expression

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

Definition

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

Primitives

data Script = Script [Def] data Def = Def Var [Var] Expn data Expn = Var Var | Num Double | Expn `Apply` Expn | Expn `Where` [Def] type Var = [Char]

prog = (many defn) `using` Script

defn = ( (some (kind Ident)) `and_then` ((lit "=") `xthen` (offside body))) `using` defnFN

body = ( expr `and_then` (((lit "where") `xthen` (some defn)) `opt` [])) `using` bodyFN

expr = (some prim) `using` (foldl1 Apply)

prim = ((kind Ident) `using` Var) `alt` ((kind Number) `using` numFN) `alt` ((lit "(") `xthen` (expr `thenx` (lit ")")))

!-- only allow a kind of tag kind !:: Tag !-> Parser (Pos Token) [Char] kind t = (satisfy ((!== t).fst)) `using` snd

— only allow a given symbol lit !:: [Char] !-> Parser (Pos Token) [Char] lit xs = (literal (Symbol, xs)) `using` snd

prog = (many defn) `using` Script

defn = ( (some (kind Ident)) `and_then` ((lit "=") `xthen` (offside body))) `using` defnFN

body = ( expr `and_then` (((lit "where") `xthen` (some defn)) `opt` [])) `using` bodyFN

expr = (some prim) `using` (foldl1 Apply)

prim = ((kind Ident) `using` Var) `alt` ((kind Number) `using` numFN) `alt` ((lit "(") `xthen` (expr `thenx` (lit ")")))

data Script = Script [Def] data Def = Def Var [Var] Expn data Expn = Var Var | Num Double | Expn `Apply` Expn | Expn `Where` [Def] type Var = [Char]

Orange functions are for transforming values.

Use data constructors to generate parse trees

Use evaluation functions to evaluate and generate a value

f x y = add a b where a = 25 b = sub x y

answer = mult (f 3 7) 5

Script [ Def "f" ["x","y"] ( ((Var "add" `Apply` Var "a") `Apply` Var "b") `Where` [ Def "a" [] (Num 25.0), Def "b" [] ((Var "sub" `Apply` Var "x") `Apply` Var "y")]), Def "answer" [] ( (Var "mult" `Apply` ( (Var "f" `Apply` Num 3.0) `Apply` Num 7.0)) `Apply` Num 5.0)]

Strategy for writing parsers

1. Identify components i.e. Lexical elements

lexer = lex [ ((some (any_of literal " \n\t")), Junk), ((string "where"), Symbol), (word, Ident), (number, Number), ((any_of string ["(", ")", "="]), Symbol)]

2. Structure these elements a.k.a. syntax

defn = ((some (kind Ident)) `and_then` ((lit "=") `xthen` (offside body))) `using` defnFN

body = (expr `and_then` (((lit "where") `xthen` (some defn)) `opt` [])) `using` bodyFN

expr = (some prim) `using` (foldl1 Apply)

prim = ((kind Ident) `using` Var) `alt` ((kind Number) `using` numFN) `alt` ((lit "(") `xthen` (expr `thenx` (lit ")")))

3. BNF notation is very helpful

4. TDD in the absence of types

Where to, next?

Monadic ParsersGraham Hutton, Eric Meijer

Introduction to FPPhilip Wadler

The Dragon BookIf your interest is in compilers

Libraries?

Haskell: Parsec, MegaParsec. ✨ OCaml: Angstrom. ✨ 🚀 Ruby: rparsec, or roll you own Elixir: Combine, ExParsec Python: Parsec. ✨

Thank you!

Twitter: @_swanand GitHub: @swanandp