Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)

@asgrim

Climbing theAbstract Syntax Tree

James TitcumbBulgaria PHP Conference 2016

Who is this guy?James Titcumb

www.jamestitcumb.com

www.roave.com

www.phphants.co.uk

www.phpsouthcoast.co.uk

@asgrim

@asgrim

How PHP works

PHP code

OpCacheExecute (VM)

Lexer + Parser

Compiler

@asgrim

The PHP Lexer

zend_language_scanner.l

@asgrim

zend_language_scanner.l<ST_IN_SCRIPTING>"exit" {

RETURN_TOKEN(T_EXIT);

}

<ST_IN_SCRIPTING>"die" {


}

<ST_IN_SCRIPTING>"function" {

RETURN_TOKEN(T_FUNCTION);

}

@asgrim



}



}



}

@asgrim



}



}



}

@asgrim



}



}



}

@asgrim

zend_language_scanner.l<ST_DOUBLE_QUOTES,ST_BACKQUOTE,ST_HEREDOC>"${" {

yy_push_state(ST_LOOKING_FOR_VARNAME);

RETURN_TOKEN(T_DOLLAR_OPEN_CURLY_BRACES);

}

<ST_LOOKING_FOR_VARNAME>{LABEL}[[}] {

yyless(yyleng - 1);

zend_copy_value(zendlval, yytext, yyleng);

yy_pop_state();

yy_push_state(ST_IN_SCRIPTING);

RETURN_TOKEN(T_STRING_VARNAME);

}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim




}


yyless(yyleng - 1);


yy_pop_state();



}

@asgrim

The PHP Lexer


@asgrim

The PHP Lexer


re2c

@asgrim

The PHP Lexer


re2c

zend_language_scanner.c

@asgrim

The PHP Parser

zend_language_parser.y

@asgrim

zend_language_parser.yif_stmt:

if_stmt_without_else %prec T_NOELSE { $$ = $1; }

| if_stmt_without_else T_ELSE statement

{ $$ = zend_ast_list_add($1, zend_ast_create(ZEND_AST_IF_ELEM, NULL, $3)); }

;

if_stmt_without_else:

T_IF '(' expr ')' statement

{ $$ = zend_ast_create_list(1, ZEND_AST_IF,

zend_ast_create(ZEND_AST_IF_ELEM, $3, $5)); }

| if_stmt_without_else T_ELSEIF '(' expr ')' statement

{ $$ = zend_ast_list_add($1,


;

@asgrim

if_stmt:




;








;


@asgrim

if_stmt:




;








;


@asgrim

if_stmt:




;








;


@asgrim

if_stmt:




;








;


@asgrim

if_stmt:




;








;


@asgrim

if_stmt:




;








;


@asgrim

if_stmt:




;








;


@asgrim

if ($a == 1)

{

a();

}

else if ($b == 1)

{

b();

}

else

{

c();

}

Using the rules to parse

@asgrim

if ($a == 1)

{

a();

}

else if ($b == 1)

{

b();

}

else

{

c();

}


if_stmt_without_else (A)

@asgrim

if ($a == 1)

{

a();

}

else if ($b == 1)

{

b();

}

else

{

c();

}



if_stmt_without_else (B)

@asgrim

if ($a == 1)

{

a();

}

else if ($b == 1)

{

b();

}

else

{

c();

}



if_stmt_without_else (B)

if_stmt

@asgrim

Zend_language_parser.y (PHP 7.0.10)if_stmt:




;








;

@asgrim

zend_language_parser.y (PHP 5.6.26)T_IF parenthesis_expr { zend_do_if_cond(&$2, &$1 TSRMLS_CC); }

statement { zend_do_if_after_statement(&$1, 1 TSRMLS_CC); }

void zend_do_if_cond(const znode *cond, znode *closing_bracket_token TSRMLS_DC)

{

int if_cond_op_number = get_next_op_number(CG(active_op_array));

zend_op *opline = get_next_op(CG(active_op_array) TSRMLS_CC);

opline->opcode = ZEND_JMPZ;

SET_NODE(opline->op1, cond);

closing_bracket_token->u.op.opline_num = if_cond_op_number;

SET_UNUSED(opline->op2);

INC_BPC(CG(active_op_array));

}

@asgrim

AST is new in PHP 7+

@asgrim

How PHP works

PHP code

OpCacheExecute (VM)

Lexer + Parser

Compiler

@asgrim

Let’s simplify!

@asgrim

First… WTF is AST?

@asgrim

AST is just a data structure

@asgrim

PHP code

<?php

echo "Hello world";

@asgrim

An AST representation

Echo statement

`-- String, value "Hello world"

@asgrim

PHP code

<?php

echo "Hello " . "world";

@asgrim

An AST representation

Echo statement

`-- Concat

|-- Left

| `-- String, value "Hello "

`-- Right

`-- String, value "world"

@asgrim

PHP code

<?php

$a = 5;

$b = 3;

echo $a + ($b * 2);

@asgrim

An AST representationAssign statement

|-- Variable $a

`-- Integer, value 5

Assign statement

|-- Variable $b


Echo statement

`-- Add operation

|-- Left

| `-- Variable $a

`-- Right

`-- Multiply operation

|-- Left

| `-- Variable $b

`-- Right


@asgrim

AST compilationStatements

EchoAssign

Scalarvalue: (int)5

Variablename: $a

Assign

Scalarvalue: (int)3

Variablename: $b Add op

Right operandLeft operand

Variablename: $a

Multiply op


Variablename: $b

Scalarvalue: (int)2

@asgrim

AST compilation: pre-order traversalStatements

EchoAssign

Scalarvalue: (int)5

Variablename: $a

Assign

Scalarvalue: (int)3

Variablename: $b Add op


Variablename: $a

Multiply op


Variablename: $b

Scalarvalue: (int)2

@asgrim

Pre-order traversal: Polish notationAssign(Variable $a, Scalar 5)

Assign(Variable $b, Scalar 3)

Echo (

Add(

Variable $a,

Multiply( $b, 2 )

)

)

@asgrim

Order of precedence

1 + 2 * 3

= 1 + (2 * 3) = 7?

= (1 + 2) * 3 = 9?

@asgrim

Order of precedence

1 + 2 * 3

= 1 + (2 * 3) = 7?

= (1 + 2) * 3 = 9?

+ 1 * 2 3

@asgrim

Order of precedence

1 + 2 * 3

= 1 + (2 * 3) = 7?

= (1 + 2) * 3 = 9?

+ 1 * 2 3

Operator Left operand Right operand

@asgrim

Order of precedence

1 + 2 * 3

= 1 + (2 * 3) = 7?

= (1 + 2) * 3 = 9?

+ 1 * 2 3



@asgrim

Reverse Polish Notation

1 2 3 * +

@asgrim


1 2 3 * + The stack

@asgrim


1 2 3 * + The stack

1

@asgrim


1 2 3 * + The stack

1

2

@asgrim


1 2 3 * + The stack

1

2

3

@asgrim


1 2 3 * + The stack

1

2

3

@asgrim


1 2 3 * + The stack

1

2

3

@asgrim


1 2 3 * + The stack

1

6

@asgrim


1 2 3 * + The stack

1

6

@asgrim


1 2 3 * + The stack

7

@asgrim

Let’s write a compiler (!!!)In three easy steps…

@asgrim

Warning: do not use in production

@asgrim

View > Sourcehttps://github.com/asgrim/basic-maths-compiler

@asgrim

Define the languageTokens

● T_ADD (+)

● T_SUBTRACT (-)

● T_MULTIPLY (/)

● T_DIVIDE (*)

● T_INTEGER (\d)

● T_WHITESPACE (\s+)

@asgrim

Step 1: Writing a simple lexer

@asgrim

Using regular expressionsprivate static $matches = [

'/^(\+)/' => Token::T_ADD,

'/^(-)/' => Token::T_SUBTRACT,

'/^(\*)/' => Token::T_MULTIPLY,

'/^(\/)/' => Token::T_DIVIDE,

'/^(\d+)/' => Token::T_INTEGER,

'/^(\s+)/' => Token::T_WHITESPACE,

];

@asgrim

Step through the input stringpublic function __invoke(string $input) : array

{

$tokens = [];

$offset = 0;

while ($offset < strlen($input)) {

$focus = substr($input, $offset);

$result = $this->match($focus);

$tokens[] = $result;

$offset += strlen($result->getLexeme());

}

return $tokens;

}

@asgrim

The matching methodprivate function match(string $input) : Token

{

foreach (self::$matches as $pattern => $token) {

if (preg_match($pattern, $input, $matches)) {

return new Token($token, $matches[1]);

}

}

throw new \RuntimeException(sprintf(

'Unmatched token, next 15 chars were: %s', substr($input, 0, 15)

));

}

@asgrim

Step 2: Parsing the tokens

@asgrim

Order tokens by operator precedence/**

* Higher number is higher precedence.

* @var int[]

*/

private static $operatorPrecedence = [

Token::T_SUBTRACT => 0,

Token::T_ADD => 1,

Token::T_DIVIDE => 2,

Token::T_MULTIPLY => 3,

];

@asgrim

Order tokens by operator precedence/** @var Token[] $stack */

$stack = [];

/** @var Token[] $operators */

$operators = [];

while (false !== ($token = current($tokens))) {

if ($token->isOperator()) {

// ...

}

$stack[] = $token;

next($tokens);

}

@asgrim


$stack = [];


$operators = [];



// ...

}

$stack[] = $token;

next($tokens);

}

@asgrim


$stack = [];


$operators = [];



// ...

}

$stack[] = $token;

next($tokens);

}

@asgrim


$stack = [];


$operators = [];



// ...

}

$stack[] = $token;

next($tokens);

}

@asgrim

Order tokens by operator precedenceif ($token->isOperator()) {

$tokenPrecedence = self::$operatorPrecedence[$token->getToken()];

while (

count($operators)

&& self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()]

> $tokenPrecedence

) {

$higherOp = array_pop($operators);

$stack[] = $higherOp;

}

$operators[] = $token;

next($tokens);

continue;

}

@asgrim



while (

count($operators)


> $tokenPrecedence

) {



}


next($tokens);

continue;

}

@asgrim



while (

count($operators)


> $tokenPrecedence

) {



}


next($tokens);

continue;

}

@asgrim



while (

count($operators)


> $tokenPrecedence

) {



}


next($tokens);

continue;

}

@asgrim

Order tokens by operator precedence// Clean up by moving any remaining operators onto the token stack

while (count($operators)) {

$stack[] = array_pop($operators);

}

return $stack;

@asgrim

Order tokens by operator precedence

1 + 2 * 3

Output stack

Operator stack

@asgrim


1 + 2 * 3

1Output stack

Operator stack

@asgrim


1 + 2 * 3

1

+

Output stack

Operator stack

@asgrim


1 + 2 * 3

1 2

+

Output stack

Operator stack

@asgrim


1 + 2 * 3

1 2

+ *

Output stack

Operator stack

@asgrim


1 + 2 * 3

1 2 3

+ *

Output stack

Operator stack

@asgrim


1 + 2 * 3

1 2 3 *

+ *

Output stack

Operator stack

@asgrim


1 + 2 * 3

1 2 3 * +

+

Output stack

Operator stack

@asgrim

Create ASTwhile ($ip < count($tokenStack)) {

$token = $tokenStack[$ip++];


// (figure out $nodeType)

$right = array_pop($astStack);

$left = array_pop($astStack);

$astStack[] = new $nodeType($left, $right);

continue;

}

$astStack[] = new Node\Scalar\IntegerValue((int)$token->getLexeme());

}

@asgrim








continue;

}


}

@asgrim








continue;

}


}

@asgrim








continue;

}


}

@asgrim

Create AST

Node\BinaryOp\Add (

Node\Scalar\IntegerValue(1),

Node\BinaryOp\Multiply (

Node\Scalar\IntegerValue(2),

Node\Scalar\IntegerValue(3)

)

)

@asgrim

Step 3: Executing the AST

@asgrim

Compile & execute ASTprivate function compileNode(NodeInterface $node)

{

if ($node instanceof Node\BinaryOp\AbstractBinaryOp) {

return $this->compileBinaryOp($node);

}

if ($node instanceof Node\Scalar\IntegerValue) {

return $node->getValue();

}

}

@asgrim

Compile & execute ASTprivate function compileBinaryOp(Node\BinaryOp\AbstractBinaryOp $node)

{

$left = $this->compileNode($node->getLeft());

$right = $this->compileNode($node->getRight());

switch (get_class($node)) {

case Node\BinaryOp\Add::class:

return $left + $right;

case Node\BinaryOp\Subtract::class:

return $left - $right;

case Node\BinaryOp\Multiply::class:

return $left * $right;

case Node\BinaryOp\Divide::class:

return $left / $right;

}

}

@asgrim

AST in userland

@asgrim

php-ast extensionhttps://github.com/nikic/php-ast

@asgrim

php-ast example usage<?php

require 'path/to/util.php';

$code = <<<'EOC'

<?php

$var = 42;

EOC;

echo ast_dump(ast\parse_code($code, $version=35)), "\n";

// Output:

AST_STMT_LIST

0: AST_ASSIGN

var: AST_VAR

name: "var"

expr: 42

@asgrim

astkithttps://github.com/sgolemon/astkit

@asgrim

astkit example usage$if = AstKit::parseString(<<<EOD

if (true) {

echo "This is a triumph.\n";

} else {

echo "The cake is a lie.\n";

}

EOD

);

$if->execute(); // First run, program is as-seen above

$const = $if->getChild(0)->getChild(0);

// Replace the "true" constant in the condition with false

$const->graft(0, false);

// Can also graft other AstKit nodes, instead of constants

$if->execute(); // Second run now takes the else path

@asgrim

PhpParserhttps://github.com/nikic/PHP-Parser

@asgrim

PHP Parser<?php

use PhpParser\ParserFactory;

$parser = (new ParserFactory)

->create(ParserFactory::PREFER_PHP7);

print_r($parser->parse(

file_get_contents('ast-demo-src.php')

));

@asgrim

Better Reflectionhttps://github.com/Roave/BetterReflection

@asgrim

Better Reflection workflow

Reflector

Source Locator

PhpParser

Reflection

@asgrim

PHP Reflection$reflection = new ReflectionClass(

\My\ExampleClass::class

);

$this->assertSame(

'ExampleClass',

$reflection->getShortName()

);

@asgrim

Better Reflection$reflection = ReflectionClass::createFromName(

\My\ExampleClass::class

);

$this->assertSame(

'ExampleClass',

$reflection->getShortName()

);

@asgrim

ReflectionClass::createFromName()// In ReflectionClass :

public static function createFromName($className)

{

return ClassReflector::buildDefaultReflector()->reflect($className);

}

@asgrim

ClassReflector::buildDefaultReflector()// In ClassReflector :

public static function buildDefaultReflector()

{

return new self(new AggregateSourceLocator([

new PhpInternalSourceLocator(),

new EvaledCodeSourceLocator(),

new AutoloadSourceLocator(),

]));

}

@asgrim

Given a class structure...<?php

class Foo

{

private $bar;

public function thing()

{

}

}

@asgrim

… we get the AST!Class, name Foo

|-- Statements

| |-- Property, name bar

| | |-- Type [private]

| | `-- Attributes [start line: 7, end line: 9]

| `-- Method, name thing

| |-- Type [public]

| |-- Parameters [...]

| |-- Statements [...]

| `-- Attributes [start line: 7, end line: 9]

`-- Attributes [start line: 3, end line: 10]

Any questions?

https://joind.in/talk/513ad

James Titcumb @asgrim

Climbing the Abstract Syntax Tree (Bulgaria PHP 2016)

Technology