Making Symfony Services async with RabbitMq (and more Symfony)

It’s all about eXperience

Gaetano Giunta

PHP LondonJuly 2015

Making Symfony Services async with

RabbitMq

(and more Symfony)

PechaKucha style!

2

A case study: generating M$Office docs

The needs

• Produce content in Microsoft Office formats• Many formats for each document

• Both Word and Excel docs

• XML content generated by CMS

• LibreOffice used to generate MS Office and PDF versions• Has anyone tried generating OOXML by hand?

3

A case study: generating M$Office docs

Reality bites

• Slooow• Rest assured that some of the docs will be pretty big

• Unreliable• Depending on version, LO crashes from a-bit to almost-

always

• Race-prone• Two conversion jobs in parallel might step on each other

4

A case study: generating M$Office docs

…and the results are:

• Ugly code: lots of polling, copying of files around, manual locking

• Does not scale at all: only one conversion process active at any given time

Web server

PHP LibreOffice

Waiting processes

5

TextBook scenario for introducing queues

Web server

Worker?

RabbitMQ

LibreOffice

PHP

6

TextBook scenario for introducing queues

Web server

Worker?

RabbitMQ

LibreOffice

PHP

7

TextBook scenario for introducing queues

Web server

PHP!

RabbitMQ

LibreOffice

PHP

8

TextBook scenario for introducing queues

Web server

PHP!

RabbitMQ

LibreOffice

PHP

9

TextBook scenario for introducing queues

Web server

Symfony!!!

RabbitMQ

LibreOffice

PHP

10

TextBook scenario for introducing queues

Web server

Symfony!!!

RabbitMQ

LibreOffice

PHP

11

The rationale

• Same library handling both sides of the connection• Easier to troubleshoot

• Sf Console component is lovely

• Everything developed in a single repository

• No need to learn new languages

• Everything which I can rewrite in PHP, I eventually will*

* = 3rd whimsical law of Gaetano

12

The details

• Json used as payload encoding• The worker has been made thread-safe

• Again, thanks to Symfony: the « Process » Component• It spins up a new php process each time it receives a command• The new process executes the command

• Bonus: a ‘watchdog’ to restart workers via crontab if they die

Symfonylistener

Symfonylistener

Symfonylistener

Symfonyworker

LibreOffice

LibreOffice

W

13

The evolution

• What is exactly a « queued command » ?

• For starters, each existing Symfony console command!

• Why not every existing Symfony Service?• Allows to run existing services asynchronously

• As long as their arguments are serializable to json

• Not a project-specific tool any more

• Could be turned into a Bundle

• …

• Profit!

14

The problems

• Assumes good security on the network

• Low throughput• Spinning up a new php worker process takes time

• Could we use php-fpm to communicate to workers instead?

• Am I reinventing the wheel?

• Others?• Suggestions are welcome

15

I’ll be here all night

THANK YOU

https://github.com/kaliop-uk/kueueingbundle

• @gggeek• www.kaliop.co.uk