Rabbits, indians and... Symfony meets queueing brokers

It’s all about eXperience

Gaetano GiuntaGilles Guirand

Forum PHP

Paris2015/11/24

Rabbits, indians and... Symfony meets

queueing brokers

Brokers, don’t they work in Wall Street?

Brokers, don’t they work in Wall Street?

Message brokers are a building block of Message oriented middleware.

[wikipedia]

Brokers, don’t they work in Wall Street?

Message brokers are a building block of Message oriented middleware.

[wikipedia]

Message oriented middleware (MOM) is software or hardware infrastructure supporting the sending and receiving of messages between distributed systems

[same as above]

• RabbitMQ• ActiveMQ• Apollo• Kafka• Kestrel• Amazon SQS• ZeroMQ*• And many more…

Let’s try again: can you name any broker?

What do they do?

Where do they differ?• Protocol support (amqp, stomp, …)

• Libraries exist for writing clients using standard protocols• Other offer their own SDK

• Routing• Amqp has exchanges and queues, Stomp only queues• Wildcards in queue names are supported by most systems

• Delivery guarantees• Possible double delivery?• Out of order dispatch

Where do they differ?• Surviving failures

• Are messages sent before the client is started buffered or lost?• What happens when a client crashes?• Support for ACK/NACK mechanisms

• Clustering• And if the server crashes?

• Performances• Batch send/receive – prefetch• Polling vs. persistent connections

Everything clear so far?

Everything clear so far?

A case study: generating MSOffice docsThe needs

• Produce content in Microsoft Office formats• Many formats for each document• Both Word and Excel docs

• XML content generated by CMS• LibreOffice used to generate MS Office and PDF versions

• Has anyone tried generating OOXML by hand?

A case study: generating MSOffice docsReality bites

• Slooow• Rest assured that some of the docs will be pretty big

• Unreliable• Depending on version, LO crashes from a-bit to

almost-always• Race-prone

• Two conversion jobs in parallel step on each other

A case study: generating MSOffice docs…and the results are:

• Ugly code: lots of polling, copying of files around, manual locking

• Does not scale at all: only one conversion process active at any given time

Web server

PHP LibreOffice

Waiting processes

TextBook scenario for introducing queues

Web server

Consumer

RabbitMQ

LibreOffice

PHP

TextBook scenario for introducing queues

Web server

Consumer?

RabbitMQ

LibreOffice

PHP

TextBook scenario for introducing queues

Web server

PHP!!!

RabbitMQ

LibreOffice

PHP

TextBook scenario for introducing queues

Web server

PHP!

RabbitMQ

LibreOffice

PHP

TextBook scenario for introducing queues

Web server

Symfony!!!

RabbitMQ

LibreOffice

Symfony

TextBook scenario for introducing queues

Web server

Symfony!!!

RabbitMQ

LibreOffice

Symfony

The rationale• Same library handling both sides of the connection

• Easier to troubleshoot

• Sf Console component is lovely

• Everything developed in a single repository

• No need to learn new languages

• Everything which I can rewrite in PHP, I eventually will*

* = 3rd whimsical law of Gaetano

There is a bundle for that

oldsound/rabbitmq-bundle• specific to RabbitMQ• supports both producers and consumers

php app/console rabbitmq:consumer <name>

class FooConsumer implements ConsumerInterface { public function execute(AMQPMessage $msg) { $foo = unserialize($msg->body); echo 'foo '.$foo->getName()." successfully downloaded!\n"; }}

Long lived php processes ?• Fatal errors

• Memory leaks

• Database connections might become broken

• Is it in your code or in a dependency?

A new hope

• kaliop/queueingbundle

• kaliop/queueingbundle-sqs

• kaliop/queueingbundle-stomp

A design focusing on safety• The consumer has been split in 2

• The listener is long lived; it spins up a php worker process each time it receives a command

• The worker process is a Symfony console command• It handles a single message then exits

Symfonylistener

Symfonyworker

LibreOffice

Queue

Does it scale ?• The listener waits for the worker to finish before staring another• Solution: run multiple listeners in parallel

W

Symfonylistener

Symfonyworker

LibreOffice

Queue

Symfonylistener

Symfonyworker

LibreOffice

Multiple protocol support

Kaliop Queueing Bundles

fusesource/stomp-phpaws/aws-sdk-php oldsound/rabbitmq

ActiveMQ ApolloSQS RabbitMQ

Application

Easy to use

• Message encoding taken care of (JSON by default)• Has events your code can listen to alter the consumption

loop

Messagereceived

Do stuff

Consumption

failed

Messageconsumed

Exception

Show me da codeold_sound_rabbit_mq: connections: default: ... producers: my_producer: connection: default exchange_options: name: my_producer.exchange type: topic consumers: my_consumer: connection: default exchange_options: name: my_producer.exchange type: topic queue_options: name: my_queue

callback: my_symphony_service

Show me da code

Worker

use Kaliop\QueueingBundle\Service\MessageConsumer;

class MyConsumer extends MessageConsumer{ /** * @param mixed $body */ public function consume($body) { // do stuff ... }}

Show me da code

Lots of console commands available

php app/console kaliop_queueing:queuemessage <queuename> <msg body>

php app/console kaliop_queueing:consumer <queuename> [-i<driver>]

php app/console kaliop_queueing:managedriver list

php app/console kaliop_queueing:managequeue list-configured [-i<driver>]

php app/console kaliop_queueing:managequeue info <queuename> [-i<driver>]

Known limitations

• Assumes good security on the network

• Low throughput• Spinning up a new php worker process takes time

• Specific features of the different protocols are not (yet) supported• The goal remains to have a uniform API

Further evolution• Instead of spinning off a console command for each message

received, send an http call to a php-fpm process

• Support for RPC-style calls• Others?

• Suggestions are welcome

Symfonylistener

Symfonyworker

LibreOfficeQueue

HTTP

Web server: phpdaemon

It’s all about eXperience

Thanks forlistening

The end