Developing highly scalable applications with Symfony and RabbitMQ

Developing highly scalable

applications with

Symfony and RabbitMQ

Alexey Petrov

[email protected]

mailto:[email protected]


Agenda

• Story 1: For those who not used queueing before

• Story 2: For those who started using queueing

• Story 3: For those who have to scale and distribute

• Story 4: Symfony2 Integration


Story 1

For those who not used queueing before


Request

ResponseTime

P

R

O

C

E

S

S

I

N

G


Time min

Muscules

(better hardware)

Brains

(better software)

Prepare

Response


Request

ResponseTime

Parsing

Request

P

R

O

C

E

S

S

I

N

G


Request

Response

Time

Parsing

Request

Prepare

Response

P

R

O

C

E

S

S

I

N

G


Request

Response

Time

Parsing

Request

Prepare

Response Task 1 Task 2


What is asynchronous?



How to run asynchronous

process from PHP


• popen

• proc_open

• shell_exec

• system

• exec


Symfony Process





Problems?


1 Request = X tasks

N Requests = N*X tasks


T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

Worker


T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

Worker

Database Cron Job


Problems?


T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

Worker

Producer

Producer

Producer


T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

T

A

S

K

WorkerProducer

Producer

Producer

Worker

Worker


Solution:

USE RIGHT TOOLS!


Use RabbitMQ



Story 2

For those who started using queueing


Basic Examples

http://www.rabbitmq.com/getstarted.html


Connection and Channels


COOL STORY BRO

But we’re in PHP :(


Exchanges To Exchanges


Exchanges and Queues

FanoutDB1

DB2

Msg Topic Stats

Email

SMSTopic


Libraries

• PhpAmqpLibhttps://github.com/videlalvaro/php-amqplib

• PECL amqphttp://php.net/manual/pl/book.amqp.php

https://github.com/alanxz/rabbitmq-c

https://github.com/bkw/pecl-amqp-official/tree/master/stubs


PECL amqp

https://bugs.php.net/bug.php?id=66661


Durability


Durability

• Exchanges

• Queues

• Messages


Durability


Transactions


Transactions


4 minutes to publish

10 000 messages


Publisher Confirms


Publisher Confirms


Publisher Confirms


Dead Letter Exchanges



• Delayed Messages (Events)

• Retries

• Errors processing

• etc.



• The message is rejected

(basic.reject or basic.nack)

with requeue=false

• The TTL for the message expires;

• The queue length limit is exceeded.




Dead Letter Messages

x-death header:

queue – name of the queue the message was in before

reason – rejected | expired | maxlen

time – timestamp the message was dead lettered

exchange – the exchange the message was published to

routing-keys – the routing keys the message was published with.

original-expiration – the original expiration message property


Delayed Pattern

MSG ExchangeQ1

TTL = 60000

DLX Q2


More Queues = Better

Queue lives in 1 Eralng process


Per-Message TTL

vs

Per-Queue TTL



Story 3

For those who have to scale and distribute


Clustering


Clustering


1 client = 1 socket

Cluster helps to scale!


Mirrored queues


Mirrored queues


Network Partitions


Network Partitions

• Ignore mode

• Pause-minority mode

• Autoheal mode


Federation


Federation


Federated Exchanges


Federated Queues


Shovel


The Symfony


Story 4

Symfony2 Integration


Installing Bundle


Configuration


Producer


Consumer


Questions?

Alexey Petrov

Senior PHP Developer / System Architect

Email: [email protected]

Twitter: @alexhelkar

mailto:[email protected]

Developing highly scalable applications with Symfony and RabbitMQ

Software