Smart Energy Systemsaiellom/pdf/5 Messaging.pdfSpring Client-server syncronous communication communication initiated by client spikes in server load messages get lost client must know

Messaging

AMQP and RabbitMQ

Ilche Georgievskiilche.georgievski@iaas.uni-stuttgart.de

Room: IAAS 0.353

2017/2018Spring

Client-server

syncronouscommunication

communicationinitiated by client

spikes in server load

messages get lost

client must knowserver‘s location

server must be up and running, ready for requests

client server

Indirect communication

Messaging

queue

broker

consumerproducer

syncronouscommunication

communicationinitiated by client

spikes in server load

messages get lost

client must knowserver‘s location

server must be up and running, ready for requests

message message

Messaging

queue

broker

consumerproducer

producer

producer

consumer

consumer

AMQPAdvanced Message Queuing Protocol

AMQP

producer consumerqueue AMQP connection

channel

channel

chan

chan

exchange

broker

virtual host

binding

RabbitMQ

• Erlang implementation of AMQP

• AMQP v0.9.1 with custom extensions

• Data persistance– In-memory/file-persisted embeded database

– Specific message storage and index files

• Clustering– Erlang´s capabilities

• Plugins– Web-based management

Tips

• Connecting to RMQ

– Your application should start irrespective ofwhether a connection to RMQ can be establishedor not

– If the connection to RMQ is lost, your applicationshould reconnect by itself

– If the connection is down, producing/consumingmessages should fail gracefully

AMQP Channels

• AMQP connection is a TCP connection

• A channel is a logical connection

– commands are sent via channels

– all channels reuse the same TCP connection

– hundreds of thousands of new channels a second

AMQP connection

channel

channel

chan

chan

Tips

• Open a channel, do something with thechannel, close the channel

• Channels instances are technically threadsafe– It is recommended to avoid having several

threads using the same channel concurrently

• Channels are terminated by exceptions– Cascade of failures: if some part of the program

tries to use a terminated channel, it will fail too

AMQP Message

properties:content-type::stringcontent-encoding::stringdelivery-mode::1|2priority::0-9correlation-id::stringreply-to::stringexpiration::stringmessage-id::stringtimestamp::timestamptype::stringuser-id::stringapp-id::stringreserved::nullheaders::map[string,object]

body:byte[]

• No information about the producer and consumer

RMQ Message

• Two parts

– Payload

• Anything

– Label

• Describes the payload

• Information about where the message should go– Exchange name

– Topic/routing key (optional)

AMQP Broker

Queue

• Place for messages ready to be consumed

• How to create a queue?– queue.declare

– Queue name• Used by consumers to subscribe to the queue• Used when creating a binding• Otherwise, RMQ assigns a random name

– Other properties• durable

• exclusive

• auto-delete

• arguments

Queue

• Who can create a queue?

– Both consumers and producers

– Consumers can’t create a queue while subscribed to another one on the same channel

• What if a queue already exists when trying to declare it?

– Idempotent if the properties are exactly the same

Tips

• Messages published into an exchange without a queue to be routed to are discarded by RMQ

• Who should create the queue?– Can you afford losing messages created by your

producers? If not, both consumers and producers should create the needed queue

• If you want to check the existence of a queue, set the passive option of queue.declare to true– check then act pattern discouraged

Queue and consumers

• Messages are sent immediately to consumers subscribed to the queue– basic.consume

– basic.get

• Consumer acknowledges a message iff it is done with processing (to RMQ and not to producer!)– basic.ack

– auto_ack parameter set to true when subscribing to a queue (acceptable risk of losing messages)

• Reject a message (rather than acknowledge it)– basic.reject

Tips

• Use basic.consume if the consumer is processing many messages out of a queue

• Use basic.consume if the consumer needs to automatically receive messages from a queue as soon as they arrive

• Don’t use basic.get in a loop as an alternative to basic.consume (performance overhead)

• Don´t forget to acknowledge a message, RMQ won´t send that consumer any more messages

Exchanges and bindings

• Exchange– Place where producers send messages

• Routing key– Rules upon which RMQ decides to which queue

it should deliver the message

• Binding– A queue is said to be bound to an exchange by a

routing key

Direct exchange

• Routing algorithm: if the routing key of a message matches therouting key of a queue, then the message is delivered to thatqueue

• If Q2 is bound to X by the routing key black, then all messageswith the routing key black will be deliverd to Q2

Direct exchange

• A default exchange

• Empty string as its name

• A newly declared queue will beautomatically bound to this exchangeusing the queue name as routing key

• channel.basic_publish(exchange=´´,

routing_key=´queue-name´, body=msg)

Fanout exchange

• Routing algorithm: a message isdelivered to all queues bound to theexchange (broadcast)

• Routing keys are ignored

Topic exchange

• Routing algorithm: if the routing key of a message matches the routing keyof one or more queues, then the message is deliverd to all the queues

• Routing key must be a list of words delimeted by dots• * can substitute for exactly one word• # can substitute for zero or more words

Topic exchange: example

• timeseries.<building>.<floor>.<room>.<area>.<type-id>.<instance-id>

• timeseries.iaas.0.353,window.light.12345

• Exchange– Name: timerseris.exchange.topic– Type: topic

• Queue– Name: timeseries.all– Type: durable– Routing key: timerseries.#

Virtual host

• Logical subdivision of RMQ´s execution space– Virtual message broker– Its own queues, exchanges and bindings– Its own persmissions

• Allow for multiple applications to one RMQ server

• Default vhost called /

• Vhosts are to RMQ what VMs are to physical servers

Tips

• Identify common functionality groups in your application and give each one itsown vhost

• Use the default one for testing

– guest/guest as credentials

– Guest has full permissions

Reliability: durability

• Queues and exchanges together with messagesdon´t survive a reboot of the RBM server

• The durable property of queues and exchanges isset to false by default

• For messages to survive reboot, queues and exchanges must be durable, but this is not sufficient– Messages must be persistent– Set the delivery-mode option of the message to 2

Tips

• Should you use persistent messaging forall messages?

– Writing a message to disk is much slower thansaving it in RAM

– Slows down RMQ in terms of the number ofmessages it can process per second

Reliability: redundancy

• Multiple RMQ instances

– Performance

– Scalability

– Fault tolerance

References

• Coulouris, G., Dollimore, J., Kindberg, T., Blair, G., Distributed Systems: Concepts and Designs. Fifth Edition.

• Videla, A. and Williams, J. W. RabbitMQ in Action: Distributed messaging for everyone. Manning Publications Co. 2012.

• Dossot, D. RabbitMQ Essentials. Packt Publishing. 2014.

• RabbitMQ Tutorial, https://www.rabbitmq.com/tutorials/.