Reliable System Integration and Scaling with WSO2 Message Broker Ramith Jayasinghe Senior Technical Lead WSO2
Reliable System Integration and Scaling with WSO2 Message Broker
Ramith JayasingheSenior Technical Lead
WSO2
Messaging Models/Concepts• RPC style
- Request/Response, Synchronous, Non-persistent - e.g. Web Services, Thrift, REST
• Point to Point (Queues)
- A message is delivered only once to a single consumer
• Publish/Subscribe (Topics)
- Broadcast a message to all the subscribers
Messaging Models/Concepts• Durable Subscriptions
- Same as topic subscriptions- Subscriber offline - Messages get queued- Subscriber back online - Messages get delivered- Un-subscription needed to remove queuing
Messaging Models/Concepts• Transactional ( send/consume -> rollback, reject, commit)• Guaranteed Delivery, QOS ( MQTT, AMQP), Persistence• Dead letter channel• Hierarchical topics
- Games.FootBall.UK, Games.FootBall.USA, - Games.Cricket.SriLanka, Games.Football.UK
Distributed Queues
• Provides strict or best effort support for inorder delivery• There are no guarantee about the global order seen
across subscribers
WSO2 Message Broker
• Design Objectives:- Higher performance (by not routing messages over the
network multiple hops)- High availability and fault tolerance.- Leaner deployment layout ( and clustering story)
• v 3.0.0 - Significant architectural changes.• v 3.1.0 - Latest version
- Minor improvements - PostgreSQL support
WSO2 MB v 3.x.x Design Objectives
• Avoid message routing over the network multiple times.• Use storage to share messages between nodes
Allow database to scale as separate layer.
• Use distributed coordination to control the behavior (i.e. Hazelcast)
WSO2 MB v 3.x.x Design Objectives
• Content and metadata are saved separately into a database shared between
all cluster nodes.
• All nodes knows about all subscribers ( - via Hazelcast coordination)
• Each node will logically divide messages into group (known as Slots).
WSO2 Message Broker Design
More information about disruptor: https://lmax-exchange.github.io/disruptor/
WSO2 Message Broker Design - Inbound
All events coming to broker are added to ‘inbound disruptor’ such as:
● Messages● Message acknowledgements ● Create/Delete queues topics● Subscribe/Unsubscribe● UI Events
Inbound disruptor:● Sequence inbound events ( - disruptor
behaviour)● Generate internal Message IDs ( length =
java - long)● persist/delete messages● Execute other control functions.
Note: #1. Internal message Ids are time based. Therefore, MB cluster needs to be time synced.
#2. Inbound path can be tuned for best performance ( broker.xml).
WSO2 Message Broker Design - Outbound
All message going out of broker are added to ‘outbound disruptor”
Slot Delivery Worker pool:1) Connects to coordinator node ( - get
messages to deliver)2) Fetches metadata of messages and add
to outbound disruptor
Outbound disruptor:1) Fetch message payloads (in parallel)2) Forward fetched messages to transport.
Note: Slot delivery thread pool and outbound disruptors can be tuned for best performance (broker.xml).
WSO2 Message Broker Design - Distributed Queues
• Content and metadata are saved separately into a database shared between
all cluster nodes.
• All nodes knows about all subscribers ( - via Hazelcast coordination)
• Each node will logically divide messages into group (known as Slots).
WSO2 Message Broker Design - Distributed Queues
• Cluster elects a leader ( known as Coordinator); Usual the first node to start in cluster.
• Coordinator keeps track of all the slots available (via Hazelcast)
• Nodes with subscriptions acquire a slot (from Coordinator) and deliver them to
subscribers.
WSO2 Message Broker v 3.1.0 - Features
● Message Queuing Telemetry Transport (MQTT)
● Shared Subscriptions
● RDBMS based message store (MSSQL, Oracle, H2, MySQL, PostgreSQL)
● Real time statistics/graphs (on each server) via carbon-matrices
● Improved internals
MQTT support in WSO2 MB• Lightweight, pub-sub protocol designed for IOT use-cases.
• Hierarchical topic structures and wildcard subscriptions
• All quality of service levels and retained messages
Shared Durable Subscriptions
Extension to JMS 1.1, until JMS 2.0 support is released. Purpose: Ability to load balance.
broker.xml: <allowSharedTopicSubscriptions>true</allowSharedTopicSubscriptions>
Back pressure / Flow Control - Inbound• Every system has its limits.
• Safeguards server from sudden bursts of message loads
• Flow control based on:
– Total inbound events
– Inbound events per connection/channel
– Global error conditions (e.g. offline databases)
<flowControl> <!-- This is the global buffer limits which enable/disable the flow control globally --> <global> <lowLimit>800</lowLimit> <highLimit>8000</highLimit> </global>
<!-- This is the channel specific buffer limits which enable/disable the flow control locally.
--> <bufferBased> <lowLimit>100</lowLimit> <highLimit>1000</highLimit> </bufferBased> </flowControl>
General Improvements• Real time statistics/graphs
– Quickly figure out bottlenecks in a production deployments.
– Database read/write latencies– Inbound/Outbound event rates– Number of Connections/Channels etc.
• Performance Improvements – Disruptor based thread model– Improved internals ( topic matching, slot based delivery,
caching)
Improvements in MB v 3.1.0• Support for PostgreSQL as the message store.• LZ4 Compression for storage
– Dramatic performance improvement large/text oriented message payloads
• Forcefully disconnect Consumers– If a subscriber misbehaves, admin can disconnect the client
forcefully.
Roadmap• Support for AMQP 1.0 and JMS 2.0
• Move to Carbon 5
• Improved GUI
• Improvements to Storage Scheme
• Performance improvements.
• Websockets + Stomp
What makes WSO2 MB different● Simplified clustering story (- Queues/Topics)
- Users don’t have to worry about how brokers transmit messages in the cluster.
● RDBMS for persistence →Easy to manage and use
● Simpler Cluster deployments → Minimum deployment effort + maintenance.
Use Case – Guaranteed Delivery with WSO2 ESB
• Guarantees In-Order Delivery.• Throttle messages ( for slow consumers/systems).• Ensure reliable messaging