Sprayer: low latency, reliable multichannel messaging
Post on 18-Dec-2014
835 Views
Preview:
DESCRIPTION
Transcript
Sprayerlow latency, reliable multichannel messaging
for Telefonica Digital
who are we?
Javier Arias@javier_arilos
Javier is a Software Architect and developer, worked in different sectors such as M2M, Telcos, Finance, Airports.
Pablo Enfedaque@pablitoev56
Pablo is a SW R&D engineer with a strong background in high performance computing, big data and distributed systems.
Telefónica is the 4th largest telco in the world
2 years ago Telefonica Digital was established to spread our business to the digital world
former Telefonica R&D / PDI was merged into this new company
some context
overview
we are developing an internal messaging service to be used by our own products
we have polyglot persistence using different NoSQL technologies
in this talk we will review Sprayer’s architecture and, for each technology, how it is used
why sprayer?
a common push messaging service. why?
➔ each project with messaging needs was implementing its own server its own way
➔ 5 push messaging systems in the company
➔ none of them supporting a wide variety of transports
➔ independent deployment and operations
the problem
cross technology push:
point to point and pubsub:
PaaS, multitenant
iOS Android Websockets HTTP
eMail SMS FirefoxOS
1 to 1 1 to N 1 to Group
inspiration
➔ Google’s Thialfi: http://research.google.com/pubs/pub37474.html
➔ Twitter Timeline: http://www.infoq.com/presentations/Twitter-Timeline-Scalability
➔ Pusher: http://www.pusher.com
➔ Pubnub: http://www.pubnub.com
➔ Amazon SNS: http://aws.amazon.com/sns/
SPRAYER!
Sprayer is a low latency, reliable messaging system supporting delivery of messages to a single receiver, to a predefined group of receivers or to a specific list of receivers over different channels (WebSockets, SMS, Email, HTTP and iOS, Android or Firefox OS native push…)
the proposal
SPRAYER!
our motto:you care about business,we deliver your messages
the proposal
server
side API
server
side API
?
server side API challenges
➔ common interface for all channels
➔ reliable, consistent, idempotent
➔ route messages efficiently
➔ simple and user oriented
◆ manage subscriptions
◆ send messages: to list or group (topic)
◆ get delivery feedback
➔ standards based (HTTP + Json)
architecture
APPLICATION <BACKEND>
sprayer backend
sms gateway
email gateway
GCM
APNs
Operationalstorage
ACCEPTERREST API
MESSAGES DISPATCHING
messagesdispatching
messagesdispatching
?
message dispatching challenges
➔ scaling horizontally
➔ reliability
➔ different channels: ◆ HTTP (outbound)◆ Websockets (inbound)◆ iOS push (APNs)◆ Android push (GCM)◆ SMS◆ eMail
architecture
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
Operationalstorage
ACCEPTERREST API
MESSAGES ROUTING
outbound-stateless dispatchers
ANDROID GCM
simple dispatchers: HTTP, iOS, Android...
➔ Take message, get msg subscribers,
dispatch to receiver, report feedback
➔ Completely stateless
Operationalstorage
ACCEPTERREST API
Operationalstorage
connection aware dispatchers
clients (websockets, HTTP long poll …)➔ messages are stored until clients connect➔ client inits a persistent connection➔ potentially, millions of clients
WEBSOCKETS
ROUTER DELIVERER
ACCEPTERREST API
inboxes
messagerouting
messagerouting
?
message routing challenges
routing (two-steps):➔ API routes messages to N dispatchers➔ Each dispatcher routes message to M
receivers (subscribers of a group)
both steps must be decoupled
The number of receivers could be thousands
architecture
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
Subscriptionsstorage
sms
android
WS
HTTP
iOS
Operationalstorage
ACCEPTERREST API
FEEDBACK
async messagedelivery feedback
?
async messagedelivery feedback
async delivery feedback challenges
make msg feedback available through API to clients
feedback must not compromise message delivery or API
feedback: msg delivery, connections, push
The number of updates could be millions
architecture
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
Subscriptionsstorage
sms
android
WS
HTTP
iOS
Operationalstorage feedback
ACCEPTERREST API
technology stack
subscriptions storage
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
Subscriptionsstorage
sms
android
WS
HTTP
iOS
Operationalstorage feedback
ACCEPTERREST API
?
subscriptions storage
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
sms
android
WS
HTTP
iOS
Operationalstorage feedback
ACCEPTERREST API
dispatcher receiver inboxes
WEBSOCKETS
ROUTER DELIVERER
ACCEPTERREST API
inboxes
?
dispatcher receiver inboxes
WEBSOCKETS
ROUTER DELIVERER
ACCEPTERREST API
inboxes
redis
Redis is an open source, advanced key-value store. It is often referred to as a data structure server (...) - (redis.io)
why redis?
- amazingly fast
- easy to use
- usage patterns: shared cache, queues, pubsub, distributed lock, counting things
redis use cases
use cases in Sprayer:
➔ group subscribers x channel➔ channels x group➔ websockets channel queues (potentially
million receivers)
limitations for our use cases:
➔ memory bound➔ queries and pagination➔ high throughput queues
redis concerns
➔ what happens when dataset does not fit in memory? two strategies◆ partition datasets to different redis clusters◆ sharding: based in tenant would be easy
➔ FT and HA◆ easy way: master-slave with virtual IPs, switch
slave’s IP when master’s out. home made daemon◆ sentinel based, some tests done, needs to be
supported by client library◆ redis cluster being implemented; limited features
operational storage
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
sms
android
WS
HTTP
iOS
Operationalstorage feedback
ACCEPTERREST API
?
operational storage
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
sms
android
WS
HTTP
iOS
feedback
ACCEPTERREST API
mongodb
mongoDB (from "humongous") is a document database (...) features: full index support, replication & HA, auto-sharding... (mongodb.org)
why mongoDB?
➔ scaling & HA➔ great performance➔ dynamic schemas➔ versatile
mongodb use cases
use cases in Sprayer:➔ operational DB, administrative data➔ message delivery feedback updates
(potentially millions of records)
limitations for our use cases:➔ operations with sets of subscribers➔ high throughput queues
mongodb concerns
no concerns about mongodb for our usecase.
maybe, in the long term, can it handle the huge amount of feedback write operations without affecting the API?
async queues
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
sms
android
WS
HTTP
iOS
feedback
ACCEPTERREST API
?
?
async queues
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
ACCEPTERREST API
rabbitmq
robust messaging for applications, easy to use
(www.rabbitmq.com)
why rabbitmq?
➔ very fast➔ reliable➔ builtin clustering
rabbitmq use cases
use cases in Sprayer:➔ jobs for dispatchers (API => dispatchers)➔ feedback status updates: message
delivery, connections, device status (dispatchers => API)
limitations for our use cases:➔ not scaling well to millions of queues
(websocket receiver inboxes)
rabbitmq concerns
no concerns!
rabbitmq is best suited to very high throughput messaging
full tech stack
APPLICATION <BACKEND>
sprayer backend
IOS
HTTP
WEBSOCKETS
ANDROID
SMS
sms gateway
email gateway
GCM
APNs
STATUS FEEDER
ACCEPTERREST API
sum up
design threats
design threats
related data in different places: redis, rabbitmq and mongo
we are not transactional, our components remain sane in case of a DB failure, idempotent operations help here
light implementation of Unit of Work architectural pattern
idempotent operations are a must here
architecture guidelines
architecture guidelines
➔ asynchronous processing / queues everywhere
➔ dedicated dispatchers for each transport
➔ common API interface
➔ used the best tool for each responsibility:
polyglot persistence
➔ processes as stateless as possible
YES, SPRAYER DOES!
thanks for coming
top related