StormMQ Introduction to AMQP, Dublin

Introduction to Message [email protected]

mailto:[email protected]

mailto:[email protected]

Agenda

A Real Application ! 20 min

Nibbles and Hands–On ! 60 min

Introduction to AMQP ! 20 min

Why Message Queuing? ! 25 min

Classic Architecture ! 10 min

History of Message Queuing

Manual Telegraphy Machine Assisted Telegraphy

1920s1911 - 192019th Century 1900s

Telegrams sent using“Store and Forward”

1900

1930s


Telcos UseElectronic Telegraphy

1950s1940s 1960s

IBM System/360 with BTAM & QTAM

Message Switching

1964

First Electronic Mail Solutions

1965

Banking Users

1970s

IBM TCAM which is the first true solution

Retired 1990!

1971

Machine Assisted Telegraphy


First Electronic Mail

FMCG & Utilities

1990s

IBM Launch MQSeries(now WebsphereMQ)

1992

Machine Assisted Telegraphy Corporates Large Websites YOU

Noughties Today

Sun Release Java JMS, Reinvigorating

Enterprise Messaging

2001

AMQP Working Group Formed by Investment Banks

2006

Cloud Enables and Drives StormMQ

Adoption

2009

The Integration Tag Soup

WS – File Transfer

!"#$%&'(#)#*#+%+

Message QueuingSOAP

EMail

SMTP

FTP

RSYNC

NTFS

!','-

!).$%&'/$.0%&1$%+

234%5)6!%$7%$

SOAMessage

OrientatedMiddleware

JMS

MQ

MQSeries

AMQP

Amazon SQS

Tibco

StormMQ

RabbitMQ

HTTP

TCP/IPDCE / RPC

CORBA

DCOM

REST

RMIXML-RPC

.NET Remoting

Remote Procedure Call

XML-RPC


WS – File Transfer

!"#$%&'(#)#*#+%+

Message QueuingSOAP

EMail

SMTP

FTP

RSYNC

NTFS

!','-

!).$%&'/$.0%&1$%+

234%5)6!%$7%$

SOAMessage

OrientatedMiddleware

JMS

MQ

MQSeries

AMQP

Amazon SQS

Tibco

StormMQ

RabbitMQ

HTTP

TCP/IPDCE / RPC

CORBA

DCOM

REST

RMIXML-RPC

.NET Remoting


XML-RPC


Message Queuing

XML-RPC

Under Examination, though

Deployment

Intimate System Knowledge

Configuration

Admin

File Transfer

!"#$%&'(#)#*#+%+


Don’t Work in the Cloud

Scaling

Ideal: Message Queuing

Systems are “loosely-coupled”

MessageQueuing

Systems don’t know each other

Why Use it: Loose Coupling

BillingCatalogue

Shipping S-a-a-S Inventory

better position: Why do people

queuing? fonts

How do we connect them, without one outage or system change taking everything down like a pack of cards?

Why Use it: Loose Coupling

BillingCatalogue

Shipping S-a-a-S Inventory

better position: Why do people

queuing? fonts

How do we connect them, without one outage or system change taking everything down like a pack of cards?

Message Queuing lets Systems and Components exchangedata, events, commands and actions with one another

with no explicit knowledge or need for them to be online

It should be ideal for the cloud

It should be ideal for the cloud

!!!Until today, messaging is …

(ZeroMQ)

(SQS uses HTTP)

(MSMQ uses COM)

(memcached)

(Java JMSuses

Source)

(MQSeries / WebsphereMQ)

(Talarian, Rendezvous, etc)

Platform Restricted BespokeProprietary

!!!Until today, messaging is …

(ZeroMQ)

(SQS uses HTTP)

(MSMQ uses COM)

(memcached)

(Java JMSuses

Source)

(MQSeries / WebsphereMQ)

(Talarian, Rendezvous, etc)

Why is it Hell?

You need more staff And moneyYou need bridge technology

Why is it Hell?

You need more staff And moneyYou need bridge technology

However, there’s a solution

A-MQPAdvanced Message Queue Protocol

However, there’s a solution

However, AMQP fixes this

A common wire-level binary format and protocol

An explicit definition of a server (aka broker)’s

semantics

Open Means

Interoperable

That is good …

“AMQP will be to Messaging what HTTP was to the Web”

MRG

Clients run on any Platform Vendors are Interoperable

That is good …

“AMQP will be to Messaging what HTTP was to the Web”

MRG

Clients run on any Platform Vendors are Interoperable

64K

From Anywhere to Anywhere

From Anywhere to Anywhere

Eddie Velez

Quick Recap

✓

The fifth male memberof the A-Team, Frankie,

was played by

Question

Message Queuing connects systems and components. Is it ideal for the cloud?

Does Loose-Coupling make individualsystems more likely to suffer outages?

Yes

No

AMQP is Open.

This makes it suitable for programmingin C and Javascript?

!

✓

Agenda





Pizza and Hands–On ! 60 min

Store and Forward

Billing

“Could you send me a duplicate of my bill please”

REST

StormMQMessaging

Billing

!

Fire and Forget

Shipping

“Too many orders. The website is running like a dog.”

SQL Push SQL Pull

StormMQMessaging

Shipping

One-To-Many

“There’s a new widget in the Catalogue: Tell all the systems.”

CatalogueShipping

S-a-a-S Inventory

File Transfer

ETL

One-To-Many

“There’s a new widget in the Catalogue: Tell all the systems.”

CatalogueShipping

S-a-a-S InventoryStormMQMessaging

StormMQMessaging

Round-Robin

Billing

“How do we easily scale a massive batch job like Billing?”

EMail BillGenerate

StormMQMessaging

Publish-Subscribe (“Topics”)

“Shipments Sent, Delivered and Received”

Shipping

S-a-a-S InventoryBilling

RIASentSent or Returned

22

Web Sockets Architecture

Copyright © 2010, Kaazing Corporation,. All rights reserved.

Agenda






Messaging: Which Jargon?

EMail(SMTP, POP3, IMAP)

VoIP(VoiceMail, XMPP)

Texting(SMS)

Instant Messaging(ICQ, MSN, Jabber)

Twitter

Enterprise Service Bus(ESB)

Dynamic OO Languages(eg Ruby)

Message Queuing(MQ)

Message Queuing(MQ)

What do wemean by

Messaging?

Essential Terms

Message Queue What’s a Message?

Envelope

Payload

Header

Essential Terms

*Strictly speaking

a receiver polls for messages

a consumer has messages pushed

≣ ≊Send Publish Enqueue

≣ ≊ DequeueConsume*Receive

Connection

AMQP Client AMQP ServerTCP / IP Network

ConnectionVirtual Host

Connections and Channels

TLS “Shielding”

Channels

Each Channel is Independent:Effectively, a Virtual Connection

Basic AMQP

Basic AMQP: Connections

Open a Connection to a Virtual Host

Open a Channel

Send a Message

Receive a Message

Close Channel

Close Connection

You only need one channel!

"

#

$

%

&

'

Exchanges Route Messages

Exchanges route Messages to Message Queues

You send messages to an exchange, not a message queue

Exchange

MQ

A

MQ

B

How do Exchanges Route?

Exchange

MQ

A

The Exchange finds a Bindingmatching the Routing Key#

A Binding connects a Routing Keyto one or more Message Queues

$Every sent message has a

Routing Key"

The Exchange delivers theMessage to the Message Queue%

A message queue can bebound more than once to

one or more exchanges(

“string”

Types of Exchange

More rarely used exchange types include amq.headers and extensions

Message Queue Name

direct

Like a ‘Map’: All MQs bound with the

routing key receive copies of the message

“” (blank)amq.direct

fanout

Empty String

All MQs bound to the Exchange receive

copies of the message

amq.fanout

topic

Dotted

Bindings useglobbing expressions (wildcards) to route messages to MQs

amq.topic

A message queue can be bound more than once to an exchange;A message queue can be bound to more than one exchange

But a message queue will only receive a message sent once

Routing Key

RoutingBehaviour

Default Definitions

Point – to – Point One – to – Many Publish – SubscribeTypical Use

And there is more!

Transactions

More Message Properties

Custom Message Properties

Immediate Delivery

Additional Exchange Types

Auto Deletion

Fine-Grained User Permissions

QueuePurging

QoS

Agenda






The Problem

Intrusion Detection

Lots of Logs

Firewall Monitoring

Performance & Failure

What’s needed

Live Debugging Live Logging Live Statistics

Audit ComplianceStatistics Mining SMS Alerts EMail Alerts

It must not get in the way

Add Hardware

Config

MaintenanceNo Extra Admins

Change Software

Design of the Clients

intrusion

api.daemon.info.charon api.memory api.daemon.sshd api.warn

checksstatisticslog

Content-Type: application/json; charset=utf-8Timestamp: 2010-11-03T16:45:56.981ZPriority: Low / HighType: statistics_2011.10.01.1203Expiry: 5 mins

api.firewall.warn.in.portscan.tcp

checks

Handling Log Messages

api.*.*.warn Queue

#Queue

Timestamp

No Acks

QoS Prefetch

WebSockets

logMessageId

Routing Key

*.warnQueue

checks

Stat

istics

Arc

hitect

ure

is th

e Sa

me

From Checks to Alerts

#Queue

emergencyQueue

#Queue

checks

alerts

alerts

Contact Detailsnot in message

Message Aggregation /Suppression by alerts

Uses MessagePriority

And debugging ?

A-MQPAdvanced Message Queue Protocol

And debugging ?

StormMQ Introduction to AMQP, Dublin

Technology

question message queuing

s inventory stormmq

message queuing systems

stormmq messaging billing

stormmq messaging shipping

s telegrams

s mqseries pheremq

s inventory billing