Top Banner
NaradaBrokering for DS-RT 2005 Grid Tutorial IEEE DS-RT 2005 Montreal Canada Oct. 9 2005 Geoffrey Fox CTO Anabas Corporation and Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 [email protected] http:// www.infomall.org
51

NaradaBrokering for DS-RT 2005 Grid Tutorial

Jan 03, 2016

Download

Documents

darryl-oneal

NaradaBrokering for DS-RT 2005 Grid Tutorial. IEEE DS-RT 2005 Montreal Canada Oct. 9 2005 Geoffrey Fox CTO Anabas Corporation and Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 [email protected] http://www.infomall.org. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: NaradaBrokering for  DS-RT 2005 Grid Tutorial

NaradaBrokering for DS-RT 2005 Grid Tutorial

IEEE DS-RT 2005 Montreal Canada Oct. 9 2005

Geoffrey Fox

CTO Anabas Corporation andComputer Science, Informatics, Physics

Pervasive Technology LaboratoriesIndiana University Bloomington IN 47401

[email protected]://www.infomall.org

Page 2: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Grid (Web Service) Messaging Build distributed systems from “interoperable” services linked by messages

(SOAP) – architect capabilities as services Grids are “just” large scale sets of such services Need to support real time streams and NOT just files (collections of

messages) consistent with WS standards (P2P and “central”) Open Source http://www.naradabrokering.org (4 downloads/day) is a

scalable distributed pub-sub system supporting multiple standards (JMS, WS) and subscription methods• Implements “Service Internet” and Notification areas of WS-*

Infrastructure Manage messaging for

• Optimize communication for bad links, firewalls etc• Collaboration (multi-cast streams)• Fault tolerance with re-transmitted messages and Replicated Services• Replay – access any message at any time• Virtualize addressing with pub-sub metaphor• Performance from protocol (UDP v Parallel TCP ..) and representation• Heterogeneous Clients – filter to and from PDA’s

Candidate for Axis2-MOM (Message Oriented Middleware) infrastructure

Page 3: NaradaBrokering for  DS-RT 2005 Grid Tutorial

NaradaBrokering

Stream

NB supports messagesand streams

NB role for Grid isSimilar toMPI role for MPP

Queues

Page 4: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Multiple protocol transport supportIn publish-subscribeParadigm with differentProtocols on each link

Transport protocols supported include TCP,  Parallel TCP streams, UDP, Multicast, SSL, HTTP and HTTPS.Communications through authenticating proxies/firewalls & NATs. Network QoS based RoutingAllows Highest performance transport

Subscription Formats Subscription can be Strings, Integers, XPath queries, Regular Expressions, SQL and tag=value pairs.

Reliable delivery Robust and exactly-once delivery in presence of failures

Ordered delivery Producer Order and Total Order over a message type. Time Ordered delivery using Grid-wide NTP based absolute time

Recovery and Replay Recovery from failures and disconnects.Replay of events/messages at any time. Buffering services.

Security Message-level WS-Security compatible security

Message Payload options

Compression and Decompression of payloadsFragmentation and Coalescing of payloads

Messaging Related Compliance

Java Message Service (JMS) 1.0.2b compliant Support for routing P2P JXTA interactions.

Grid Feature Support NaradaBrokering enhanced Grid-FTP. Bridge to Globus GT3.

Web Services supported

Implementations of WS-ReliableMessaging, WS-Reliability and WS-Eventing.

Traditional NaradaBrokering Features

Page 5: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Features for March—June 2005 Releases Production implementations of WS-Eventing, WS-

Notification, WS-RM and WS-Reliability. SOAP message support and NaradaBrokers viewed as SOAP

Intermediaries Active replay support: Pause and Replay live streams. Stream Linkage: can link permanently multiple streams –

using in annotating real-time video streams Replicated storage support for fault tolerance and resiliency

to storage failures. Management: HPSearch Scripting Interface to streams and

services Broker Discovery: Locate appropriate brokers

Page 6: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Summary NaradaBrokering provides a fully distributed queue manager

where queues buffer streams with overheads of a few milliseconds per broker

• << 30 ms frame interval

• << 100’s ms network delay

• Much faster than using databases or writing files Collaboration is implemented by sharing synchronizing

streams Compatible with Grids, Web Services, Java Message Service Streams are “first class entities” with rich set of features

• Don ‘t open a socket; hand data to NaradaBrokering Software Overlay Network or Message Oriented Middleware

Page 7: NaradaBrokering for  DS-RT 2005 Grid Tutorial

NaradaBrokering Services

Page 8: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Reliable Delivery Service Guaranteed delivery in multiple producer/ consumer

settings. Guarantees hold true in the presence of• Node/Link Failures• Links can lose messages and garble message order.• Storage failures: Stores need to recover after failure.• Prolonged entity disconnects

Exactly-Once and Ordered delivery of events Uses both positive& negative acknowledgements Supports Replay and Fast Recovery from failures Independent of underlying archival system. Was used to enhance fault tolerance in Grid-FTP. Uses “Reliable Storage” to keep messages temporarily

Page 9: NaradaBrokering for  DS-RT 2005 Grid Tutorial

0

2

4

6

8

10

12

14

16

0 1000 2000 3000 4000 5000 6000 7000 8000 900010000

Tim

e (

Mill

iseco

nds)

Content Payload Size in Bytes

Transit delays/Standard deviations in a 3 broker network.NB-BestEffort(BE)(TCP) Vs NB-ReliableDelivery(RD)(UDP)

Mean delay (NBRD-UDP) Mean delay (NBBE-TCP)

Std Dev (NBRD-UDP) Std Dev (NBBE-TCP)

Page 10: NaradaBrokering for  DS-RT 2005 Grid Tutorial

0

2

4

6

8

10

12

0 1000 2000 3000 4000 5000 6000 7000 8000 900010000

Tim

e (

Mill

iseco

nds)

Content Payload Size in Bytes

Transit delays/Standard deviations in a single broker network.NB-Best Effort(TCP) Versus NB-Reliable Delivery(UDP)

Mean delay (NBRD-UDP) Mean delay (NB-BETCP)

Std Dev (NBRD-UDP) Std Dev (NBBE-TCP)

Page 11: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Dealing with large payload sizes To cope with large payloads, the substrate incorporate

2 sets of services. Compression/Decompression service: The substrate

incorporate support for zlib based compression and decompression of payloads.

Fragmentation/Coalescing Service: These service can break up a large payload into smaller fragments. The coalescing service can take these smaller fragments and coalesce them into the original large payload.• This was used to deal with transfer of large payloads (up to 1

GB) in the NB enhanced Grid-FTP application.

Page 12: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Replay Services Replay requestors can specify replays based on several

parameters• A range of sequence numbers can be specified.

• Additionally, constraints on an event’s content synopsis can be specified.

• Based on a specified time range. Replay services have been tested with applications such

as Audio/Video conferencing, Whiteboards etc. Essential for recording and replay of collaborative

sessions Important special case supports rewind and similar

operations on a real-time stream

Page 13: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Buffering Service This service is incorporated into the system to facilitate

the buffering of events prior to releasing them. Buffering service time orders events and releases event

based on three metrics• Number of events in the buffer

• Size of the buffer

• Time spent by event in a buffer.

Page 14: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Time Differential Service This service is essential to reduce jitters in large

distributed environments.• Networks introduce unpredictable delays that increase jitter.

Service takes events released by buffering service, and ensures that it preserves time spacing between events.

TDS can provide time spacing resolution of up to 1 millisecond between events.

Page 15: NaradaBrokering for  DS-RT 2005 Grid Tutorial

-2

0

2

4

6

8

10

12

0 100 200 300 400 500 600 700 800 900 1000

Jitte

r (M

illis

econ

ds)

Sample Number

Jitter values comparing the Input to the Buffering Service and the Output of the TDS

Buffering InputTDS Output

Trans-Atlantic Settings

Page 16: NaradaBrokering for  DS-RT 2005 Grid Tutorial

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0 100 200 300 400 500 600 700 800 9001000

Jitte

r (M

illis

econ

ds)

Sample Number

Jitter values from the output of the TDS

TDS Output

Trans-Atlantic Settings

Page 17: NaradaBrokering for  DS-RT 2005 Grid Tutorial

NaradaBrokering NTP Service NaradaBrokering includes an implementation of the Network

Time Protocol (NTP) All entities within the system use NTP to communicate with

atomic time servers maintained by organizations like NIST and USNO to compute offsets• Offset is the computed difference between global time and the

local time.• The offset is computed based on the time returned from

multiple atomic time servers.• The NTP algorithms weighs results from individual time

clocks based on the distance of the atomic server from the entity.

This ensures that all entities are within 1 ms of each other. The timestamps account for clock drifts on machines

• Time returned corrects software clocks which can slow down with increased computing load on the machine.

Page 18: NaradaBrokering for  DS-RT 2005 Grid Tutorial

-1

-0.5

0

0.5

1

0 20 40 60 80 100 120 140 160

Offs

et c

hang

e (M

illis

econ

ds)

Elapsed time in 100s of seconds

NTP Offset variations over a period of 4 hoursIndiana Linux machine with

a native NTP daemon process

Offset Variation

Page 19: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Broker Discovery Service Locates the nearest available broker that a client can

connect to.• Incorporates specialized nodes – broker discovery nodes – to

maintain broker info.

• Depending on load or security issues, brokers may decide to respond/ignore discovery requests.

• If available the scheme can exploit IP multicast for discovery.

• Nearest broker determined by ping times, loss rates and available bandwidth.

Page 20: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Broker Discovery: Brokers at Indianapolis, NCSA, UMN, FSU, IU & San Diego Supercomputing Center. Broker at Indy selects IU, NCSA and UMN for pings.

Page 21: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Broker Discovery: Brokers at Indianapolis, NCSA, University of Minnesota, FSU & San Diego Supercomputing Center. Cardiff selects Indy, NCSA and UMN for pings.

Page 22: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Topic Discovery Service Allows publishers and subscribers to advertise topics. Creator of topic possesses credentials to indicate

ownership of the topic. Discovery of topics takes into account credentials of

client trying to discover topic.• Topic owner may restrict discovery to a limited authorized set

of clients. Discovery requests can be made using simple strings or

regular expression queries.

Page 23: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Based on Message Level SecurityMessages organized into topicsEach topic has a separate key; Topics can be organized into sessions

Security Service

Page 24: NaradaBrokering for  DS-RT 2005 Grid Tutorial

NaradaBrokeringSupport for SOAP and Web Services

Page 25: NaradaBrokering for  DS-RT 2005 Grid Tutorial

SOAP Support I The broker can receive SOAP messages (over HTTP)

from any entity. • This removes any client dependence in client-broker

interaction The broker can function as an intermediary performing

multiple roles which could just be routing but also involve mapping using filters

There can be multiple filter-pipelines, each comprising multiple filters, available at the broker node.• Some of these would be system filter-pipelines configured

statically.

• Filter-Pipelines can also be configured by users, dynamically, at run-time.

Page 26: NaradaBrokering for  DS-RT 2005 Grid Tutorial

SOAP Support II Multiple roles could be associated with

• Different servlets hosted by a broker.• A given servlet hosted by a broker.

Scheme will allow filters to be registered for individual roles.• A filter could be part of multiple roles.

There is a dedicated filter pipeline per role. This implies that a NaradaBroker can be used as a Web

Service container although full container support is not yet available

Filters are used internally by NB to implement performance monitoring

Page 27: NaradaBrokering for  DS-RT 2005 Grid Tutorial

The FilterPipeline-Filter model I The filter and filter-chain facilitate many of the

interactions that are missing in JAX-RPC handlers.• Filters are NaradaBrokering approach to the handlers used in

Web Service containers Filters can inject messages at any time

• These messages can be sent either to the application or over the network.

• No limit on the number of messages that can be triggered because of a single message from application.

Messages can be injected into a Filter Pipeline from either directions.

Filters can generate responses automatically. No need to route to application.

Page 28: NaradaBrokering for  DS-RT 2005 Grid Tutorial

The FilterPipeline-Filter model II Applications have access to individual filters and

filter-pipeline at all times. Explicitly direct which filters need to be skipped or added.

Filters have access to position within Filter Pipeline, and can specify message injection at a specific location.

Dynamic reconfiguration possible for Filter Chain.

Allow different networking substrates to be registered. This can be dynamically changed. • Network substrate is last filter and is

responsible ONLY for routing SOAP message.

Page 29: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Web Services Support I Currently we have incorporated support for the

following Web Service specifications• WS-Eventing (WSE): This is a publish/subscribe based

notification framework from Microsoft and IBM.• WS-ReliableMessaging WSRM): This is a protocol for

ensuring the guaranteed delivery of SOAP messages between 2 Web Service endpoints. This specification is from IBM and Microsoft.

• WS-Reliability (WSR)- This is a competing specification from Oracle and Sun in the area of reliable messaging between Web Services.

These handlers are available for use in Axis1.2 or exploiting NB SOAP Intermediary support without a container• Axis1.2 version can be used inside container or as a Proxy

Page 30: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Web Services Support - II We are also working on implementing support for the

WS-Notification (WSN) suite of specification that is part of the Web Services Resource Framework (WSRF).

WS-Notification explicitly adds brokers to Eventing Note that almost all these specifications leverage the

WS-Addressing (WSA) specification. • We have incorporated support for all the rules associated

with WSA.

Page 31: NaradaBrokering for  DS-RT 2005 Grid Tutorial

NaradaBrokering in Web Services a) WSM WSR WSN WSE support for Axis1.2 which is

available as standalone handlers without need for any NaradaBrokers

b) The support described in a) implemented as a separate proxy and inside containers

c) NaradaBrokers used as SOAP Intermediaries d) NaradaBrokers can support filters in SOAP

intermediaries forming limited light-weight containers e) NaradaBrokers can be Brokers defined in WSN

Specification f) One can use NaradaBrokers in non-brokered

publish-subscribe such as WS-Eventing to make it scalable

Page 32: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Operation Mean StdDev

StdError

Outlier

Min Max Mem (Bytes)

Create an XMLBeans based Envelope Document

121.29

25.77 2.65 6 110 333 2192

Create an Axis based SOAPMessage

85.76 79.36 8.22 7 34 540 1824

Convert an EnvelopeDocument to a SOAPMessage

3503.8

758.48

80.85

12 2632

5406

57152

Convert SOAPMessage to EnvelopeDocument

730.08

392.35

41.58

11 327 1911

34424

Create a WS-Addressing EPR(Contains just a URL address)

84.61 25.61 2.67 8 72 301 2072

Create a WS-Addressing EPR(Contains WSA ReferenceProperties)

133.13

35.64 3.71 8 114 354 2648

Create an Envelope targeted to a specific WSA EPR

157.98

12.19 1.27 8 140 219 7184

Create an Envelope targeted to a specific WSA EPR with most WSA message information headers

263.20

35.73 3.74 9 240 471 13880

Implementation of WS-Reliable Messaging (WSRM) I

Page 33: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Implementation of WS-Reliable Messaging (WSRM) II

Operation Mean StdDev

StdError

Outlier Min Max Mem(Bytes)

Parse an EnvelopeDocument to retrieve WSA Headers

711.74

231.61

23.76

5 555 1317

61024

Create a Wsrm Fault 413.80

239.17

25.07

9 271 1212

18096

Create a Wsrm SequenceRequest

268.95

37.93 3.97 9 212 374 16392

Create a Wsrm SequenceResponse

234.97

17.40 1.81 8 212 324 18160

Create a Wsrm SequenceDocument

43.812

2.99 0.30 4 42 53 2424

Add a WsrmSequenceDocument to an existing envelope. (Contains sequence identifier and message number)

13.01

0.57 0.05 4 11 15 464

Create a WSRM SequenceAcknowledgement based on a set of message numbers

461.17

172.40

18.27

11 301 1043

20624

Create a WSRM TerminateSequence

20.95

1.30 0.13 4 20 25 2072

Page 34: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Transport Layer in

NaradaBrokering

Page 35: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Transport Layer Support for multiple network protocols such as TCP,

UDP, Multicast, SSL, RTP, HTTP and Parallel TCP.• Support for both blocking and non-blocking IO in the TCP

support.

• The UDP support manages payloads greater than 64K datagram limit. Also incorporates pinging mechanism to detect connection losses in connectionless setting.

Tunnel through firewalls/proxies • Microsoft’s ISA, Checkpoint, Apache

Page 36: NaradaBrokering for  DS-RT 2005 Grid Tutorial

hop-3

0

1

2

3

4

5

6

7

8

9

100 1000

Tra

nsit

Del

ay

(Mill

isec

onds

)

Message Payload Size (Bytes)

Mean transit delay for message samples in NaradaBrokering: Different communication hops

hop-2

hop-5 hop-7

Pentium-3, 1GHz, 256 MB RAM100 Mbps LAN

JRE 1.3 Linux

Page 37: NaradaBrokering for  DS-RT 2005 Grid Tutorial

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1000 1500 2000 2500 3000 3500 4000 4500 5000

Sta

nd

ard

De

via

tion

(M

illis

eco

nd

s)

Message Payload Size (Bytes)

Standard Deviation for message samples in NaradaBrokering Different communication hops - Internal Machines

hop-2hop-3hop-5hop-7

Page 38: NaradaBrokering for  DS-RT 2005 Grid Tutorial
Page 39: NaradaBrokering for  DS-RT 2005 Grid Tutorial
Page 40: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Performance of NaradaBrokering in collaborative settings

Page 41: NaradaBrokering for  DS-RT 2005 Grid Tutorial

0.1

1

10

100

0 200 400 600 800 1000 1200 1400 1600

Tim

e (

Mill

ise

con

ds)

Number of Users

Average Latencies and Jitters for Audio Conferencing Clients. Single Broker, Single Meeting

Average LatencyAverage Jitter

Page 42: NaradaBrokering for  DS-RT 2005 Grid Tutorial

0.1

1

10

100

1000

0 100 200 300 400 500 600 700 800 900

Tim

e (

Mill

ise

con

ds)

Number of Users

Average Latencies and Jitters for Video Conferencing Clients. Single Broker, Single Meeting

Average LatencyAverage Jitter

Page 43: NaradaBrokering for  DS-RT 2005 Grid Tutorial

10

100

1000

200 300 400 500 600 700 800 900

La

ten

cy (

Mill

ise

con

ds)

Number of Users per broker

Average Latencies for Video Conferencing Clients at different Brokers. 4 Brokers, Single Meeting

Latency at B1Latency at B2Latency at B3Latency at B4

Page 44: NaradaBrokering for  DS-RT 2005 Grid Tutorial

1

10

100

20 30 40 50 60 70 80 90 100

La

ten

cy (

Mill

ise

con

ds)

Number of Meetings

Average Latencies for Video Conferencing Clients at different Brokers. 4 Brokers, Multiple Meetings (20 Users per Meeting)

Latency at B1Latency at B2Latency at B3Latency at B4

Page 45: NaradaBrokering for  DS-RT 2005 Grid Tutorial

1

10

100

0 20 40 60 80 100 120 140 160

La

ten

cy (

Mill

ise

con

ds)

Number of Users per Site

Average Latencies for Video Conferencing Clients at different locations. Sites in Indiana, Florida, New York and Cardiff

IndianaNew York

FloridaCardiff UK

Page 46: NaradaBrokering for  DS-RT 2005 Grid Tutorial

“GridMPI” v. NaradaBrokering In parallel computing, MPI and PVM provided “all the features

one needed’ for inter-node messaging NB aims to play same role for the Grid but the requirements and

constraints are very different• NB is not MPI ported to a Grid/Globus environment

Typically MPI aiming at microsecond latency but for Grid, time scales are different• 100 millisecond quite normal network latency• 30 millisecond typical packet time sensitivity (this is one audio or video

frame) but even here can buffer 10-100 frames on client (conferencing to streaming)

• 1 millisecond is time for a Java server to “think” Jitter in latency (transit time through broker) due to routing,

processing (in NB) or packet loss recovery is important property Grids need and can use software supported message functions and

trade-offs between hardware and software routing different from parallel computing

Page 47: NaradaBrokering for  DS-RT 2005 Grid Tutorial

HPSearch Management Engine HPSearch is an engine for orchestrating distributed

Web Service interactions• It uses an event system and supports both file transfers and

data streams. HPSearch flows can be scripted with JavaScript

• HPSearch engine binds the flow to a particular set of services and executes the script.

HPSearch can access and set NaradaBrokering features (create topics, display performance data)

ProxyWebService: a wrapper class that adds notification and streaming support to a remote Web Service.

HPSearch is a streaming sensitive workflow engine

Page 48: NaradaBrokering for  DS-RT 2005 Grid Tutorial

WMS GIS service and a data Layer

Page 49: NaradaBrokering for  DS-RT 2005 Grid Tutorial

Data Filter(Danube)

Pattern Informatics(Danube) Accumulate Data Run PI Code Create Graph Convert RAW -> GML

GPS Database(Gridfarm001)

WMS

HPSearch(TRex)

HPSearch(Danube)

HPSearch hosts an AXIS service for remote deployment of scripts

GML(Danube)

WS Context(Tambora)

NaradaBroker network: Used by HPSearch engines as well as for data transfer

Actual Data flow

HPSearch controls the Web services

Final Output pulled by the WMS

HPSearch Engines communicate using NB Messaging infrastructure

Virtual Data flow

Data can be stored and retrieved from the 3rd part repository (Context Service)

WMS submits script execution request (URI of script, parameters)

Workflow (BPEL) Fragment

Page 50: NaradaBrokering for  DS-RT 2005 Grid Tutorial

SensorML and NaradaBrokering OGC defined a set SensorML of specifications

indicating how to integrate Sensors with its GIS Services

We are using Southern California SCIGN GPS data to prototype this

RYO Binary

Text

GML

Sensor Source

Filter

Filter

NaradaBrokeringTopics

You can access whicheverversion you want!

Page 51: NaradaBrokering for  DS-RT 2005 Grid Tutorial

NaradaBrokering Futures Support for replicated storages within the system.

• In a system with N replicas the scheme can sustain the loss of N-1 replicas. Clarification and expansion of NB Broker to act as a WS

container Integration with Axis 2.0 as Message Oriented Middleware

infrastructure Support for High Performance transport and representation for

Web Services• Needs Context catalog under development

Performance based routing• The broker network will dynamically respond to changes in the network

based on metrics gathered at individual broker nodes. Replicated publishers for fault tolerance Pure client P2P implementation (originally we linked to JXTA) Security Enhancements for fine-grain topic authorization, multi-

cast keys, Broker attacks