June 18 th 2003. ACM Middleware 2003 NaradaBrokering: A Middleware Framework and Architecture for.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering: A Middleware Framework and Architecture for Enabling P2P Grids

Middleware 2003Rio de Janeiro, Brazil16-20 June 2003

Shrideep Pallickara and Geoffrey Foxspallick, gcf@indiana.edu

Community Grid Computing Laboratory,Pervasive Technology Labs

Indiana University.http://www.naradabrokering.org

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Talk Outline• P2P Grids• Messaging Infrastructure requirements• NaradaBrokering Overview• P2P support• Extensible transport framework• Security and Monitoring Infrastructure• Conclusions• Future Work

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

P2P Grids• Grids - Typified by infrastructure for seamless access to

high-end computing resources.– Job submissions, data management services.

• P2P systems – Sophisticated resource sharing environments. – Search, discovery & sharing of resources (CPU/files).

• P2P Grid comprises services of both Grids and P2P systems.

• Integrates ideas of computational grids, web services, P2P systems and message-oriented-middleware.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Messaging Infrastructure for P2P Grids• Scaling – Support large number of devices/users.• Efficient disseminations of interactions• Guaranteed Delivery Mechanisms• Location Independence• Support for P2P interactions – JXTA, Gnutella• Interoperate with Messaging clients• Performance Monitoring and Security services.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering: Features• Based on a network of cooperating broker nodes

– Cluster based architecture allows system to scale to arbitrary size

• Originally to provide uniform software multicast to support real-time collaboration linked to publish-subscribe for asynchronous systems.

• Now has four major core functions– Message transport (based on performance) in multi-link

fashion– General publish-subscribe including JMS, JXTA and Gnutella

(started) – Support for RTP-based audio/video conferencing.– Federation of multiple instances (just starting) of Grid services

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NB: Engineering Issues Addressed• Tunnel through firewalls/proxies

– Microsoft’s ISA, Checkpoint, Apache• Support for multiple network protocols such as

TCP, UDP, Multicast, SSL, RTP and HTTP.• Support for both blocking and non-blocking IO.• Scaling of software multicast

– Efficient calculation of destinations and routes..• Supports local broker accesses• Transparently replace single server JMS systems

with a distributed solution.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering: Organization

SSC-A SC-1

SC-2

SC-3

l13 14

15

n20

21

i4 5

6

j7 8

9

m16 17

18

k10 1112

h1 23

19

Broker Node

Service Provider

End Client

1, 10 Super-super-cluster controller

5, 9, 10, 16 Super-cluster controller 2,4, 6,8, 12,14,18,20 Cluster controller

k

10 11

12

SP

SPSP

SPSP

11a10a

12a

EC EC

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Organization of Profiles and Routing• Client subscriptions are stored hierarchically within the

system.– A broker maintains client subscriptions, cluster-controller

maintains broker subscriptions and so on.• When an event is received, the event is matched against

stored profiles and destinations are computed– A cluster-controller computes broker destinations. A broker

computes client destinations.• Every broker node, when supplied with a set of

destinations, computes the best broker-hops to take to reach these destinations.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering: Distributed Results

Transit Delays under different matching rates:22 Brokers 102 ClientsMatch Rate=100%

Match Rate=50% Match Rate=33% Match Rate=4%

0100200300400500600700Publish Rate (Events/sec) 0 50100150200250300350400450500

Event Size (Bytes)

050100150200250300350400450

Transit Delay (MilliSeconds)

i4 56 l

13 1415

j7 89

h1 2

3

k10 11

12

m16 17

18

n20

2119

22

MeasuringSubscriber

Publisher

Every broker – SPARC Ultra-5 (128 MB, 333 MHz)105 Clients – SPARC-Ultra-60 (512MB, 360 MHz)JRE-1.2.1, 100 Mbps Network

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering: Matching Engines• Matching engines are responsible for matching

events against stored profiles.• NaradaBrokering supports a variety of matching

engines supporting– “/” separated String based topics– Equality based <tag,value> topics– Integer based topic– SQL (from JMS) – XPath based queries

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Matching Engine Performance

0

2

4

6

8

10

20 30 40 50 60 70 80 90 100

Del

ay (

Mic

rose

cond

s)

Number of subscriptions (in thousands) being matched

Average delay to match event with String-based Matching for subscriptions with different sizes

String size=16String size=24String size=32

0

10

20

30

40

50

60

70

20 30 40 50 60 70 80 90 100

Del

ay (

Sec

onds

)

Number of subscriptions (in thousands) being matched

Average delay to match event to Xpath and SQL-based subscriptions

SQL Delay XPath Delay

<?xml version="1.0" encoding="ISO-8859-1"?><menu> <softdrinks> <brand>Minute Maid</brand> <fruit>Apple</fruit> <source>Brazil</source> <company>Coca Cola</company> <price>2.90</price> <year>2003</year> </softdrinks></menu>

XPath Query type: /menu/softdrinks[price>1.80]

Stand alone processPentium-3 1 GHZ 256MB RAMJRE 1.4

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Why Support P2P?• Core features – Resource Sharing & Discovery

• CPU cycles: SETI@home, Folding@HOME • File Sharing: Napster, Gnutella

• Deployments user driven – No dedicated management• Management of resources

– Expose resources & specify security strategy– Replicate resources based on demand

• Dynamic peer groups, fluid group memberships• Sophisticated search mechanisms

– Peers respond to queries based on their interpretations– Responses do not conform to traditional templates.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

P2P Systems: The Downsides• Routing not very sophisticated

– Inefficient network utilization • Usually relying on simple forwarding of interactions

– Peer Traces (to eliminate echoing)– Attenuations (to suppress propagation)

• TTL’s associated with interactions.

• Interactions are attenuated– Resulting in localized interactions and a fragmented

world of multiple P2P subsystems

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering & JXTA Interaction Model• Based on proxy model

– Acts as both Rendezvous peer (JXTA routers) and NaradaBrokering client.

• No changes to JXTA core or straitjacketing of interactions– Change made to Rendezvous

layer• Peers are not aware that they

interact with a Narada-JXTA proxy or Rendezvous peer.

• Narada-JXTA provides JXTA guaranteed long distance delivery

NARADA-JXTA proxy

NARADAbroker cloud

Peers

JXTARendezvousPEER

Dynamic/fluidpeer groups

High end "long lived"/persistent resources

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering-JXTA Proxy• Glean relevant information from JXTA interactions.

– Peer group advertisements (XML Doc describing resource)– Requests/Responses to be part of peer group.– Messages sent to a peer group.– Queries and responses to these queries.

• Subscribe to relevant topics to ensure delivery• Construct corresponding Narada-JXTA event from

interactions.– These events lend themselves to efficient routing.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering-JXTA Apps and Setups• Applications

– Integrated NaradaBrokering-JXTA environment tested under JXTA shell and myJxta (InstantP2P)

– Plan to integrate myJxta into Anabas (distance education software) with NaradaBrokering managing P2P and middle-tier (JMS) style interactions.

• Experimental Setup– Sender/receiver - (Pentium-3, 1 GHz, 256 MB RAM). – Every node (broker/router) hosted on a different machine

(Pentium-3, 1 GHz, 256 MB RAM). – Machines reside on a 100 Mbps LAN– Run-time environment for all the processes is

JDK-1.3 build Blackdown-1.3.1, Red Hat Linux 7.3

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

0

50

100

150

200

250

300

500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500

Tran

sit D

elay

(M

illise

cond

s)

Message Payload Size (Bytes)

Transit delay for message samples in Narada, JXTA and Narada-JXTA Topology - III (8 routers) Internal Machines

NaradaBrPure JXTA

Narada-JXTAR R

R

R

R R

R

R

(a)

R R

R

R

R R

R

R

N

(b)

N N

N

N

N N

N

N

(c)

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering Communications• Applications interface to NaradaBrokering through

UserChannels.• UserChannels have publish/subscribe semantics• Links implement a single conventional “data” protocol.

– Different links can have different underlying transport implementations

– Addition of new transport protocols within the Framework is easy to achieve.

– Administrative channel negotiates the best available communication protocol

• Link implementations can incorporate their own handshaking protocols to facilitate communications and data exchange.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering Communications - II

Transport Interfaces LinkPerformance

Data

TransportHandler Link

Factory

LinkFactory

LinksSpecific to a transport

Link Monitors

Data accumulated byMonitoring Service

Brokernode Administrative Link (HTTP)

Optimal Transport

Alternate Link

TransportInterfaces

(Application andContent Dependent)

Negotiated with infoexchanged over

Administrative Link

Brokernode

MonitoringService

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Performance Monitoring• Every broker incorporates a Monitoring service

that monitors links originating from the node.• Every link measures and exposes a set of metrics

– Average delays, jitters, loss rates, throughput.• Individual links can disable measurements for

individual or the entire set of metrics.• Measurement intervals can also be varied• Monitoring Service, returns measured metrics to

Performance Aggregator.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Performance Aggregation• Aggregated information will be used to

– Circumvent bottlenecks– Aid routing algorithms– Facilitate Dynamic Load-balancing

BrokerNode

LinkData

BrokerNode

LinkData

Performance AggregationService

Control MessageExchange

Aggregates infofrom nodes in acertain domain

MonitoringService

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

NaradaBrokering: Security Framework• Based on Message Level Security• Authentication – Confirm whether a user is really who he says he

is.• Authorization – Identify if the user is authorized to receive

certain events• Key distribution – Based on authentication & authorization

distribute keys, which ensure that only the valid clients are able to decrypt encrypted data.

• Digital Signing – Have the ability to verify the source of the event and whether the source is authorized to publish events conforming to the specified template.

• Communication Protocol Independent • Detection and Response to Security Compromise

– Clients required to respond to queries (stored during initializations) issued at random intervals.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

2 Respond back with topickey if authorized to publish

NaradaBrokering Broker Cloud

KeyManagementCenter (KMC)

1 2 3

4

5

6

78

Broker Node

Entity (Publisher or Subscriber)

SSL encryptedcommunications

6 Respond back with topic key ifauthorized to subscribe

7

Create subscription requestCompute Message DigestSign MD and message IDIssue Subscription request Message

4Verify Signature & PermissionsCheck integrity by verifying MDCheck ID for replay attacks

3

Encrypt message with topic keyCompute Message Digest(MD)Sign MD and message IDPublish Message

1 Request permission to publish

5 Request permission to subscribe

8

Verify SignatureVerify Permissions for SubscribingCheck integrity by verifying MDCheck ID for replay attacks

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Security Results• Experiments performed on a Pentium-3, 1.5GHz,

512 MB RAM.• JRE 1.4.1, Cryptographic provider is IAIK• Points in the graphs represent the average value

of the operation being performed 1000 times. • Results indicate that the security scheme does

not introduce unacceptable delays.– Since messages are encrypted only once, costs are

amortized during traversal over multiple broker hops.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

0

2

4

6

8

10

12

14

16

18

500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Tim

e in

Mill

isec

onds

Data Size in Bytes

Encryption/decryption times for symmetric key algorithms

192-bit 3DES Decryption 128-bit AES Decryption

192-bit 3DES Encryption 128-bit AES Encryption

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

1

10

100

1000

10000

500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Tim

e in

Mic

rose

cond

s

Data Size in Bytes

Time to compute Message Digests and Signing of messages using SHA-1 and RSA algorithm

Compute Digest RSA Sign

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Conclusions• NaradaBrokering is a messaging infrastructure

that is appropriate for P2P Grids.• Results demonstrate that it can be used for

synchronous and asynchronous applications.• Availability of multiple matching engines

provides for sophisticated interactions between entities within the system.

June 18th 2003.ACM Middleware 2003

http://www.naradabrokering.orgspallick,gcf@indiana.edu

Future Work• Federation of P2P systems and accompanying

search/query/response mechanisms. – Ongoing work with Limewire (Gnutella)

• Distributed A/V conferencing management.• Dynamic Resource Management• Management of lightweight XML databases

– Ongoing investigations with Apache Xindice and Source Forge eXist.