Reliable Multicasting with JGroups

Bela Ban, Jan 2004belaban@yahoo.comhttp://www.jgroups.org

EBIG, Oakland Jan 21 2004 2

OverviewAPI, architectureProtocolsBuilding BlocksPerformanceFuture, Conclusion

What Is It ?

Toolkit for reliable multicasting Fragmentation Message retransmission Ordering Group membership, membership

change notificationLAN or WAN based

License

JGroups is a toolkit (JAR), to be linked against an application

Open Source under LGPL Commercial products can use JGroups

without having to LGPL their code Modifications to JGroups itself need to

be LGPL'ed (if distributed)Dual licensing in the future

Channel: similar to java.net.MulticastSocket plus group membership, reliability

Operations: Create a channel with a set of properties Connect to a group X. Everyone that

connects to X will see each other Send a message to all members of X Send a message to a single member

Receive a message Retrieve membership Be notified when members join, leave

(including crashes) Disconnect from the group Close the channel

JChannel channel=new JChannel("file://home/bela/default.xml");

channel.connect("demo-group");

System.out.println("members are: " + channel.getView().getMembers());

Message msg=new Message(null, null, "Hello world");

channel.send(msg);

Message m=(Message)channel.receive(0);

System.out.println("received msg from " + m.getSrc() + ": " + m.getObject());

ch.disconnect();

ch.close();

Group topology

Architecture of JGroups

C ha nne l

U N IC A S T

N A K A C K

C ha nne l

U N IC A S T

N A K A C K

C ha nne l

U N IC A S T

N A K A C K

N e tw o rk

A p p lic a tio n A p p lic a tio n A p p lic a tio n

B u ild ingB lo c k s

DrawReplicatedTree: shared state

JGroups has ~ 90KLOC 30KLOC protocols 45KLOC main + building blocks 15KLOC unit tests

~ 90 protocols shipped with JGroupsSet of well-tested stacks (in XML

files)

Available protocols I

Transport UDP, TCP, TCP_NIO, TUNNEL, JMS,

LOOPBACKDiscovery

PING, TCPPING, TCPGOSSIP, UDPPINGGroup membershipReliable delivery & FIFO

NAKACK, SMACK, UNICAST

Available protocols II

Failure detection FD, FD_SOCK, FD_PID, FD_SIMPLE,

FD_PROB, VERIFY_SUSPECTSecurity

ENCRYPT, SSL ConnectionTable (n/a)Fragmentation (FRAG)State transfer (STATE_TRANSFER)

Available protocols III

Ordering FIFO, CAUSAL, TOTAL, TOTAL_TOKEN

Virtual Synchrony FLUSH, QUEUE, VIEW_ENFORCER

Probabilistic Broadcast PBCAST

Merging: MERGE(2), MERGEFAST

Available protocols IV

Distributed message garbage collection STABLE

Debugging PERF, TRACE, PRINTOBJS, SIZE, BSH

Simulation SHUFFLE, DELAY, DISCARD, DEADLOCK,

LOSS, PARTITIONER

Available protocols V

Dynamic configuration AUTOCONF

Flow control FLOW_CONTROL, FC

Misc PIGGYBACK, COMPRESS

Transport

Task Send messages from above to all members in

the group, or to a single member Receive messages from NW, pass up stack

UDP: multicast and multiple UDP unicastTCP: mcast done by multiple TCP unicastsTUNNEL: send to external router, e.g.

through firewall

Discovery

Task Initial discovery of members Used by GMS to determine coordinator to

send JOIN request toEach member returns its own addr,

plus the addr of the coordinator Typical response ({A,A}, {B,A}, {C,A})

Wait for n milliseconds or m responses

Discovery - UDP

Multicast discovery requestEach member responds with a

unicast UDP datagram (local-addr, coord-addr), back to the sender

Discovery - TCPGOSSIPCan be used by both UDP and TCPExternal GossipServer

org.jgroups.stack.GossipServer Maintains table of <group, members> Each member registers (groupname, own

addr) Lease based - members have to

periodically renew registration Multiple GossipServers possible

Discovery - TCPGOSSIP

To obtain initial membership for a given group, TCPGOSSIP contacts the GossipServer

Membership info does not need to be accurate - only goal is to determine coord to send JOIN request to

Discovery - TCPPING

Give a set of well known membersFor discovery, those members are

pingedIf at least 1 responds, we can find

the coordinatorDoes not require additional process

Group Membership

Task Maintain a list of members Notify members when a new member

joins, or an existing member leaves (or crashes)

Each member has the same ordered list List can be retrieved by Channel.getView() First (= oldest) member is coordinator If coord crashes, 2nd oldest takes over

Group Membership - JOIN

New member uses discovery to find coord If first member -> become coord Else: sends JOIN to coord

Coord adds new member to list, multicasts new view (member list) to all members

If 2 initial members are started at the same time, MERGE protocol merges them into a single group

Group Membership - LEAVE

Member sends LEAVE to coordCoord multicasts new view to all

members

Group membership - CRASH

Failure detection protocol sends up SUSPECT event

VERIFY_SUSPECT double checksGMS multicasts new view (not

containing crashed member)If member resurfaces, it will be

shunned Has to leave and rejoin group

Failure detection

Task Detect if a member has crashed and

send SUSPECT event up the stack (to be handled by GMS)

Logical ring over membershipEach member pings its neighbor to the right

Failure detection - FD

Reliable delivery & FIFO

Lossless and FIFO delivery for multicast and unicast messages Multicast: NAK and ACK Unicast: ACK

Missing messages (gaps) are retransmitted Sender resends or Receiver requests retransmission

Encryption

Uses public/private encryption to join new member and get shared group key

Shared key is used to encrypt all messages

Group key is recomputed on joins/leavesSSL ConnectionTable

As alternative, to be used in TCP Uses SSLSocket rather than Socket

Properties configuration

Plain string format "UDP(mcast_addr=228.8.8.8;mcast_port=45566;ip_ttl=32;" +

"mcast_send_buf_size=64000;mcast_recv_buf_size=64000):" + "PING(timeout=2000;num_initial_members=3):" + "MERGE2(min_interval=5000;max_interval=10000):" + "FD_SOCK:" + "VERIFY_SUSPECT(timeout=1500):" +

"pbcast.NAKACK(max_xmit_size=8096;gc_lag=50;retransmit_timeout=600,1200,2400):" + "UNICAST(timeout=600,1200,2400,4800):" + "pbcast.STABLE(desired_avg_gossip=20000):" + "FRAG(frag_size=8096;down_thread=false;up_thread=false):" + "pbcast.GMS(join_timeout=5000;join_retry_timeout=2000;" + "shun=false;print_local_addr=true)"

URL / XML

Advantages of protocol stacks

Each property is implemented by 1 prot Fragmentation, retransmission, ordering

Protocols are assembled into a stackStack has exactly the properties needed

by the appl / required by the networkCan‘t get this with java.net.Socket,

always comes with full TCP/IP

Advantages of protocol stacks

Small scope: a protocol does just one job, but does it well

Protocol stacks are fashionable: Servlet 2.3 filters Interceptors (Corba, JBoss) AOP: separation of concerns, e.g.

fragmentation should not be an application concern

Benefits

Same application code, different protocol stacks (deployment issue)

Application requirements reflected in protocol stack specification

App focuses on domain specific issues

Building Blocks

Replicated CacheNotificationBusGroup RPC

Replicated Cache

Shared state across a groupAny change is replicated to all membersNew members acquire initial state from

coordStructures supported

Tree Hashmap Queues

NotificationBus

Thin layer on ChannelNotifications sent to all membersCallback when notification is

receivedHook for state sharing

Group RPC

Invoke a method call in all membersGet a list of responsesWait for all responses, majority, first,

or none response (use optional timeout)

Handles crashed members correctly (no blocking)

Serverless JMS

JMS based on JGroupsPeer-to-peer architecture rather than C/SClient publishing to a topic

Instead of sending msg to server, and server distributes to multiple clients: publisher multicasts message

JMS Server just another member Handles persistent messages (DB)

Serverless JMS

Publisher

JMS Server

Subscriber

Client/Server Model

Publisher

JMS Server

Subscriber

Serverless Model

Multicast

(discard) (accept)

(accept)

(discard)

Cost: 4 unicasts Cost: 1 multicast

Serverless JMS

Clients are still able to publish even when server is down

Caveat: works in scenario where client and server are in same multicast-reachable NW

Status Topics/Queues available No TX/XA, no durable subscriptions, no

persistent messages Download (standalone) beta at jboss.org

Where is JGroups used ?

JBoss Clustering

Replication of entity beans, SLSBs and SFSBsHA-JNDICache invalidationSession repl (integrated Tomcat, Jetty)

Serverless JMS Cache

Replicated transactional clustered cache

Jonas appserver (clustering)GroupPac (FT-CORBA impl)GCT: port to .NETReplicated Caching

OpenSyphony OSCache Jakarta Turbine's JCS Swarmcache

Session replication Jetty Tomcat 4.x Work in progress on plugin architecture

for Tomcat 5.xUnofficial ones...

Performance

4 nodes, 1 or 2 senders750MHz SunBlade 1000 512MB, 100MB

switched ethernetJGroups 2.18000 10K msgs, in 200 bursts of 20 (2

senders), sleep after burst = 5ms 451 msgs/s == 4.5MB/s throughput Resident heap size 35MB max (-Xmx128m)

Performance

1.4 billion messages total4 nodes, 2 sendersMessage size = 10KAverage msgs/s: 350Max resident mem: 35M (-Xmx128m)Tests available as part of JG distro

Includes gnuplot scripts to generate graphs

Current and future projects

JBossCache, Serverless JMSPort to J2ME (first version available on

www.jgroups-me.org)hsqldb (HyperSonic) database replicationJCache JSR 107 compliant impl (JBoss

Cache)Potential work on GroupComm JSR

jcluster project on dev.java.net

www.jgroups.org "Papers and Articles": link to IBM

devworks

Questions ?

Reliable Multicasting with JGroups

group topologyebig

multiple tcp unicaststunnel

apijchannel channel

members of xsend

single memberreceive

new messagenull

new jchannelfile

getmembersmessage msg

Documents

Overcast: Reliable Multicasting with an Overlay Network...

Multithreading, Multicasting, Multichallenging - GIS-SCADA.....

IPV6 - MULTICASTING

Intégration du protocole des Trains à JGroups

Ip multicasting 02 multiple unicasting vs multicasting

Application Layer Multicasting

Групповое вещание - Multicasting

Jnp - LTM Chuong8 Multicasting

Chapter 16 Exterior Routing Protocols and Multicasting 1...

with JGroups 3.x Reliable group...

Presentation of 'Reliable Rate-Optimized Video Multicasting....

Reliable Multicasting in the Xpress Transport...

with JGroups 3.x Reliable group...

IPv6 multicasting

Reliable Multicasting in MANETs - Defence Research...

JGroups Manual