Reliable Multicasting with JGroups

Reliable Multicasting with JGroups

Bela Ban, Jan [email protected]://www.jgroups.org

EBIG, Oakland Jan 21 2004 2

OverviewAPI, architectureProtocolsBuilding BlocksPerformanceFuture, Conclusion


What Is It ?

Toolkit for reliable multicasting Fragmentation Message retransmission Ordering Group membership, membership

change notificationLAN or WAN based


License

JGroups is a toolkit (JAR), to be linked against an application

Open Source under LGPL Commercial products can use JGroups

without having to LGPL their code Modifications to JGroups itself need to

be LGPL'ed (if distributed)Dual licensing in the future


API

Channel: similar to java.net.MulticastSocket plus group membership, reliability

Operations: Create a channel with a set of properties Connect to a group X. Everyone that

connects to X will see each other Send a message to all members of X Send a message to a single member


API

Receive a message Retrieve membership Be notified when members join, leave

(including crashes) Disconnect from the group Close the channel


API

JChannel channel=new JChannel("file://home/bela/default.xml");

channel.connect("demo-group");

System.out.println("members are: " + channel.getView().getMembers());

Message msg=new Message(null, null, "Hello world");

channel.send(msg);

Message m=(Message)channel.receive(0);

System.out.println("received msg from " + m.getSrc() + ": " + m.getObject());

ch.disconnect();

ch.close();


Group topology

Architecture of JGroups

C ha nne l

G M S

U N IC A S T

U D P

F D

N A K A C K

C ha nne l

G M S

U N IC A S T

U D P

F D

N A K A C K

C ha nne l

G M S

U N IC A S T

U D P

F D

N A K A C K

N e tw o rk

A p p lic a tio n A p p lic a tio n A p p lic a tio n

B u ild ingB lo c k s




Demo

DrawReplicatedTree: shared state


Stats

JGroups has ~ 90KLOC 30KLOC protocols 45KLOC main + building blocks 15KLOC unit tests

~ 90 protocols shipped with JGroupsSet of well-tested stacks (in XML

files)


Available protocols I

Transport UDP, TCP, TCP_NIO, TUNNEL, JMS,

LOOPBACKDiscovery

PING, TCPPING, TCPGOSSIP, UDPPINGGroup membershipReliable delivery & FIFO

NAKACK, SMACK, UNICAST


Available protocols II

Failure detection FD, FD_SOCK, FD_PID, FD_SIMPLE,

FD_PROB, VERIFY_SUSPECTSecurity

ENCRYPT, SSL ConnectionTable (n/a)Fragmentation (FRAG)State transfer (STATE_TRANSFER)


Available protocols III

Ordering FIFO, CAUSAL, TOTAL, TOTAL_TOKEN

Virtual Synchrony FLUSH, QUEUE, VIEW_ENFORCER

Probabilistic Broadcast PBCAST

Merging: MERGE(2), MERGEFAST


Available protocols IV

Distributed message garbage collection STABLE

Debugging PERF, TRACE, PRINTOBJS, SIZE, BSH

Simulation SHUFFLE, DELAY, DISCARD, DEADLOCK,

LOSS, PARTITIONER


Available protocols V

Dynamic configuration AUTOCONF

Flow control FLOW_CONTROL, FC

Misc PIGGYBACK, COMPRESS


Transport

Task Send messages from above to all members in

the group, or to a single member Receive messages from NW, pass up stack

UDP: multicast and multiple UDP unicastTCP: mcast done by multiple TCP unicastsTUNNEL: send to external router, e.g.

through firewall


Discovery

Task Initial discovery of members Used by GMS to determine coordinator to

send JOIN request toEach member returns its own addr,

plus the addr of the coordinator Typical response ({A,A}, {B,A}, {C,A})

Wait for n milliseconds or m responses


Discovery - UDP

Multicast discovery requestEach member responds with a

unicast UDP datagram (local-addr, coord-addr), back to the sender


Discovery - TCPGOSSIPCan be used by both UDP and TCPExternal GossipServer

org.jgroups.stack.GossipServer Maintains table of <group, members> Each member registers (groupname, own

addr) Lease based - members have to

periodically renew registration Multiple GossipServers possible


Discovery - TCPGOSSIP

To obtain initial membership for a given group, TCPGOSSIP contacts the GossipServer

Membership info does not need to be accurate - only goal is to determine coord to send JOIN request to


Discovery - TCPPING

Give a set of well known membersFor discovery, those members are

pingedIf at least 1 responds, we can find

the coordinatorDoes not require additional process


Group Membership

Task Maintain a list of members Notify members when a new member

joins, or an existing member leaves (or crashes)

Each member has the same ordered list List can be retrieved by Channel.getView() First (= oldest) member is coordinator If coord crashes, 2nd oldest takes over


Group Membership - JOIN

New member uses discovery to find coord If first member -> become coord Else: sends JOIN to coord

Coord adds new member to list, multicasts new view (member list) to all members

If 2 initial members are started at the same time, MERGE protocol merges them into a single group


Group Membership - LEAVE

Member sends LEAVE to coordCoord multicasts new view to all

members


Group membership - CRASH

Failure detection protocol sends up SUSPECT event

VERIFY_SUSPECT double checksGMS multicasts new view (not

containing crashed member)If member resurfaces, it will be

shunned Has to leave and rejoin group


Failure detection

Task Detect if a member has crashed and

send SUSPECT event up the stack (to be handled by GMS)

Logical ring over membershipEach member pings its neighbor to the right


Failure detection - FD


Reliable delivery & FIFO

Lossless and FIFO delivery for multicast and unicast messages Multicast: NAK and ACK Unicast: ACK

Missing messages (gaps) are retransmitted Sender resends or Receiver requests retransmission


Encryption

Uses public/private encryption to join new member and get shared group key

Shared key is used to encrypt all messages

Group key is recomputed on joins/leavesSSL ConnectionTable

As alternative, to be used in TCP Uses SSLSocket rather than Socket


Properties configuration

Plain string format "UDP(mcast_addr=228.8.8.8;mcast_port=45566;ip_ttl=32;" +

"mcast_send_buf_size=64000;mcast_recv_buf_size=64000):" + "PING(timeout=2000;num_initial_members=3):" + "MERGE2(min_interval=5000;max_interval=10000):" + "FD_SOCK:" + "VERIFY_SUSPECT(timeout=1500):" +

"pbcast.NAKACK(max_xmit_size=8096;gc_lag=50;retransmit_timeout=600,1200,2400):" + "UNICAST(timeout=600,1200,2400,4800):" + "pbcast.STABLE(desired_avg_gossip=20000):" + "FRAG(frag_size=8096;down_thread=false;up_thread=false):" + "pbcast.GMS(join_timeout=5000;join_retry_timeout=2000;" + "shun=false;print_local_addr=true)"

URL / XML


Advantages of protocol stacks

Each property is implemented by 1 prot Fragmentation, retransmission, ordering

Protocols are assembled into a stackStack has exactly the properties needed

by the appl / required by the networkCan‘t get this with java.net.Socket,

always comes with full TCP/IP


Advantages of protocol stacks

Small scope: a protocol does just one job, but does it well

Protocol stacks are fashionable: Servlet 2.3 filters Interceptors (Corba, JBoss) AOP: separation of concerns, e.g.

fragmentation should not be an application concern


Benefits

Same application code, different protocol stacks (deployment issue)

Application requirements reflected in protocol stack specification

App focuses on domain specific issues


Building Blocks

Replicated CacheNotificationBusGroup RPC


Replicated Cache

Shared state across a groupAny change is replicated to all membersNew members acquire initial state from

coordStructures supported

Tree Hashmap Queues


NotificationBus

Thin layer on ChannelNotifications sent to all membersCallback when notification is

receivedHook for state sharing


Group RPC

Invoke a method call in all membersGet a list of responsesWait for all responses, majority, first,

or none response (use optional timeout)

Handles crashed members correctly (no blocking)


Serverless JMS

JMS based on JGroupsPeer-to-peer architecture rather than C/SClient publishing to a topic

Instead of sending msg to server, and server distributes to multiple clients: publisher multicasts message

JMS Server just another member Handles persistent messages (DB)


Serverless JMS

Publisher

JMS Server

Subscriber

Subscriber

Subscriber

Client/Server Model

Publisher

JMS Server

Subscriber

Subscriber

Subscriber

Serverless Model

Multicast

(discard) (accept)

(accept)

(accept)

(accept)

(discard)

Cost: 4 unicasts Cost: 1 multicast


Serverless JMS

Clients are still able to publish even when server is down

Caveat: works in scenario where client and server are in same multicast-reachable NW

Status Topics/Queues available No TX/XA, no durable subscriptions, no

persistent messages Download (standalone) beta at jboss.org


Where is JGroups used ?

JBoss Clustering

Replication of entity beans, SLSBs and SFSBsHA-JNDICache invalidationSession repl (integrated Tomcat, Jetty)

Serverless JMS Cache

Replicated transactional clustered cache



Jonas appserver (clustering)GroupPac (FT-CORBA impl)GCT: port to .NETReplicated Caching

OpenSyphony OSCache Jakarta Turbine's JCS Swarmcache



Session replication Jetty Tomcat 4.x Work in progress on plugin architecture

for Tomcat 5.xUnofficial ones...


Performance

4 nodes, 1 or 2 senders750MHz SunBlade 1000 512MB, 100MB

switched ethernetJGroups 2.18000 10K msgs, in 200 bursts of 20 (2

senders), sleep after burst = 5ms 451 msgs/s == 4.5MB/s throughput Resident heap size 35MB max (-Xmx128m)


Performance

1.4 billion messages total4 nodes, 2 sendersMessage size = 10KAverage msgs/s: 350Max resident mem: 35M (-Xmx128m)Tests available as part of JG distro

Includes gnuplot scripts to generate graphs


Current and future projects

JBossCache, Serverless JMSPort to J2ME (first version available on

www.jgroups-me.org)hsqldb (HyperSonic) database replicationJCache JSR 107 compliant impl (JBoss

Cache)Potential work on GroupComm JSR

jcluster project on dev.java.net


Links

www.jgroups.org "Papers and Articles": link to IBM

devworks

Questions ?

Reliable Multicasting with JGroups

Documents

group topologyebig

multiple tcp unicaststunnel

apijchannel channel

members of xsend

single memberreceive

new messagenull

new jchannelfile

getmembersmessage msg