Sib: Alternatives of stateful replication in applicati on servers Huaigu WU Mcgill University February, 2003
Dec 19, 2015
Sib:Alternatives of stateful replication
in application servers
Huaigu WU
Mcgill University
February, 2003
Purpose of Sib
• An alternative approach to implement replication in application server– Abstract level (derived from the J2EE specific
ation)– Focus on Stateful Session Bean– Will extend to Stateless Session Bean and En
tity Bean
Outline
• Sib VS. JBoss implementation– Simple Calls– Calling Hierarchies– Direct Database Acce
ss
• Compare– Normal process– Site crash and failove
r
Application Server
Client
SFB 1
1 2
Application Server
Client
SFB 1
1 4
Data
2 3
Application Server
Client
SFB 1
1 4
SFB 22
3
Assumption
• We don’t consider partition.• Abstract component architecture
– Component in this talk consists of client, Stateful Session Bean (SFB) and third-party database
• Third-party database will be used without modification
• Deterministic component for basic analysis• One primary, one (or more) backup• So far, we assume every component is single thr
eaded
Normal Processing in Simple Calls (JBoss)
• Only primary receives and processes request
• Replicate primary’s up-to-date state to backups at the end of every request before returning its reply
• Backup updates state after every request
Backup Pri mary
Client
Cluster
SFB 1 SFB 1
1 43 6
2 5
Normal Processing in Simple Calls (Sib)
• At the beginning, states of all replicas are the same
• request is broadcast to both primary and backup
• Primary processes requests• Backup receives requests• Occasionally (periodically),
Primary’s state is replicated to backup– Replication Point
(operational quiescence )• Independent for
component• Deterministic component:
it could be any time point
Pri mary
Client
Cluster
SFB 1 SFB 1
Backup
4'
1
2
3
4
0
Comparison• Network overhead for one component
– Message types:• Request (req), Reply (rep), Replicated state information (s)
– Message size: s >> req • JBoss analysis: s = 2-3 KByte
– Message number (N is number of sites)• JBoss: 1 request: 1 req + 1 rep + N*s• Sib: 1 requset: N req + 1 rep + q * N * s (q is the frequency of repli
cation, q<=1)• Compare: JBoss > Sib
• Backup’s CPU overhead– JBoss: receive and install state– Sib: receive requests– Assume install state > install request, so JBoss > Sib
Site Crash and Failover for Simple Calls (JBoss)
• Characters– When primary fails, client resends
request to the new primary• Issues
– Resend must be repeatable • Content
– Duplicated requests must be identified by SFB
Cluster
Client
SFB 1
Primary
1
SFB 1
Backup
Resend 1
3
2
Cluster
Client
SFB 1
Primary
1
SFB 1
Backup
Resend 1
3
2
Cluster
Client
SFB 1
Primary
1
SFB 1
Backup2
Resend 1
3
Identify duplicated request 1(Does JBoss do it ?)
Site Crash and Failover for Simple Calls (Sib)
• Characters– When primary fails, the new
primary automatically re-executes all requests starting from last replication point
– Client does not need to resend request
• Issues– Duplicated replies must be
identified by client– Replies must be deterministic
Cluster
Client
SFB 1
Primary
SFB 1
Backup
2
2
Identify duplicated reply 2(testable result)
Execute request 1
1
Cluster
Client
SFB 1
Primary
SFB 1
Backup
2 Execute request 1
1
Normal Processing in Calling Hierarchies
JBoss Sib
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
1 6
2 4
3
5
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
4
3
12
0
0
3'
Site Crash and Failover for Calling Hierarchies (JBoss)
• Assumption– Node fails (all beans in the node fail)
• Characters– Similar to simple calls for one
component
• Issues– Duplicated internal
requests must be identified by SFB
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
1
2
Resend 1
6
3
45
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
1
2
Resend 1
2
6
5
3
4
Duplicatedinternal
request 2
4
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
1
2
Resend 1
2 4
3
6
5
Site Crash and Failover for Calling Hierarchies (Sib)
• Assumption:– Node fails (all beans in the node fail)
• Character:– Every backup SFB re-executes
requests starting from its own last replication point
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
4
3 7 7632
8
3'
Duplicatedreply 8
2 6
0
0
Discardduplicated
internalrequests
Identifyduplicated
internalrequests
Duplicatedreply 4
1 5
• Issues– Duplicated internal requests
must be identified by SFB
– Duplicated internal replies must be identified by SFB
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
4
3 7 7
8
Duplicatedreply 81 5
2 6
0
0 4'
Re-executerequest 2,6
Re-executerequest 5
3 6
Duplicatedinternalrequest
Duplicatedinternalreplies
BackupPrimary
Client
Cluster
SFB 1 SFB 1
SFB 2 SFB 2
4
3 7 7
8
Duplicatedreply 81 5
2 6
0
0 4'
Re-executerequest 6
Re-executerequest 5
6
Duplicatedinternalrequest
3'
Direct Database Access (JBoss)
• Normal process
Cluster
Client
SFB 1
Primary
1
SFB 1
Backup
Data
Write and com
mit
Ack
Resend 1
3
2
Possible duplicateddatabase accessW
rite
and
com
mit
• Failover– NO “Exactly-once” Semantic
Cluster
Client
SFB 1
Primary
1
SFB 1
Backup
Data
Write and com
mit
2
3
Ack
Resource Manager(Resource Adapter)
Direct Database Access (Sib)
• Normal process– Data bind to specified
Resource Manager
– All requests to some data is forwarded to its specified RM
– Focus on write operation
Data
BackupPrimary
Client
Cluster
SFB 1 SFB 1
RM RM
2
Ack
1
0
0
SQ
L
Direct Database Access (Sib)• Failover suggestion 1
– Characters• Write bind to commit• Converse Undo, start from
the last write.• If original write is
successful, undo rollback it.• If original write is failed,
undo has no effect.– Issues
• Tables must have primary key to be able to generate undo
• In case primary committed, other access between commit and undo might be possible
– Conclusion• Simple during normal
process• Complex during recovery
Data
BackupPrimary
Client
Cluster
SFB 1 SFB1
RM RM
10
0
Write &
Com
mit U
ndo
toro
llbac
k(r
epea
tabl
e)A
ckW
rite
& C
omm
itA
ck
AckSQL
2
Re-executerequest 1
Duplicated internalSQL request
Re-executeSQL request
SQ
L
Generateundo
Direct Database Access (Sib)
• Failover Suggestion 2– Characters
• Insert Additional mark before every commit
• If mark exists, redo is not required
• If mark does not exist, redo
– Issues• additional insert
overhead In normal process
• RM must maintain global transaction ids
– Conclusion• Complex during normal
process• Simple and correct
during recovery Data
BackupPrimary
Client
Cluster
SFB 1 SFB1
RM RM
1
0
0
Write A
ckR
edo,
if no
mar
k
AckSQL
2
Re-executerequest 1
Duplicated internalSQL request
SQ
L
Insert
Mark
comm
it
Che
ck M
ark
MarkExist?
Summary• Sib’s Advantage
– Better performance• State information (including response information) is much
bigger than request• Infrequent replication points enough since EJB is short-living
so that failure is unlikely
– More powerful for “Exactly-once” database access
• Sib’s Disadvantage– More complicated recovery– Duplicated external responses
• Client needs to remove duplicated responses
Current Work• Compare schemas for complex transactions
– A transaction crosses several requests– Entity beans replication in complex transactions
• Multi-Thread: Typical Non-deterministic environment– Multi-Thread Client / Single-Thread SFB
• JBoss: – Repeatable request might be problematic (e.g. resend in different orde
r)
• Sib– Avoid resend requests– Total order o requests in all replicas
– Multi-Thread Client / Multi-Thread SFB• Both JBoss and Sib might be problematic
– Sib: special concurrency control and synchronous mechanism
• Component Failure
Problematic Example under Multi-Thread Environment
Cluster
Client
Primary
1
Backup
Resend 2
2
2
X+3X-3
Resend 1
S2
X=0
X=3
X=-3
X=0
S2 S1
X-3
Reference• [1] W.Zhao, L.E.Moser and P.M.Melliar-smith, “Unification of Replication and Transaction P
rocessing in the Three-Tier Architectures”, 22nd International Conference on Distributed Computing Systems (ICDCS'02), Vienna, Austria, July 02 - 05, 2002.
• [2] Marcia Pasin, Michel Riveill and Taisy Silva Weber, “High-Available Enterprise JavaBeans Using Group Communication System Support”, Proceedings of the Fourth European Research Seminar on Advances in Distributed Systems - ERSADS 2001, pages 161-166, University Residential Center of University of Bologna, Berliner (Frolic), Italy, 14-18 May 2001.
• [3] Roger Barga, David Lomet, Gerhard Weikum, “Recovery Guarantees for General Multi-Tier Aplications”, Proceedings of the 18th International Conference on Data Engineering (ICDE'02)
• [4] P.Narasimhan, L.E.Moser and P.M.Melliar-smith, “State Synchronization and Recovery for Strongly Consistent Replicated CORBA Objects”, International Conference on Dependable Systems and Networks, Göteborg, Sweden (July 2001).
• [5] N.Narasimhan, L.E.Moser and P.M.Melliar-smith, “Transparent Consistent Replication of Java RMI Object”, DOA 2000: 17-26
• [6] Svend Frølund, Rachid Guerraoui, “e-Transactions: End-to-End Reliability for Three-Tier Architectures”, TSE 28(4): 378-395 (2002)
• [7] Svend Frølund, Rachid Guerraoui, “Implementing e-Transactions with Asynchronous Replication”, IEEE Transactions on Parallel and Distributed Systems 12(2): 133-146 (2001)
• [8] Object Management Group, “Fault Tolerant CORBA Specification”. December 1999, OMG Technical Committee Document (orbos/00-04-04)
Reference• [9] IONA Technologies PLC, “White Paper Orbix E2A Application Load Balancing and
Fault Tolerance”, online documentation, http://www.iona.com/forms/wprequest.htm, April, 2002.
• [10] BEA WebLogic Server™, “Programming WebLogic Enterprise JavaBeans”, Copyright © 2000 BEA Systems, Inc. All rights reserved.
• [11] Gerhard Weikum, Gpttfried Vossen, “Transactional Information Systems: Theory. Algorithms, and the Practice of Concurrency Control and Recover”, Morgan Kaufmann Publishers, January 2002.
• [12] Ed Roman, Scott Ambler and Tyler Jewell, “Mastering Enterprise JavaBeans second edition”, Wiley Computer Publishing, John Wiley & Sons, Inc., ISBN: 0-471-41711-4, December 2001.
• [13] Sun Microsystems Inc., “Java™ 2 Platform Enterprise Edition Specification, v1.3”, online documentation, http://java.sun.com/j2ee/, Oct. 20, 2000.
• [14] Sun Microsystems Inc., “Enterprise JavaBeansTM Specification, Version 2.0”, online documentation, http://java.sun.com/j2ee/, Oct. 23, 2000.
• [15] R. Jimenez-Peris, M. Patino-Martinez, “Deterministic Scheduling and Online Recovery for Replicated Multithreaded Transactional Servers”, WDMS 2002, June 26, 2002.
• [16] Roman Vitenberg, Idit Keidar, Gregory V.Chockler and Danny Dolev, “Group Communication Specifications: A Comprehensive Study”, In ACM Computing Surveys 33(4), pages 1-43, December 2001.
• [17] Sacha Labourey, Bill Burke, “JBoss Clustering”, The JBoss Group, November 2002
• [18] Andreas Schaefer, “JBoss 3.0 Quick Start Guide”, The JBoss Group, July 2002