Replication in Databases and Distributed Systems - …cs655/lectures/CS655-Louis_Rabiet...Replication in Databases and Distributed Systems Course: CS655 ... Trans Per sec Nb Actions

Eager and lazy replications Databases and distributed systems Streaming application CRDT Hybrid Replication Thoughts CCL

Replication in Databases and Distributed SystemsCourse: CS655

Rabiet Louis

Colorado State University

Thursday 5 December 2013

1 / 43


1 Eager and lazy replications

2 Databases and distributed systems

3 Streaming application

4 CRDTTreedoc: Cooperative editing

5 Hybrid Replication

6 Thoughts

7 CCL

2 / 43


Plan






6 Thoughts

7 CCL

3 / 43


Analysis of replication in DB [GH96]

Hypothesis

All the nodes are suppose to have a copy of the data (nopartition, no scaling up)

Eager and lazy replication are not appropriate solutions [nextslides].

Eager: Synchronization then commit.Lazy: Asynchronous propagation.

Fixed number of objects in the database.

Fixed number of nodes (but may be unavailable).

No delay in communications.

4 / 43



Hypothesis







4 / 43



Hypothesis



Eager: Synchronization then commit.

Lazy: Asynchronous propagation.




4 / 43



Hypothesis







4 / 43



Hypothesis







4 / 43



Hypothesis







4 / 43



Hypothesis







4 / 43


Eager Replication

Just one node.

Node Deadlock Rate ≈ Trans Deadlock Rate ∗ Nb Trans

Prob Deadlock

Lifetime OneTrans

Trans Per sec ∗ Nb Actions ∗ OneAction Time

Prob Deadlock ≈ (Prob Waiting)2 ∗ Nb Trans

5 / 43


Eager Replication

Just one node.


Prob Deadlock

Lifetime OneTrans



5 / 43


Eager Replication

Just one node.


Prob Deadlock

Lifetime OneTrans



5 / 43


Eager Replication

Just one node.


Prob Deadlock

Lifetime OneTrans



5 / 43


Eager Replication

Generalization.

Nb Trans’ = Trans Per sec’ ∗ Lifetime OneTrans’

Trans Per sec ∗ Nb Nodes

Nb Actions ∗ OneAction Time ∗ Nb Nodes

Scalability issue

Nb Trans’ in Nb Nodes2 order

6 / 43


Eager Replication

Generalization.




Scalability issue


6 / 43


Eager Replication

Generalization.




Scalability issue


6 / 43


Eager Replication

Generalization.




Scalability issue


6 / 43


Eager Replication

Generalization.

Eager Deadlock =Prob Deadlock Eager

Lifetime OneTrans’∗ Nb Trans’

Prob Deadlock Eager ≈ (Prob Waiting)2 ∗ Nb Trans’

Scalability issue

Eager Deadlock in Nb Nodes3 order

7 / 43


Eager Replication

Generalization.




Scalability issue


7 / 43


Eager Replication

Generalization.




Scalability issue


7 / 43


Lazy Group Replication

Motivation

scalable.

support disconnection of nodes.

Reminder

Any node can update data (no master) = Group Replication.

Reconciliation of conflict (timestamps are used) is needed (nodeadlock in lazy group schema).

8 / 43



Motivation

scalable.


Reminder



8 / 43



Motivation

scalable.


Reminder



8 / 43



Motivation

scalable.


Reminder



8 / 43



Motivation

scalable.


Reminder



8 / 43



Lazy Reconcialiation Rate = Prob Collision ∗ Nb Nodes

Disconnection Time

Inbound Updates ∗ Outbound Updates

DB Size

(in o(Nb Nodes))

in o(Nb Nodes)

9 / 43



Lazy Reconcialiation Rate = Prob Collision ∗ Nb Nodes

Disconnection Time

Inbound Updates ∗ Outbound Updates

DB Size

(in o(Nb Nodes))

in o(Nb Nodes)

9 / 43


Lazy Master

A master is assigned to each object.

Conflict are in o(Nb Nodes2).

No support for disconnection.

10 / 43


Lazy Master




10 / 43


Lazy Master




10 / 43


Two-Tier Replication

Slight modification of the lazy master replication schema.

Restriction: at most a mobile node is involved in thetransaction.

A mobile (in the case where it is the master) while connect toa base node (that contains a replica of the data) and willupdate his content with this new data.

11 / 43






11 / 43






11 / 43


Plan






6 Thoughts

7 CCL

12 / 43


Motivation

Databases and distributed systems use replication.

One for Performance. One for failure resilience.

Idea is to compare them.

13 / 43


Motivation




13 / 43


Motivation




13 / 43


5 Phases in replication

Request.

Server Coordination.

Execution.

Agreement Coordination.

Response.

14 / 43



Request.


Execution.


Response.

14 / 43



Request.


Execution.


Response.

14 / 43



Request.


Execution.


Response.

14 / 43



Request.


Execution.


Response.

14 / 43


Distributed System replication

Active replication.

Passive replication.

Semi-active replication.

15 / 43



Active replication.



15 / 43



Active replication.



15 / 43


Database replication One operation.

Eager primary copy.

Eager update everywherewith distributed locking.

Eager update everywherebased on atomic broadcast.

Lazy update everywhere.

16 / 43



Eager primary copy.




16 / 43



Eager primary copy.




16 / 43



Eager primary copy.




16 / 43


Database replication Several operations.

Eager primary copy.


Certification replication.

17 / 43



Eager primary copy.



17 / 43



Eager primary copy.



17 / 43


Plan






6 Thoughts

7 CCL

18 / 43


Application [hHBR+04]

Interested in recovery.

Precise > Rollback > Gap.

Streams of data.

Pairs of primary/secondary.

19 / 43





Streams of data.


19 / 43





Streams of data.


19 / 43





Streams of data.


19 / 43


Gap Recovery

In case of failure or partial communication, just ignore thequeries.

Called amnesia.

No recovery.

20 / 43


Gap Recovery


Called amnesia.

No recovery.

20 / 43


Gap Recovery


Called amnesia.

No recovery.

20 / 43


Passive standby

Sending the input queues, operators and output queues to thesecondary node.

To avoid suspension during check-pointing: sweep line.

Strongest property for recovery.

21 / 43


Passive standby




21 / 43


Passive standby




21 / 43


Upstream backup

Upstream nodes act as backup for downstream nodes.

22 / 43


Upstream backup


22 / 43


Upstream backup


22 / 43


Active standby

Same as passive standby except that upstream secondarynodes receive the tuples and are processing it.

Because of non deterministic operators, input must berecorded to keep the relation input,output.

23 / 43


Active standby

Same as passive standby except that upstream secondarynodes receive the tuples and are processing it.

Because of non deterministic operators, input must berecorded to keep the relation input,output.

23 / 43


Some extensions to make precise recovery

Passive standby: verification with downstream nodes.

Active standby: distribute information to replay operations.

Upstream Backup: using the watermarks.

24 / 43






24 / 43






24 / 43


Experimentation

recovery time bandwith overhead processing overhead

Amnesia 0 0 0

Passive K + Qp f1(1/M,C) f2(1/M,C)

Upstream K + Q ′p f3(1/M, c) f4(1/M, ops, paths)

Active ε 100% + f3(1/M, c) 100% + 2 ∗ f4(1/M, ops, paths)

25 / 43


Plan






6 Thoughts

7 CCL

26 / 43


CRDT

Commutativity =⇒ Convergence simpler

27 / 43


CRDT

SEC (Strong eventual Consistency).

Eventual delivery ∀i , j , f ∈ ci =⇒ 3f ∈ cjStrong convergence ∀i , j , ci ∈ cj =⇒ si ≡ sjTermination

Without any conflict resolution.

n − 1 failures possibles.

28 / 43


CRDT


Eventual delivery ∀i , j , f ∈ ci =⇒ 3f ∈ cj

Strong convergence ∀i , j , ci ∈ cj =⇒ si ≡ sjTermination



28 / 43


CRDT


Eventual delivery ∀i , j , f ∈ ci =⇒ 3f ∈ cjStrong convergence ∀i , j , ci ∈ cj =⇒ si ≡ sj

Termination



28 / 43


CRDT





28 / 43


CRDT





28 / 43


CRDT





28 / 43


Examples of CDRT

Vectors counters.

(Nn, [0, ..., 0],≥n, value, inc,maxn)

if pi can only update the i-th element in the vector ≡ vector-clock

29 / 43


Sufficient condition

Eventual delivery

Termination

Monotonic semi-lattice

=⇒ SEC

30 / 43


Treedoc: Cooperative editing

Treedoc [PMSL09]

c

b

a

0

0

e

d

0

f

1

1

31 / 43



Treedoc [PMSL09]

c

b

a

0

0

e

d

0

f

1

1

dC

dB

dA

dE

dD dF

32 / 43



Treedoc [PMSL09]

33 / 43


Plan






6 Thoughts

7 CCL

34 / 43


Hybrid Quorum [CML+06]

Use a quick quorum if no contention.

BFT [CL02] if contention.

Combine quorum and agreement based state-machinereplication.

35 / 43






35 / 43






35 / 43


Normal Case

Write operations = 2 phases.

Read operations = 1 phase

3f + 1 replica.

Phase Write1 = timestamp t obtained using a quorum.

Phase Write2 = Execution (2f + 1 must agree to execute theoperation at t).

36 / 43


Normal Case



3f + 1 replica.



36 / 43


Normal Case



3f + 1 replica.



36 / 43


Normal Case



3f + 1 replica.



36 / 43


Normal Case



3f + 1 replica.



36 / 43


Contention

if Write2 has been completed, it is still assigned.

if Write1 has obtained t then the operation will be executedat t ′ ≥ t

37 / 43


Contention

if Write2 has been completed, it is still assigned.

if Write1 has obtained t then the operation will be executedat t ′ ≥ t

37 / 43


Plan






6 Thoughts

7 CCL

38 / 43


CRDT

3 Applicable only on commutative operations.

3 Not many use cases?

39 / 43


CRDT

3 Applicable only on commutative operations.

3 Not many use cases?

39 / 43


Streaming

3 Number of replica = 2.

Checkpointing.

Load Balancing between the two nodes.

Several loops (image processing).

Path exploration.

40 / 43


Streaming


Checkpointing.



Path exploration.

40 / 43


Streaming


Checkpointing.



Path exploration.

40 / 43


Streaming


Checkpointing.



Path exploration.

40 / 43


Streaming


Checkpointing.



Path exploration.

40 / 43


Plan






6 Thoughts

7 CCL

41 / 43


Conclusion

Thank You for your attention !

42 / 43

Bibliography

Miguel Castro and Barbara Liskov.

Practical byzantine fault tolerance and proactive recovery.ACM Trans. Comput. Syst., 20(4):398–461, November 2002.

James Cowling, Daniel Myers, Barbara Liskov, Rodrigo Rodrigues, and Liuba Shrira.

Hq replication: A hybrid quorum protocol for byzantine fault tolerance.In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, OSDI ’06, pages177–190, Berkeley, CA, USA, 2006. USENIX Association.

Jim Gray and Pat Helland.

The dangers of replication and a solution.In In Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, pages173–182, 1996.

Jeong hyon Hwang, Magdalena Balazinska, Er Rasin, Michael Stonebraker, and Stan Zdonik.

High-availability algorithms for distributed stream processing.Technical report, 2004.

Nuno Preguica, Joan Manuel Marques, Marc Shapiro, and Mihai Letia.

A commutative replicated data type for cooperative editing.In Proceedings of the 2009 29th IEEE International Conference on Distributed Computing Systems, ICDCS’09, pages 395–403, Washington, DC, USA, 2009. IEEE Computer Society.

Marc Shapiro, Nuno Preguica, Carlos Baquero, and Marek Zawirski.

Conflict-free replicated data types.In Proceedings of the 13th International Conference on Stabilization, Safety, and Security of DistributedSystems, SSS’11, pages 386–400, Berlin, Heidelberg, 2011. Springer-Verlag.

Matthias Wiesmann, Fernando Pedone, Andre Schiper, Betina Kemme, and Gustavo Alonso.

Understanding replication in databases and distributed systems.pages 264–274, 2000.

Replication in Databases and Distributed Systems - …cs655/lectures/CS655-Louis_Rabiet...Replication in Databases and Distributed Systems Course: CS655 ... Trans Per sec Nb Actions

Documents