Ordered Communication
Feb 01, 2016
Ordered Communication
Ordered Communication
Define guarantees about the order of deliveries inside group of processes
Type of ordering: Deliveries respect the FIFO ordering of the corresponding sendingsDeliveries respect the Causal ordering of the corresponding sendingsDelivery respects a total ordering of deliveries (atomic communication)
Advantages of ordered communication
Orthogonality wrt reliable communication. Reliable broadcast does not have any property on ordering deliveries of messagesThis can cause anomalies in many applicative contexts
“Reliable ordered communication” are obtained adding one or more ordering properties to reliable communication
Example: flight booking system. Consider the message pattern depicted in the figure. The server cansel a reservation that has never been done!
tclient
server
“reserve” “cancel”
“Prices 15% off”
FIFO Broadcast\ specificationMessages sent by the a process has to be delivered in the sending order.
FIFO Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of SafetySafety that captures the notion of order (example):
FIFO Order: if a process sends a broadcast message m before m’, then no correct process delivers m’ if it has not already delivered m.
FIFO order can be uniform/non uniform
FIFO Broadcast = Reliable Broadcast + FIFO Order
Each process q holds: S p
a count of messages broadcast by p Rp the sequence number of the latest message sent by p and delivered by q
For p to FO-multicast a message to g, it piggybacks S p
on the message, rbBroadcasts it and increments S p
by 1
On receipt of a message from q sent by p with sequence number S, p checks whether
S = Rp + 1. If so, q FO-delivers it
if S > Rp + 1 then q places message
in hold-back queue until intervening messages have been delivered. (note that rbBroadcast does eventually deliver messages unless the sender crashes)
Messageprocessing
Delivery queueHold-back
queue
deliver
Incomingmessages
When delivery guarantees aremet
FIFO Broadcast\algorithm
Advantages of Ordered Communication (2)
Prof.
Student 2
m1: “Fri exam cancelled”
Student 1m2:
“let’s party on Thu night”
m3: “but we have an exam on Fri!”
Causal Order FIFO Order, But FIFO OrderCausal Orderthus, Causal Order = FIFO Order + ?
Causal Broadcast\specification
Causal Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of SafetySafety that captures the notion of order (example):
Causal Order: if the sending of a message m causally precedes the sending of a message m’, then every correct process has to deliver m before delivering m’.
Causal Broadcast = Reliable Broadcast+Causal Order
Causal Broadcast\specification
Causal Order = FIFO Order + Local Order.
Local Order: if a process delivers a message m before sending a msg m’, then no correct process deliver m’ if it has not already delivered m.
Example:
p
q
r
tm
m’
The delivery of m’ is delaied till the arrival and the delivery of m
Causal Broadcast\implementations
Two implementations blocking algorthm using vector clocks (already discussed)non-blocking algorithm using piggybacking of causal past
p1
p2
p3
COBcast(m1)
CObcast(m2)
COBcast(m3)
m1
m2
m1,m2 , m3
COdelv(m1)
COdelv(m3)m2 già COdelivered!
COdelv(m1)
COdelv(m1)COdelv(m2)
COdelv(m2) COdelv(m3)
COdelv(m3)
COdelv(m2)Filter out
Advantages of Ordered Communication (3)
Causal Order is not enough strong to avoid anomaliesEs. banking. Bank account replicated on two sites
R1
R2
A:£100
A:£100
Deposit £20
Add 10% interest
A:£120
A:£110
A:£132
A:£130
Despite the fact that replicas initially share the same state, the state reaches a different value in the two sites at the end of the exexution as shown in Figure. Note that the computation is Causally Ordered
To guarantee values of replicas be the same, one has to ensure that the order of delivery be the same at each process. In the above example R1 delivers m1 before m2 and R2 delivers m2 before m1.
Note that ensuring the same delivery order at each replicas does not look at the sending order of messages
m1
m2
Atomic Broadcast\specification
Atomic (Total) Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of SafetySafety that captures the notion of total order (example):
Total Order: if two correct processes p and q deliver m and m’, then p delivers m before m’ if, and only if, q delivers m before m’
Total order is orthogonal with respect to FIFO and Causal Order. Total order would accept indeed a computation in which a process sends n messages to a group, and each of the processes of the group delivers such messages in the reverse order of their sendings. The computation is totally ordered but it is not FIFO.
Causal Atomic broadcast
Hierarchy of Broadcast Specifications
Reliable broadcast
FIFO broadcast
Causal broadcast
FIFO Atomic broadcast
Atomic broadcast
Total Order
Causal Order
Total Order
Total Order
FIFO Order FIFO Order
Local OrderCausal OrderLocal Order
System model
Static set of processes Π = {p1 … pn}
Message passing over perfect channels (message exchanging between correct processes is reliable)AsynchronousCrash fault model for processesWe characterize the system in terms of its possible runs R
Rp1
p2
pn
TOcast(m)
m
m
m
TOdeliver(m)
crash
r
A few notation
Property P: predicate on the system, identifying a set of runs RP R
P P’ iff RP RP’
Specification S(P1,…,Pm): logical and of m properties, identifying a set of runs RS=RP1
∩ … ∩ RPm R
S → S’ iff RS RS’
RP RP’
RS RS’
RP1RPnRS
R
R
R
TO specifications
Total order specifications are usually composed by four properties, namely Validity, Integrity,Agreement, and Order. A Validity property guarantees that messages sent by correct processes will eventually be delivered at least by correct processes; An Integrity property guarantees that no spurious or duplicate messages are delivered; An Agreement property ensures that (at least correct) processes deliver the same set of messages; An Order property constrains (at least correct) processes delivering the same messages to deliver them in the same order.
TO specifications
Total Order Broadcast = S(V,I,A,O)V = ValidityI = IntegrityA = AgreementO = Order
Distinct specifications arise from distinct formulations of each property
uniform vs non-uniformA uniform property imposes restrictions on the behavior of (at least) correct processes on the basis of events occurred in some process
NUVUI
TO(A,O)
TO Specifications
Crash failure + Perfect channels NUV. if a correct process TOCAST a message m then some correct process will eventually deliver mUI. For any message m, every process p delivers m at most once and only if m was previously tocast by some (correct or not) process.
The Agreement property
(Uniform Agreement, UA) If a process (correct or not) todelivers a message m, then all correct processes will eventually todeliver m;(Non-uniform Agreement, NUA) If a correct process todelivers a message m, then all correct processes will eventually todeliver m
The Agreement property
Constrains the set of delivered messages
Correct processes always deliver the same set of messages MEach faulty process p delivers a set Mp
UA: Mp M
NUA: Mp can be s.t. Mp - M ≠
m2
m4
p1
p2
p3
m2m4
m1
m1
m3
m3
m3
m4
m1
m2
UAm4
p1
p2
p3
m2m4
m1
m1
m3
m3
m3
m4
m1
m2
m5
NUA
The Order property
Constrains the order of message deliveries and possibly the set of delivered messages
SUTO: if p delivers m<m’, q delivers m’ only after m
same ordersame prefix of the set of delivered messagesafter an omission, disjoint sets of delivered messages
WUTO: if p,q deliver m,m’, they get the same order
no restrictions on the set of delivered messages
p1
p2
p3
m2
m2
m2
m1
m1
m1m4
m3
m3
m7
m6
m5
p1
p2
p3
m2
m2m1
m1
m1m4
m3
m3
m7
m6
m5
SUTO WUTO
The Order property (2)SUTO and WUTO are uniformThey both have a non-uniform counterparts: SNUTO and WNUTO (Strong Non-uniform Total Order, SNUTO). If some correct process todelivers some message m before message m', then a correct process todelivers m‘ only after it has todelivered m.
(Weak Non-uniform Total Order, WNUTO) If correct processes p and q both todeliver messages m and m', then p todelivers m before m' if and only if q todelivers m before m‘
The Order property (2)
SUTO WUTOSNUTO WNUTO
p1
p2
p3
m1
m2
m2
m1
m1
m2m4
m3
m3
m7
m6
m5
SNUTO
p1
p2
p3
m1
m2m1
m1
m2m4
m3
m3
m7
m6
m5
WNUTOm2
TO specifications
TO(UA,SUTO)The strongest TO spec.
p1
p2
p3
m2
m2
m2
m1
m1
m1
m3
m3
m6
m6
p1
p2
p3
m2
m2
m2
m1
m1
m1m4
m3
m3
m6
m6
m5
TO(NUA,SUTO) TO(UA,SUTO)(Strongest total order)
TO(NUA,SUTO)
TO specifications (2)
TO(UA,WUTO)
m3
p1
p2
p3
m2
m2
m1
m1
m1
m3
m3
m4
m4
m4
m3
p1
p2
p3
m2m1
m1
m1
m3
m4
m4
m2m3 m4
m5
m6
m6
m6
m2
m2
TO(NUA,WUTO)
TO(UA,WUTO)
TO(UA,SUTO)(Strongest total order)
TO(NUA,SUTO)
TO(NUA,WUTO)
TO specifications (3)
TO(UA,WNUTO)
m4
p1
p2
p3
m2
m2
m1
m1
m1
m3
m3
m3
m4
m4
m2
TO(NUA,WNUTO)
m4
p1
p2
p3
m2m1
m1
m1
m3
m3
m4
m2m3 m4
m5
m6
m6
m6m2 TO(NUA,WNUTO)
TO(UA,WNUTO)
TO(UA,SUTO)(Strongest total order)
TO(NUA,SUTO)TO(UA,WUTO)
TO(NUA,WUTO)