Minimizing Coordination in Replicated systems
Cheng Li
Max Planck Institute for Software Systems
Joint work with João Leitão, Allen Clement, Nuno Preguiça, and Rodrigo Rodrigues
MPI-SWS/NOVALINCS
Fast Internet services
Build replicated systems that are fast and correct
4/13/2015 Cheng Li @ PaPoC'15 2
[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]
MPI-SWS/NOVALINCS
Fast Internet services
Build replicated systems that are fast and correct
4/13/2015 Cheng Li @ PaPoC'15
Blue ops: no coordination, weak consistency
3
[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]
MPI-SWS/NOVALINCS
Fast Internet services
Build replicated systems that are fast and correct
4/13/2015 Cheng Li @ PaPoC'15
State convergence
Invariant preservation
Blue ops: no coordination, weak consistency
4
[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]
MPI-SWS/NOVALINCS
Fast Internet services
Build replicated systems that are fast and correct
4/13/2015 Cheng Li @ PaPoC'15
Blue ops: no coordination, weak consistency
Red ops: coordination, stronger consistency
5
State convergence
Invariant preservation
[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]
MPI-SWS/NOVALINCS
How to coordinate red ops?
• Totally order all of them– simple but sometimes too conservative/costly
• Observation: a red op only needs to be ordered w.r.t other relevant red ops.
• E.g., auction service– Bid + close auction operations
– Invariant: highest accepted bid is declared winner
4/13/2015 Cheng Li @ PaPoC'15 6
MPI-SWS/NOVALINCS
bidder price
Replicated auction service
7Cheng Li @ PaPoC'154/13/2015
UKUS
bid($10)
winner bidder price
Bob 10
winner bidder price
MPI-SWS/NOVALINCS
bidder price
Replicated auction service
8Cheng Li @ PaPoC'154/13/2015
UKUS
winner bidder price
Bob 10
winner bidder price
bid($15)
bidder price
Alice 15
MPI-SWS/NOVALINCS
bidder price
Replicated auction service
9Cheng Li @ PaPoC'154/13/2015
UKUS
winner bidder price
Bob 10
winner bidder pricebidder price
Alice 15
close()
winner
Bob
MPI-SWS/NOVALINCS
Replicated auction service
10Cheng Li @ PaPoC'154/13/2015
UKUS
Bob won even with a lower bid than Alice.
bidder price
Bob 10
winner bidder price
Alice 15
winner
Bob
bidder price
Alice 15
Bob 10
bidder price
Alice 15
Bob 10
winner
Bob
MPI-SWS/NOVALINCS
Less coordination is better!
4/13/2015 Cheng Li @ PaPoC'15 11
bid
bid
close
bid
bid
bid
close
bid
bid
MPI-SWS/NOVALINCS
Less coordination is better!
4/13/2015 Cheng Li @ PaPoC'15 12
bid
bid
close
bid
bid
bid
close
bid
bid
Visibility restriction
A partial order enforcing visibility
restrictions
Pros: fine-grained tuning in consistency semantics (the amount of coordination) using restrictions
MPI-SWS/NOVALINCS
Outline
• Background
• Partial order-Restriction (PoR) consistency– Visibility restrictions
– Consistency model
• Restriction inference
• Conclusion
4/13/2015 Cheng Li @ PaPoC'15 13
MPI-SWS/NOVALINCS
Outline
• Background
• Partial order-Restriction (PoR) consistency– Visibility restrictions
– Consistency model
• Restriction inference
• Conclusion
4/13/2015 Cheng Li @ PaPoC'15 14
MPI-SWS/NOVALINCS
Visibility restrictions
• A restriction between two operations implies that one must see effects introduced by the other.
• For operation 𝑎, 𝑏, the restriction 𝑟 𝑎, 𝑏 implies that 𝑎 ≺ 𝑏 𝑏 ≺ 𝑎 w.r.t any partial order ≺.
4/13/2015 Cheng Li @ PaPoC'15 15
a0
b0
a1
a2
b1
b2
If 𝑎 ≺ 𝑏 𝑏 ≺ 𝑎,
then 𝑟 𝑎, 𝑏 is met in ≺.
MPI-SWS/NOVALINCS
Partial order-restrictions (PoR) Consistency
• A replicated system 𝑆 is associated with a set of restrictions 𝑅𝑠.
– e.g., 𝑅𝑠 = {𝑟(𝑎, 𝑏), 𝑟(𝑐, 𝑑)}
• 𝑆 is PoR Consistent if, for any its executions, there exists an admissible partial order, where all restrictions in 𝑅𝑠 are met.
4/13/2015 Cheng Li @ PaPoC'15 16
MPI-SWS/NOVALINCS
Outline
• Background
• Partial order-Restriction (PoR) consistency– Visibility restrictions
– Consistency model
• Restriction inference
• Conclusion
4/13/2015 Cheng Li @ PaPoC'15 17
MPI-SWS/NOVALINCS
Challenges
• Which restrictions are sufficient?– They must ensure relevant properties.
• Does the set of restrictions comprise a minimal coordination plan?– i.e., we cannot remove any restriction from the set.
4/13/2015 Cheng Li @ PaPoC'15 18
MPI-SWS/NOVALINCS
Outline
• Background
• Partial order-Restriction (PoR) consistency
• Restriction inference– State convergence
– Invariant preservation
• Conclusion
4/13/2015 Cheng Li @ PaPoC'15 19
MPI-SWS/NOVALINCS
State convergence
• Definition: all replicas reach the same final state when the system becomes quiescent.
• Insight: any pair of operations must be ordered if they don’t commute.
4/13/2015 Cheng Li @ PaPoC'15 20
MPI-SWS/NOVALINCS
Invariant preservation
• Insight: for any violation, add restrictions among concurrent operations
4/13/2015 Cheng Li @ PaPoC'15 21
bid(“Bob”, 10)
bid(“Alice”, 15)
close()
winner bidder price
winner
Bob
bidder price
Alice 15
Bob 10
MPI-SWS/NOVALINCS
Invariant preservation (cont.)
• Insight: for any violation, add restrictions among concurrent operations
4/13/2015 Cheng Li @ PaPoC'15 22
bid(“Bob”, 10)
bid(“Alice”, 15)
close()
winner bidder price
winner
Bob
bidder price
Alice 15
Bob 10
Q: Can we remove any concurrentop while maintaining the violation?
MPI-SWS/NOVALINCS
Invariant preservation (cont.)
• Insight: for any violation, add restrictions among concurrent operations
4/13/2015 Cheng Li @ PaPoC'15 23
bid(“Bob”, 10)
bid(“Alice”, 15)
winner bidder price
winner bidder price
Alice 15
Bob 10
Q: Can we remove any concurrentop while maintaining the violation?A: close must be there.
MPI-SWS/NOVALINCS
Invariant preservation (cont.)
• Insight: for any violation, add restrictions among concurrent operations
4/13/2015 Cheng Li @ PaPoC'15 24
bid(“Bob”, 10)
bid(“Alice”, 15)
close()
winner bidder price
winner
Bob
bidder price
Alice 15
Bob 10
Q: Can we remove any concurrentop while maintaining the violation?
MPI-SWS/NOVALINCS
Invariant preservation (cont.)
• Insight: for any violation, add restrictions among concurrent operations
4/13/2015 Cheng Li @ PaPoC'15 25
bid(“Alice”, 15)
close()
winner bidder price
Bob 10
winner
Bob
bidder price
Alice 15
Bob 10
Q: Can we remove any concurrentop while maintaining the violation?A: Yes, bid(“Bob”, 10).
MPI-SWS/NOVALINCS
Invariant preservation (cont.)
• Insight: for any violation, add restrictions among concurrent operations
4/13/2015 Cheng Li @ PaPoC'15 26
bid(“Alice”, 15)
close()
winner bidder price
Bob 10
winner
Bob
bidder price
Alice 15
Bob 10
This is a minimal violating execution.Solution: 𝑟(𝑏𝑖𝑑, 𝑐𝑙𝑜𝑠𝑒)
Analysis:Step 1: For any such an execution, we add a restriction over a pair of its maximaloperations. Step 2: Iterate all these executions
MPI-SWS/NOVALINCS
Take-home points
• Consistency semantics can be captured by visibility restrictions over operations.
• Invariant and commutativity analysis finds the minimal set of restrictions to preserve convergence and invariants.
• PoR Consistency enforces all restrictions throughout all executions of a replicated system.
4/13/2015 Cheng Li @ PaPoC'15 27
Minimizing Coordination in Replicated systems
Thanks for your attention!