Minimizing Coordination in Replicated Systems

Minimizing Coordination in Replicated systems

Cheng Li

Max Planck Institute for Software Systems

Joint work with João Leitão, Allen Clement, Nuno Preguiça, and Rodrigo Rodrigues

MPI-SWS/NOVALINCS

Fast Internet services

Build replicated systems that are fast and correct

4/13/2015 Cheng Li @ PaPoC'15 2

[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]

MPI-SWS/NOVALINCS

Fast Internet services

Build replicated systems that are fast and correct

4/13/2015 Cheng Li @ PaPoC'15

Blue ops: no coordination, weak consistency

3

[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]

MPI-SWS/NOVALINCS

Fast Internet services

Build replicated systems that are fast and correct

4/13/2015 Cheng Li @ PaPoC'15

State convergence

Invariant preservation

Blue ops: no coordination, weak consistency

4

[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]

MPI-SWS/NOVALINCS

Fast Internet services

Build replicated systems that are fast and correct

4/13/2015 Cheng Li @ PaPoC'15

Blue ops: no coordination, weak consistency

Red ops: coordination, stronger consistency

5

State convergence

Invariant preservation

[Making Geo-Replicated Systems Fast as Possible, Consistent when necessary, OSDI’12]

MPI-SWS/NOVALINCS

How to coordinate red ops?

• Totally order all of them– simple but sometimes too conservative/costly

• Observation: a red op only needs to be ordered w.r.t other relevant red ops.

• E.g., auction service– Bid + close auction operations

– Invariant: highest accepted bid is declared winner

4/13/2015 Cheng Li @ PaPoC'15 6

MPI-SWS/NOVALINCS

bidder price

Replicated auction service

7Cheng Li @ PaPoC'154/13/2015

UKUS

bid($10)

winner bidder price

Bob 10

winner bidder price

MPI-SWS/NOVALINCS

bidder price

Replicated auction service

8Cheng Li @ PaPoC'154/13/2015

UKUS

winner bidder price

Bob 10

winner bidder price

bid($15)

bidder price

Alice 15

MPI-SWS/NOVALINCS

bidder price

Replicated auction service

9Cheng Li @ PaPoC'154/13/2015

UKUS

winner bidder price

Bob 10

winner bidder pricebidder price

Alice 15

close()

winner

Bob

MPI-SWS/NOVALINCS

Replicated auction service

10Cheng Li @ PaPoC'154/13/2015

UKUS

Bob won even with a lower bid than Alice.

bidder price

Bob 10

winner bidder price

Alice 15

winner

Bob

bidder price

Alice 15

Bob 10

bidder price

Alice 15

Bob 10

winner

Bob

MPI-SWS/NOVALINCS

Less coordination is better!

4/13/2015 Cheng Li @ PaPoC'15 11

bid

bid

close

bid

bid

bid

close

bid

bid

MPI-SWS/NOVALINCS

Less coordination is better!

4/13/2015 Cheng Li @ PaPoC'15 12

bid

bid

close

bid

bid

bid

close

bid

bid

Visibility restriction

A partial order enforcing visibility

restrictions

Pros: fine-grained tuning in consistency semantics (the amount of coordination) using restrictions

MPI-SWS/NOVALINCS

Outline

• Background

• Partial order-Restriction (PoR) consistency– Visibility restrictions

– Consistency model

• Restriction inference

• Conclusion

4/13/2015 Cheng Li @ PaPoC'15 13

MPI-SWS/NOVALINCS

Outline

• Background

• Partial order-Restriction (PoR) consistency– Visibility restrictions

– Consistency model

• Restriction inference

• Conclusion

4/13/2015 Cheng Li @ PaPoC'15 14

MPI-SWS/NOVALINCS

Visibility restrictions

• A restriction between two operations implies that one must see effects introduced by the other.

• For operation 𝑎, 𝑏, the restriction 𝑟 𝑎, 𝑏 implies that 𝑎 ≺ 𝑏 𝑏 ≺ 𝑎 w.r.t any partial order ≺.

4/13/2015 Cheng Li @ PaPoC'15 15

a0

b0

a1

a2

b1

b2

If 𝑎 ≺ 𝑏 𝑏 ≺ 𝑎,

then 𝑟 𝑎, 𝑏 is met in ≺.

MPI-SWS/NOVALINCS

Partial order-restrictions (PoR) Consistency

• A replicated system 𝑆 is associated with a set of restrictions 𝑅𝑠.

– e.g., 𝑅𝑠 = {𝑟(𝑎, 𝑏), 𝑟(𝑐, 𝑑)}

• 𝑆 is PoR Consistent if, for any its executions, there exists an admissible partial order, where all restrictions in 𝑅𝑠 are met.

4/13/2015 Cheng Li @ PaPoC'15 16

MPI-SWS/NOVALINCS

Outline

• Background

• Partial order-Restriction (PoR) consistency– Visibility restrictions

– Consistency model

• Restriction inference

• Conclusion

4/13/2015 Cheng Li @ PaPoC'15 17

MPI-SWS/NOVALINCS

Challenges

• Which restrictions are sufficient?– They must ensure relevant properties.

• Does the set of restrictions comprise a minimal coordination plan?– i.e., we cannot remove any restriction from the set.

4/13/2015 Cheng Li @ PaPoC'15 18

MPI-SWS/NOVALINCS

Outline

• Background

• Partial order-Restriction (PoR) consistency

• Restriction inference– State convergence

– Invariant preservation

• Conclusion

4/13/2015 Cheng Li @ PaPoC'15 19

MPI-SWS/NOVALINCS

State convergence

• Definition: all replicas reach the same final state when the system becomes quiescent.

• Insight: any pair of operations must be ordered if they don’t commute.

4/13/2015 Cheng Li @ PaPoC'15 20

MPI-SWS/NOVALINCS

Invariant preservation

• Insight: for any violation, add restrictions among concurrent operations

4/13/2015 Cheng Li @ PaPoC'15 21

bid(“Bob”, 10)

bid(“Alice”, 15)

close()

winner bidder price

winner

Bob

bidder price

Alice 15

Bob 10

MPI-SWS/NOVALINCS

Invariant preservation (cont.)

• Insight: for any violation, add restrictions among concurrent operations

4/13/2015 Cheng Li @ PaPoC'15 22

bid(“Bob”, 10)

bid(“Alice”, 15)

close()

winner bidder price

winner

Bob

bidder price

Alice 15

Bob 10

Q: Can we remove any concurrentop while maintaining the violation?

MPI-SWS/NOVALINCS

Invariant preservation (cont.)

• Insight: for any violation, add restrictions among concurrent operations

4/13/2015 Cheng Li @ PaPoC'15 23

bid(“Bob”, 10)

bid(“Alice”, 15)

winner bidder price

winner bidder price

Alice 15

Bob 10

Q: Can we remove any concurrentop while maintaining the violation?A: close must be there.

MPI-SWS/NOVALINCS

Invariant preservation (cont.)

• Insight: for any violation, add restrictions among concurrent operations

4/13/2015 Cheng Li @ PaPoC'15 24

bid(“Bob”, 10)

bid(“Alice”, 15)

close()

winner bidder price

winner

Bob

bidder price

Alice 15

Bob 10

Q: Can we remove any concurrentop while maintaining the violation?

MPI-SWS/NOVALINCS

Invariant preservation (cont.)

• Insight: for any violation, add restrictions among concurrent operations

4/13/2015 Cheng Li @ PaPoC'15 25

bid(“Alice”, 15)

close()

winner bidder price

Bob 10

winner

Bob

bidder price

Alice 15

Bob 10

Q: Can we remove any concurrentop while maintaining the violation?A: Yes, bid(“Bob”, 10).

MPI-SWS/NOVALINCS

Invariant preservation (cont.)

• Insight: for any violation, add restrictions among concurrent operations

4/13/2015 Cheng Li @ PaPoC'15 26

bid(“Alice”, 15)

close()

winner bidder price

Bob 10

winner

Bob

bidder price

Alice 15

Bob 10

This is a minimal violating execution.Solution: 𝑟(𝑏𝑖𝑑, 𝑐𝑙𝑜𝑠𝑒)

Analysis:Step 1: For any such an execution, we add a restriction over a pair of its maximaloperations. Step 2: Iterate all these executions

MPI-SWS/NOVALINCS

Take-home points

• Consistency semantics can be captured by visibility restrictions over operations.

• Invariant and commutativity analysis finds the minimal set of restrictions to preserve convergence and invariants.

• PoR Consistency enforces all restrictions throughout all executions of a replicated system.

4/13/2015 Cheng Li @ PaPoC'15 27

Minimizing Coordination in Replicated systems

Thanks for your attention!