System Reliability and Fault Tolerance Reliable Communication Byzantine Fault Tolerance.

System Reliability and Fault Tolerance

Reliable Communication

Byzantine Fault Tolerance

Recap: Replication

Write is handled only by the remote primary server, and the backups are updated accordingly; Read is performed locally.

Replicated services:– Sun Network Info Service (NIS, formerly Yellow Pages)– FAB: Building distributed enterprise disk arrays from commodity

components, ASPLOS’04

© C. Xu, 1998-2007

Reliable Point-to-Point Comm. Failure Models

– Process failure: Sender vs Receiver» Fail-stop: a process crash which can be detected by other processes» How to detect such crash? Timeout can indicate only that a process is

not responding

– Comm. failure» send failure: A process completes a send, but the msg is not put in its

outgoing msg buffer» receive failure: A msg is in incoming buf, but is not received by a

process» Channel failure: fail while msg is transmitted from outgoing buf to

incoming buf

– Arbitrary failure (Byzantine failure)» Any type of error may occur. E.g. return wrong value.

Reliable comm:– Validity: Any msg in the outgoing buf is eventually delivered to the

incoming buf– Integrity: The msg received is identical to the one sent and no msgs

are delivered twice.

© C. Xu, 1998-2007

RPC Failure Semantics Five Possible Failures:

– The client is unable to locate the server» Server is down or Stub mismatches with Skeleton» Throw UnknownHostException

– The request message is lost» start with a timer » retransmission of the request message

– The server crashes after receiving a request– The reply message is lost– The client crashes after sending a request

In Java, all remote methods must be prepared to catch RemoteException

© C. Xu, 1998-2007

Server Crashes A server in client-server communication

a) Normal caseb) Crash after execution c) Crash before execution

• At least once semantics• At most once semantics• Exactly once semantics

java.rmi.ServerRuntimeException

© C. Xu, 1998-2007

Server Crash (Cont’) Assume client requests server to print a msg

– Send a completion msg (M) before print (P), or

– Send a completion msg (M) after print (P)

Combinations– MPC: crash after ack and print

– MC(P):

– PMC:

– PC(M)

– C(PM): crash before print and ack

– C(MP)

© C. Xu, 1998-2007

Client Server

Strategy M P Strategy P M

Reissue strategy MPC MC(P) C(MP) PMC PC(M) C(PM)

Always DUP OK OK DUP DUP OK

Never OK ZERO ZERO OK OK ZERO

Only when ACKed DUP OK ZERO DUP OK ZERO

Only when not ACKed OK ZERO OK OK DUP OK

When a crashed server recovers, the client can never reissue a request (Never)

always reissue a request (Always)

reissue only if it received ack

reissue only if it received no ack

ok: Text is printed once; dup: printed twice; zero: no printout

© C. Xu, 1998-2007

Lost Reply Messages– Some requests can be re-executed with side-effects (e.g.

Read 1024 bytes of a file); some not (idempotent).– Solutions:

» Structure requests in an idempotent way» Assign request a sequence number to be checked by server

Client Crashes leading to orphan computation – extermination: client side logging of RPC about what to

do; the log is checked after a reboot.– reincarnation: client bcasts a new epoch when it reboots;

server detects orphan computations based on epochs. » kill orphan remote computation or locate their owners

– expiration: set a time quantum for each RPC request; if it cannot finish, more quanta are asked.

© C. Xu, 1998-2007

Reliable Multicast Basic properties:

– Validity: If a correct process multicasts message m, then it will eventually deliver m.

– Integrity: a correct process delivers the msg at most once

Atomic messages (aka agreement)– A message is delivered to all members of a group, or to

none Message ordering guarantees

– within group– across groups

© C. Xu, 1998-2007

Message Ordering Different members may see messages in different

orders Ordered group communication requires that all

members agree about the order of messages Within each group, assign global ordering to

messages Hold back messages that arrive out of order (delay

their delivery)

(I) Unordered Multicasts

Process P1 Process P2 Process P3

mcast m1 receives m1 receive m2

mcast m2 receives m2 receives m1

Process P1 Process P2 Process P3 Process P4

mcast m1 receive m1 receives m3 mcast m3

mcast m2 receives m3 receives m1 mcast m4

receives m2 receives m2

receives m4 receives m4

(II) FIFO-ordered Multicasts

If a process multicasts two msgs m and m’ in order, then every process in the group will deliver the msgs in the same order

(III) Causally-order Multicasts

(1) (2)

(1) Delayed (2)

A B

C

D

(1)

(VI) Totally-ordered multicasts

If mcast(g, m) mcast(g, m’)Then any process in the group

should deliver m before m’

If a process delivers msg m before m’, then any other processthat delivers m’ will deliver m before m’.

© C. Xu, 1998-2007

Centralized Impl of Total Ordering Central ordering server (sequencer) assigns global

sequence numbers Hosts apply to ordering server for numbers, or

ordering server sends all messages itself Hold-back easy, since sequence numbers are

sequential– Msgs will remain in hold-back queue until they can be

delivered according to their sequence numbers. Sequencer: bottleneck and single point of failure

– tricky protocol to deal with case where ordering server fails

© C. Xu, 1998-2007

Atomic Messages Each recipient acks message, and sender retransmits if ack

not received– Sender could crash before msg is delivered!!

» Simple approach: if sender crashes, a recipient volunteers to be “backup sender” for the message

» re-sends message to everybody, waits for acks

» use simple algorithm to choose volunteer

» apply method again if backup sender crashes

No single best solutions exist!

Reliability due to Replication Blocking update, waiting till backups are updated

– Blocking update of backup servers must be atomic so as to implement sequential consistency as the primary can serialize all incoming writes (in order) and all processes see all writes in the same order from any backup servers.

Total ordering due to the use of primary for centralized sequencer

Atomic: – What happens if some W4 are Postive Ack and some are NAck?

– Two-phase commit protocol: » W3: “prepare” msg from primary to other replicas

» W3+: ack to “prepare” (If in a prepared state, related objects be preserved in permanent storage; will eventually be able to commit it.)

» W4: “commit” or “abort” msg

» W4+: ack to “commit/abort”

2PC Protocol in the Presence of Failures If ack to “prepare” msg is timed out, primary can send “abort” to

replicas and safely abort itself If replica waiting for “commit” or “abort” is timed out,

– If its ack to “prepare” was negative, simply abort itself

– If its ack to “prepare” was positive, it cannot “commit”, nor “abort”. Block, waiting for primary or network recovery

How to handle crash/reboot, particularly primary failure?– Cannot back out of a commit if already decided– Semantics of failure: store commit; cannot commit before store– Recovery protocol w/ non-volatile memory

2PC causes a long waiting time if primary fails after “prepare” msg is sent out– Three-phase commit protocol: Pre-Prepare, Prepare, Commit– Replica times out waiting for Commit msg will commit the trans– 2PC: execute transaction when everyone is willing to commit– 3PC: execute transaction when everyone knows it will commit

Recovery from Primary Failure Need to pick up a new primary defining a new “view.” It

could be set by human operator OR autonomic – Suppose the lowest-numbered live server is the primary

– Replicas need to ping each other

– Ping msg lose or delayed may lead to more than one primary

Paxos protocol for fault-tolerant consensus – At most a single value is chosen

– Agreement reached despite lost msgs and crashed nodes

– Paxos protocol: eventually succeeds if a majority of replicas are reachable

– See Lamport’98 (submitted to TOCS in 90) and Chandra-Toueg’96 for details

Handling Byzantine Failure Byzantine Failure

– Failed replicas are not necessarily failure-stop

– Failed replicas may generate arbitrary results!!

The Byzantine Generals Problem

Leslie Lamport, Robert Shostak, and Marshall Pease in 1982

Byzantine Generals Problem N divisions of Byzantine army surround city

– Each division commanded by a general

– Some of the N generals are traitors

Generals communicate via messages– Traitors can send different values to different generals

Requirements:– All loyal generals decide upon same plan of action

– A “small” number of traitors cannot cause loyal generals to adopt a bad plan

– NOT required to identify traitors

Restricted BGP Restate problems as:

– 1 commanding general

– N-1 lieutenants

Interactive consistency requirements– IC1: All loyal lieutenants obey the same order

– IC2: If the commander is loyal, every loyal lieutenant obeys his/her order

If we can solve this problem…– Original BGP problem reduces to N instances of this problem;

one instance per general acting as commander

3-General Impossibility Result Assume 2 loyal generals and 1 traitor (shaded)

– Two messages: ATTACK or RETREAT

If Lt.1 sees {Attack, “He said Retreat”} what to do?– If Lt2 is traitor (Fig1), L1 must attack to satisfy IC2

– If Commander is traitor (Fig2), L1 and L2 must make same decision(always obeying commanders order over lieutenant’s violates IC1)

Commander

lieutenant lieutenant

AttackAttack

He said Retreat

Commander

lieutenant lieutenant

Retreat

Attack

He said Retreat

General Impossibility Result In general, no solutions with fewer than 3m+1 generals

if there are m traitors

Proof by contradiction:– Assume there is a solution for 3m Albanians, including m

traitors

– Let a Byzantine general simulate m Albanian generals

– The problem is then reduced to 3-general problem

Solution Example With one faulty process: f=1, N=4

1st round: the cmd sends a value to each Lt

2nd round: each Lt copies the value to all other Lts

p1 (Commander)

p2 p3

1:v1:v

2:1:v

3:1:u

Faulty processes are shown coloured

p4

1:v

4:1:v

2:1:v 3:1:w

4:1:v

p1 (Commander)

p2 p3

1:w1:u

2:1:u

3:1:w

p4

1:v

4:1:v

2:1:u 3:1:w

4:1:v

Practical Byzantine Fault Practical Byzantine Fault ToleranceTolerance

Miguel Castro and Barbara Liskov

OSDI’99

Assumptions Asynchronous distributed systems Faulty nodes may behave arbitrarily

– Due to malicious attacks or software errors

Independent node failures Network may fail to deliver, delay, duplicate or deliver

them out of order An adversary may coordinate faulty nodes, delay comm,

or delay correct nodes in order to cause the most damage to the service. BUT it cannot delay correct nodes indefinitely, nor subvert the cryptographic techniques– Any network fault will be eventually repaired

– E.g. cannot forge a valid signature of non-faulty node

– E.g. Cannot find two msgs with the same digests

Objectives

To be used for the implementation of any deterministic replicated service with a state and some operations– Clients issue requests and block waiting for a reply

Safety if no more than [(n-1)/3] faulty replicas (i.e. to tolerate f faulty nodes, at least n=3f+1 needed)– Safety: the replicated service satisfies linearizability

– Behaves like a centralized implementation that executes ops atomically one at a time

– Why 3f+1 the optimal resiliency?

Liveness: clients eventually receive replies to their requests,– At most [(n-1)/3] faulty replicas

– Comm delay is bounded with unknown bounds; delay is the latency from the time of first sending to the time of receipt by the destination

Algorithm in a nutshell

Client Primary

Backup

Backup

Backup

f + 1 Match (OK)

Replicas and Views

R0 R1 R2 R|R-1|

Set of replicas (R): |R| ≥ 3f + 1

For view v primary p assigned such that p= v For view v primary p assigned such that p= v mod |R|mod |R|

R0

View 0

R0 R1

View 1

………

Normal Case Operation

Client Primary{REQUEST{REQUEST, , o, t, o, t, c}c}

o – o – OperationOperation

t – t – TimestampTimestamp

c - Client c - Client Timestamps are totally ordered such that later requests have higher timestamps than earlier ones

Normal Case Operation state of each replica is stored in a message log

Primary p receives a client request m , it starts a three-phase protocol to atomically multicast the request to the replicas– Pre-prepare, Prepare, Commit

Pre-Prepare and Prepare phases are for total ordering of requests sent in the same view, even when the primary is faulty

Prepare and Commit phases are to ensure committed requests are totally ordered across views

Pre-Prepare Phase

v – view number

n – sequence number

m – message

d – digest of the message

Primary

Backup

Backup

Backup

<<PRE-PREPARE, v, n, d> , m>

Prepare Phase

If replica i accepts the PRE-PREPARE message it enters prepare phase by multicasting

<PREPARE, v, n, d, i>to all other replicas and adds both messages to its log

Otherwise does nothing

A replica accepts the PRE-PREPARE message provided,– The signatures are valid and the digest matches m

– It is in view v

– It has not accepted a PRE-PREPARE for the same v and n

– Sequence number is within accepted bounds

Commit Phase When replica i receives 2f matched PREPARE msg,

the replica gets into Commit Phase by multicasting

<COMMIT, v, n, d , i> to other replicas

Replica i executes required operation after it has accepted 2f+1 matched commit msgs from different replicas.

Replica i’s state reflects the seq execution of all requests with lower sequence numbers. This ensures all non-faulty replicas execute requests in same order.

To guarantee exactly-once semantics, replicas discard requests whose timestamp is lower than the timestamp in the last reply they sent to the client.

Normal Operation Reply All replicas sends the reply <REPLY, v, t, c, i, r>,

directly to the client

v = current view number

t = timestamp of the corresponding request

i = replica index

r = execution result

Client waits for f+1 replies with valid signatures from different replicas, and with same t and r, before accepting the result r

Normal Case Operation: Summery

C

Primary: 0

2

1

Faulty: 3

Request Pre-prepare Prepare Commit Reply

X

Safeguards If the client does not receive replies soon enough, it

broadcasts the request to all replicas

If the request has already been processed, the replicas simply re-send the reply

If the replica is not the primary, it relays the request to the primary

If the primary does not multicast the request to the group, it will eventually be suspected to be faulty by enough replicas to cause a view change

View ChangesView Changes Timer is set when a request is received, recording

the waiting time for the request to execute. If the timer of replica expires in view v, the replica

starts a view change to move to view v+1 by,– Stop accepting pre-prepare/prepare/commit messages

– Multicasting a VIEW-CHANGE message

<VIEW-CHANGE, v+1, n, C, P, i>

n = #seq of last stable checkpoint s known to i

C = 2f + 1 checkpoint msgs proving correctness of s

P = {Pm: for each m prepared, its #seq >n} Pm = pre-prepare and 2f matching prepare

msgs

New PrimaryNew Primary When primary p’ of view v+1 receives 2f valid

VIEW-CHANGE messages– It multicasts a <NEW-VIEW, v+ 1, V, O> message to

all other replicas where» V = set of 2f valid VIEW-CHANGE messages

» O = set of reissued PRE-PREPARE messages

– Moves to view v+1

For a replica that accepts NEW-VIEW– Sends PREPARE messages for every pre-prepare in

set O

– Moves to view v+1

References See OSDI’99 for

– Optimization, implementation, and evaluation

– 3% overhead in NFS daemon

See tech report for– Formal presentation of the algorithm in I/O

automation model

– Proof of safety and liveness

See OSDI’00 for– Proactive recovery

Further ReadingsFurther Readings In synchronous systems, assume msg exchanges take place in

rounds and processes can detect the absence of a msg through a timeout

– At least f+1 rounds of msgs are needed (Fisher-Lynch, 82) In async systems with unbounded delay, a crashed process

becomes indistinguishable from a slow one. – [Impossibility] No algorithm can guarantee to reach consensus in such

systems, even with one process crash failure. (Fisher-Lynch-Paterson, J. ACM’85)

Approaches to working around the impossibility– In partially async systems with bounded but unknown delay

» Practical Byzantine Fault Tolerant Alg (Castro-Liskov’99)

– Using failure detectors: unresponsive process be treated as failure and discard their subsequent msgs.

– Consensus can be reached, even with an unreliable failure detector, if fewer than N/2 processes crashes and comm is reliable. (Chandra-Toueg’96)

– Statistical consensus: “no guarantee” doesn’t mean “cannot”. » Introduce an element of chance in processes’ behaviors so that the

adversary cannot exercise its thwarting strategy effectively.