Top Banner
Fault Tolerance
22

9 fault-tolerance

Jul 14, 2015

Download

Engineering

4020132038
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: 9 fault-tolerance

Fault Tolerance

Page 2: 9 fault-tolerance

Basic Concepts• Availability The system is ready to work immediately

• Reliability The system can run continuously

• Safety When the system fails, nothing catastrophic happens

• Maintainability

A failed system can be easily repaired.

Fault types: transient, intermittent, permanent

Page 3: 9 fault-tolerance

Failure Models

Different types of failures.

Type of failure Description

Crash failure A server halts, but is working correctly until it halts

Omission failure Receive omission Send omission

A server fails to respond to requestsA server fails to receive incoming messagesA server fails to send messages

Timing failure A server's response lies outside the specified time interval

Response failure Value failure State transition failure

The server's response is incorrectThe value of the response is wrongThe server deviates from the correct flow of control

Arbitrary failure A server may produce arbitrary responses at arbitrary times

Page 4: 9 fault-tolerance

Failure Masking by Redundancy

•Information redundancy (extra bits)

•Time redundancy (extra operations)

• Physical redundancy (extra equipment or processes)

Page 5: 9 fault-tolerance

Failure Masking by Redundancy

Triple modular redundancy (TMR).An electronic circuit example

Page 6: 9 fault-tolerance

Process failures

To tolerate a faulty process, identical processes organized into a group

When one process of the group fails, some other process in the group takes care of the work

Process groups may be dynamic

Mechanisms are needed for managing groups membership•Group server maintains information on membership (centralized)•Distributed management (less simple and time consuming)

Page 7: 9 fault-tolerance

Flat Groups versus Hierarchical Groups

a) Communication in a flat group (voting mechanism, slow decision) Replicated write protocolsb) Communication in a simple hierarchical group (single point of failure) Primary based protocols

Page 8: 9 fault-tolerance

Client-server communication failuresUsing a reliable transport protocol (TCP) masks omission failures, but many failures are not masked.

Classes of failure • The client is unable to locate the server – exception is a solution, but we loose in transparency

•The request message from the client to the server is lost – retransmission

•The server crashes after receiving a request

•The reply message from the server to the client is lost – retransmission, but…

•The client crashes after sending a request – orphan is generated. (extermination, reincarnation with epoch #, gentle reincarnation, expiration…)

Page 9: 9 fault-tolerance

Server Crashes (1)

A server in client-server communicationa) Normal caseb) Crash after execution c) Crash before execution

At least once semantic: after server reboot, to try until a request is obtained At most once semantic: immediate failure report Exactly once semantic: no way

Page 10: 9 fault-tolerance

Server Crashes (2)

Different combinations of client and server strategies in the presence of server crashes.

Client Server

Strategy M -> P Strategy P -> M

Reissue strategy MPC MC(P) C(MP) PMC PC(M) C(PM)

Always DUP OK OK DUP DUP OK

Never OK ZERO ZERO OK OK ZERO

Only when ACKed DUP OK ZERO DUP OK ZERO

Only when not ACKed OK ZERO OK OK DUP OK

Example: a client send a message to a server for printing (P) it, having a completion message back (M). The server can crash (C)

Page 11: 9 fault-tolerance

Group CommunicationBasic Reliable-Multicasting Schemes

A simple solution to reliable multicasting when all receivers are known and are assumed not to fail

a) Message transmission b) Reporting feedback

Efficient only for little # of receivers ( only nack, timer etc..)

Important for messaging in process group

Page 12: 9 fault-tolerance

Nonhierarchical Feedback Control

Several receivers have scheduled a request for retransmission, but the first retransmission request leads to the suppression of others (Scalable

Reliable Multicasting protocol).It leads to timing problems, useless retransmissions or a complicated

organization of the group membership

To scale, we need to reduce the number of messages,with feedback suppression

Page 13: 9 fault-tolerance

Hierarchical Feedback Control

The essence of hierarchical reliable multicasting. A tree of receivers partitions is formed

• Each local coordinator forwards the message to its children.• A local coordinator handles retransmission requests.Acknowledge between coordinators

Page 14: 9 fault-tolerance

Atomic Multicast In presence of process failures, the guarantee that a message is delivered to all or none

of the receivers is needed. This lead to the atomic multicast problemAtomic multicasting ensures that group members maintain consistency

The logical organization of a distributed system to distinguish between message receipt and message delivery

In atomic multicasting a multicast message is uniquely associated to a list of receiving processes ( Group view )

A view change takes place when a process joins or leaves the group

Page 15: 9 fault-tolerance

Virtual Synchrony

The principle of virtual synchronous multicast (view change similar to synchronization variable)

We need an ordered reliable multicasting.Virtual Synchrony guarantees that a message sent to a group view is delivered to each

non-faulty member of the group.If the sender crashes, the message may be either delivered to all the other processes or

ignored by each of them.

Page 16: 9 fault-tolerance

Message Ordering

Four different type of ordering of multicasts:

• Reliable, unordered multicast no guarantees is given on the order in which messages are delivered

• FIFO ordered multicast messages from the same process are delivered in the order as they are sent

• Causally ordered multicast causality between messages is preserved

• Totally-ordered multicast messages are delivered in the same order to all members of the group

Page 17: 9 fault-tolerance

Message Ordering

Four processes in the same group with two different senders, and a possible delivery order of messages under FIFO-ordered multicasting

Process P1 Process P2 Process P3 Process P4

sends m1 receives m1 receives m3 sends m3

sends m2 receives m3 receives m1 sends m4

receives m2 receives m2

receives m4 receives m4

Process P1 Process P2 Process P3

sends m1 receives m1 receives m2

sends m2 receives m2 receives m1

Unordered multicast:Three communicating processes in the same group. The ordering of events per process is shown

along the vertical axis.

Page 18: 9 fault-tolerance

Message Ordering

Six different versions of virtually synchronous reliable multicasting.

Multicast Basic Message Ordering Total-ordered Delivery?

Reliable multicast None No

FIFO multicast FIFO-ordered delivery No

Causal multicast Causal-ordered delivery No

Atomic multicast None Yes

FIFO atomic multicast FIFO-ordered delivery Yes

Causal atomic multicast Causal-ordered delivery Yes

Virtually synchronous reliable multicasting offering totally ordered delivery is called atomic multicasting

Page 19: 9 fault-tolerance

Distributed Commit

a) The finite state machine for the coordinator in two phase commit.b) The finite state machine for a participant.The first phase is the vote phase, the second is the decision phaseTimeout mechanisms are necessary, coordinator can crash

Distributed commit means that an operation has to be performed by each member of a group or none at all

One phase distributed commit is performed using a coordinator ( if a participant cannot perform the operation, no means to advise the coordinator)

Page 20: 9 fault-tolerance

Two Phase Commit

• The coordinator send a vote_request to all participants• A participant returns a vote-commit (it is ready to commit its part of transaction) or a vote-abort• The coordinator collects the votes and send a global_commit or a global_abort (if one of the participants has sent a vote_abort)• A participant receive a global_commit and locally commits the transaction, or receive a global_abort and locally aborts the transaction

1 – voting phase2 – decision phase

1

2

Page 21: 9 fault-tolerance

Three-Phase Commit

It avoids blocking processes in case of coordinator crash

• There is no state from which it is possible to make a transition directly to COMMIT or ABORT

• There is no state in which it is not possible to make a final decision and from which a transition to a COMMIT can be made

Page 22: 9 fault-tolerance

Recovery

• Backward recovery brings the system to the previous correct state. It is necessary to record the state (check-pointing)

• Forward recovery attempt to bring the system in a correct new state to continue the execution.