2. Fault Tolerance
Dec 26, 2015
2. Fault Tolerance
2
Fault - Error - Failure
Fault = physical defect or flow occurring in some component (hardware or software)
Error = incorrect behavior caused by a fault• – manifestation of fault
Failure = inability of the system to perform its specified service
Latent Fault = which has not yet produced error Latent Error = which has not yet produced failure
3
Fault - Error - Failure
Note: presents of fault does not ensure that error will occur, e.g. memory stuck-at-0
4
Origin of Defects in Objects (HW/SW)
Good object wearing out with age• – Hardware (software can age too)• – Incorrect maintenance/operation
Good object, unforeseen hostile environment• – Environmental fault
Marginal object: occasionally fails in target environment
• – Tight design/bad inputs Implementation mistakes Specification mistakes
Note: From Top to Down-> Increasing human responsibility
5
Bathtub Curve
Three phases of system lifetime• – Infant mortality• – Normal lifetime• – Wear-out period
6
Life-time of a Software system
7
Faults Characteristics
1- Cause– Specification errors
very dangerousgeneric fault
– Implementation errors• very hard to formally verify
– Random component faults• random, not manufacturing defects
– External disturbance• noise, EMP, vibration, radiation• much like random component
8
Faults Characteristics
2- Origin– software or hardware
• • Physical device level (HW)• • Logic level (HW)• • Chip level (HW)• • System level (HW/SW)
– interfacing, specifications, …
– don’t care, except: • hardware can be analog
• indeterminate voltage level
9
Faults Characteristics
3- Duration– Permanent fault
• occurs and doesn’t go away• easiest to diagnose
– Transient fault• occurs once and disappears• 10 times as expected as permanent fault
– Intermittent fault• occurs occasionally• may appear to be transient (if long period)• hard and expensive to detect
10
Faults Characteristics
4- Extent– Global
• A power supply fault
– Local• A memory fault
5- Value– Determinate
• memory stuck-at-0
– Indeterminate• A sensitive fault to data or time
11
What to do about Faults
Finding & identifying faults:• • Fault detection: is a fault there?• • Fault location: where?• • Fault diagnosis: which fault it is?
Automatic handling of faults• • Fault containment: blocking error flow
– Fault masking: fault has no effect• • Fault recovery: back to correct operation
12
System Response to faults
Error on output: may be acceptable in non-critical systems if happens only rarely
Fault masking: output correct even when fault from a specific class occurs
• – Critical applications: air/space/manufacturing
Fault-secure: output correct or error indication• – Retryable: banking, telephony
Fail safe: output correct or in safe state• – Flashing red traffic light, disabled ATM
13
What is Fault-Tolerance?
A fault-tolerant system is one that continues to perform at desired level of service according to their specification, in the presence of faults.
There are no failures in a fault-tolerant system.
Fault-tolerance is the ability of a system to provide a service complying with the specification in spite of faults.
A better title might have been Dependable or Reliable or Available computing
14
Fault Tolerance In the physical universe:
• - Fault detection• - Fault location• - Fault containment• - Fault recovery• - Continue servicing
In the informational universe:• - Error detection• - Error location• - Error containment• - Error recovery• - Continue servicing
15
Fault Recovery
How quickly is the fault detected? How soon can recovery begin?
• – Does is require human intervention• – How is the system admin notified?
How long does recovery take?• – Restore from backup?• – Purchase new HW?
16
Fault Coverage (C) Measure of system’s ability to perform:
• – fault detection• – fault location• – fault containment• – (and/or fault recovery)
C = P (fault detection | fault occurrence) C = P (fault recovery | fault occurrence) Note:
• – recovery implies that the system as a whole is operational• – this does not imply that a repair occurred• – e.g. duplex system with benign fault can recover to
continue operation on one non-faulty processor
17
Design Philosophies to Combat Faults
Fault avoidance (off-line)• Attempts to prevent fault in the:
• Design review• Component selecting• Quality control• Shielding• Testing
Fault masking (on-line)• Attempts to prevent a fault in a system from introducing
errors• Error correcting memory• Majority voting
Fault tolerance (on-line)• Attempts to provide a system to continue performing its
expected tasks after the occurrence of faults
18
Design Philosophies to Combat Faults
Fault avoidance Fault masking Fault tolerance
19
Fault Avoidance vs. Tolerance
Fault avoidance: eliminate problem sources• – Remove defects: Testing and debugging• – Robust design: reduce probability of defects• – Minimize environmental stress: Radiation shielding etc• – Impossible to avoid faults completely
Fault tolerance: add redundancy to mask effect• – Additional resources needed (more later)• – Examples:
• Error correction coding
• Backup storage
• Spare tire etc
20
Fault Forecasting vs. Tolerance Fault Tolerance
• Execution-time techniques that handle with the effects of faults
Fault Forecasting• Estimate current number, future incidence and likely
consequences• You can’t tolerate what you don’t expect• But if we expected it, we would avoid or eliminate the fault!• In general: We can itemize the classes of faults that can
occur• We can define what we want done if the fault occurs and if
the error is detected• Example: Automobile tire
• Lose air• Do not expect it to experience electrical overload
21
Fault Tolerant computing
Deterministic approaches• – Based on simplifying assumptions: “fault model”• – Obtain methods using the models: test generation• – Evaluation of effectiveness• – Used for Testing & combinatorial fault-tolerance
Probabilistic approaches• – We can’t predict exactly when a person will die, but
we can still get “life expectancy = 77.2”, if we have data• – Used for evaluating, achieving and optimizing
reliability• – Random testing
22
Fault Tolerant vs. Performance There are many Fault-tolerance approaches that
sacrifice performance to tolerate faults Ex. 1:
• – Periodically stop the system and checkpoint its state to disk.
• * If fault occurs, recover state from checkpoint and resume Ex. 2:
• – Log all changes made to system state in case recovery is needed
• * During recovery, undo the changes from the log Ex. 3:
• – Run two identical systems in parallel, compare their results before using them
Ex. 4:• – Run software with lots of error checking
23
Fault Tolerant vs. Cost
There are many Fault-tolerance approaches that sacrifice cost to tolerate faults
Ex. 1:• – Replicate the hardware 3 times and vote to determine
correct output
Ex. 2:• – Mirror the disks (RAID-1) to tolerate disk failures
Ex. 3:• – Use multiple independent versions of software to
tolerate bugs (Called N-version programming)
24
Fault Tolerant vs. Power There are many Fault-tolerance approaches
that sacrifice power to tolerate faults Ex. 1, 2 & 3 (same as previous slide)
• – Replicate the hardware 3 times and vote to determine correct output
• – Mirror the disks (RAID-1) to tolerate disk failures• – Use multiple independent versions of software to
tolerate bugs Ex. 4
• – Add continuously running checking hardware to system
Ex. 5• – Add extra code to check for software faults
25
Need for Fault Tolerance: Universal
Natural objects:• • Fat deposits in body: survival in starvation• • Duplication of eyes: graceful degradation
upon failure
Man-made objects• • Redundancy in ordinary text• • Asking for password twice during initial set-
up• • Duplicate tires in trucks
26
Forms of Redundancy
Hardware redundancy• – add extra hardware for detection or tolerating
faults Software redundancy
• – add extra software for detection and possibly tolerating faults
Information redundancy• – extra information, i.e. codes
Time redundancy• – extra time for performing tasks for fault
tolerance
27
Redundancy base
Time
Try Retry Retry
Space
Try
Try
Try