Theme Feature Fault Injection Techniques and Toolsece749/docs/faultInjection... · 2004-11-17 · detection, fault isolation, and reconfiguration and recovery capabilities. Fault

0018-9162/96/$10.00 © 1997 IEEE April 1997 75

Fault InjectionTechniques and Tools

D ependability evaluation involves the study offailures and errors. The destructive nature ofa crash and long error latency make it difficult

to identify the causes of failures in the operationalenvironment. It is particularly hard to recreate afailure scenario for a large, complex system.

To identify and understand potential failures, weuse an experiment-based approach for studying thedependability of a system. Such an approach isapplied not only during the conception and designphases, but also during the prototype and opera-tional phases.1,2

To take an experiment-based approach, we mustfirst understand a system’s architecture, structure,and behavior. Specifically, we need to know its tol-erance for faults and failures, including its built-indetection and recovery mechanisms,3 and we needspecific instruments and tools to inject faults, createfailures or errors, and monitor their effects.

DIFFERENT PHASES, DIFFERENT TECHNIQUESEngineers most often use low-cost, simulation-

based fault injection to evaluate the dependabilityof a system that is in the conceptual and designphases. At this point, the system under study is onlya series of high-level abstractions; implementationdetails have yet to be determined. Thus the systemis simulated on the basis of simplified assumptions.

Simulation-based fault injection, which assumesthat errors or failures occur according to predeter-mined distribution, is useful for evaluating the effec-tiveness of fault-tolerant mechanisms and a system’sdependability; it does provide timely feedback to sys-tem engineers. However, it requires accurate inputparameters, which are difficult to supply: Design andtechnology changes often complicate the use of past

measurements. Testing a prototype, on the otherhand, allows us to evaluate the system without anyassumptions about system design, which yields moreaccurate results. In prototype-based fault injection,we inject faults into the system to

• identify dependability bottlenecks,• study system behavior in the presence of faults,• determine the coverage of error detection and

recovery mechanisms, and• evaluate the effectiveness of fault tolerance

mechanisms (such as reconfiguration schemes)and performance loss.

To do prototype-based fault injection, faults areinjected either at the hardware level (logical or elec-trical faults) or at the software level (code or datacorruption) and the effects are monitored. The sys-tem used for evaluation can be either a prototype ora fully operational system. Injecting faults into anoperational system can provide information aboutthe failure process. However, fault injection is suit-able for studying emulated faults only. It also failsto provide dependability measures such as meantime between failures and availability.

Instead of injecting faults, engineers can directlymeasure operational systems as they handle realworkloads.2 Measurement-based analysis uses actualdata, which contains much information about nat-urally occurring errors and failures and sometimesabout recovery attempts. Analyzing these data canprovide understanding of actual error and failurecharacteristics and insight for analytical models.However, measurement-based analysis is limited todetected errors. Furthermore, data must be collectedover a long time because errors and failures occur

Fault injection is important to evaluating the dependability of computersystems. Researchers and engineers have created many novel methods toinject faults, which can be implemented in both hardware and software.

Mei-ChenHsueh,Timothy K. Tsai, andRavishankar K. IyerUniversity ofIllinois atUrbana-Champaign

Th

em

e F

ea

ture

76 Computer

infrequently. Field conditions can vary widely, thuscasting doubt on the statistical validity of the result.

Although each of the three experimental methodshas its limitations, their unique values complementone another and allow for a wide spectrum of depend-ability studies.

FAULT INJECTION TECHNIQUESEngineers use fault injection to test fault-tolerant

systems or components. Fault injection tests faultdetection, fault isolation, and reconfiguration andrecovery capabilities.

Fault injection environmentFigure 1 shows a fault injection environment, which

typically consists of the target system plus a fault injec-tor, fault library, workload generator, workload library,controller, monitor, data collector, and data analyzer.

The fault injector injects faults into the target systemas it executes commands from the workload generator(applications, benchmarks, or synthetic workloads).The monitor tracks the execution of the commands andinitiates data collection whenever necessary. The datacollector performs online data collection, and the dataanalyzer, which can be offline, performs data process-ing and analysis. The controller controls the experiment.

Physically, the controller is a program that can runon the target system or on a separate computer. Thefault injector can be custom-built hardware or soft-ware. The fault injector itself can support differentfault types, fault locations, fault times, and appropri-ate hardware semantics or software structure—thevalues of which are drawn from a fault library. Thefault library in Figure 1 is a separate component,which allows for greater flexibility and portability.

The workload generator, monitor, and other compo-nents can be implemented the same way.

Injection method and implementationChoosing between hardware and software fault

injection depends on the type of faults you are inter-ested in and the effort required to create them. Forexample, if you are interested in stuck-at faults (faultsthat force a permanent value onto a point in a circuit),a hardware injector is preferable because you can con-trol the location of the fault. The injection of perma-nent faults using software methods either incurs a highoverhead or is impossible, depending on the fault.However, if you are interested in data corruption, thesoftware approach might suffice. Some faults, such asbit-flips in memory cells, can be injected by eithermethod. In a case like this, additional requirements,such as cost, accuracy, intrusiveness, and repeatabil-ity may guide the choice of approach. Table 1 sum-marizes commonly studied faults and injectionmethods.

HARDWARE FAULT INJECTIONHardware-implemented fault injection uses addi-

tional hardware to introduce faults into the target sys-tem’s hardware. Depending on the faults and theirlocations, hardware-implemented fault injection meth-ods fall into two categories:

• Hardware fault injection with contact. The injec-tor has direct physical contact with the target sys-tem, producing voltage or current changesexternally to the target chip. Examples are meth-ods that use pin-level probes and sockets.

• Hardware fault injection without contact. The

Figure 1. Basic components of a faultinjectionenvironment.

Fault injection system

Controller

Fault injector

Target system

Workloadgenerator Monitor

Data collector

Data analyzer

Faultlibrary

Workloadlibrary

injector has no direct physical contact with thetarget system. Instead, an external source pro-duces some physical phenomenon, such as heavy-ion radiation and electromagnetic interference,causing spurious currents inside the target chip.

These methods are well suited for studying thedependability characteristics of prototypes thatrequire high time-resolution for hardware triggeringand monitoring (fault latency in the CPU, for exam-ple) or require access to locations that cannot be eas-ily reached by other fault injection methods.

Engineers generally model hardware methods onlow-level fault models; for example, a bridging faultmight be a short circuit. Hardware also triggers faultsand monitors their impact, thus providing high time-resolution and low perturbation. Normally, the hard-ware triggers faults after a specified time has expiredon a hardware timer or after it has detected an event,such as a specified address on the address bus.

Injection with contactHardware fault injection using direct contact with

circuit pins, often called pin-level injection, is prob-ably the most common method of hardware-implemented fault injection. There are two maintechniques for altering electrical currents and volt-ages at the pins:

• Active probes. This technique adds current viathe probes attached to the pins, altering their elec-trical currents. The probe method is usually lim-ited to stuck-at faults, although it is possible toattain bridging faults by placing a probe acrosstwo or more pins. Care must be taken when usingactive probes to force additional current into thetarget device, as an inordinate amount of currentcan damage the target hardware.

• Socket insertion. This technique inserts a socketbetween the target hardware and its circuitboard. The inserted socket injects stuck-at, open,or more complex logic faults into the target hard-ware by forcing the analog signals that representdesired logic values onto the pins of the targethardware. The pin signals can be inverted,ANDed, or ORed with adjacent pin signals oreven with previous signals on the same pin.

Both of these methods provide good controllabil-ity of fault times and locations with little or no per-turbation to the target system. Note that becausefaults are modeled at the pin level, they are not iden-tical to traditional stuck-at and bridging fault modelsthat generally occur inside the chip. Nonetheless, youcan achieve many of the same effects, like the exercise

of error detection circuits, using these injection meth-ods. Active probes attached to the power supply hard-ware inject power supply disturbance faults. However,this can damage the injected device or increase the riskof destructive injection.

Injection without contactThese faults are injected by creating heavy-ion radi-

ation. An ion passes through the depletion region ofthe target device and generates current. Placing thetarget hardware in or near an electromagnetic fieldalso injects faults. Engineers like these methodsbecause they mimic natural physical phenomena.However, it is difficult to exactly trigger the time andlocation of a fault injection using this techniquebecause you cannot precisely control the exactmoment of heavy-ion emission or electromagneticfield creation.

Selected toolsMessaline,4 developed at LAAS-CNRS, in Toulouse,

France, uses both active probes and sockets to con-duct pin-level fault injection. Figure 2 on the next pageshows Messaline’s general architecture and its envi-ronment. Messaline can inject stuck-at, open, bridg-ing, and complex logical faults, among others. It canalso control the length of fault existence and the fre-quency. Signals collected from the target system canprovide feedback to the injector. Also, a device is asso-ciated with each injection point to sense when and ifeach fault is activated and produces an error. It canalso inject up to 32 injection points simultaneously.This tool has been used in experiments on a central-ized, interlocking system employed in a computerizedrailway control system and on a distributed systemfor the Esprit Delta-4 Project.

FIST5 (Fault Injection System for Study of TransientFault Effect), developed at the Chalmers University ofTechnology in Sweden, employs both contact and con-tactless methods to create transient faults inside thetarget system. This tool uses heavy-ion radiation tocreate transient faults at random locations inside achip when the chip is exposed to the radiation andcan thus cause single- or multiple-bit-flips. The radi-

April 1997 77

Table 1. Fault-injection implementation methods by fault model.

Hardware Software

Open Storage data corruptionBridging (such as register, memory, and disk)Bit-flip Communication data corruption Spurious current (such as bus and communication network)Power surge Manifestation of software defectsStuck-at (such as machine level and higher levels)

78 Computer

ation source is mounted inside a vacuum chambertogether with a small two-processor computer sys-tem. The computer is positioned so that one of theprocessors is exposed directly under the radiation.The other processor is used as a reference for detect-ing whether the radiation results in any bit-flips.Figure 3 illustrates the FIST environment.

FIST can inject faults directly inside a chip, whichcannot be done with pin-level injections. It can pro-duce transient faults at random locations evenly in achip, which leads to a large variation in the errors seenon the output pins. In addition to radiation, FISTallows for the injection of power disturbance faults.This is done by placing a MOS transistor between thepower supply and the Vcc pin of the processor chip tocontrol the amplitude of the voltage drop. Power sup-ply disturbances usually affect multiple locations withina chip and can cause gate propagation delay faults. Theexperimental results show that the errors resulting fromboth methods cause similar effects on program con-trol-flow and data errors. However, heavy-ion radia-tion causes mostly address bus errors, while powersupply disturbances affect mostly control signals.

MARS6 (Maintainable Real-Time System) is a dis-tributed, fault-tolerant architecture developed at theTechnical University of Vienna. In addition to usingheavy-ion radiation as is used in FIST, MARS useselectromagnetic fields to conduct contactless faultinjection: A circuit board placed between two chargedplates or a chip placed near a charged probe causesfault injection. Dangling wires that act as antennasplaced on individual chip pins accentuate the electro-magnetic field effect on those pins. Researchers com-pared these three methods (heavy-ion radiation,pin-level injection, and electromagnetic interference)

in terms of their capability to exercise the MARS errordetection mechanisms. Results showed that the threemethods are complementary and generate differenttypes of errors. Pin-level injections cause error detec-tion mechanisms outside the CPU to be exercised moreeffectively than heavy-ion radiation or electromag-netic interference. The latter two methods were bet-ter suited for exercising software and application-levelerror detection mechanisms.

SOFTWARE FAULT INJECTIONIn recent years, researchers have taken more inter-

est in developing software-implemented fault injec-tion tools. Software fault-injection techniques areattractive because they don’t require expensive hard-ware. Furthermore, they can be used to target appli-cations and operating systems, which is difficult to dowith hardware fault injection.

If the target is an application, the fault injector isinserted into the application itself or layered betweenthe application and the operating system. If the targetis the operating system, the fault injector must beembedded in the operating system, as it is very difficultto add a layer between the machine and the operatingsystem.

Although the software approach is flexible, it hasits shortcomings.

• It cannot inject faults into locations that are inac-cessible to software.

• The software instrumentation may disturb theworkload running on the target system and evenchange the structure of original software. Carefuldesign of the injection environment can minimizeperturbation to the workload.

Figure 2. Generalarchitecture of Messaline.

Operator

Management of the test sequence

Control of the experiment

Target system

Activation

Environmentsimulation Data collectionInjection

Input files Output files

Initialization Fault

Synchronization ReadoutsInputs/outputs

• The poor time-resolution of the approach maycause fidelity problems. For long latency faults,such as memory faults, the low time-resolutionmay not be a problem. For short latency faults,such as bus and CPU faults, the approach may failto capture certain error behavior, like propagation.Engineers can solve this problem by taking ahybrid approach, which combines the versatilityof software fault injection and the accuracy ofhardware monitoring. The hybrid approach is wellsuited for measuring extremely short latencies.However, the hardware monitoring involved cancost more and decrease flexibility by limitingobservation points and data storage size.

We can categorize software injection methods onthe basis of when the faults are injected: during com-pile-time or during runtime.

Compile-time injectionTo inject faults at compile-time, the program

instruction must be modified before the programimage is loaded and executed. Rather than injectingfaults into the hardware of the target system, thismethod injects errors into the source code or assem-bly code of the target program to emulate the effectof hardware, software, and transient faults. The mod-ified code alters the target program instructions, caus-

ing injection. Injection generates an erroneous soft-ware image, and when the system executes the faultimage, it activates the fault.

This method requires the modification of the pro-gram that will evaluate fault effect, and it requires noadditional software during runtime. In addition, itcauses no perturbation to the target system duringexecution. Because the fault effect is hard-coded, engi-neers can use it to emulate permanent faults. Thismethod’s implementation is very simple, but it doesnot allow the injection of faults as the workload pro-gram runs.

Runtime injectionsDuring runtime, a mechanism is needed to trigger

fault injection. Commonly used triggering mecha-nisms include:

• Time-out. In this simplest of techniques, a timerexpires at a predetermined time, triggering injec-tion. Specifically, the time-out event generates aninterrupt to invoke fault injection. The timercan be a hardware or software timer. Thismethod requires no modification to the applica-tion or workload program. A hardware timermust be linked to the system’s interrupt handlervector. Since it injects faults on the basis of timerather than specific events or system state, it pro-

April 1997 79

Figure 3. FIST environment.

Errordata

Error dataReset

Reference CPU

Comparatorerror flip-flops

Logicanalyzer

Hostcomputer

Monitoringcomputer

Errordata

Commands andprogram loading

Serialport Memory

Test CPU

Reset

Data

Externalbus

Externalbus

Externalbus

Trigger

Inside vacuum chamber

80 Computer

duces unpredictable fault effects and programbehavior. It is, however, suitable for emulatingtransient faults and intermittent hardware faults.

• Exception/trap. In this case, a hardware excep-tion or a software trap transfer control to thefault injector. Unlike time-out, exception/trap caninject the fault whenever certain events or con-ditions occur. For example, a software trapinstruction inserted into a target program willinvoke the fault injection before the program exe-cutes a particular instruction. When the trap exe-cutes, an interrupt is generated that transferscontrol to an interrupt handler. A hardwareexception invokes injection when a hardware-observed event occurs (when a particular mem-ory location is accessed, for example). Bothmechanisms must be linked to the interrupt han-dler vector.

• Code insertion. In this technique, instructions areadded to the target program that allow faultinjection to occur before particular instructions,much like the code-modification method. Unlikecode modification, code insertion performs faultinjection during runtime and adds instructionsrather than changing original instructions. Unlikethe trap method, the fault injector may exist aspart of the target program and run at user moderather than system mode.

Selected toolsFerrari7 (Fault and Error Automatic Real-Time

Injection), developed at the University of Texas atAustin, uses software traps to inject CPU, memory,and bus faults. Ferrari consists of four components:the initializer and activator, the user information, thefault-and-error injector, and the data collector andanalyzer.

The fault-and-error injector uses software trap andtrap handling routines. Software traps are triggeredeither by the program counter when it points to thedesired program locations or by a timer. When thetraps are triggered, the trap handling routines injectfaults at the specific fault locations, typically by chang-ing the content of selected registers or memory loca-tions to emulate actual data corruptions. The faultsinjected can be those permanent or transient faultsthat result in an address line error, a data line error,and a condition bit error.

Experiments conducted on Sun SparcStationsshowed that error detection is highly dependent onthe fault type. Faults in the task memory resulted inthe highest level of detection, due mainly to therepeated injection of faults when trap instructionswere placed in program loops. Also, many faultsinjected into I/O routines and system libraries wentundetected because these routines were less frequentlyexercised.7

The Fault Tolerance and Performance Evaluator(Ftape),8 developed at the University of Illinois, con-sists of the components shown in Figure 4. Engineerscan inject faults into user-accessible registers in CPUmodules, memory locations, and the disk subsystem.

Figure 4. Ftape environment.

Faultinjection

Workloadgenerator

Workloadspecifications

Faults Workload

Normalized

1

0Measure

Workloadactivity

Workloadspecifications

Fault injectionspecifications

Injection strategy: Random Stress-based

CPU parameters: Register set

Memory parameters: Kernel/user space Text/data/heap/ stack space

I/O parameters: Disk controller error set

Composition(relative mix of): CPU function Memory function I/O function

Level of dataflow between functions

Intensity of functions

CPU Memory I/O

Disk

network

video

The faults are injected as bit-flips to emulate error asa result of faults.

Disk system faults are injected by executing a rou-tine in the driver code that emulates I/O errors (buserror and timer error, for example). Fault injection dri-vers added to the operating system inject the faults,so no additional hardware or modification of appli-cation code is needed. A synthetic workload genera-tor creates a workload containing specified amountsof CPU, memory, and I/O activity, and faults areinjected with a strategy that considers the character-istics of the workload at the time of injection (whichcomponents are experiencing the greatest amount ofworkload activity, for example). Ftape has been usedon several Tandem fault-tolerant computers and servesas the basis of a benchmark for fault tolerance, whichmeasures the occurrence of system failures and theamount of performance degradation under fault con-ditions.

Doctor9 (Integrated Software Fault InjectionEnvironment), developed at the University ofMichigan, allows injection of CPU faults, memoryfaults, and network communication faults. It uses threetriggering methods—time-out, trap, and code modifi-cation—to trigger fault injection. Time-out triggersmemory fault injection. Once time-out occurs, it trig-gers the fault injector to overwrite the memory con-tent to emulate occurrence of a memory fault. Fornonpermanent CPU faults, traps trigger fault injection.For permanent CPU faults, fault injection is done bychanging program instructions during compilation toemulate instruction and data corruptions due to thefaults. Doctor has been used on Harts, a distributed,real-time system, to investigate the effect of intermit-tent message losses between two adjacent nodes andthe effect of routing using failure data. The researchersused experimental results to validate a message deliv-

ery model and to evaluate different message deliverymethods.

Xception,10 developed at the University ofCoimbra in Portugal, takes advantage of theadvanced debugging and performance monitoringfeatures present in many modern processors to injectmore realistic faults. It requires no modification inapplication software and no insertion of softwaretraps. Xception, in fact, uses a processor’s built-inhardware exception triggers to trigger fault injection.The fault injector is implemented as an exceptionhandler and requires modification of the interrupthandler vector. Xception faults are triggered basedon access to specific addresses (rather than on a timeperiod following an event), so the experiments arereproducible. The following events can trigger faultinjection:

• opcode fetch from a specified address, • operand load from a specified address, • operand store to a specified address, • a specified time passed since start-up, and • a combination of the above fault triggers.

Each fault has a specified fault mask: a set of bitsthat determines which corresponding bits in the tar-get location will be injected. Bits in the fault mask setto 1 can use several bit-level operations: stuck-at-zero,stuck-at-one, bit-flip, and bridging. Xception has beenimplemented on a Parsytec parallel machine based onthe PowerPC 601 processor. Experiments revealed thedeficiency in the error detection mechanisms by show-ing that up to 73 percent of injected faults resulted inincorrect results that were undetected for certainprocessor functional units.

Table 2 classifies the hardware and software faultinjection methods.

April 1997 81

Table 2. Characteristics of fault injection methods.

Hardware SoftwareWith contact Without contact Compilation Runtime

Cost High High Low Low

Perturbation None None Low High

Risk of damage High Low None None

Monitoring High High High Lowtime- resolution

Accessibility of Chip pin Chip internal Register memory Register memoryfault injection software I/O controller/portpoints

Controllability High Low High High

Trigger Yes No Yes Yes

Repeatability High Low High High

82 Computer

The contrast between the hardware and softwaremethods lies mainly in the fault injection pointsthey can access, cost, and level of perturbation.

Hardware methods can inject faults into chip pinsand internal components, such as combinational cir-cuits and registers that are not software-addressable.On the other hand, software methods are convenientfor directly producing changes at the software-statelevel (memory, register, for example). Thus, we usehardware methods to evaluate low-level error detec-tion and masking mechanisms and software meth-ods to test higher level mechanisms. Softwaremethods are less expensive, but they also incur ahigher perturbation overhead because they executesoftware on the target system. ❖

AcknowledgmentsThis research was supported in part by the US

Department of Defense Advanced Research ProjectsAgency under contract DABT63-94-C-0045, ONRcontract N00014-91-J-1116, and by NASA GrantNAG 1-613, in cooperation with the IllinoisComputer Laboratory for Aerospace Systems andSoftware (ICLASS).

References1. J.A. Clark and D.K. Pradhan, “Fault Injection: A

Method for Validating Computing-System Dependabil-ity,” Computer, June 1995, pp. 47-56.

2. D. Tang and R.K. Iyer, “Experimental Analysis of Com-puter System Dependability,” in Fault-Tolerant Com-puter System Design, D.K. Pradhan, ed., Prentice-HallProf. Tech. Ref., Upper Saddle River, N.J., pp. 282-392.

3. J.A. Abraham, “Challenges in Fault Detection,” Proc.25th Ann. Int’l Symp. Fault-Tolerant Computing, IEEECS Press, Los Alamitos, Calif., 1995, pp. 96-114.

4. J. Arlat, Y. Crouzet, and J.C. Laprie, “Fault Injectionfor Dependability Validation of Fault-Tolerant Com-puter Systems,” Proc. 19th Ann. Int’l Symp. Fault-Tolerant Computing, IEEE CS Press, Los Alamitos,Calif., 1989, pp. 348-355.

5. O. Gunnetlo, J. Karlsson, and J. Tonn, “Evaluation ofError Detection Schemes Using Fault Injection byHeavy-ion Radiation,” Proc. 19th Ann. Int’l Symp.Fault-Tolerant Computing, IEEE CS Press, Los Alami-tos, Calif., 1989, pp. 340-347.

6. J. Karlsson, J. Arlat, and G. Leber, “Application of ThreePhysical Fault Injection Techniques to the Experimen-tal Assessment of the MARS Architecture,” Proc. FifthAnn. IEEE Int’l Working Conf. Dependable Comput-ing for Critical Applications, IEEE CS Press, Los Alami-tos, Calif., 1995, pp. 150-161.

7. G.A. Kanawati, N.A. Kanawati, and J.A. Abraham,“FERRARI: A Tool for the Validation of SystemDependability Properties,” Proc. 22nd Ann. Int’l Symp.

Fault-Tolerant Computing, IEEE CS Press, Los Alami-tos, Calif., 1992, pp. 336-344.

8. T.K. Tsai and R.K. Iyer, “An Approach to Benchmark-ing of Fault-Tolerant Commercial Systems,” Proc. 26thAnn. Int’l Symp. Fault-Tolerant Computing, IEEE CSPress, Los Alamitos, Calif., 1996, pp. 314-323.

9. S. Han, K.G. Shin, and H.A. Rosenberg, “Doctor: AnIntegrated Software Fault-Injection Environment for Dis-tributed Real-Time Systems,” Proc. Second Annual IEEEInt’l Computer Performance and Dependability Symp.,IEEE CS Press, Los Alamitos, Calif., 1995, pp. 204-213.

10. J. Carreira, H. Madeira, and J.G. Silva, “Xception: Soft-ware Fault Injection and Monitoring in Processor Func-tional Units,” Proc. Fifth Ann. IEEE Int’l Working Conf.Dependable Computing for Critical Applications, IEEECS Press, Los Alamitos, Calif., pp. 135-149.

Mei-Chen Hsueh is a visiting research associate pro-fessor at the Coordinated Science Laboratory in theUniversity of Illinois at Urbana-Champaign. She is ona leave of absence from Digital Equipment Corpora-tion. Her interests are in high-performance anddependable system design, fault-tolerance computing,and large complex system design. Hsueh received aPhD in computer science from the University of Illi-nois at Urbana-Champaign. She is a member of theIEEE and the IEEE Computer Society.

Timothy K. Tsai is a member of the technical staff atBell Labs, Lucent Technologies. His research interestsinclude fault-tolerant computing, software engineer-ing, and distributed systems. Tsai received a BS in elec-trical engineering from Brigham Young University andan MS and a PhD in electrical engineering from theUniversity of Illinois at Urbana-Champaign. He is amember of the IEEE and the IEEE Computer Society.

Ravishankar K. Iyer is a professor in the departmentsof Electrical and Computer Engineering and Com-puter Science, and at the Coordinated Science Labo-ratory, at the University of Illinois at Urbana-Champaign. He is also codirector of the Center forReliable and High-Performance Computing and theIllinois Computing Laboratory for Aerospace Systemsand Software, a NASA Center for Excellence in Aero-space Computing. Iyer’s research interests are in thearea of reliable computing, measurement and evalua-tion, and automated design. He is an associate fellowof the AIAA and a fellow of the IEEE.

Readers can contact Hsueh and Iyer at the Coordi-nated Science Laboratory, University of Illinois,Urbana, IL 61801; hsueh, [email protected]. Con-tact Tsai at Lucent Technologies, 600 Mountain Ave.,Murray Hill, NJ 07954; [email protected].