P R E S E N T A T I O N International Conference On Software Testing, Analysis & Review NOV 8-12, 1999 • BARCELONA, SPAIN Presentation Paper Bio Return to Main Menu F9 Friday, Nov 12, 1999 Les Hatton Testing, Complexity, Coupling, Diagnosis and Repetitive Failure
45
Embed
Presentation Bio Return to Main Menu F9Presentation Bio Return to Main Menu Paper Bio F9 Friday, Nov 12, 1999 Les Hatton Testing, Complexity, ... Airbus A320 AF319, ... • Fault in
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Note the following quotations:-“Our students graduate and move into industry without anysubstantial knowledge of how to go about testing a program.Moreover, we rarely have any advice to provide in ourintroductory courses on how a student should go abouttesting and debugging his or her exercises”.
“Every programmer and programming organisation couldimprove immensely by performing a detailed analysis of thedetected errors, or at least a subset of them of duty of care”
“An efficient program debugger should be able to pinpointmost errors without going near a computer”
The most depressing aspect about these is thatthey were made in 1979 by Glen Myers.
This is called a stack trace. It points unerringly at theresponsible code line and usually takes a matter ofmoments to fix. This is why pointer failuresamount for a relatively small amount of failure inreleased systems.
This is a comparison of real valued variables. It isbroken in most programming languages. In 1982,the author and a colleague spent 14 weeks tryingto find this in the middle of 70,000 lines of signal-processing software, because it occasionallybehaved slightly differently on one machine thananother. An acceptance test found the symptoms.
In those systems where excessive complexity has been restricted, thecurve is truncated. Here a component specification issue is closelyinvolved with deciding the eventual optimal test strategy.
– It is overwhelmingly likely that systems will fail
– Systems are dominated by repetitive failure
– Relating failure to a responsible fault or faults isgetting much harder leading to a substantial“don’t know” category - the diagnosis problem.
All this leads us inevitably to recognise thatfailure is an inevitable property of softwaresystems and we should assess the risk by testquantification and plan for its management.
• The Patriot missile missed an incoming Scud inthe 1991 Gulf War, which then killed 29 people.
• The tracking software failed because therewere anomalously two different representationsof the constant “0.1”. This in turn was multipliedby system uptime to give a spatial error.
• The defect was found in systems testing toolate to fix.
• The system carried the message, “System mustbe rebooted every 8 hours”. This keeps spatialerror small enough. System was left up 48hours.
! Just to show that reluctance to learn isn’t solelythe preserve of software engineers ...
– The DC-10 cargo door saga" In the six months prior to the dreadful crash of the Turkish
Airlines DC-10 in Paris in March 1974, there had been no lessthan 1000 cargo door incidents amongst the then 100 strongfleet of DC-10s. They were disregarded.
" In this incident, the cargo door fell off, and the resulting de-pressurisation caused the cabin floor to collapse severing vitalcontrol cables..
! In essence poor diagnosis has the followingimplications for testing
– Testing priority switches in favour of riskmanagement. Tests find defects which cannotbe corrected.
– Testing is not only assessing the reliability of theproduct but its sensitivity. Well-designedproducts respond easily to correctivemaintenance, (and don’t require much of it !).(Think of car engine evolution)
– One of the responsibilities of testing is to assessdiagnosability
The point at which testing stops is the point at which it is economicallyviable to stop given the trade-off between early availability and therisk of failure. It is manifestly NOT an engineering decision.
! Testing is increasingly concerned with riskassessment
– Diagnosis is much harder
– Systems are much bigger and more tightly-coupled
! Testing is itself a diagnostic for thedevelopment process
! Testing should play a much bigger role indesign
Les Hatton
Les Hatton is an independent consultant in software reliability. He isalso Professor of Software Reliability at the Computing Laboratory,University of Kent, U.K. He holds a B.A. (1970) from King's College,Cambridge, an M.Sc. (1971) and Ph.D. (1973) from the University ofManchester, all in mathematics, an A.L.C.M. (1980) in guitar from theLondon College of Music, and an LL.M. in IT law from the University ofStrathclyde (1999). He received a number of international prizes forgeophysics in the 1970's and '80s culminating in the 1987 EuropeanConrad Schlumberger prize for his work in computational geophysics.
Shortly afterwards, he became interested in software reliability, andchanged careers to study the design of high-integrity and safety-criticalsystems on which he has been a keynote speaker at a number of softwareconferences. He is the author of numerous technical papers and booksand is finally nearing completion of another book entitled "SoftwareFailure: avoiding the avoidable and living with the rest. In October1998, he was voted amongst the “world’s leading scholars of systems andsoftware engineering” for the period 1993-1997 by the US Journal ofSystems and Software.