SENG 637 SENG 637 Dependability Reliability & Dependability Reliability & Dependability, Reliability & Dependability, Reliability & Testing of Software Testing of Software Systems Systems Cl Cl S ft D l t S ft D l t Cleanroom Cleanroom Software Development Software Development (Chapter 11) (Chapter 11) Department of Electrical & Computer Engineering, University of Calgary B.H. Far ([email protected]) http://www enel ucalgary ca/People/far/Lectures/SENG637/ SENG635 (Winter 2007) [email protected]1 http://www.enel.ucalgary .ca/People/far/Lectures/SENG637/
78
Embed
SENG 637 Dependability Reliability & Dependability ...people.ucalgary.ca/~far/Lectures/SENG637/PDF/SENG637-11.pdf · SENG 637 Dependability Reliability & Dependability, Reliability
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
SENG 637SENG 637Dependability Reliability & Dependability Reliability & Dependability, Reliability & Dependability, Reliability & Testing of Software Testing of Software SystemsSystems
ClCl S ft D l tS ft D l tCleanroomCleanroom Software DevelopmentSoftware Development(Chapter 11)(Chapter 11)
Department of Electrical & Computer Engineering, University of Calgary
B.H. Far ([email protected])http://www enel ucalgary ca/People/far/Lectures/SENG637/
Based on data representing 8 380 SE projects only 16 2% of projects Based on data representing 8,380 SE projects, only 16.2% of projects met the delivery date, the budget and with all of the specified features and functions. 31% of projects were cancelled before they were completed 52 7% were delivered with over-budget over-schedule orcompleted, 52.7% were delivered with over budget, over schedule or with fewer features and functions than specified.
Software Productivity Research [Chapman 2000]Software Productivity Research [Chapman 2000] %60 of the United State’s software work force is dedicated to fixing
software errors that could have been avoided. In addition, there are only 47 days in a calendar year dedicated to doing development oronly 47 days in a calendar year dedicated to doing development or enhancement of software applications. The rest is spent mainly on fixing the bugs.
The First Computer Bug!The First Computer Bug!The First Computer Bug!The First Computer Bug! On September 9th 1945,
Grace Murray Hopper wasGrace Murray Hopper was working on the Harvard University Mark II Aiken Relay Calculator when theRelay Calculator when the machine was experiencing problems.
An investigation showed An investigation showed that there was a moth trapped between the points of Relay #70 in Panel Fof Relay #70, in Panel F.
Cleanroom SECleanroom SECleanroom SECleanroom SE Cleanroom software engineering (CSE) is an engineering
process for the development of high quality software withprocess for the development of high quality software with certified reliability with the emphasis on design with no defects and test based on software reliability engineeringconcepts.concepts.
CSE focuses on defect prevention instead of defect correction, and certification of reliability for the intended environment of use.
CSE yields software that is correct by mathematically sound design, and software that is certified by statistically valid testing.g
CSE represents a paradigm shift from traditional, craft-based SE practices to rigorous, engineering-based practices.
CSE: HistoryCSE: HistoryCSE: HistoryCSE: History 1983: Original idea of Cleanroom came from one of Dr. Harlan Mills’
published papers1987 P d b D Mill SE h d l Th 1987: Proposed by Dr. Mills as a SE methodology. The name “Cleanroom” was borrowed from the electronics industry
1988: Defense Advanced Research Projects Agency (DARPA) Software Technology for Adaptable Reliable Systems (STARS) focus on gy p y ( )Cleanroom
1991-1992: Prototyping of Cleanroom Process Guide 1992: A book of CSE published, foundation of CSE
1992 1993 A d Ai F D i f Cl 1992-1993: Army and Air Force Demonstration of Cleanroom Technology
1993-1994: Prototyping of Cleanroom tools 1995: Commercialization of a Cleanroom Certification Tool 1995: Commercialization of a Cleanroom Certification Tool 1995: Cleanroom and CMM Consistency Review …
Informal load or coverage testing Statistical usage testing
Cleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: Technologies Development practices are based on mathematical function
theorytheory Test practices are based on applied statistics.
Analysis and design models are based on incremental software model and created using box structure
t ti A b l t th t (representation. A box encapsulates the system (or some aspect of the system) at a specific level of abstraction.
Correctness verification is applied once the box structureCorrectness verification is applied once the box structure design is complete.
Cleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: TechnologiesCleanroom SE: Technologies Software is tested by defining a set of usage scenarios (i.e.,
i i l d ) d i i hoperations or operational modes), determining the probability of use for each scenario (i.e., operational profile), and then defining random tests that conform to theand then defining random tests that conform to the probabilities.
Error records are checked No corrective actions are taken Error records are checked. No corrective actions are taken. Only certification test is conducted to check whether errors (i.e., current failure intensity) meet the projected reliability ( , y) p j y(i.e., failure intensity objective) for the software component.
CSE: Specification Process /1CSE: Specification Process /1CSE: Specification Process /1CSE: Specification Process /1 Requirements AnalysisRequirements Analysis
Elicitation and analyzes of requirements Elicitation and analyzes of requirements Define requirements for the software product Obtain agreement with the customer on the requirements
Requirements are reconfirmed or clarified throughout the Requirements are reconfirmed or clarified throughout the incremental development and certification process.
Function SpecificationFunction SpecificationB th lt f R i t A l i Base on the result of Requirements Analysis
Specify the complete functional behavior of the software in all possible modes of use
Obtain agreement with the customer on the specified function as the basis for software development and certification
CSE: Specification Process /2CSE: Specification Process /2CSE: Specification Process /2CSE: Specification Process /2 Usage SpecificationUsage Specification
Identify and classify software users usage scenarios and Identify and classify software users, usage scenarios, and environments of use (operational modes)
Establish and analyze the probability distribution for ft d lsoftware usage models
Obtain agreement with the customer on the specified usage as the basis for software certification
Architecture SpecificationArchitecture Specification Define the conceptual model, the structural organization,
and the execution characteristics of the softwareand the execution characteristics of the software Architecture definition is a multi-level activity that spans
CSE: Specification Process /3CSE: Specification Process /3CSE: Specification Process /3CSE: Specification Process /3 Increment PlanningIncrement Planning
Allocate customer requirements defined in the Function Specification to a series of softwarei h i f h S f A hiincrements that satisfy the Software Architecture,
Define schedule and resource allocations for i t d l t d tifi tiincrement development and certification
Obtain agreement with the customer on the increment planincrement plan
A set of correctness verification activities on the design and moves later to code. First level verification is via application of a set of “correctness questions”.pp q
Code generation, inspection & verification (CG & Code generation, inspection & verification (CG & CI)CI) The box structure transformed to a programming language.
Walkthrough and code inspection techniques are used to ensure semantic conformance with the box structureensure semantic conformance with the box structure.
Cleanroom Strategy /3Cleanroom Strategy /3Cleanroom Strategy /3Cleanroom Strategy /3 Statistical test planning (TP)Statistical test planning (TP)
Planning the test based on operational modes, operational profiles and reliability.
Statistical use testing (SUT)Statistical use testing (SUT) Statistical use testing (SUT)Statistical use testing (SUT) Creating test case, execute them and collecting error data.
Certification (C)Certification (C) Certification (C)Certification (C) Conducting certification test rather than reliability growth
to accept/reject developed software components (using reliability demonstration chart, etc).
S ifi th b h i f t t f t Specifies the behavior of a system or a part of a system. The system responds to specific stimuli (events) by applying a set of transition rules that map the stimuli to response.
State boxState box Encapsulates state data and services (operations) Input to Encapsulates state data and services (operations). Input to
the state box and outputs are represented. Clear boxClear box
Transition function that are implied by the state box. It contains the procedural design of the state box.
CSE: Certification Process /1CSE: Certification Process /1CSE: Certification Process /1CSE: Certification Process /1 Usage modeling and test planningUsage modeling and test planning
A usage model represents a possible usage scenario of the software A usage model represents a possible usage scenario of the software Usage model is based on usage specification and is used for testing
CSE: Certification Process /2CSE: Certification Process /2CSE: Certification Process /2CSE: Certification Process /2 Statistical Testing and CertificationStatistical Testing and Certification
Testing is conducted in a formal statistical design under experimental control.
The software is demonstrated to perform correctly with The software is demonstrated to perform correctly with respect to its specification.
Statistically valid estimates of the properties addressed by the certification goals are derived for the software.
Management decisions on continuation of testing and certification of the software are based on statisticalcertification of the software are based on statistical estimates of software quality.
Cleanroom TestingCleanroom TestingCleanroom TestingCleanroom Testing Using statistical usage concept for testing. Determine a usage probability distribution via the
following steps:1) A l h ifi i id if f i li1) Analyze the specification to identify a set of stimuli
(direct and indirect input variables).2) Create usage scenarios (operational modes).2) Create usage scenarios (operational modes).3) Assign probability to use of each stimuli (operational
profile).4) Generate test cases for each stimuli according to the
Certification TestCertification TestCertification TestCertification Test Cleanroom approach DOES NOT emphasize on
Unit or integration testing Bug fixing as a result of test and regression
C tifi ti d i l th f ll i Certification procedure involves the followings: Create usage scenarios Specify a usage profile Specify a usage profile Generate test cases from the profile Execute test cases and record failure data Compute reliability and certify the component or system
M th ti l d l i l f d ti f d fi i Mathematical and logical foundation for defining requirements accurately with precise notation.
Proactive versus reactive approach with regards to requirements validation.
Ambiguous, inconsistent and conflicting requirements are caught before the system test.g y
Box structure uses black, state, and clear box and it is a stepwise approach to refine requirements.Usage models define how the software is to be used by the Usage models define how the software is to be used by the customer.
Quick and clean development in Cleanroom Engineering Quick and clean development in Cleanroom Engineering Continuous validation Provides measurable progress Manage higher risk requirements (i e prototype) Manage higher risk requirements (i.e. prototype). Tracking of requirements Stepwise building functionalities that satisfies stakeholders’
requirementsrequirements Allows for fast delivery on important parts Focus on planning and discipline at management level and technical
level Statistical testing make the project quality control in proper level Verifiable specifications
Incomplete or conflicting requirements cannot be resolved at the beginning to determine increments
Risk analysis has not been incorporated explicitly Risk analysis has not been incorporated explicitly Need more care about configuration management Requires extra planning at both the management and q p g g
technical levels Stable requirements for each increment is needed, i.e.,
cannot adapt q ickl to “rapidl changing” req irementscannot adapt quickly to “rapidly changing” requirements
Evaluation: Certification TestEvaluation: Certification TestEvaluation: Certification TestEvaluation: Certification Test DisadvantagesDisadvantages
Testing is derived from a usage model that must be exhaustive in order to select a subset for testing
Statistical testing and verification will be more reliable if it Statistical testing and verification will be more reliable if it is based on the some history data
It would be effective if it could be integrated with other testing methods
Testing is not suitable for bug-huntingH man resid al coding errors ma not be addressed Human residual coding errors may not be addressed
Cleanroom SE: Case StudyCleanroom SE: Case StudyCleanroom SE: Case StudyCleanroom SE: Case Study Cleanroom software development relies on a
th ti ll d d l f d i tmathematically sound model of design to ensure that no defects are introduced into the softwaresoftware.
Cleanroom Software Specification and Design begins with an external view (black box) andbegins with an external view (black box), and is transformed into a state machine view (state box), and is fully developed into a procedurebox), and is fully developed into a procedure (clear box).
Box StructureBox StructureBox StructureBox Structure Box structures map system inputs and the
ti l hi t i ( i i t ) i tstimulus histories (previous inputs) into outputs.I Bl k B t t ffi i t t t Is Black-Box construct sufficient to represent this? e.g. Jackson modelN No
Cleanroom SE: Process /1Cleanroom SE: Process /1Cleanroom SE: Process /1Cleanroom SE: Process /11) Define the system requirements2) S if d lid t th bl k b2) Specify and validate the black box
Define the system boundary and specify all stimuli and responsesp
Specify the black box mapping rules Validate the black box with owners and users
) if d if h b3) Specify and verify the state box Specify the state data and initial state values Specify the state box transition function Specify the state box transition function Derive the black box behavior of the state box and
Detailed definition of the calculator function and what it does must be given and verifiedand what it does must be given and verified with the customer Various formal methods can be used: graph Various formal methods can be used: graph
State Box Structure /2State Box Structure /2State Box Structure /2State Box Structure /2 Several state boxes can be generated
d di th bi ti f t bldepending on the combination of acceptable (unacceptable) inputs and historiesE l Example: If 1st operand is non-numeric and calc symbol are
typed the next state is error statetyped the next state is error state If 1st operand is numeric and any other key other
than calc symbol is typed the next state is error y ypstate
Cleanroom Testing /3Cleanroom Testing /3Cleanroom Testing /3Cleanroom Testing /3 We must generate a sequence of usage test cases that conform
to the usage probability distribution.to the usage probability distribution. A series of random numbers are generated between 0 and 99
that corresponds to the probability of stimuli occurrence. F l th f ll i d b For example, the following random number sequences are generated: 14 – 95 – 26 – 44 : A1; D1; B1; B1 81 – 19 – 31 – 69 38 – 21 – 52 – 84
The testing team executes the test cases noted above (and Test case
g (others) and verifies software behavior against the specification for the system.
Cleanroom CertificationCleanroom CertificationCleanroom CertificationCleanroom Certification The verification and testing techniques lead to certification of
software components p Certification implies that the reliability can be specified for
each component. Each component would have a certified reliability under the p y
usage scenario and testing regime. This information is needed for future use of the components.
The certification approach involves five steps: pp p 1. Usage scenario is created. 2. A usage profile is specified. 3. Test cases are generated from the profile. 4. Tests are executed and failure data are recorded and analyzed. 5. Reliability is computed and certified.
CSE: Overall AdvantagesCSE: Overall AdvantagesCSE: Overall AdvantagesCSE: Overall Advantages Suitable for iterative and incremental software
d l tdevelopment. Uses formal specification that defines more
t l fli t d l taccurate, less conflict and complete requirements.C ti ifi ti f ft lit i Continuous verification of software quality is possible.S ft lit b tifi d i Software quality can be certified using software reliability engineering method.
CSE: Overall DisadvantagesCSE: Overall DisadvantagesCSE: Overall DisadvantagesCSE: Overall Disadvantages Cleanroom advocates the use of sequence-based
specifications These are better suited to problemsspecifications. These are better suited to problems with a high degree of logical interactions. Not suitable for black boxes used in numerical or highly computational applications.
Non-functional requirements (real-time, security constraints) and a significant portion of algorithmicconstraints) and a significant portion of algorithmic requirements are hard to be represented by Box structure.
After requirements changes, rework of the box-structure is a time-consuming process.
ReferencesReferencesReferencesReferences Linger, R. and Trammel, C. (1996). Cleanroom
Software Engineering Reference Model Version 1 0Software Engineering Reference Model Version 1.0. http://www.sei.cmu.edu/pub/documents/96.reports/pdf/tr022.96.pdfp
Wolack, C. (2001). Taking The Art Out of Software Development – An In-Depth Review of Cleanroom S ft E i iSoftware Engineering. http://www.scisstudyguides.addr.com/papers/cwdiss725paper1.htm
Pressmen and Associates (2000). Cleanroom Engineering Resources.