Slides of Test and SQA

Kiểm thử và Đảm bảo Chất lượng Phần mềm

Software Testing and SQA

Chapter 9

Quản lý chất lượng PhầnMềm

9.2Kiểm thử và Đảm bảo Chất lượng Phần mềm

Mục tiêu

Giới thiệu quá trình quản lý chất lượng vàcác hoạt động QLCL chínhGiải thích vai trò của chuẩn trong QLCLGiải thích khái niệm về độ đo phần mềm, đọđo dự báo và độ đo điều khiểnGiải thích độ đo có thể sử dụng ntn để đánhgiá chất lượng phần mềm và hạn chế củacách đo phần mềm


Topics covered

Process and product qualityQuality assurance and standardsQuality planningQuality control


Software quality management

Concerned with ensuring that the required level of quality is achieved in a software product.Involves defining appropriate quality standards and procedures and ensuring that these are followed.Should aim to develop a ‘quality culture’where quality is seen as everyone’s responsibility.


What is quality?

Quality, simplistically, means that a product should meet its specification.This is problematical for software systems

There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.);Some quality requirements are difficult to specify in an unambiguous way;Software specifications are usually incomplete and often inconsistent.


The quality compromise

We cannot wait for specifications to improve before paying attention to quality management.We must put quality management procedures into place to improve quality in spite ofimperfect specification.


Scope of quality management

Quality management is particularly important for large, complex systems. The quality documentation is a record of progress and supports continuity of development as the development team changes.For smaller systems, quality management needs less documentation and should focus on establishing a quality culture.


Quality management activities

Quality assuranceEstablish organisational procedures and standards for quality.

Quality planningSelect applicable procedures and standards for a particular project and modify these as required.

Quality controlEnsure that procedures and standards are followed by the software development team.

Quality management should be separate from project management to ensure independence.


Quality management and software development


The quality of a developed product is influenced by the quality of the production process.This is important in software development as some product quality attributes are hard to assess.However, there is a very complex and poorly understood relationship between software processes and product quality.

Process and product quality


Chất lượng căn cứ vào tiến trình (1)

There is a straightforward link between process and product in manufactured goods.More complex for software because:

The application of individual skills and experience is particularly important in software development;External factors such as the novelty of an application or the need for an accelerated development schedule may impair product quality.

Care must be taken not to impose inappropriate process standards - these could reduce rather than improve the product quality.


Chất lượng căn cứ vào tiến trình (2)


Define process standards such as how reviews should be conducted, configuration management, etc.Monitor the development process to ensure that standards are being followed.Report on the process to project management and software procurer.Don’t use inappropriate practices simply because standards have been established.

Practical process quality


Standards are the key to effective quality management.They may be international, national, organizational or project standards.Product standards define characteristics that all components should exhibit e.g. a common programming style.Process standards define how the software process should be enacted.

Đảm bảo chất lượng và chuẩn chất lượng


Encapsulation of best practice- avoids repetition of past mistakes.They are a framework for quality assurance processes - they involve checking compliance to standards.They provide continuity - new staff can understand the organisation by understanding the standards that are used.

Tầm quan trọng về chuẩn


Chuẩn sản phẩm và chuẩn tiến trình

Product standards Process standards

Design review form Design review conduct

Requirements document structure Submission of documents to CM

Method header format Version release process

Java programming style Project plan approval process

Project plan format Change control process

Change request form Test recording process


Các vấn đề về chuẩn

They may not be seen as relevant and up-to-date by software engineers.They often involve too much bureaucratic form filling.If they are unsupported by software tools, tedious manual work is often involved to maintain the documentation associated with the standards.


Involve practitioners in development. Engineers should understand the rationale underlying a standard.Review standards and their usage regularly. Standards can quickly become outdated and this reduces their credibility amongst practitioners.Detailed standards should have associated tool support. Excessive clerical work is the most significant complaint against standards.

Standards development


ISO 9000

An international set of standards for quality management.Applicable to a range of organisations from manufacturing to service industries.ISO 9001 applicable to organisations which design, develop and maintain products.ISO 9001 is a generic model of the quality process that must be instantiated for each organisation using the standard.


ISO 9001

Management responsibility Quality system

Control of non-conforming products Design control

Handling, storage, packaging anddelivery

Purchasing

Purchaser-supplied products Product identification and traceability

Process control Inspection and testing

Inspection and test equipment Inspection and test status

Contract review Corrective action

Document control Quality records

Internal quality audits Training

Servicing Statistical techniques


ISO 9000 certification

Quality standards and procedures should be documented in an organisational quality manual.An external body may certify that an organisation’s quality manual conforms to ISO 9000 standards.Some customers require suppliers to be ISO 9000 certified although the need for flexibility here is increasingly recognised.


ISO 9000 and quality management


Documentation standards

Particularly important - documents are the tangible manifestation of the software.Documentation process standards

Concerned with how documents should be developed, validated and maintained.

Document standardsConcerned with document contents, structure, and appearance.

Document interchange standardsConcerned with the compatibility of electronic documents.


Documentation process


Document standards

Document identification standardsHow documents are uniquely identified.

Document structure standardsStandard structure for project documents.

Document presentation standardsDefine fonts and styles, use of logos, etc.

Document update standardsDefine how changes from previous versions are reflected in a document.


Document interchange standards

Interchange standards allow electronic documents to be exchanged, mailed, etc. Documents are produced using different systems and on different computers. Even when standard tools are used, standards are needed to define conventions for their use e.g. use of style sheets and macros.Need for archiving. The lifetime of word processing systems may be much less than the lifetime of the software being documented. An archiving standard may be defined to ensure that the document can be accessed in future.


Quality planning

A quality plan sets out the desired product qualities and how these are assessed and defines the most significant quality attributes.The quality plan should define the quality assessment process.It should set out which organisational standards should be applied and, where necessary, define new standards to be used.


Quality plans

Quality plan structureProduct introduction;Product plans;Process descriptions;Quality goals;Risks and risk management.

Quality plans should be short, succinct documents

If they are too long, no-one will read them.


Software quality attributes

Safety Understandability Portability

Security Testability Usability

Reliability Adaptability Reusability

Resilience Modularity Efficiency

Robustness Complexity Learnability


Quality control

This involves checking the software development process to ensure that procedures and standards are being followed.There are two approaches to quality control

Quality reviews;Automated software assessment and software measurement.


Quality reviews

This is the principal method of validating the quality of a process or of a product.A group examines part or all of a process or system and its documentation to find potential problems.There are different types of review with different objectives

Inspections for defect removal (product);Reviews for progress assessment (product and process);Quality reviews (product and standards).


Types of review

Review type Principal purpose

Design or programinspections

To detect detailed errors in the requirements, design or code. A checklist ofpossible errors should drive the review.

Progress reviews To provide information for management about the overall progress of theproject. This is b oth a process and a product review and is concerned withcosts, plans and schedules.

Quality reviews To carry out a t echnical analysis of product components or documentation tofind mismatches between the specification and the component design, code ordocumentation and to ensure that defined quality standards have been followed.


A group of people carefully examine part or all of a software system and its associated documentation.Code, designs, specifications, test plans, standards, etc. can all be reviewed.Software or documents may be 'signed off' at a review which signifies that progress to the next development stage has been approved by management.

Quality reviews


Review functions

Quality function - they are part of the general quality management process.Project management function - they provide information for project managers.Training and communication function -product knowledge is passed between development team members.


Quality reviews

The objective is the discovery of system defects and inconsistencies.Any documents produced in the process may be reviewed.Review teams should be relatively small and reviews should be fairly short.Records should always be maintained of quality reviews.


Comments made during the review should be classified

No action. No change to the software or documentation is required;Refer for repair. Designer or programmer should correct an identified fault;Reconsider overall design. The problem identified in the review impacts other parts of the design. Some overall judgement must be made about the most cost-effective way of solving the problem;

Requirements and specification errors may have to be referred to the client.

Review results


Software measurement and metrics

Software measurement is concerned with deriving a numeric value for an attribute of a software product or process.This allows for objective comparisons between techniques and processes.Although some companies have introduced measurement programmes, most organisations still don’t make systematic use of software measurement.There are few established standards in this area.


Any type of measurement which relates to a software system, process or related documentation

Lines of code in a program, the Fog index, number of person-days required to develop a component.

Allow the software and the software process to be quantified.May be used to predict product attributes or to control the software process.Product metrics can be used for general predictions or to identify anomalous components.

Software metric


Predictor and control metrics


A software property can be measured.The relationship exists between what we can measure and what we want to know. We can only measure internal attributes but are often more interested in external software attributes.This relationship has been formalised and validated.It may be difficult to relate what can be measured to desirable external quality attributes.

Metrics assumptions


Internal and external attributes


The measurement process

A software measurement process may be part of a quality control process.Data collected during this process should be maintained as an organisational resource.Once a measurement database has been established, comparisons across projects become possible.


Product measurement process


Data collection

A metrics programme should be based on a set of product and process data.Data should be collected immediately (not in retrospect) and, if possible, automatically.Three types of automatic data collection

Static product analysis;Dynamic product analysis;Process data collation.


Data accuracy

Don’t collect unnecessary data The questions to be answered should be decided in advance and the required data identified.

Tell people why the data is being collected. It should not be part of personnel evaluation.

Don’t rely on memory Collect data when it is generated not after a project has finished.


A quality metric should be a predictor of product quality.Classes of product metric

Dynamic metrics which are collected by measurements made of a program in execution;Static metrics which are collected by measurements made of the system representations;Dynamic metrics help assess efficiency and reliability; static metrics help assess complexity, understandability and maintainability.

Product metrics


Dynamic and static metrics

Dynamic metrics are closely related to software quality attributes

It is relatively easy to measure the response time of a system (performance attribute) or the number of failures (reliability attribute).

Static metrics have an indirect relationship with quality attributes

You need to try and derive a relationship between these metrics and properties such as complexity, understandability and maintainability.


Software product metricsSoftware metric Description

Fan in/Fan-out Fan-in is a measure of the number of functions or methods that call some other functionor method (say X). Fan-out is the number of functions that are called by function X. Ahigh value for fan-in means that X i s tightly coupled to the rest of the design andchanges to X will have extensive knock-on effects. A high value for fan-out suggeststhat the overall complexity of X m ay be high because of the complexity of the controllogic needed to coordinate the called components.

Length of code This is a measure of the size of a program. Generally, the larger the size of the code of acomponent, the more complex and error-prone that component is likely to be. Length ofcode has been shown to be one of the most reliable metrics for predicting error-proneness in components.

Cyclomatic complexity This is a measure of the control complexity of a p rogram. This control complexity maybe related to program understandability. I discuss how to compute cyclomaticcomplexity in Chapter 22.

Length of identifiers This is a measure of the average length of distinct identifiers in a p rogram. The longerthe identifiers, the more likely they are to be m eaningful and hence the moreunderstandable the program.

Depth of conditionalnesting

This is a measure of the depth of nesting of if-statements in a program. Deeply nested ifstatements are hard to understand and are potentially error-prone.

Fog index This is a measure of the average length of words and sentences in documents. The higherthe value for the Fog index, the more difficult the document is to understand.


Object-oriented metricsObject-orientedmetric

Description

Depth of inheritancetree

This represents the number of discrete levels in the inheritance tree where sub-classes inherit attributes and operations (methods) from super-classes. Thedeeper the inheritance tree, the more complex the design. Many different objectclasses may have to be understood to understand the object classes at the leavesof the tree.

Method fan-in/fan-out

This is directly related to fan-in and fan-out as described above and meansessentially the same thing. However, it may be appropriate to make adistinction between calls from other methods within the object and calls fromexternal methods.

Weighted methodsper class

This is the number of methods that are included in a class weighted by thecomplexity of each method. Therefore, a simple method may have a complexityof 1 and a large and complex method a much higher value. The larger the valuefor this metric, the more complex the object class. Complex objects are morelikely to be more difficult to understand. They may not be logically cohesive socannot be reused effectively as super-classes in an inheritance tree.

Number ofoverridingoperations

This is the number of operations in a super-class that are over-ridden in a sub-class. A high value for this metric indicates that the super-class used may not bean appropriate parent for the sub-class.


Measurement analysis

It is not always obvious what data means Analysing collected data is very difficult.

Professional statisticians should be consulted if available.Data analysis must take local circumstances into account.


Measurement surprises

Reducing the number of faults in a program leads to an increased number of help desk calls

The program is now thought of as more reliable and so has a wider more diverse market. The percentage of users who call the help desk may have decreased but the total may increase;A more reliable system is used in a different way from a system where users work around the faults. This leads to more help desk calls.


Key points

Software quality management is concerned with ensuring that software meets its required standards.Quality assurance procedures should be documented in an organisational quality manual.Software standards are an encapsulation of best practice.Reviews are the most widely used approach for assessing software quality.


Key points

Software measurement gathers information about both the software process and the software product.Product quality metrics should be used to identify potentially problematical components.There are no standardised and universally applicable software metrics.

Danh sách các test tools (nc opensource) • jUnit (test unit cho Java) • csUnit (test unit cho .NET) • Nunit (.Net framework) • jTestcase • Jfunc (test function) • Ant/Nant

Cách thực hiện:

• Nghiên cứu các công cụ kiểm thử • Viết một phần mềm đơn giản bằng java hoặc .NET • Sử dụng các công cụ kiểm thử để test cho phần mềm trên.

Các loại kiểm thử • Unit test • Regrestion test • Function test • Integrated test • Test UML test • DB test




GVC Thạc Bình Cường, [email protected]@mail.hut.edu.vn

Mob: 0913226660Dept. of SE, Faculty of IT, HUT

Acknowledgments: Many thanks to Prof. Ian Sommerville and Prof. Roger S. Pressman

for providing the materials for this course


Vài nét về môn học KT&BĐCL PM

Tổng quan về môn họcMục tiêu Đối tượngTổ chức môn họcYêu cầu kỹ năng và hiểu biếtSách giáo trìnhTự học

Giới thiệu về đối tượng


Course goalsA. Knowledge of testing process, classification

scheme, terminologyB. Ability to classify test terminologyC. Knowledge of/skill with several test generation

methodsD. Understanding of possibilities, limitations, required

effort, mutual relations of test generation methodsE. Ability to determine given a situation which test

generation on method is suitableF. Verification and validationG. Software cost estimationH. SQA Management


Course overview

1 Introduction: Concepts 2 V&V 3 Testing4 Data-based black box testing 5 Structure-based white box testing6 Structure-based black box testing7 Integration testing8 Software reliability engineering9 Testing in industry10 SW cost estimation11 SW quality management12 Configuration management13 Additional: Real-time & Embedded systems design & Test


Required knowledge, skills

Language (speak/understand/write) [Vietnamese/English/Japanese/…]

IT Fundamentals (Computer science):Skill in programming (making+correcting errors)SE Fundamentals, software development processBehavioural specification languages: process algebra, state machines, petri nets

Mathematics:Logic, sets, reasoningStatistics


Textbooks and References

1. Ian Sommerville: “Software Engineering”, 7th Ed., 2004.2. Roger S. Pressman: “Software Engineering: A Practitioner's Approach”, 6th

Ed., McGraw-Hill, 2004.3. John Musa: “Software Reliability Engineering”, McGraw-Hill, 1998. 4. Barry W. Boehm et al.: “Software Cost Estimation with COCOMO II”,

Prentice Hall PTR, 2000.5. Websites on SW Testing and SQA6. Materials, Handouts7. David E. Simon: “An Embedded Software Primer”, Addison-Wesley, 1999.8. Frank Vahid and Tony Givargis: “Embedded System Design. A Unified

Hardware/Software Introduction”, John Wiley & Sons, Inc.2002.9. Phillip A. Laplante: “Real-time Systems Design and Analysis. An

Engineer’s Handbook”, 2nd Ed., IEEE Press, 1997.10. A.W. Leigh: “Real-time Software for Small Systems”, Sigma Press, 1987.


Outline

Course overviewChapter 1: Introduction to the subject

What is... error/bug/fault/failure/testing?Overview of the testing processClassifying aspects of testing


Chapter 1Introduction: Concepts

What is... error/bug/fault/failure/testing?Overview of the testing processClassifying aspects of testing


What is...

error: something wrong in software itself

fault: manifestation of an error(observable in software behaviour)

failure: something wrong in software behaviour

(deviates from requirements)requirements:for input i,give output 2*(i^3)

(so 6 yields 432)

software:i=input(STDIN);i=double(i);i=power(i,3);output(STDOUT,i);

output (verbose):input: 6doubling input..computing power..output: 1728

bug

error fault

fault + failure


What is...

Testing: by experiment,

find errors in software (Myers, 1979)establish quality of software (Hetzel, 1988)

a successful test: finds at least one errorgives quality judgment with maximum confidence with minimum effort


What’s been said?

Dijkstra:Testing can show the presence of bugs, but not the absence

Beizer:1st law: Every method you use to prevent or find bugs leaves a residue

of subtler bugs, for which other methods are needed2nd law: Software complexity grows to the limits of our ability to manage

it

Beizer:Testers are not better at test design than programmers are at code

design

...Let’s start with a global look at the testing process


Concept map of the testing process


Dimensions of software testing

1. What is tested?a. Software characteristics (design/embedded?/language/...)

b. Requirements (what property/quality do we want from the software)

c. Purpose (what development phase, what kind of errors, what next step)

2. How to test?a. Method (adequacy criterion: how to generate how many tests)

b. Assumptions (limitations, simplifications, heuristics)

c. Organization (documentation, implementation, execution:platform, interactive, batch)

d. Who tests? (programmer, user, third party)

3. How are the results evaluated?


Dimensions + concept map

1a

1b

1c2a2b

2c

2d

3


1b. Requirements

functional: the behaviour of the system is correctprecise

non-functional:performance, reliability, compatibility, robustness (stress/volume/recovery), usability, ...possibly quantifiable, always vague


1c. Test purpose

Software development phases (V-model):

implementationcode

detaileddesign

specification

requirements acceptancetest

systemtest

integrationtest

unittest


1c. Test purpose

What errors to find?typical typesfunctional mistakesunimplemented required features‘software does not do all it should do’

implemented non-required features‘software does things it should not do’


2a. Test method dimensions

data-based

structure-based

black box white boxerror seedingtypical errors

efficiency...


2b. Assumptions, limitations

Single/multiple fault:clustering/dependency of errors

Testability: can software be tested?

time, resources, accessibility,

Perfect repair...


2c. Test organization

Documentationfor reuse!

Implementationplatformbatch?inputs, coordination, ...

Executionduration, timing, interruptionsmanual/interactive or automatedin parallel on several systems


Dimensions + concept map classify aspects of testing

smoke testingrequirements: only major functionspurpose: often system testingmethod: error guessingassumption: (regression testing) a bit of testing suffices

capture/replay tooltest implementation & execution: the tool records test input

behaviour and saves it for automatic reusetraceability matrix

requirements, test set: the matrix shows relationship between these, gives some coverage confidence


Chapter 2Verification and Validation

Thẩm tra và Phê Chuẩn


GVC Thạc Bình CườngDept. of SE, Faculty of IT, HUT


Objectives

To introduce software verification and validation and to discuss the distinction between them.To describe the program inspection process and its role in V & V.To explain static analysis as a verification technique.To describe the Cleanroom software development process.


Topics covered

Verification and validation planningSoftware inspectionsAutomated static analysisCleanroom software development


Verification: "Are we building the product right?”.

The software should conform to its specification.

Validation:"Are we building the right product?”.

The software should do what the user really requires.

Verification vs validation


Is a whole life-cycle process - V & V must be applied at each stage in the software process.Has two principal objectives

The discovery of defects in a system;The assessment of whether or not the system is useful and useable in an operational situation.

The V & V process


V& V goals

Verification and validation should establish confidence that the software is fit for purpose.This does NOT mean completely free of defects.Rather, it must be good enough for its intended use and the type of use will determine the degree of confidence that is needed.


V & V confidence

Depends on system’s purpose, user expectations and marketing environment

Software functionThe level of confidence depends on how critical the software is to an organisation.

User expectationsUsers may have low expectations of certain kinds of software.

Marketing environmentGetting a product to market early may be more important than finding defects in the program.


Software inspections. Concerned with analysis of the static system representation to discover problems (static verification)

May be supplement by tool-based document and code analysis

Software testing. Concerned with exercising and observing product behaviour (dynamic verification)

The system is executed with test data and its operational behaviour is observed

Static and dynamic verification


Static and dynamic V&V


Can reveal the presence of errors NOT their absence.The only validation technique for non-functional requirements as the software has to be executed to see how it behaves.Should be used in conjunction with static verification to provide full V&V coverage.

Program testing


Defect testingTests designed to discover system defects.A successful defect test is one which reveals the presence of defects in a system.

Validation testingIntended to show that the software meets its requirements.A successful test is one that shows that a requirements has been properly implemented.

Types of testing


Defect testing and debugging are distinct processes.Verification and validation is concerned with establishing the existence of defects in a program.Debugging is concerned with locating and repairing these errors.Debugging involves formulating a hypothesis about program behaviour then testing these hypotheses to find the system error.

Testing and debugging


The debugging process


Careful planning is required to get the most out of testing and inspection processes.Planning should start early in the development process.The plan should identify the balance between static verification and testing.Test planning is about defining standards for the testing process rather than describing product tests.

V & V planning


The V-model of development


The structure of a software test plan

The testing process.Requirements traceability.Tested items.Testing schedule.Test recording procedures.Hardware and software requirements.Constraints.


Lập kế hoạch kiểm thử

Quá trình kiểm thửMô tả các giai đoạn chính của quá trình kiểm thử .

Lần theo dấu vếtNgười sử dụng hầu như chỉ quan tâm đến hệ thống có đáp ứng các yêu cầu và kiểm thử cần lập kế hoạch sao cho các yêu cầu được kiểm thửmọt cách độc lập .

Tested itemsThe products of the software process that are to be tested should be specified.

Testing scheduleAn overall testing schedule and resource allocation for this schedule. This, obviously, is linked to the more general project development schedule.


Lập kế hoạch kiểm thử(cont.)

Thủ tục ghi chép lại kiểm thử. It is not enough simply to run tests. The results of the tests must be systematically recorded. It must be possible to audit the testing process to check that it been carried out correctly.

Các yêu cầu về phần cứng và phần mềmThis section should set out software tools required and estimated hardware utilisation.

Ràng buộcConstraints affecting the testing process such as staff shortages should be anticipated in this section.


Software inspections

These involve people examining the source representation with the aim of discovering anomalies and defects.Inspections not require execution of a system so may be used before implementation.They may be applied to any representation of the system (requirements, design,configuration data, test data, etc.).They have been shown to be an effective technique for discovering program errors.


Inspection success

Many different defects may be discovered in a single inspection. In testing, one defect ,may mask another so several executions are required.The reuse domain and programming knowledge so reviewers are likely to have seen the types of error that commonly arise.


Inspections and testing

Inspections and testing are complementary and not opposing verification techniques.Both should be used during the V & V process.Inspections can check conformance with a specification but not conformance with the customer’s real requirements.Inspections cannot check non-functional characteristics such as performance, usability, etc.


Program inspections

Formalised approach to document reviewsIntended explicitly for defect detection (not correction).Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an uninitialised variable) or non-compliance with standards.


Inspection pre-conditionsA precise specification must be available.Team members must be familiar with the organisation standards.Syntactically correct code or other system representations must be available. An error checklist should be prepared.Management must accept that inspection will increase costs early in the software process.Management should not use inspections for staff appraisal ie finding out who makes mistakes.


The inspection process


Inspection procedure

System overview presented to inspection team.Code and associated documents are distributed to inspection team in advance.Inspection takes place and discovered errors are noted.Modifications are made to repair discovered errors.Re-inspection may or may not be required.


Inspection rolesAuthor or ownerThe programmer or designer responsible for producing the program or document. Responsible for fixing defects discovered during the inspection process.

InspectorFinds errors, omissions and inconsistencies in programs and documents. May also identify broader issues that are outside the scope of the inspection team.

ReaderPresents the code or document at an inspection meeting.


Inspection roles (cont.)

Scribe (Secretary)Records the results of the inspection meeting.

Chairman or moderator Manages the process and facilitates the inspection. Reports process results to the Chief moderator.

Chief moderatorResponsible for inspection process improvements, checklist updating, standards development etc.


Inspection checklists

Checklist of common errors should be used to drive the inspection.Error checklists are programming language dependent and reflect the characteristic errors that are likely to arise in the language.In general, the 'weaker' the type checking, the larger the checklist.Examples: Initialisation, Constant naming, loop termination, array bounds, etc.


Inspection checks 1a

•Data faults• Are all program variables initialised before

their values are used?• Have all constants been named?• Should the upper bound of arrays be

equal to the size of the array or Size -1?• If character strings are used, is a

delimiter explicitly assigned?• Is there any possibility of buffer overflow?


Inspection checks 1b•Control faults

• For each conditional statement, is the condition correct?

• Is each loop certain to terminate?• Are compound statements correctly

bracketed?• In case statements, are all possible cases

accounted for?• If a break is required after each case in

case statements, has it been included?


Inspection checks 1c

•Input/output faults•Are all input variables used?•Are all output variables assigned a value before they are output?

•Can unexpected inputs cause corruption?


Inspection checks 2a•Interface faults

• Do all function and method calls have the correct number of parameters?

• Do formal and actual parameter types match?

• Are the parameters in the right order? • If components access shared memory, do

they have the same model of the shared memory structure?


Inspection checks 2b•Storage management faults

• If a linked structure is modified, have all links been correctly reassigned?

• If dynamic storage is used, has space been allocated correctly?

• Is space explicitly de-allocated after it is no longer required?

•Exception management faults• Have all possible error conditions been

taken into account?


Inspection rate

500 statements/hour during overview.125 source statement/hour during individual preparation.90-125 statements/hour can be inspected.Inspection is therefore an expensive process.Inspecting 500 lines costs about 40 man/hours effort - about £2800 at UK rates.


Automated static analysis

Static analysers are software tools for source text processing.They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team.They are very effective as an aid to inspections - they are a supplement to but not a replacement for inspections.


Static analysis checks 1Fault class and Static analysis checkData faults

• Variables used before initialisation• Variables declared but never used• Variables assigned twice but never used

between assignments• Possible array bound violations • Undeclared variables

Control faults• Unreachable code• Unconditional branches into loops


Static analysis checks 2

Input/output faults• Variables output twice with no intervening

assignment• Interface faults• Parameter type mismatches• Parameter number mismatches• Non-usage of the results of functions• Uncalled functions and procedures

Storage management faults• Unassigned pointers• Pointer arithmetic


Stages of static analysis

Control flow analysis. Checks for loops with multiple exit or entry points, finds unreachable code, etc.Data use analysis. Detects uninitialised variables, variables written twice without an intervening assignment, variables which are declared but never used, etc.Interface analysis. Checks the consistency of routine and procedure declarations and their use


Stages of static analysisInformation flow analysis. Identifies the dependencies of output variables. Does not detect anomalies itself but highlights information for code inspection or reviewPath analysis. Identifies paths through the program and sets out the statements executed in that path. Again, potentially useful in the review processBoth these stages generate vast amounts of information. They must be used with care.


LINT static analysis138% more lint_ex.c#include <stdio.h>printarray (Anarray) int Anarray;{ printf(“%d”,Anarray); }

main (){ int Anarray[5]; int i; char c; printarray (Anarray, i, c); printarray (Anarray) ;}

139% cc lint_ex.c140% lint lint_ex.c

lint_ex.c(10): warning: c may be used before setlint_ex.c(10): warning: i may be used before setprintarray: variable # of args. lint_ex.c(4) :: lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(11)printf returns value which is always ignored


Use of static analysis

Particularly valuable when a language such as C is used which has weak typing and hence many errors are undetected by the compiler,Less cost-effective for languages like Java that have strong type checking and can therefore detect many errors during compilation.


Verification and formal methods

Formal methods can be used when a mathematical specification of the system is produced.They are the ultimate static verification technique.They involve detailed mathematical analysis of the specification and may develop formal arguments that a program conforms to its mathematical specification.


Arguments for formal methods

Producing a mathematical specification requires a detailed analysis of the requirements and this is likely to uncover errors.They can detect implementation errors before testing when the program is analyzed alongside the specification.


Arguments against formal methods

Require specialized notations that cannot be understood by domain experts.Very expensive to develop a specification and even more expensive to show that a program meets that specification.It may be possible to reach the same level of confidence in a program more cheaply using other V & V techniques.


The name is derived from the 'Cleanroom' process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal.This software development process is based on:

Incremental development;Formal specification;Static verification using correctness arguments;Statistical testing to determine program reliability.

Cleanroom software development


The Cleanroom process


Cleanroom process characteristics

Formal specification using a state transition model.Incremental development where the customer prioritises increments.Structured programming - limited control and abstraction constructs are used in the program.Static verification using rigorous inspections.Statistical testing of the system.


Formal specification and inspections

The state based model is a system specification and the inspection process checks the program against this model.The programming approach is defined so that the correspondence between the model and the system is clear.Mathematical arguments (not proofs) are used to increase confidence in the inspection process.


Specification team. Responsible for developing and maintaining the system specification.Development team. Responsible for developing and verifying the software. The software is NOT executed or even compiled during this process.Certification team. Responsible for developing a set of statistical tests to exercise the software after development. Reliability growth models used to determine when reliability is acceptable.

Cleanroom process teams


The results of using the Cleanroom process have been very impressive with few discovered faults in delivered systems.Independent assessment shows that the process is no more expensive than other approaches.There were fewer errors than in a 'traditional' development process.However, the process is not widely used. It is not clear how this approach can be transferred to an environment with less skilled or less motivated software engineers.

Cleanroom process evaluation


Key points

Verification and validation are not the same thing. Verification shows conformance with specification; validation shows that the program meets the customer’s needs.Test plans should be drawn up to guide the testing process.Static verification techniques involve examination and analysis of the program for error detection.


Key points

Program inspections are very effective in discovering errors.Program code in inspections is systematically checked by a small team to locate software faults.Static analysis tools can discover program anomalies which may be an indication of faults in the code.The Cleanroom development process depends on incremental development, static verification and statistical testing.



Chương 3Kiểm thử phần mềm


Mục tiêu

Thảo luận sự Phân biệt giữa kiểm thửchấp nhận và kiểm thử phát hiện lỗi Mô tả các nguyên lý của kiểm thử hệthống và kiểm thử thành phần Mô tả chiến lược tạo ra các trường hợp kiểm thử hệ thống Hiểu các đặc trưng cần thiết của công cụ sử dụng cho tự động hoá kiểm thử


Các chủ đề

Kiểm thử Hệ thống Kiểm thử thành phần Thiết kế trường hợp kiểm thửTự động hoá kiểm thử


The testing process

Component testing Testing of individual program components;Usually the responsibility of the component developer (except sometimes for critical systems);Tests are derived from the developer’s experience.

System testingTesting of groups of components integrated to create a system or sub-system;The responsibility of an independent testing team;Tests are based on a system specification.


Testing phases


Defect testing

The goal of defect testing is to discover defects in programsA successful defect test is a test which causes a program to behave in an anomalous wayTests show the presence not the absence of defects


Testing process goals

Validation testingTo demonstrate to the developer and the system customer that the software meets its requirements;A successful test shows that the system operates as intended.

Defect testingTo discover faults or defects in the software where its behaviour is incorrect or not in conformance with its specification;A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.


The software testing process


Only exhaustive testing can show a program is free from defects. However, exhaustive testing is impossible,Testing policies define the approach to be used in selecting system tests:

All functions accessed through menus should be tested;Combinations of functions accessed through the same menu should be tested;Where user input is required, all functions must be tested with correct and incorrect input.

Testing policies

ngovi

Note

Chien luoc kiem thu


System testing

Involves integrating components to create a system or sub-system.May involve testing an increment to be delivered to the customer.Two phases:

Integration testing - the test team have access to the system source code. The system is tested as components are integrated.Release testing - the test team test the complete system to be delivered as a black-box.


Integration testing

Involves building a system from its components and testing it for problems that arise from component interactions.Top-down integration

Develop the skeleton of the system and populate it with components.

Bottom-up integrationIntegrate infrastructure components then add functional components.

To simplify error localisation, systems should be incrementally integrated.


Incremental integration testing


Testing approaches

Architectural validationTop-down integration testing is better at discovering errors in the system architecture.

System demonstrationTop-down integration testing allows a limited demonstration at an early stage in the development.

Test implementationOften easier with bottom-up integration testing.

Test observationProblems with both approaches. Extra code may be required to observe tests.


Release testing

The process of testing a release of a system that will be distributed to customers.Primary goal is to increase the supplier’s confidence that the system meets its requirements.Release testing is usually black-box or functional testing

Based on the system specification only;Testers do not have knowledge of the system implementation.


Black-box testing

IeDu lieu kiem thu

OeOutput test results

SHe thong

Inputs causinganomalousbehaviour

Outputs which revealthe presence ofdefects


Testing guidelines

Testing guidelines are hints for the testing team to help them choose tests that will reveal defects in the system

Choose inputs that force the system to generate all error messages;Design inputs that cause buffers to overflow;Repeat the same input or input series several times;Force invalid outputs to be generated;Force computation results to be too large or too small.


Testing scenario

A student is studying Vietnam History and has been asked to write a paper on ‘Vietnamese people's anti-American war during 1954-1975". To do this, she needs to find sources from a range of libraries, including the US ones. She logs on to the LIBSYS system and uses the search facility to discover if she can access original documents from that time. She discovers sources in various US university libraries and downloads copies of some of these. However, for one document, she needs to have confirmation from her university that she is a genuine student and that use is for non-commercial purposes. The student then uses the facility in LIBSYS that can request such permission and registers her request. If granted, the document will be downloaded to the registered library’s server (e.g. HUT's e-Library) and printed for her. She receives a message from LIBSYS telling her that she will receive an e-mail message from the Department of Information-Documentation Services when the printed document is available for collection.


System tests

1. Test the login mechanism using correct and incorrect logins to checkthat valid users are accepted and invalid users are rejected.

2. Test the search facility using different queries against known sources tocheck that the search mechanism is actually finding documents.

3. Test the system presentation facility to check that information aboutdocuments is displayed properly.

4. Test the mechanism to request permission for downloading.

5. Test the e-mail response indicating that the downloaded document isavailable.


Use cases

Use cases can be a basis for deriving the tests for a system. They help identify operations to be tested and help design the required test cases.From an associated sequence diagram, the inputs and outputs to be created for the tests can be identified.


Collect weather data sequence chart


Performance testing

Part of release testing may involve testing the emergent properties of a system, such as performance and reliability.Performance tests usually involve planning a series of tests where the load is steadily increased until the system performance becomes unacceptable.


Stress testing

Exercises the system beyond its maximum design load. Stressing the system often causes defects to come to light.Stressing the system test failure behaviour.. Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data.Stress testing is particularly relevant to distributed systems that can exhibit severe degradation as a network becomes overloaded.


Component testing

Component or unit testing is the process of testing individual components in isolation.It is a defect testing process.Components may be:

Individual functions or methods within an object;Object classes with several attributes and methods;Composite components with defined interfaces used to access their functionality.


Object class testing

Complete test coverage of a class involvesTesting all operations associated with an object;Setting and interrogating all object attributes;Exercising the object in all possible states.

Inheritance makes it more difficult to design object class tests as the information to be tested is not localised.


Weather station object interface


Weather station testing

Need to define test cases for reportWeather, calibrate, test, startup and shutdown.Using a state model, identify sequences of state transitions to be tested and the event sequences to cause these transitionsFor example:

Waiting -> Calibrating -> Testing -> Transmitting -> Waiting


Objectives are to detect faults due to interface errors or invalid assumptions about interfaces.Particularly important for object-oriented development as objects are defined by their interfaces.

Interface testing


Interface testing


Interface types

Parameter interfacesData passed from one procedure to another.

Shared memory interfacesBlock of memory is shared between procedures or functions.

Procedural interfacesSub-system encapsulates a set of procedures to be called by other sub-systems.

Message passing interfacesSub-systems request services from other sub-systems.


Interface errorsInterface misuse

A calling component calls another component and makes an error in its use of its interface e.g. parameters in the wrong order.

Interface misunderstandingA calling component embeds assumptions about the behaviour of the called component which are incorrect.

Timing errorsThe called and the calling component operate at different speeds and out-of-date information is accessed.


Interface testing guidelines

Design tests so that parameters to a called procedure are at the extreme ends of their ranges.Always test pointer parameters with null pointers.Design tests which cause the component to fail.Use stress testing in message passing systems.In shared memory systems, vary the order in which components are activated.


Test case design

Involves designing the test cases (inputs and outputs) used to test the system.The goal of test case design is to create a set of tests that are effective in validation and defect testing.Design approaches:

Requirements-based testing;Partition testing;Structural testing.


Requirements based testing

A general principle of requirements engineering is that requirements should be testable.Requirements-based testing is a validation testing technique where you consider each requirement and derive a set of tests for that requirement.


LIBSYS requirements

• The user shall be able to search either all of the initial set of databases or select a subset from it.

• The system shall provide appropriate viewers for the user to read documents in the document store.

• Every order shall be allocated a unique identifier (ORDER_ID) that the user shall be able to copy to the account’s permanent storage area.


LIBSYS tests• Initiate user search for searches for items that are known to

be present and known not to be present, where the set ofdatabases includes 1 database.

• Initiate user searches for items that are known to be presentand known not to be present, where the set of databasesincludes 2 databases

• Initiate user searches for items that are known to be presentand known not to be present where the set of databasesincludes more than 2 databases.

• Select one database from the set of databases and initiateuser searches for items that are known to be present andknown not to be present.

• Select more than one database from the set of databasesand initiate searches for items that are known to be presentand known not to be present.


Partition testing

Input data and output results often fall into different classes where all members of a class are related.Each of these classes is an equivalence partition or domain where the program behaves in an equivalent way for each class member.Test cases should be chosen from each partition.


Equivalence partitioning


Equivalence partitions


Search routine specificationprocedure Search (Key : ELEM ; T: SEQ of ELEM;

Found : in out BOOLEAN; L: in out ELEM_INDEX) ;

Pre-condition-- the sequence has at least one elementT’FIRST <= T’LAST

Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)

or-- the element is not in the array( not Found andnot (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))


Inputs which conform to the pre-conditions.Inputs where a pre-condition does not hold.Inputs where the key element is a member of the array.Inputs where the key element is not a member of the array.

Search routine - input partitions


Testing guidelines (sequences)

Test software with sequences which have only a single value.Use sequences of different sizes in different tests.Derive tests so that the first, middle and last elements of the sequence are accessed.Test with sequences of zero length.


Search routine - input partitions

Sequence ElementSingle value In sequenceSingle value Not in sequenceMore than 1 value First element in sequenceMore than 1 value Last element in sequenceMore than 1 value Middle element in sequenceMore than 1 value Not in sequence

Input sequence (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 29, 21, 23 17 true, 141, 18, 9, 31, 30, 16, 45 45 true, 717, 18, 21, 23, 29, 41, 38 23 true, 421, 23, 29, 33, 38 25 false, ??


Sometime called white-box testing.Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases.Objective is to exercise all program statements (not all path combinations).

Structural testing


Structural testing


Pre-conditions satisfied, key element in array.Pre-conditions satisfied, key element not in array.Pre-conditions unsatisfied, key element in array.Pre-conditions unsatisfied, key element not in array.Input array has a single value.Input array has an even number of values.Input array has an odd number of values.

Binary search - equiv. partitions


Binary search equiv. partitions


Binary search - test cases

Input array (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 21, 23, 29 17 true, 19, 16, 18, 30, 31, 41, 45 45 true, 717, 18, 21, 23, 29, 38, 41 23 true, 417, 18, 21, 23, 29, 33, 38 21 true, 312, 18, 21, 23, 32 23 true, 421, 23, 29, 33, 38 25 false, ??


Path testing

The objective of path testing is to ensure that the set of test cases is such that each path through the program is executed at least once.The starting point for path testing is a program flow graph that shows nodes representing program decisions and arcs representing the flow of control.Statements with conditions are therefore nodes in the flow graph.


Binary search flow graph


1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 141, 2, 3, 4, 5, 141, 2, 3, 4, 5, 6, 7, 11, 12, 5, …1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …Test cases should be derived so that all of these paths are executedA dynamic program analyser may be used to check that paths have been executed

Independent paths


Test automation

Testing is an expensive process phase. Testing workbenches provide a range of tools to reduce the time required and total testing costs.Systems such as Junit support the automatic execution of tests.Most testing workbenches are open systems because testing needs are organisation-specific.They are sometimes difficult to integrate with closed design and analysis workbenches.


A testing workbench


Testing workbench adaptation

Scripts may be developed for user interface simulators and patterns for test data generators.Test outputs may have to be prepared manually for comparison.Special-purpose file comparators may be developed.


Key points

Testing can show the presence of faults in a system; it cannot prove there are no remaining faults.Component developers are responsible for component testing; system testing is the responsibility of a separate team.Integration testing is testing increments of the system; release testing involves testing a system to be released to a customer.Use experience and guidelines to design test cases in defect testing.


Key pointsInterface testing is designed to discover defects in the interfaces of composite components.Equivalence partitioning is a way of discovering test cases - all cases in a partition should behave in the same way.Structural analysis relies on analysing a program and deriving tests from this analysis.Test automation reduces testing costs by supporting the test process with a range of software tools.

1Kiểm thử và Đảm bảo Chất lượng Phần mềm


Chương 4

Phương pháp kiểm thử


Nội dung chính

Data-based black box testingclassificationmethods

boundary-value testingequivalence partitioningdecision tableserror guessing, random testing

ExampleStructure-based white box testingData/structure-based white box testingStructure-based black box testing

ngovi

Note

phan hoach tuong duong - ko kiem thu het tat ca gia tri dụuoc - kiem thu bang cach lay mot so thanh phan













1a

1b

1c2a2b

2c

2d

3


Test purpose

Software development phase:

implementationcode

detaileddesign

specification


systemtest

integrationtest

unittest


Test method dimensions

data-based

structure-based


efficiency...


Classifying data-based black box testing

allrequirements: some logical/mathematical input/output relationpurpose: unit testing, functional mistakesmethod: black box, data-basedassumption: both single-/multiple-fault assumption

boundary value testingmethod: typical errors (boundaries of domains)

equivalence partitioning/decision tablesmethod: efficiency


Running example: mortgage

input: gender [m,f], age [18-55], monthly salary [0-10000]output: mortgage total for one personrequirements:

mortgage= salary*factor

program: (41-50 years) 35(46-55 years) 30old(31-40 years) 50(36-45 years) 55middle(18-30 years) 70(18-35 years) 75young

femalemalecategory

Mortgage (male:Boolean, age:Integer, salary:Integer): Integerbeginif (male) thenreturn ((18<=age<35)?(75*salary):(31<=age<40)?(55*salary):(30*salary))

else /* female */return ((18<=age<30)?(75*salary):(31<=age<40)?(50*salary):(35*salary))

fi;end


Testing the mortgage program

each test case = input combination + expected output valuecomplete testing requires all input combinations: infeasibleselect some combinationsassumptions:

single/multiple faultinput type: quantity, enumerable, boundedtypical errorsefficiency: redundant input combinations


Boundary value testing

detect errors to do with input domain boundsfor integer input x with domain [a,b], test input values around a and b# tests = for n inputs, 4n+1 input combinationsassumes:

independent quantity inputssingle-fault assumption

d

c

a b

y

x


Robustness boundary value

also test values outside the domain# tests = for n input variables, 6n+1 input combinations

a test: <x_nom, y_max+1, expected output>

d

c

a b

y

x

maximal value

minimal value

nominal value


Worst-case boundary value

multiple-fault assumption# tests = for n input variables, 5n input combinations

d

c

a b

y

x


Worst-case robustness bv

multiple-fault assumption, tests also outside the domain# tests =for n input variables, 7n input combinations

d

c

a b

y

x


Subsume relations

Method A subsumes method B if A tests at least what B tests (possibly more)Which subsume relations for boundary value variants?(Assume one fixed nominal value for each input)

d

c

a b

y

x

boundary value

A B

worst-case robustness

worst-case robustness


Boundary value testing Mortgage

ObservationsInput gender is not a quantityBoundaries are not fixed:Inputs gender and age are not independent!

Variant ‘boundary value’ may not detect error(if nominal value for gender is ‘male’)

Many more errors in the program than we can expect any boundary value variant to detect


Boundary value testing summary

Coverage: not good#tests: moderate to very manyUsage: straightforward, easy to implementWhen to use:

independent inputsenumerable quantities, e.g. age(obviously) when suspecting boundary errors

See literature:ZhuJorgensen



detect errors to do with computational mistakesfor integer input x with domain [a,b] partitioned in s_x subdomains, test input values from each subdomainassumes:

independent variablesredundancy in subdomain

f

e

a b

y

x

g

c d

weak normal variant:• assumes:

– independent variables– single-fault assumption

• #tests = max s_x for any input x



strong normal variant:assumes:

multiple-fault assumption#tests = Π s_x for each input x

f

e

a b

y

x

g

c d



weak robust variant:tests outside domainassumes:

independent variablessingle-fault assumption

#tests =max s_x for any input x+ 2*(#inputs)

f

e

a b

y

x

g

c d



strong robust variant:tests outside domainassumes:

multiple-fault assumption#tests = Π (s_x+2) for each

input xf

e

a b

y

x

g

c d


Subsume relations

Which subsume relations for equivalence partition variants?(Assume same value selected for each input, for

each subdomain)

x

weak normal

strong robust

strong normal weak robustf

e

a b

y

x

g

c d


Equivalence partitioning summary

Coverage: moderate#tests: small to moderateUsage: some study of requirements neededWhen to use:

independent inputsenumerable quantitieswhen suspecting computational errorswhen redundancy can be assumedmay easily be combined with boundary value


Decision table testing

also known as cause-effect graphingdetect errors to do with computational mistakesinput are partitioned through conditionsassumes: dependent variables#tests = depends on granularity of conditions/actions

xa3: faulty inputsx

xxa: correct answer type 1a2: correct answer type 2

ynn_c3: z even?-n-yc2: x<=y?-ynyc1: x>=0?

rule 5-8rule 4rule 2, 3rule 1conditions/actions - : don’t care_ : fixed value


Decision table testing Mortgage

a5: 50*agea4: 55*age


c5: < young?

c2: young?c3: middle?c4: old?


a1: wrong inputc7: 0 <= salary <= 10000?1c6: > old?

c1: male?conditions/actions rule 1 rule 2


Decision table testing summary

Coverage: good#tests: moderate to largeUsage: much study of requirement neededWhen to use:

dependent inputswhen suspecting computational errorsmay be combined with boundary testing (tricky)

See literature:ZhuJorgensen


Error guessing & Random Testing

Error guessingNo method involved, just experiment and see if something goes wrongSome people have a knack for exposing failures (e.g. young children!)

Random testingSelect input combinations randomly

Both can be very effective, but should be used in addition to methodic testing


Guidelines data-based black box testing

Inputs dependent on each other?decision table

Suspect computational errors?decision table or equivalence partition (see 1st bullet)

Suspect boundary errors?boundary value

Single-fault assumption?strong/worst-case variant

Suspect errors outside the domain?robust variant

Want to know what a non-average user does?error guessing, random testing

Combinations can and should be constructed

sensibly!


Outline

Data-based black box testingStructure-based white box testing

classificationexampleadequacy criteriagenerate tests?guidelines

Data/structure-based white box testingStructure-based black box testing



1a

1b

1c2a2b

2c

2d

3


Test purposeSoftware development phase:

implementationcode

detaileddesign

specification


systemtest

integrationtest

unittest



data-based

structure-based


efficiencycoverage


Classifying structure-based white box testing

all criteriarequirements: some logical/mathematical

input/output relationsoftware characteristics: available, imperative

programming language purpose: unit testing, functional mistakesmethod: white box, structure-based, coverage,

efficiencyassumption: both single and multiple-fault

assumption


Running example: triangle

input: a,b,c: [1-..]output: equilateral/isosceles/

scalene/not a trianglerequirements: mathematically

correcttesting: input combination,

expected outputstructure: each test is a pathcoverage: when ...

paths/branches/... executed

1. Triangle (a, b, c: Integer): String2. IsATriangle: Boolean3. begin4. if (a<=b+c or b<=a+c or c<=b+c)5. then IsATriangle := true6. else IsATriangle := false fi;7. if (IsATriangle)8. then if (a=b and b=c)9. then return “Equilateral”10. else if (a≠b and a≠c and b≠c)11. then return “Scalene”12. else return “Equilateral” fi; fi;13. else return “Not a triangle” fi;14. end

5 (6?) errors!

isosceles equilateral scalene


Testing the triangle program

5. 6.

7.

8.

9.

13.

10.

11. 12.

14.

4. 1. Triangle (a, b, c: Integer): String2. IsATriangle: Boolean3. begin4. if (a<=b+c or b<=a+c or c<=b+c)5. then IsATriangle := true6. else IsATriangle := false fi;7. if (IsATriangle)8. then if (a=b and b=c)9. then return “Equilateral”10. else if (a≠b and a≠c and b≠c)11. then return “Scalene”12. else return “Equilateral” fi; fi;13. else return “Not a triangle” fi;14. end

programgraph



each test represents a path: a=b=c=1

5. IsATriangle := true 6. IsATriangle := false

7. IsATriangle?

8. (a=b and b=c)?

9. “Equilateral”

13. “Not a triangle”

10. (a≠b and a≠c and b≠c)?

11. “Scalene” 12. “Equilateral”

14. end

4. (a<=b+c or b<=a+c or c<=b+c)?yes

no

yes

yes

yes

no

no

no



each test represents a path: a=b=c=1: “Equilateral”coverage:

nodesedges...

all paths feasible? no! (this is in general

undecidable)


7. IsATriangle?

8. (a=b and b=c)?





14. end


no

yes

yes

yes

no

no

no


Program graphs & testing

infeasible paths: combinations that cannot occur (dependent nodes)dead code

# paths: possibly infinite (loops!)

coverage criteria:...


Program graphs & testing

coverage criteria:statement: each node is executedbranch/decision: each edge is executed(multiple) condition:

let p_1, p_2, ... be the smallest parts in condition c (atomic predicates)condition coverage: each condition is executed so often that each atomic predicate p_i has evaluated to both truth valuesmultiple condition coverage: each condition is executed so often that the atomic predicates have evaluated to all possible truth value combinations

all paths

single fault

multiple fault


Subsume relations

A B: Criterion A subsumes criterion B if A tests at least what B tests

branch coverage

multiple condition coverageall paths

statement coverage

branch+condition coverage

condition coverage

Note: multiple condition coverage tests more, but may show fewer failures than branch+condition coverage! (Frankl & Weyuker, Provable Improvements on Branch Testing, IEEE Trans. Softw. Eng.:19(10), 1993)



each test represents a path: a=b=c=1: “Equilateral”coverage:

edges


7. IsATriangle?

8. (a=b and b=c)?





14. end


no

yes

yes

yes

no

no

no

Exercise (5 min): Construct input combinations,

such that each edge is executed.Compare program output values

to required output values.Which path is not feasible?


Observations

How to construct tests & get everything covered?Guidelines:

Start with test generation based on black box methodMeasure white box coverage for black box test setUse tools!Which coverage criterion? How many errors may remain?A bit stronger than branch coverage is very effective, e.g. branch + condition coverageIndustrial practice:

varies from statement coverage (100%?) to multiple condition coverage (?%)


What we haven’t discussed

Loops:once and zero timesmaximal + minimal number of times...

Alternative program graphs:LCSAJ blocks per node

Stepwise testing: graph condensationDiscrepancies between graph and requirements:

feasible paths not requiredrequired paths not present/feasible


Outline (Complementary)

Data-based black box testingStructure-based white box testingData/structure-based white box testingStructure-based black box testing


Data/structure-based criteria

variable x of the program:define occurrence: x gets a value (assignment)use occurrence: the value of x is used

computational (in the value for assignment)predicate (in a condition)

criteria:for each definition occurrence of x at node n, for

each/some use occurrence that can be reached from n, each/some path is executed

all definitions criterion







all uses criterion







all definition-use paths criterion


Structure-based white box testing summary

Coverage: poor to very goodbut even for weak criteria 100% not always possible!

#tests: few to very many

Usage: hard to generate tests, easier to measure coverage

When to use:in combination with black box generation methodselaborate methods only for vital parts of softwareif tools are available


Outline

Data-based black box testingStructure-based white box testingData/structure-based white box testingStructure-based black box testing













1a

1b

1c2a2b

2c

2d

3


Test purpose

Software development phase:

implementationcode

detaileddesign

specification


systemtest

integrationtest

unittest



data-based

structure-based


efficiencycoverage


Classifying structure-based black box testing

all criteriarequirements: behaviour, defined in FSMsoftware characteristics: can be modelled as

an FSM purpose: unit testing, functional mistakesmethod: black box, structure-based,

coverage, efficiencyassumption: ...


Running example: musical toy

input:event: a button is

pushed

output: event: sound changes

requirements:FSM (behaviour)

testing: ...

Rhythm

Quit

Song

1. PlayMusic2. inputs pushRhythm, pushSong,

pushQuit;3. outputs startRhythm, startSong, Quit;4. state Rhythm, Song: Boolean;5. begin6. while (true) do7. if (pushQuit) then8. Rhythm, Song := false, false;9. if (Rhythm and Song) then output

Quit fi10. elsif (pushRhythm) then11. if (Rhythm) then output Rhythm fi;12. Rhythm := true13. elsif (pushSong) then14. if (Song) then output Song fi; 15. Song := true fi16. od;17. end


FSM of musical toy requirements

1. PlayMusic2. inputs pushRhythm, pushSong,

pushQuit;3. outputs startRhythm, startSong, Quit;4. state Rhythm, Song: Boolean;5. begin6. while (true) do7. if (pushQuit) then8. Rhythm, Song := false, false;9. if (Rhythm and Song) then output

Quit fi10. elsif (pushRhythm) then11. if (Rhythm) then output Rhythm fi;12. Rhythm := true13. elsif (pushSong) then14. if (Song) then output Song fi; 15. Song := true fi16. od;17. end

Rhythm Song

Both

Off

PushRhythm?/StartRhythm!


PushRhythm?/-

PushRhythm?/-

PushSong?/StartSong!


PushSong?/-

PushQuit?/Quit!

PushQuit?/-

PushSong?/-


What is an FSM?

Rhythm Song

Both

Off



PushRhythm?/-

PushRhythm?/-



PushSong?/-

PushQuit?/Quit!

PushQuit?/-

PushSong?/-

FSM (Finite State Machine), or Mealy machine: <S,I,O,δ,λ>

states S (usually has a start state)alphabets: inputs I, outputs Otransfer function: δ:S×I→Soutput function λ:S×I→Ospecial no-output label: ‘-’assumptions!

suitable for: precise, abstract modelsinterfaces, protocols, embedded software, ...formal test generation


Assumptions for FSM modelling

Rhythm Song

Both

Off



PushRhythm?/-

PushRhythm?/-



PushSong?/-

PushQuit?/Quit!

PushQuit?/-

PushSong?/-

FSM must bedeterministic + input-enabled:

δ and λ are proper functionsfinite:

I,O,S all finitestrongly connected:

each state can be reached from any state

reduced:non-equal states have different observable behaviour


Two FSM models:

SpecificationFSM model of requirementssatisfies restrictions previous slideis used for test generation

ImplementationFSM model of software to be testedsatisfies restrictions except ‘reduced’not really needed, but:

justification for FSM-based testing#states must be estimated

implementation?


Testing Impl versus Spec:

test = sequence of inputs: (1) a? b? a? (2) b? c? a?expected outcome = sequence of outputs

expected: (1) e! e! e! (2) e! - -actually: (1) e! e! e! (2) e! e! e!

Impl implements Spec if paths(Impl) = paths(Spec)

?

s0

s1

t0

t1t2a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/- c?/-

a?/e!

a?/-

b?/e!

b?/e!c?/e!

c?/-

a?/-c?/-


Testing Impl versus Spec:

Impl implements Spec if paths(Impl) = paths(Spec)possible errors:1. wrong output (computational mistake)2. extra/missing states (extra/missing features)3. transition to wrong state (computational mistake)which paths to test??

?

s0

s1

t0

t1t2a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/- c?/-

a?/e!

a?/-

b?/e!

b?/e!c?/e!

c?/-

a?/-c?/-


Test generation from Spec

test each transition: <s,i?>1. go to state s from arbitrary state

(synchronizing sequence + transferring sequence)2. apply input i?

(test transition)3. check output λ(s,i)

(test transition)4. check if in state δ(s,i)

(UIO-sequence/distinguishing sequence/W-set/...)

test fails iff something wrong in step 3. or 4.

s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-









s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

a? b?









s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

for s0: (empty)for s1: a?









s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

e.g. for <s1,b?>: b?









s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

e.g. for <s1,b?>: e!









s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

a?output for state s1: -



The total set of tests:

s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

<s0,a>

...<s0,b>

state verification

expected output

transferring sequence

synchronisingsequence

transition

Homework: to be completed!


What we haven’t (yet) discussed

Why is it correct?Non-deterministic systemsData parametersLiterature: Lee and Yannakakis, “Principles and Methods of Testing Finite State machines—A survey”, Proc. of the IEEE:84(8), 1996.Separate input or output transitionsOther implementation criteria (subsume rel)Multi-process/distributed testingSummary, guidelines









s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-


Test generation assumptions

FSM = <S,I,O,δ,λ>Spec FSM:

is deterministic, complete, reducedhas a synchronizing sequencehas an UIO sequence for each state

Impl FSM :is deterministic, completehas no more states than Spec

Generalised δ,λ functions:δ(s, i_1 i_2 ... i_n) = t (target state of a sequence of inputs)λ(s, i_1 i_2 ... i_n) = o_1 o_2 ... o_n (sequence of outputs)


Synchronising sequence

sequence of inputsbrings FSM to one designated statedetermine through successor graph:

node: set of states S’={s,t,...} (initially: S)edge: from S’ to S’’ with i/o where

S’’ = { δ(s,i) | s∈S’ and λ(s,i)=o }synchronising sequence σ

all paths for σ from S end in same state {s}σ does not always exist!!often there is a reliable resetif exists, easy to determine in successor graph

s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

{s0,s1}

a?/e!

{s1}

b?/e! c?/-

a?/-


Successor graph for musical toy

synchronising sequence: PushQuit?

PR?/R!

{Rhythm,Both}PS?/S!

{Song,Both}

PS?/-PR?/-

{Both}

{Off,Rhythm,Song,Both}

PQ?/Q!{Off}

PQ?/-

PR?/R!

PR?/-PR?/-

PQ?/Q!PQ?/Q!

PS?/S!PS?/-

PS?/-

{Off,Rhythm,Song,Both}

PQ?/Q!{Off}

PQ?/-


Another FSM + successor graph

synchronising sequence? no!(see Lee & Yannakakis for proof construction)

s1

s2 s3

a?/0!

b?/1!

b?/0!

a?/1! a?/0!

b?/1!

FSM:

a?/0!

{s1,s3}

b?/1!

{s2,s3}

b?/0!a?/1!

{s3}

{s1,s2,s3}

b?/0!

{s1}

a?/1!

b?/1!

a?/0!

a?/0!

b?/1!{s2}

b?/0!

successor graph:


Relax: homing vs. synchronising

synchronising sequence:ignore outputsalways brings FSM to same end statedoes not always exist

homing sequence:check outputsoutputs determine which end statealways existsextra test effort:

check outputs, determine end statetransfer to initial state (or directly to state-under-test)

s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-


UIO sequence

Unique Input/Output sequence determines which first stateinputs (+ expected outputs per state)determine through successor graph with initial states:

node: set of states S’={s,t,...} (initial: S)path from S’ to S’’ with i1/o1 i2/02 ...

S’’={ s∈S’ | λ(s,i1 i2...) = o1 o2... }UIO sequence for s: path to {s}does not always exist!!

s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-

{s0,s1}

{s0}

a?/e!

{s1}

b?/e! c?/-

a?/-


For musical toy

UIO sequence for any state: PushRhythm? PushSong?

PS?/S!

{S,B}{S,B}

PS?/-PQ?/Q!

{O}{R,S,B}

PQ?/-

{O}{O}

{current states}{possible initial states}

{S,B}{O,R}

PR?/R!

{R,B}{O,S}

PR?/-

{B}{O}

{O,R,S,B}{O,R,S,B}

PS?/S! PS?/-PS?/-

{R,B}{R,B}

{B}{S}

{B}{R}

{B}{B}

PS?/S!

PR?/R!

{R,B}{O,S}

PR?/-

{B}{O}

{O,R,S,B}{O,R,S,B}

PS?/S! PS?/-PS?/-

{R,B}{R,B}

{B}{S}

{B}{R}

{B}{B}

PS?/S!


Yet another FSM

UIO sequences?s1: b?/0! s3: a?/1! s2: no UIO sequence!

s1

s2 s3

a?/0!

b?/1!

b?/0!

b?/1! a?/1!

a?/0!

FSM:

a?/0!

{s1}{s1,s2}

b?/1!

{s2}{s2,s3}

{s1}{s2,s3}

b?/1!a?/0!a?/0!

b?/0!a?/1!

{s1,s2,s3}{s1,s2,s3}

{s3}{s1}

{s3}{s3}

b?/0!

successor graphwith initial states:

{s3}{s1,s2}

b?/0!a?/1!

{s1,s2,s3}{s1,s2,s3}

{s3}{s1}

{s3}{s3}


Relax: UIO vs. W-set

UIO:1 sequence of inputs/outputs per statedoes not always exist

W-set:set W(s) of input (+ output) sequences for state sfor each distinct pair s,s’: W(s) contains a sequence that distinguishes s from s’always existsextra test effort:

large number of sequencesto establish state verification for s, repeatedly go back to s to test all sequences in W(s)

s0

s1a?/-

a?/e!b?/e!

b?/e!

c?/-

c?/-


Other relaxations (1)

Spec/Impl FSM:has data parameters (EFSM):

only test control structure (mediocre coverage)compute complete FSM (huge effort, very many tests)combine with data-based black box methods

is non-deterministic: randomize testing algorithm

is incomplete: test only specified parts

sta?/e!

ua?/e!

s in(y)? [NOT a]/e! [x:=y] t

s

u

a?/0! t

b?/1!


Other relaxations (2)

Impl FSM :has n more states than Spec:

for complete coverage: extra test effort factor |I|n

No FSM model possible:Labelled Transition System (LTS) modelinput/output conformance (ioco) methodsee KUN/UTwente courses on testing:

http://www.cs.kun.nl/~tretmans/testtechnieken/

sa? t

b?f!

ub?


Structure-based black box testing summary

Coverage: complete (formal) to rather good (relaxations)

#tests: moderate to very manyUsage: hard to model FSM and implement testsWhen to use:

when requirements = (non-trivial) behaviourwhen complete coverage is essentialunit/system/acceptance testingtools for test generation (commercially) available



Chapter 5

Kiểm thử tích hơp


Nội dung chính

Kiểm thử tích hơpPhân loại kiểm thửCách tiếp cận tích hợpNgữ nghĩa của hệ sinh kiểm thử và tínhphủ (coverage)Thí dụ


Mục đích

Giai đoạn phát triển phần mềm:

Lập trình

Thiết kế chi tiết

Đặc tả

Các yêu cầu Kiểm thử chấp nhận

Kiểm thử hệ thống

Kiểm thử tích hợp

Kiểm thử đơn vị


Phân loại kiểm thử tích hợp

Các yêu cầu: “Giao diện giữa các đơn vị CT đươc thực hiện đúng đắn “

Đặc trưng phần mềm : Truy cập phần mềm tạimức mô đun, không phải tới chính các đơn vịchương trình

Mục đích: Kiểm thử tích hơp, giao diện/các lỗiphối hợp

Giả định: Các đơn vị đã được kiểm thử trước đó


Sự tích hơp phần mềm được kiểmthử

Sự hơp thành các đơn vị chương trìnhCác đơn vị chương trình đã được kiểmthử trướcGiao diện phải được kiểm thử

Không có liên quan đến hành vi mức hệthống ! Lời gọi thủ tục : Cấu tạo đồ thị gọi

Để đảm bảo kiểm thử thực tế, mọi đơnvị CT cần bao gồm biểu diễn về

Các con của nó có mặt trong đồ thị lời gọiÍt ra có một cha trong đồ thị gọi

Main

GF

A C

E

D

B


Tiếp cận tích hợp: big-bang

Đặt các đơn vị cùng với nhau khichúng cùng một phần (1 sessionDựa vào các kết quả kiểm thử củađơn vị CT (relatively) Có ít các kiểm thửRất khó cô lập các lỗi

Main

GF

A C

E

D

B


Tiếp cận tích hợp: top-down

start with top unit: Mainreplace each unit called by Main with a stub:

dummy softwarereacts as original unit wouldeasy to program

1st session: test interfaces for Main

Main

GFE

s s

s

s


Tiếp cận tích hợp: top-down

second session: unit A...many stubs requiredmany test sessions required

1 session per unit

easy to isolate errors

Main

GFs

A s

s

s


Tiếp cận tích hợp: bottom-up

start with lowest-level unit: Ereplace unit calling E with a driver:

dummy softwaremakes calls as original unit wouldharder to program

many drivers requiredusually fewer than #stubs for top-down

many test sessions required1 session per unit


Main

GF

d C

E

D

B


Tiếp cận tích hợp: sandwich

mixture of top-down and bottom-upsome stubs, some drivers requiredtest interfaces between portions already testedmoderate #test sessions requirederrors harder to isolate

Main

GF

A C

E

D

B

top-down bottom-up


Tiếp cận tích hợp: pair-wise

per session, test a pair of units: C/Dsome drivers, some stubs required

fewer than bottom-up or top-down

many test sessions required1 session per pair


d

ss

A C

E

D

B


What is an integration test?

input values + driver/stub/unit set-upin the composed program graph: unit-paths with call/return markersm1 <Main calls C> c1 <C calls F> f1

<F returns C> c2 ...

Main

GF

A C

E

D

B

Main1

5

2

3

46

F

5

2

3

4

6

C1

2

3

4

m1

m2

c1

c2

f1


Sinh và phủ các kiểm thử tích hợp

Coverage in the call-graph:minimally/usually: all edges (each call-return interface is tested at least once)maximally: all feasible paths

Test generation requires(white box) the composed program graph(grey box) composition of units’ call/return models, e.g. FSMs

Main

GF

A C

E

DB

Main 1

5

23

46

F

52

3

4

6

C 1

234


Thực hiện ví dụ : bank machine

1

Bank Balance Machine

WELCOME

insert your card

2 3

4 5 6

7 8 9

0 Cancel

card


Đặc tả mức hệ thống FSM

Savings?/Screen6,CardOut!

ValidCardIn?/Screen2!

InvalidPIN?/Screen4!

Cancel?/Screen7,CardOut!

CardRemoved?/Screen1!

ValidPIN?/Screen5!

InvalidCardIn?/CardOut! idle

process PIN

process balance

end session

Checking?/Screen6,CardOut!


Thông báo màn hình của máyNgân hàng

Bank Balance Machine

WELCOME

Insert your card.

Enter your PIN:(Personal

IndentificationNumber)

____

(press Cancel to clear)

Invalid PIN!Please enter your

PIN again:

____

(press Cancel to clear)

Identification failed!

Your card is retained. Contact your bank.

Operation cancelled!

Please take your card.Thank you.

Your balance isbeing printed on

a receipt.

Please take yourreceipt and card.

Thank you.

Select your account:

checkingssavings

(press Cancel to abort)

Screen 1 Screen 4Screen 3Screen 2

Screen 5 Screen 7Screen 6


Phân rã chức năng ...

... Không giúp đỡ gì vì đồ thị-gọi nhìn kháchoàn toàn !

Bank Machine

PIN ValidationCard TypeValidation

Central BankCommunication

ScreenManagement

CardInput/Output

ReceiptsCardManagement

BalanceManagement


Đồ thị - gọi máy ngân hàng

Main

PrintScreen

CardIn

ValidateCard

CardOut

ValidatePIN

PrintReceipt

GetPIN

ContactBank

ManageBalance



Chapter 6Phần mềm khả hiện

(More Reliable Software)Một cách nhìn tổng quan về Nhanh

hơn và Rẻ hơn


Most Important Software Problem

1. Customers demand (in order):A. More reliable softwareB. Faster

C. Cheaper2. Your success in meeting demands affects

market share, profitability, your career3. Demands conflict, causing risk and

overwhelming pressure

Introduction


Software Reliability Engineering –Developed to Address the Problem

1. It differs from other approaches by being primarily quantitative.

2. You add and integrate software reliability engineering (SRE) with other good processes and practices; you do not replace them.

3. With SRE you control the development process, it doesn’t control you:A. Development process is not externally imposed.B. You use quantitative information to choose the most

cost-effective software reliability strategies for your situation.

Introduction


Outline

1. Nature of software reliability engineering (SRE)

2. SRE process (with illustration)

Introduction


Reliability and Availability Definitions1. Reliability: the probability that a system or a

capability of a system will continue to function without failure for a specified period in a specified environment. The period may be specified in natural or time units.A. Natural unit: unit other than time related to amount of

processing performed by software-based product, such as runs, pages of output, transactions, telephone calls, jobs, semiconductor wafers, queries, or API calls

B. Failure intensity (FI): failures per natural or time unit, an alternative way of expressing reliability

2. Availability: the average (over time) probability that a system or a capability of a system is currently functional in a specified environment

Nature of SRE


How Does SRE Work?

Nature of SRE – How Does SRE Work?

1. Increase effective resourcesA. Quantitatively characterize

expected useB. Focus resources on most

used and most critical functions

C. Maximize test effectiveness by making test highly representative of field

Increase in Effective

Resources

OriginalResources


How Does SRE Work?


2. Apply resources to maximize customer valueA. Set quantitative

objectives for major quality characteristics (reliability and/or availability, delivery time, price)

Rel / Avail

Time Price


How Does SRE Work?


B. Choose software reliability strategies to meet objectives

C. Track reliability in system test against objective as one of the release criteria

Added CustomerValue - Focus

Added CustomerValue -

Matching Needs

Original CustomerValue

OriginalCustomer Value


SRE - A Proven, Standard, Widespread Best Practice

Nature of SRE - Proven, Standard, Widespread Best Practice

1. Proven practiceA. Example: AT&T International Definity PBX

[6, pp 167-8]a. Reduced customer-reported problems by factor of

10b. Reduced system test interval by factor of 2c. Reduced total development time by 30%d. No serious service outages in 2 years of

deployment




B. AT&T Best Current Practice since 5/91 (based on widespread practice, documented strong benefit/cost ratio, probing review) [6, pp 219-254]

2. Standard practiceA. McGraw-Hill handbook [6]B. AIAA standard since 1993C. IEEE standards under development




3. Widespread practiceA. Users include Alcatel, AT&T, Bellcore, CNES

(France), ENEA (Italy), Ericsson Telecom, Hewlett Packard, Hitachi, IBM, NASA’s Jet Propulsion Laboratory, Lockheed-Martin, Lucent Technologies, Microsoft, Mitre, Nortel, Saab Military Aircraft, Tandem Computers, the US Air Force, and the US Marine Corps.

B. Over 50 published articles by users of software reliability engineering, continuing to grow ([1]; [3], pp. 371 - 374)


SRE Is Widely Applicable

Nature of SRE - Widely Applicable

1. Technically speaking, you can apply SRE to any software-based product, beginning at start of any release cycle.

2. Economically speaking, the complete SRE process may be impractical for small components (involving perhaps less than 2 staff months of effort), unless used in a large number of products. It may still be worthwhile to implement it in abbreviated form.

3. Independent of development technology and platform.

4. SRE requires no changes in architecture, design, or code - but it may suggest changes that would be beneficial.


Activities of SRE Process and Relationto Software Development Process

SRE Process

Design andImplementation

Requirementsand Architecture Test

5. Execute Test4. Prepare for Test

6. Guide Test

1. List AssociatedSystems

2. Implement OperationalProfiles

3. Engineer “Just Right”Reliability


Illustration - FONE FOLLOWER (FF) -Product Description

1. Subscriber calls FF, enters planned phone numbers (forwardees) to which calls are to be forwarded vs time.

2. FF forwards incoming calls (voice or fax) from network to subscriber as per program. Incomplete voice calls go to pager (if subscriber has one) and then voice mail.

SRE Process



SRE Process - List Associated Systems




6. Guide Test





List Associated Systems of Product

SRE Process - List Associated Systems

1. Base product2. Major variations of base product (for

substantially different environments, platforms, or configurations)

3. Frequently used supersystems of base product or variations



SRE Process - Implement OPs




6. Guide Test





Operation

Operation: major system logical task performed for initiator, which returns control to system when complete.Illustrations - FF:

Process fax call, Phone number entry, Audit section of phone number database

SRE Process - Implement OPs - Concepts


Operational Profile

Operational profile (OP): complete set of operations with probabilities of occurrenceIllustration - FF:

SRE Process - Implement OPs - Concepts

OperationOccur.Prob.

Process voice call, no pager, ans. 0.21Process voice call, pager, ans. 0.19Process fax call 0.17Process voice call, pager, ans. on page 0.13

•••

1


Develop Operational Profiles –Step by StepDevelop operational profile for base product and each variation (supersystems have same operational profiles as those of their related base product or variations).

SRE Process - Implement OPs – Develop OPs


Determine Operational Profile

1. Identify initiators of operations [1]2. Create operations list [1]3. Review operations list [1]4. Determine occurrence rates [1]5. Determine occurrence probabilities [1]Steps 1, 2, 3 are mostly the same across base product and variations. New release often requires only slight change from previous release, all steps.

SRE Process - Implement OPs – Develop OPs


Apply Operational Profiles

Apply operational profile and criticality information to increase efficiency of:1. Development of all developed software2. Test of all associated systems

SRE Process - Implement OPs - Apply OPs


Apply Operational Profiles to Increase Development Efficiency

For developed software in base product and each variation:1. Review functionality to be implemented

(Reduced Operation Software or ROS)2. Suggest operations where looking for

opportunities for reuse will be most cost-effective3. Plan a more competitive release strategy

(operational development)



Operational Development -Illustration

Proportion of operations developed

Proportion of use represented


Release 1

Release 2

Release 3


Apply Operational Profiles to Increase Development Efficiency

4. Allocate development resources: A. Among operations - for system engineering,

architectural design, requirements reviews, design reviews

B. Among modules - for code, code reviews, unit test ([3], pp. 118 - 119)



Apply Operational Profiles to Increase Test Efficiency

1. Allocate new test cases of release among new operations of base product and variations (Prepare for Test activity)

2. Allocate test time (Prepare for Test and Execute Test activities)




SRE Process - Engineer “Just Right” Reliability




6. Guide Test





Failure and Fault


Failure Fault

Departure of systembehavior in executionfrom user needs

Defect in systemimplementation thatcauses the failure whenexecuted

User-oriented Developer-oriented

Failure Fault

Departure of systembehavior in executionfrom user needs

Defect in systemimplementation thatcauses the failure whenexecuted

User-oriented Developer-oriented


Engineer “Just Right” Reliability -Step by Step

1. Define failure consistently over life of product release and clarify with examples [1]

2. Choose common reference measure for all failure intensities [1]

3. Set system FIO for each associated system [1]4. For any software you develop:

A. Find developed software FIO [1]B. Choose software reliability strategies to meet developed

software FIO and schedule objectives with lowest development cost [1]




SRE Process - Prepare for Test

Test



Requirementsand Architecture


6. Guide Test




Prepare for Test1.Specify new test cases for new operations

A. Distribute new test cases to new operations based on operational profile [1]Illustration - FF:

Allocate 17% of test cases to Proc. fax call operationB. Detail new test cases for each new operation by

selecting from possible choices of input variable values with equal probability [1]Illustration - FF:

Forwardee = Local calling area

2.Specify test procedure, based on the test operational profile [1]

SRE Process - Prepare for Test



SRE Process - Execute Test




6. Guide Test





Execute Test

1.Determine and allocate test time among associated systems and types of test (feature, load, regression) [1]

*2. Invoke test in accordance with operational profile [1]

3. Identify system failures and when they occurred - use data in Guide Test [1]



Invoke Test


F

Track reliability growthF = Feature testR = Regression test

Certify reliability

Supersystem

Variations

SupersystemLoadTestF

Base ProductLoadTestF

RR RRRRR RRR

RRRR



SRE Process - Guide Test




6. Guide Test





Guide Test

Process system failure data gathered in test to:

*1. Track reliability growth of developed software of base product and variations

*2. Certify reliability of:A. Base product and variations that customers

will acceptance testB. Supersystems

3.Guide product release

SRE Process - Guide Test


Track Reliability Growth

1. Execute CASRE software reliability estimation program to obtain FI / FIO ratio

2. Plot FI / FIO ratio against time*3. Interpret plot

SRE Process - Guide Test - Track Reliability Growth

6.38Kiểm thử và Đảm bảo Chất lượng Phần mềm SRE Process - Guide Test - Track Reliability Growth

Interpret Plot : Illustration - FF

FI/FIO

Mcalls

0

2

4

6

8

10

12

14

16

18

0 0.1 0.2 0.3 0.4 0.5

Conventional test

Operational-profile-driven test



0

2

4

6

8

10

12

14

16

18

0 0.1 0.2 0.3 0.4 0.5

FI/FIO

Mcalls

Solutions:1. Defer features2. Rebalance major

quality characteristics3. Increase work hours

Scheduledtest

completion



0

2

4

6

8

10

12

14

16

18

0 0.1 0.2 0.3 0.4 0.5

FI/FIO

Mcalls

Investigatesignificant

upwardtrends

Possible causes:1. Poor change control2. Poor control of test

execution, resultingin test selectionprobabilities varyingin time


Mcalls


0

2

4

6

8

10

12

14

16

18

0 0.1 0.2 0.3 0.4 0.5

FI/FIO

Terminatetest


Certify Reliability – Dem. Chart:Illus. - FF – BP Supersystem

SRE Process - Guide Test - Certify Reliability

12

0 10862 4

1614

0246

810

Normalized units

Failurenumber

Continue

Accept

Reject

Fail.No.

Mcalls at

FailureNormalized

Units

123

0.003750.006250.025

0.751.25

5

Failure intensity objective:200 failures / Mcalls


SRE and You

1. SRE gives you a powerful way to engineer software-based products so you can be confident in the availability and reliability of the software-based product you deliver as you deliver it in minimum time with maximum efficiency.

2. With SRE you control the process; it doesn’t control you.

3. SRE is a vital skill for being competitive.

Conclusion - SRE and You


To Explore Further1. More Reliable Software Faster and Cheaper, two

day course, described on internet athttp://members.aol.com/JohnDMusa/FLweb.htm

2. Software Reliability Engineering website: http://members.aol.com/JohnDMusa/Overview, briefing for managers, bibliography of articles by software reliability engineering users, course information, useful references, Question of the Month.

3. Musa, J. D., Software Reliability Engineering: More Reliable Software, Faster Development and Testing, ISBN 0-07-913271-5, McGraw-Hill, 1998. Detailed, extensive treatment of practice.

Conclusion - Explore


To Explore Further4. Musa, Iannino, Okumoto; Software

Reliability: Measurement, Prediction, Application, ISBN 0-07-044093-X, McGraw-Hill, 1987. Practice plus extensive theoretical background.

5. Musa, J.D., More Reliable Software Faster and Cheaper. Overview of software reliability engineering, suitable for managers and anyone wanting a fast and broad but not detailed understanding of the topic. May be downloaded from:http://members.aol.com/JohnDMusa/ARTweb.htm



To Explore Further

6. Lyu, M. (Editor), Handbook of Software Reliability Engineering , ISBN 0-07-039400-8, McGraw-Hill, 1996. Particularly useful for CD/ROM of CASRE program.

7. IEEE Computer Society Technical Committee on Software Reliability Engineering. Professional organization of the field: publishes newsletter, sponsors ISSRE annual international conference): membership application at http://www.tcse.org




Chapter 7Software Testing in Industry

(By Maurice Siteur)



1a

1b

1c2a2b

2c

2d

3


Program

TMapTesting in embedded software

CMM/TMM (process improvement)Test tools


Testing

Testing is a process of planning, preparation and measuring, with the aim to view the characteristics of a product, that makes it possible to make judgements about the quality of that product.


Test structure

OrganizationOrganization

Life cycleLife cycle

InfrastructureInfrastructure

TechniquesTechniquesTT

LL

II OO


TMap

Master test plan

Unit testUnit

integrationtest

System testFunctionalacceptance

test

ProductionAcceptance

test


What makes testing difficult

Specifications are not always rightThere are bugs in the software (you should test for that)Testing everything is impossible!!Reproducing tests


What makes embedded systems different

Hardware and software (not available when you would like it to be)Mechanics, electronics, optics, ….

Stubs and drivers needed

Performance, timing, synchronisation, resource usageRegulatory requirements


Embedded software testing - example

Unit testing programs/componentsUnit integration testingSystem testSoftware acceptance testHardware component testHardware integration testSoftware/Hardware integration testManufacturers acceptance testClient acceptance test


TMap essentials

Test planning‘Risk based’ testingTest designTest reporting


Test planning

Looks like normal plan of approach

Approach/strategybased on risk'sUse of quality attributes

Here test coverage is determinedCoverage levelsTheory of finite state machines


Test design

Logical and physical

Determination of test cases

Test caseDescriptionStarting pointInputExpected result

Regression test / re use of test cases


Test reporting

Record findings of test executionReport on progressEnd report

ConclusionsFindings


Guidelines

System development 75%, testing 25%

Preparation 20%Specification 35%Execution 40%Completion 5%

Planning and control 10% on top

Test automation can turn these figures up side down.


Process improvement

Software process improvementCapability Maturity Model

Test process improvementTest Maturity ModelTPI (Homework)


CMM


Level 5: Optimization, Defect Prevention andQuality Control

Level 4: Management and Measurement

Level 3: Integration

Level 2: Phase Definition

Level 1: Initial

- Test proces optimization- Quality control- Application of process data for defect prevention

- Software quality evaluation- Establish a test measurement program- Establish an organization-wide review program

- Control and monitor the test process- Integrate testing into the software lifecycle- Establish a technical training program- Establish a software test organization

- Institutionalize basic testing techniques and methods- Initiate a test planning process- Develop testing and debugging goals

TMM


Tool Support of testing activitiesSystem development

RequirementsAcceptance criteria

Specifications

Coding

Test execution• Capture and playback• Test drivers• Load en performance testing• Monitoring

Test planning

Generating testsetsfor system- and acc.testing

Generating testsetson the basis of design

Test management & control• Test planning• Configuration management• Defect tracking / incident management• Progress control

Design

White box testing• Debugging• Test coverage

Generating testsetsfor progr.test

Static testing

Dynamic testing

Test repository

Static testing

Static testing

Aids• Test design• Test data manipulation


Tool selection and implementation

Identify your test activities and see where tools can be supportiveCreation of long listCreation of a shortlistArrange demo’sEvaluation of selected toolsReview and select toolPilot projectRoll out

Are you ready for CAST?













1a

1b

1c2a2b

2c

2d

3


Homework

Which CMM and TMM level has the following organization?

- 300 employees, 20 IT- Working on getting documentation done- Project plans are made- They do a survey on testing tool- Test planning is on its way- Test design is made during acceptance testWhat would you suggest to improve testing using TMM?


Homework

Try to make a case for yourself what automated testing would cost and manual testing. (Assume the organisation made the initial investment)



Chương 8

Ước lượng chi phíphần mềm


Mục tiêu

Giới thiệu các khái niệm và nền tảng về chi phí và giá cả phần mềmMô tả 3 độ đo để đánh giá năng suất phầnmềmGiải thích tại sao nên sử dụng các kỹ thuậtkhác nhau để đánh giá ước lượng phần mềmMô tả các nguyên lý của giải thuật cho môhình ước lượng chi phí COCOMO 2


Các chủ đề chính

Năng suất phần mềm Software productivityKỹ thuật ước lượng Estimation techniquesMô hình hoá chi phí giải thuật Algorithmic cost modellingKhoảng thời gian và thực hiện Project duration and staffing


Các vấn đề ước lượng cơ bản

Mất bao nhiêu chi phí cần thiết để hoànthành một hoạt động?Mất bao nhiêu thời gian để hoàn thành mộthoạt động?Tổng chi phí cho một hoạt động là gì?ước lượng dự án và lịch biểu là các hoạt độngquản lý có liên hệ với nhau.


Các thành phần chi phí phần mềm

Các chi phí về phần cứng và phần mềm.Chi phí đi lại và đào tạo.Các chi phí nỗ lực (Nhân tố nổi trội trong hầu hếtcác dự án)

Tiền lương các kỹ sư làm việc trong dự án;Chi phí bảo hiểm và xã hội.

Các chi phí phải trả trướcChi phí xây nhà, lò sưởi, chiếu sáng.Chi phí mạng và truyền thông.Chi phí các phương tiện công trình công cộng dùngchung (thư viện, nhà ăn, v.v...).


Chi phí và giá cả

Ước lượng được đánh giá để phát hiện giá cảcho các nhà phát triển làm ra các hệ thốngphần mềm.Không có mối quan hệ đơn giản nào giữa chi phí phát triển và giá cả mà khách hàng phảitrả..Hội đồng tổ chức, nền kinh tế, chính trị vàcác điều kiện kinh doanh có ảnh hưởng đángkể đến giá phải trả.


Các nhân tố giá cả phần mềmCơ hội thị trường A development organisation may quote a low price because it

wishes to move into a new segment of the software market. Accepting a low profit on one project may give the opportunity of more profit later. The experience gained may allow new products to be developed.

Sự không chắc chắn của luợng lượng chi phí

If an organisation is unsure of its cost estimate, it may increase its price by some contingency over and above its normal profit.

Các điều khoản hơp đồng

A customer may be willing to allow the developer to retain ownership of the source code and reuse it in other projects. The price charged may then be less than if the software source code is handed over to the customer.

Thay đổi yêu cầu If the requirements are likely to change, an organisation may lower its price to win a contract. After the contract is awarded, high prices can be charged for changes to the requirements.

Sức mạnh về tài chính

Developers in financial difficulty may lower their price to gain a contract. It is better to make a smaller than normal profit or break even than to go out of business.


Độ đo tỷ lệ mà các kỹ sư phát triển phầnmềm đưa ra ra phần mềm và các tài liệu liênquan.Không định hướng tới chất luợng mà thôngqua đảm bảo chất lượng là một nhân tố đánhgiá năng suất.Một cách cần thiết, chúng ta muốn đo chứcnăng của sản phẩm đưa ra trên đơn vị thờigian.

Năng suất phần mềm


Độ đo liên quan đến kích cỡ : Dựa vào đầu racủa quá trình làm phần mềm. Có thể là sódòng của mã nguồn, các lệnh mã đích v.v....Độ đo liên quan đến chức năng: Dựa vào ướclượng các chức năng của sản phẩm phânphối. Điểm chức năng là cách tốt nhất để đocho loại này.

Độ đo năng suất


Estimating the size of the measure (e.g. how many function points).Estimating the total number of programmer months that have elapsed.Estimating contractor productivity (e.g. documentation team) and incorporating this estimate in overall estimate.

Measurement problems


What's a line of code?The measure was first proposed when programs were typed on cards with one line per card;How does this correspond to statements as in Java which can span several lines or where there can be several statements on one line.

What programs should be counted as part of the system?This model assumes that there is a linear relationship between system size and volume of documentation.

Lines of code


The lower level the language, the more productive the programmer

The same functionality takes more code to implement in a lower-level language than in a high-level language.

The more verbose the programmer, the higher the productivity

Measures of productivity based on lines of code suggest that programmers who write verbose code are more productive than programmers who write compact code.

Productivity comparisons


System development times

Analysis Design Coding Testing Documentation

Assembly codeHigh-level language

3 weeks3 weeks

5 weeks5 weeks

8 weeks4 weeks

10weeks

6 weeks

2 weeks2 weeks

Size Effort Productivity

Assembly codeHigh-level language

5000 lines1500 lines

28 weeks20 weeks

714 lines/month300 lines/month


Function points

Based on a combination of program characteristicsexternal inputs and outputs;user interactions;external interfaces;files used by the system.

A weight is associated with each of these and the function point count is computed by multiplying each raw count by the weight and summing all values.

UFC = ∑(number of elements of given type) × (weight)


Function points

The function point count is modified by complexity of the projectFPs can be used to estimate LOC depending on the average number of LOC per FP for a given language

LOC = AVC * number of function points; AVC is a language-dependent factor varying from 200-300 for assemble language to 2-40 for a 4GL;

FPs are very subjective. They depend on the estimator

Automatic function-point counting is impossible.


Object points

Object points (alternatively named application points) are an alternative function-related measure to function points when 4Gls or similar languages are used for development.Object points are NOT the same as object classes.The number of object points in a program is a weighted estimate of

The number of separate screens that are displayed;The number of reports that are produced by the system;The number of program modules that must be developed to supplement the database code;


Object point estimation

Object points are easier to estimate from a specification than function points as they are simply concerned with screens, reports and programming language modules.They can therefore be estimated at a fairly early point in the development process.At this stage, it is very difficult to estimate the number of lines of code in a system.


Real-time embedded systems, 40-160 LOC/P-month.Systems programs , 150-400 LOC/P-month.Commercial applications, 200-900 LOC/P-month.In object points, productivity has been measured between 4 and 50 object points/month depending on tool support and developer capability.

Productivity estimates


Factors affecting productivity

Applicationdomainexperience

Knowledge of the application domain is essential for effectivesoftware development. Engineers who already understand adomain are likely to be the most productive.

Process quality The development process used can have a s ignificant effect onproductivity. This is covered in Chapter 28.

Project size The larger a project, the more time required for teamcommunications. Less time is available for development soindividual productivity is reduced.

Technologysupport

Good support technology such as C ASE tools, configurationmanagement systems, etc. can improve productivity.

Workingenvironment

As I discussed in Chapter 25, a q uiet working environment withprivate work areas contributes to improved productivity.


All metrics based on volume/unit time are flawed because they do not take quality into account.Productivity may generally be increased at the cost of quality.It is not clear how productivity/quality metrics are related.If requirements are constantly changing then an approach based on counting lines of code is not meaningful as the program itself is not static;

Quality and productivity


Estimation techniques

There is no simple way to make an accurate estimate of the effort required to develop a software system

Initial estimates are based on inadequate information in a user requirements definition;The software may run on unfamiliar computers or use new technology;The people in the project may be unknown.

Project cost estimates may be self-fulfillingThe estimate defines the budget and the product is adjusted to meet the budget.


Changing technologies

Changing technologies may mean that previous estimating experience does not carry over to new systems

Distributed object systems rather than mainframe systems;Use of web services;Use of ERP or database-centred systems;Use of off-the-shelf software;Development for and with reuse;Development using scripting languages;The use of CASE tools and program generators.


Estimation techniques

Algorithmic cost modelling.Expert judgement.Estimation by analogy.Parkinson's Law.Pricing to win.


Estimation techniquesAlgorithmiccost modelling

A model based on historical cost information that relates some softwaremetric (usually its size) to the project cost is used. An estimate is madeof that metric and the model predicts the effort required.

Expertjudgement

Several experts on the proposed software development techniques andthe application domain are consulted. They each estimate the projectcost. These estimates are compared and discussed. The estimationprocess iterates until an agreed estimate is reached.

Estimation byanalogy

This technique is applicable when other projects in the same applicationdomain have been completed. The cost of a new project is estimated byanalogy with these completed projects. Myers (Myers 1989) gives avery clear description of this approach.

ParkinsonÕsLaw

ParkinsonÕs Law states that work expands to fill the time available. Thecost is determined by available resources rather than by objectiveassessment. If the software has to be delivered in 12 months and 5people are available, the effort required is estimated to be 60 person-months.

Pricing to win The software cost is estimated to be whatever the customer hasavailable to spend on the project. The estimated effort depends on thecustomerÕs budget and not on the software functionality.


Pricing to win

The project costs whatever the customer has to spend on it.Advantages:

You get the contract.

Disadvantages: The probability that the customer gets the system he or she wants is small. Costs do not accurately reflect the work required.


Top-down and bottom-up estimation

Any of these approaches may be used top-down or bottom-up.Top-down

Start at the system level and assess the overall system functionality and how this is delivered through sub-systems.

Bottom-upStart at the component level and estimate the effort required for each component. Add these efforts to reach a final estimate.


Top-down estimation

Usable without knowledge of the system architecture and the components that might be part of the system.Takes into account costs such as integration, configuration management and documentation.Can underestimate the cost of solving difficult low-level technical problems.


Bottom-up estimation

Usable when the architecture of the system is known and components identified.This can be an accurate method if the system has been designed in detail.It may underestimate the costs of system level activities such as integration and documentation.


Estimation methods

Each method has strengths and weaknesses.Estimation should be based on several methods.If these do not return approximately the same result, then you have insufficient information available to make an estimate.Some action should be taken to find out more in order to make more accurate estimates.Pricing to win is sometimes the only applicable method.


Pricing to win

This approach may seem unethical and un-businesslike.However, when detailed information is lacking it may be the only appropriate strategy.The project cost is agreed on the basis of an outline proposal and the development is constrained by that cost.A detailed specification may be negotiated or an evolutionary approach used for system development.


Algorithmic cost modelling

Cost is estimated as a mathematical function of product, project and process attributes whose values are estimated by project managers:

Effort = A × SizeB × M

A is an organisation-dependent constant, B reflects the disproportionate effort for large projects and M is a multiplier reflecting product, process and people attributes.

The most commonly used product attribute for cost estimation is code size.Most models are similar but they use different values for A, B and M.


Estimation accuracy

The size of a software system can only be known accurately when it is finished.Several factors influence the final size

Use of COTS and components;Programming language;Distribution of system.

As the development process progresses then the size estimate becomes more accurate.


Estimate uncertainty


The COCOMO model

An empirical model based on project experience.Well-documented, ‘independent’ model which is not tied to a specific software vendor.Long history from initial version published in 1981 (COCOMO-81) through various instantiations to COCOMO 2.COCOMO 2 takes into account different approaches to software development, reuse, etc.


COCOMO 81

Projectcomplexity

Formula Description

Simple PM = 2.4 (KDSI)1.05 × M Well-understood applicationsdeveloped by small teams.

Moderate PM = 3.0 (KDSI)1.12 × M More complex projects whereteam members may have limitedexperience of related systems.

Embedded PM = 3.6 (KDSI)1.20 × M Complex projects where thesoftware is part of a stronglycoupled complex of hardware,software, regulations andoperational procedures.


COCOMO 2COCOMO 81 was developed with the assumption that a waterfall process would be used and that all software would be developed from scratch.Since its formulation, there have been many changes in software engineering practice and COCOMO 2 is designed to accommodate different approaches to software development.


COCOMO 2 models

COCOMO 2 incorporates a range of sub-models that produce increasingly detailed software estimates.The sub-models in COCOMO 2 are:

Application composition model. Used when software is composed from existing parts.Early design model. Used when requirements are available but design has not yet started.Reuse model. Used to compute the effort of integrating reusable components.Post-architecture model. Used once the system architecture has been designed and more information about the system is available.


Use of COCOMO 2 models


Application composition model

Supports prototyping projects and projects where there is extensive reuse.Based on standard estimates of developer productivity in application (object) points/month.Takes CASE tool use into account.Formula is

PM = ( NAP × (1 - %reuse/100 ) ) / PROD

PM is the effort in person-months, NAP is the number of application points and PROD is the productivity.


Object point productivity

DeveloperÕs experienceand capability

Very low Low Nominal High Very high

ICASE maturity andcapability

Very low Low Nominal High Very high

PROD (NOP/month) 4 7 13 25 50


Early design model

Estimates can be made after the requirements have been agreed.Based on a standard formula for algorithmic models

PM = A × SizeB × M where

M = PERS × RCPX × RUSE × PDIF × PREX × FCIL × SCED;A = 2.94 in initial calibration, Size in KLOC, B varies from 1.1 to 1.24 depending on novelty of the project, development flexibility, risk management approaches and the process maturity.


MultipliersMultipliers reflect the capability of the developers, the non-functional requirements, the familiarity with the development platform, etc.

RCPX - product reliability and complexity;RUSE - the reuse required;PDIF - platform difficulty;PREX - personnel experience;PERS - personnel capability;SCED - required schedule;FCIL - the team support facilities.


The reuse model

Takes into account black-box code that is reused without change and code that has to be adapted to integrate it with new code.There are two versions:

Black-box reuse where code is not modified. An effort estimate (PM) is computed.White-box reuse where code is modified. A size estimate equivalent to the number of lines of new source code is computed. This then adjusts the size estimate for new code.


Reuse model estimates 1

For generated code:PM = (ASLOC * AT/100)/ATPRODASLOC is the number of lines of generated codeAT is the percentage of code automatically generated.ATPROD is the productivity of engineers in integrating this code.


Reuse model estimates 2

When code has to be understood and integrated:

ESLOC = ASLOC * (1-AT/100) * AAM.ASLOC and AT as before.AAM is the adaptation adjustment multiplier computed from the costs of changing the reused code, the costs of understanding how to integrate the code and the costs of reuse decision making.


Post-architecture levelUses the same formula as the early design model but with 17 rather than 7 associated multipliers.The code size is estimated as:

Number of lines of new code to be developed;Estimate of equivalent number of lines of new code computed using the reuse model;An estimate of the number of lines of code that have to be modified according to requirements changes.


This depends on 5 scale factors (see next slide). Their sum/100 is added to 1.01A company takes on a project in a new domain. The client has not defined the process to be used and has not allowed time for risk analysis. The company has a CMM level 2 rating.

Precedenteness - new project (4)Development flexibility - no client involvement - Very high (1)Architecture/risk resolution - No risk analysis - V. Low .(5)Team cohesion - new team - nominal (3)Process maturity - some control - nominal (3)

Scale factor is therefore 1.17.

The exponent term


Exponent scale factorsPrecedentedness Reflects the previous experience of the organisation with this type of

project. Very low means no previous experience, Extra high meansthat the organisation is completely familiar with this applicationdomain.

Developmentflexibility

Reflects the degree of flexibility in the development process. Verylow means a prescribed process is used; Extra high means that theclient only sets general goals.

Architecture/riskresolution

Reflects the extent of risk analysis carried out. Very low means littleanalysis, Extra high means a complete a thorough risk analysis.

Team cohesion Reflects how well the development team know each other and worktogether. Very low means very difficult interactions, Extra highmeans an integrated and effective team with no communicationproblems.

Process maturity Reflects the process maturity of the organisation. The computationof this value depends on the CMM Maturity Questionnaire but anestimate can be achieved by subtracting the CMM process maturitylevel from 5.


Product attributes Concerned with required characteristics of the software product being developed.

Computer attributes Constraints imposed on the software by the hardware platform.

Personnel attributes Multipliers that take the experience and capabilities of the people working on the project into account.

Project attributes Concerned with the particular characteristics of the software development project.

Multipliers


Effects of cost drivers

Exponent value 1.17System size (including factors for reuseand requirements volatility)

128, 000 DSI

Initial COCOMO estimate withoutcost drivers

730 person-months

Reliability Very high, multiplier = 1.39Complexity Very high, multiplier = 1.3Memory constraint High, multiplier = 1.21Tool use Low, multiplier = 1.12Schedule Accelerated, multiplier = 1.29Adjusted COCOMO estimate 2306 person-months

Reliability Very low, multiplier = 0.75Complexity Very low, multiplier = 0.75Memory constraint None, multiplier = 1Tool use Very high, multiplier = 0.72Schedule Normal, multiplier = 1Adjusted COCOMO estimate 295 person-months


Algorithmic cost models provide a basis for project planning as they allow alternative strategies to be compared.Embedded spacecraft system

Must be reliable;Must minimise weight (number of chips);Multipliers on reliability and computer constraints > 1.

Cost componentsTarget hardware;Development platform;Development effort.

Project planning


Management options


Management option costs

Option RELY STOR TIME TOOLS LTEX Total effort Software cost Hardwarecost

Total cost

A 1.39 1.06 1.11 0.86 1 63 949393 100000 1049393B 1.39 1 1 1.12 1.22 88 1313550 120000 1402025C 1.39 1 1.11 0.86 1 60 895653 105000 1000653D 1.39 1.06 1.11 0.86 0.84 51 769008 100000 897490E 1.39 1 1 0.72 1.22 56 844425 220000 1044159F 1.39 1 1 1.12 0.84 57 851180 120000 1002706


Option choice

Option D (use more experienced staff) appears to be the best alternative

However, it has a high associated risk as experienced staff may be difficult to find.

Option C (upgrade memory) has a lower cost saving but very low risk.Overall, the model reveals the importance of staff experience in software development.


Project duration and staffing

As well as effort estimation, managers must estimate the calendar time required to complete a project and when staff will be required.Calendar time can be estimated using a COCOMO 2 formula

TDEV = 3 × (PM)(0.33+0.2*(B-1.01))

PM is the effort computation and B is the exponent computed as discussed above (B is 1 for the early prototyping model). This computation predicts the nominal schedule for the project.

The time required is independent of the number of people working on the project.


Staffing requirements

Staff required can’t be computed by diving the development time by the required schedule.The number of people working on a project varies depending on the phase of the project.The more people who work on the project, the more total effort is usually required.A very rapid build-up of people often correlates with schedule slippage.


Key points

There is not a simple relationship between the price charged for a system and its development costs.Factors affecting productivity include individual aptitude, domain experience, the development project, the project size, tool support and the working environment.Software may be priced to gain a contract and the functionality adjusted to the price.


Key pointsDifferent techniques of cost estimation should be used when estimating costs. The COCOMO model takes project, product, personnel and hardware attributes into account when predicting effort required.Algorithmic cost models support quantitative option analysis as they allow the costs of different options to be compared.The time to complete a project is not proportional to the number of people working on the project.



Chapter 10

Quản lý cấu hình


Mục tiêu

Diễn tả tầm quan trọng của quản lý cấu hìnhQLCH (CM)Mô tả các hoạt động chính của QLCH: Lập kếhoạch, quản lý thay đổi, quản lý phiên bản vàXD kiến trúcThảo luận về các công cụ CASE để hỗ trợ tiến trình QLCH


Các chủ đề chính của QLCH

Lập kế hoạch QLCHQuản lý thay đổiQuản lý phiên bản và phát hànhKiến trúc hệ thốngCac công cụ CASE cho QLCH


Các phiên bản mới của HT phần mềm được tạo ra khi HT thay đổi :

Khi thay đổi máy tính/Hệ điều hành OS;Yêu cầu các chức năng khác đi;Các yêu cầu thay đổi do khách hàng đưa ra cho phù hợp (may đo).

QLCH liên quan đến quản lý hệ thống phần mềm tiến hoá:

Sự thay đổi hệ thống là hoạt động của một nhóm đội ngũ;QLCH nhằm điều khiển chi phí và nỗ lực thay đổi một hệthống.

Quản lý cấu hình


Quản lý cấu hình (Tiếp)

Liên quan đến sự phát triển và ứng dụng của các thủ tục và các chuẩn nhằm quản lý sản phẩm phần mềm tiến hoá.QLCH có thể xem như một phần của quátrình quản lý chất lượng (tổng quát hơn).Khi đưa ra QLCH, Các hệ thống phần mềm đôi khi còn được gọi đường cơ sở như chúng đang ở thời điểm xuất phát cho việc phát triển sau này.


Những họ hệ thống


Các chuẩn QLCH

CM should always be based on a set of standards which are applied within an organisation.Standards should define how items are identified, how changes are controlled and how new versions are managed.Standards may be based on external CM standards (e.g. IEEE standard for CM).Some existing standards are based on a waterfall process model - new CM standards are needed for evolutionary development.


Concurrent development and testing

A time (say 2pm) for delivery of system components is agreed.A new version of a system is built from these components by compiling and linking them.This new version is delivered for testing using pre-defined tests.Faults that are discovered during testing are documented and returned to the system developers.


Frequent system building

It is easier to find problems that stem from component interactions early in the process.This encourages thorough unit testing -developers are under pressure not to ‘break the build’.A stringent change management process is required to keep track of problems that have been discovered and repaired.


All products of the software process may have to be managed:

Specifications;Designs;Programs;Test data;User manuals.

Thousands of separate documents may be generated for a large, complex software system.

Configuration management planning


Defines the types of documents to be managed and a document naming scheme.Defines who takes responsibility for the CM procedures and creation of baselines.Defines policies for change control and version management.Defines the CM records which must be maintained.

The CM plan


The CM plan

Describes the tools which should be used to assist the CM process and any limitations on their use.Defines the process of tool use.Defines the CM database used to record configuration information.May include information such as the CM of external software, process auditing, etc.


Large projects typically produce thousands of documents which must be uniquely identified.Some of these documents must be maintained for the lifetime of the software.Document naming scheme should be defined so that related documents have related names.A hierarchical scheme with multi-level names is probably the most flexible approach.

PCL-TOOLS/EDIT/FORMS/DISPLAY/AST-INTERFACE/CODE

Configuration item identification


Configuration hierarchy


All CM information should be maintained in a configuration database.This should allow queries about configurations to be answered:

Who has a particular system version?What platform is required for a particular version?What versions are affected by a change to component X?How many reported faults in version T?

The CM database should preferably be linked to the software being managed.

The configuration database


CM database implementation

May be part of an integrated environment to support software development.

The CM database and the managed documents are all maintained on the same system

CASE tools may be integrated with this so that there is a close relationship between the CASE tools and the CM tools.More commonly, the CM database is maintained separately as this is cheaper and more flexible.


Software systems are subject to continual change requests:

From users;From developers;From market forces.

Change management is concerned with keeping track of these changes and ensuring that they are implemented in the most cost-effective way.

Change management


Request change by completing a change request formAnalyze change request if change is valid then Assess how change might be implemented Assess change cost Submit request to change control board if change is accepted then repeat make changes to software submit changed software for quality approval until software quality is adequate create new system version else reject change request else reject change request

The change management process


The definition of a change request form is part of the CM planning process.This form records the change proposed, requestor of change, the reason why change was suggested and the urgency of change (from requestor of the change).It also records change evaluation, impact analysis, change cost and recommendations (System maintenance staff).

Change request form


Change request form


A major problem in change management is tracking change status.Change tracking tools keep track the status of each change request and automatically ensure that change requests are sent to the right people at the right time.Integrated with E-mail systems allowing electronic change request distribution.

Change tracking tools


Changes should be reviewed by an external group who decide whether or not they are cost-effective from a strategic and organizational viewpoint rather than a technical viewpoint. Should be independent of project responsible for system. The group is sometimes called a change control board.The CCB may include representatives from client and contractor staff.

Change control board


This is a record of changes applied to a document or code component.It should record, in outline, the change made, the rationale for the change, who made the change and when it was implemented.It may be included as a comment in code. If a standard prologue style is used for the derivation history, tools can process this automatically.

Derivation history


Component header information

// BANKSEC project (IST 6087)//// BANKSEC-TOOLS/AUTH/RBAC/USER_ROLE//// Object: currentRole// Author: N. Perwaiz// Creation date: 10th November 2002//// © Lancaster University 2002//// Modification history// Version ModifierDate Change Reason// 1.0 J. Jones 1/12/2002 Add header Submitted to CM// 1.1 N. Perwaiz 9/4/2003New field Change req. R07/02


Invent an identification scheme for system versions.Plan when a new system version is to be produced.Ensure that version management procedures and tools are properly applied.Plan and distribute new system releases.

Version and release management


Version An instance of a system which is functionally distinct in some way from other system instances.Variant An instance of a system which is functionally identical but non-functionally distinct from other instances of a system.Release An instance of a system which is distributed to users outside of the development team.

Versions/variants/releases


Version identification

Procedures for version identification should define an unambiguous way of identifying component versions.There are three basic techniques for component identification

Version numbering;Attribute-based identification;Change-oriented identification.


Simple naming scheme uses a linear derivation

V1, V1.1, V1.2, V2.1, V2.2 etc.

The actual derivation structure is a tree or a network rather than a sequence.Names are not meaningful. A hierarchical naming scheme leads to fewer errors in version identification.

Version numbering


Version derivation structure


Attributes can be associated with a version with the combination of attributes identifying that version

Examples of attributes are Date, Creator, Programming Language, Customer, Status etc.

This is more flexible than an explicit naming scheme for version retrieval; However, it can cause problems with uniqueness - the set of attributes have to be chosen so that all versions can be uniquely identified.In practice, a version also needs an associated name for easy reference.

Attribute-based identification


Attribute-based queries

An important advantage of attribute-based identification is that it can support queries so that you can find ‘the most recent version in Java’ etc.The query selects a version depending on attribute values

AC3D (language =Java, platform = XP, date = Jan 2003).


Change-oriented identification

Integrates versions and the changes made to create these versions.Used for systems rather than components.Each proposed change has a change set that describes changes made to implement that change.Change sets are applied in sequence so that, in principle, a version of the system that incorporates an arbitrary set of changes may be created.


Releases must incorporate changes forced on the system by errors discovered by users and by hardware changes.They must also incorporate new system functionality.Release planning is concerned with when to issue a system version as a release.

Release management


System releases

Not just a set of executable programs.May also include:

Configuration files defining how the release is configured for a particular installation;Data files needed for system operation;An installation program or shell script to install the system on target hardware;Electronic and paper documentation;Packaging and associated publicity.

Systems are now normally released on optical disks (CD or DVD) or as downloadable installation files from the web.


Customer may not want a new release of the system

They may be happy with their current system as the new version may provide unwanted functionality.

Release management should not assume that all previous releases have been accepted. All files required for a release should be re-created when a new release is installed.

Release problems


Release decision making

Preparing and distributing a system release is an expensive process.Factors such as the technical quality of the system, competition, marketing requirements and customer change requests should all influence the decision of when to issue a new system release.


System release strategyFactor Description

Technical quality ofthe system

If serious system faults are reported which affect the way in whichmany customers use the system, it may be necessary to issue a faultrepair release. Howeve r, minor system faults may be repaired by issuingpatches (often distributed over the Internet) that can be applied to thecurrent release of the system.

Platform changes You may have to create a new release of a software application when anew version of the operating system platform is released.

LehmanÕs fifth law(see Chapter 21)

This suggests that the increment of functionality that is included in eachrelease is approximately constant. Therefore, if there has been a systemrelease with significant new functionality, then it may have to befollowed by a repair release.

Competition A new system release may be necessary because a co mpeting product isavailable.

Marketingrequirements

The marketing department of an organisation may have made acommitment for releases to be available at a particular date.

Customer changeproposals

For customised systems, customers may have made and paid for aspecific set of system change proposals and they expect a system releaseas soon as these have been implemented.


Release creation

Release creation involves collecting all files and documentation required to create a system release.Configuration descriptions have to be written for different hardware and installation scripts have to be written.The specific release must be documented to record exactly what files were used to create it. This allows it to be re-created if necessary.


The process of compiling and linking software components into an executable system.Different systems are built from different combinations of components.This process is now always supported by automated tools that are driven by ‘build scripts’.

System building


Do the build instructions include all required components?

When there are many hundreds of components making up a system, it is easy to miss one out. This should normally be detected by the linker.

Is the appropriate component version specified?

A more significant problem. A system built with the wrong version may work initially but fail after delivery.

Are all data files available?The build should not rely on 'standard' data files. Standards vary from place to place.

System building problems


Are data file references within components correct?

Embedding absolute names in code almost always causes problems as naming conventions differ from place to place.

Is the system being built for the right platformSometimes you must build for a specific OS version or hardware configuration.

Is the right version of the compiler and other software tools specified?

Different compiler versions may actually generate different code and the compiled component will exhibit different behaviour.

System building problems


�System building


CASE tools for configuration management

CM processes are standardised and involve applying pre-defined procedures.Large amounts of data must be managed.CASE tool support for CM is therefore essential.Mature CASE tools to support configuration management are available ranging from stand-alone tools to integrated CM workbenches.


CM workbenches

Open workbenchesTools for each stage in the CM process are integrated through organisational procedures and scripts. Gives flexibility in tool selection.

Integrated workbenchesProvide whole-process, integrated support for configuration management. More tightly integrated tools so easier to use. However, the cost is less flexibility in the tools used.


Change management tools

Change management is a procedural process so it can be modelled and integrated with a version management system.Change management tools

Form editor to support processing the change request forms;Workflow system to define who does what and to automate information transfer;Change database that manages change proposals and is linked to a VM system;Change reporting system that generates management reports on the status of change requests.


Version management tools

Version and release identificationSystems assign identifiers automatically when a new version is submitted to the system.

Storage management.System stores the differences between versions rather than all the version code.

Change history recordingRecord reasons for version creation.

Independent development Only one version at a time may be checked out for change. Parallel working on different versions.

Project supportCan manage groups of files associated with a project rather than just single files.


Delta-based versioning


System building

Building a large system is computationally expensive and may take several hours.Hundreds of files may be involved.System building tools may provide

A dependency specification language and interpreter;Tool selection and instantiation support;Distributed compilation;Derived object management.


Component dependencies


Configuration management is the management of system change to software products.A formal document naming scheme should be established and documents should be managed in a database.The configuration data base should record information about changes and change requests.A consistent scheme of version identification should be established using version numbers, attributes or change sets.

Key points


Key points

System releases include executable code, data, configuration files and documentation.System building involves assembling components into a system…CASE tools are available to support all CM activitiesCASE tools may be stand-alone tools or may be integrated systems which integrate support for version management, system building and change management.

Slides of Test and SQA

Documents

vo tin trnh

cht lng cn

organisational

successful

initiate user

software development

successful

test cases