Software Project Management Lecture # 13. Outline Software Reliability Relationship of Software Reliability & Failure Measures of Reliability & Availability.

Software Project Management

Lecture # 13

Outline

Software Reliability Relationship of Software Reliability &

Failure Measures of Reliability & Availability Software Safety Quality Standards SQA Plan

Software Reliability

Definition The probability of failure free operation of

a computer program in a specified environment for a specified time.

Reliability is the probability of not failing in a specified length of time

Software Reliability (Contd.)

It is a quality factor that can be directly measured and estimated using historical and development data.

It measures how often s/w encounters a data input or other condition that it does not correctly process to produce correct answer

If programX has reliability of 0.96 (over 8 processing hours) then it means, if programX runs 100 times – it will operate correctly 96 times

Relationship of Reliability & Failure

When we discuss reliability, we consider the term ‘failure’.

Non-conformance to s/w requirements leads to failures

Negative results or in worst case no output is failure

Failure may be of different levels of complexity. Some failures can be corrected in seconds, some

in weeks and others in months Correction of a failure may introduce new errors

(and eventually new failures).

Relationship of Reliability & Failure

Mathematical representation F(n) = 1 - R(n)

where, F(n) = probability of failing in a specified length

of time R(n) = probability of reliability (i.e. not failing) n = no. of time units,

if time unit is assumed in days then probability of not failing in 1 day is R(1)

Measures of Reliability & Availability

Early work in software reliability attempted to extrapolate the mathematics of hardware reliability theory to prediction of software reliability. But, Most hardware reliability models are based on

failure due to physical wear (corrosion effects, shock, temperature, etc.) rather than design defects.

The opposite is true for softwares - All software failures can be traced to design or implementation problems.

The following concepts apply to both systems

Measures of Reliability & Availability Measure of Reliability

Consider a computer-based system. A simple measure of reliability for such a system is

mean-time-between-failure (MTBF) MTBF = MTTF + MTTR

MTTF = Mean-time-to-failureMTTR = Mean-time-to-repair

Many researchers argue that MTBF is more useful term than defects/KLOC or defects/FP as user is more concerned with failure rate as compared to defect count.

Each defect does not have same failure rate and the total defect count gives little indication of the reliability of a system

Measures of Reliability & Availability

Measure of Availability Software availability is the probability that

a program is operating according to requirements at a given point in time.

It is defined asAvailability = [MTTF / (MTTF + MTTR)] * 100%

Availability measure is sensitive to MTTR

Software Safety This SQA activity focuses on identification &

assessment of potential hazards that may affect software negatively & cause an entire system to fail.

Early identification of hazards can help to have design features that either eliminate or control potential hazards.

A modeling & analysis process is conducted as part of s/w safety.

1) Initially – hazards identified & categorized by criticality & risk

2) Next – analysis techniques are used to assign severity & probability of occurrence (similar to risk analysis methods but different as the emphasis in this case is on technology issues rather than project )

Software Safety (Contd.)

3) After hazards identification & analysis, the next step is to specify safety related requirements, i.e., to find

o A list of undesirable events & the desired system responses to these events

o Role of s/w in managing undesirable events is then indicated

Following analysis techniques can be used: Fault tree analysis Real-time logic Petri Net model

Software Safety (Contd.)

Although software reliability and software safety are closely related but it is important to understand the subtle difference between them Software reliability uses statistical analysis to

determine likelihood of software failure – the failure does not necessarily result in a mishap or hazard

Software safety examines ways in which failures result in conditions that lead to a mishap or hazard – failures are considered in context of the entire computer system

The ISO 9000 Quality Standard A quality assurance system may be defined as

the organizational structure, responsibilities, procedures, processes, & resources for implementing quality management.

ISO 9000 is a family of standards that describe a quality assurance system in generic terms which can applied to any business regardless of the products/services offered

ISO 9000 is maintained by ISO (International Organization for Standardization) and is administered by accreditation and certification bodies.

The ISO 9000 Quality Standard

ISO 9000 does not describe how an organization should implement these quality system elements.

Consequently, the challenge lies in designing and implementing a quality assurance system that meets the standard and fits the company’s products, services, and culture.

Getting Certified…

To become registered to one of the quality assurance system models contained in ISO 9000, a company’s quality system & operations are scrutinized by third-party auditors for compliance to the standard for effective operation.

Upon successful registration, the company is issued a certificate from a registration body represented by the auditors.

Semi-annual surveillance audits ensure continued compliance to the standard.

ISO 9001:2000 The ISO 9001:2000 standard contains 20 requirements

that must be present for an effective QA system.

These requirements address topics such as management responsibility, quality system, contract review, design control, document and data control, process control, etc.

For a software organization to become registered to this standard, it must establish policies & procedures to address each of these requirements

A special set of ISO guidelines (ISO 9000-3) have been developed to help interpret the standard for use in software process.

Capability Maturity Model

It is a ‘model of the maturity’ of the capability of certain business processes.

A maturity model can be described as a ‘structured collection of elements’ that describe certain aspects of maturity in an organization,

and aids in the definition and understanding of an

organization's processes. CMM was originally developed as a tool for

objectively assessing the ability of government contractors' processes (to perform a contracted software project).

Capability Maturity Model

The Capability Maturity Model is based on the Process Maturity Framework first described in the 1989 book "Managing the Software Process" by Watts Humphrey.

Humphrey based this framework on the earlier Quality Maturity Grid developed by Phil Crosby in his book "Quality Is Free".

However, Humphrey's approach differed because of his unique insight that organizations mature their processes in stages based on solving process problems in a specific order.

Humphrey based his approach on the staged evolution of a system of software development practices within an organization, rather than measuring the maturity of each separate development process independently.

Capability Maturity Model

The model identifies 5-levels of process maturity for an organization: Level 1 - Ad hoc (Chaotic)

At this level, the processes are (typically) undocumented and in a state of dynamic change, tending to be driven in an ad hoc, uncontrolled and reactive manner by users or events. This provides a chaotic or unstable environment for the processes.

Level 2 - Repeatable At this level, some processes are repeatable,

possibly with consistent results. Process discipline is unlikely to be rigorous, but where it exists it may help to ensure that existing processes are maintained during times of stress.

Capability Maturity Model Level 3 - Defined

At this level, there are sets of defined and documented standard processes established and subject to some degree of improvement over time. These standard processes are in place (i.e., they are the AS-IS processes) and used to establish consistency of process performance across the organization.

Level 4 - Managed At this level, using process metrics, management can

effectively control the AS-IS process (e.g., for software development ). In particular, management can identify ways to adjust and adapt the process to particular projects without measurable losses of quality or deviations from specifications. Process Capability is established from this level.

Level 5 - Optimizing At this level the focus is on continually improving

process performance through both incremental and innovative technological changes/improvements.

Quality Standards

Reading Assignment CMM CMMI

The SQA Plan Provides a roadmap for establishing SQA

Developed by SQA group (or software team if SQA group does not exist)

A standard structure for SQA plans by IEEE recommends the following:

1) Scope & purpose of the plan.2) Description of all s/w engg. work products that

fall within range of SQA.3) All applicable standards & practices that are

applied during the software process.

The SQA Plan

4) SQA actions & tasks (including reviews & audits) and their placement throughout the software process.

5) Tools and methods that support SQA actions & tasks.

6) Software configuration management procedures for managing change.

7) Methods for assembling, safeguarding, and maintaining all SQA-related records.

8) Organizational roles and responsibilities relative to product quality