Software Reliability
Software Reliability
Organization of this Lecture:
• Introduction. • Reliability metrics• Reliability growth modelling• Statistical testing• Summary
Software Reliability – Alternate Definitions
• Informally denotes a product’s trustworthiness or dependability.
• Probability of the product working “correctly” over a given period of time.
Software Reliability
• a software product having a large number of defects is unreliable.
• reliability of a system improves if the number of defects is reduced.
Reliability
• User’s concern– An important attribute determining
the quality of the product.
• Demand for quantitative estimation of reliability before making buying decision.
Software Reliability Estimation
• Tricky problem– Several factors contribute to making
measurement of software reliability difficult.
Problems in Reliability Measurements
• Errors do not cause failures at the same frequency and severity.– measuring latent errors alone not
enough
• The failure rate is observer-dependent
Difficulties in Software Reliability Measurement (1)
• No simple relationship observed between system reliability and the number of latent software defects.
• Removing errors from parts of software which are rarely used makes little difference to the perceived reliability.
The 90-10 Rule
• Experiments from analysis of behavior of a large number of programs:– 90% of the total execution time is
spent in executing only 10% of the instructions in the program.
– This 10% part is the core of the program.
Effect of 90-10 Rule on Reliability
• removing 60% defects from least used parts would lead to only about 3% improvement to product reliability.
• Reliability improvements from correction of a single error depends on whether the error belongs to the core or the non-core part of the program.
Difficulty in Software Reliability Measurement (2)
• The perceived reliability depends to a large extent upon how the product is used
• In technical terms on its operation profile.
Effect of Operational Profile on Software Reliability Measurement• If we select input data so that only
“correctly” implemented functions are executed none of the errors will be exposed perceived reliability of the product will be high.
• On the other hand, if we select the input data such that only functions containing errors are invoked perceived reliability of the system will be low.
Software Reliability
• Different users use a software product in different ways. – defects which show up for one user
may not show up for another.
• Reliability of a software product– clearly observer-dependent – cannot be determined absolutely
Difficulty in Software Reliability Measurement (3)
• Software reliability keeps changing through out the life of the product each time an error is detected and corrected
Hardware vs. Software Reliability
• Hardware failures inherently different from software failures.
• Most hardware failures are due to component wear and tear:– some component no longer functions
as specified.
Hardware vs. Software Reliability
• A logic gate can be stuck at 1 or 0,– or a resistor might short circuit.
• To fix hardware faults:– replace or repair the failed part.
Hardware vs. Software Reliability
• Software faults are latent:– system will continue to fail unless
changes are made to the software design and code.
Hardware vs. Software Reliability
• Because of the difference in effect of faults, metrics appropriate for hardware reliability measurements are not good software reliability metrics
Hardware vs. Software Reliability
• When a hardware is repaired:– its reliability is maintained
• When software is repaired:– its reliability may increase or
decrease.
Hardware vs. Software Reliability
• Goal of hardware reliability study :– stability (i.e. interfailure times
remains constant)
• Goal of software reliability study – reliability growth (i.e. interfailure
times increases)
Reliability Metrics
• Different categories of software products have different reliability requirements: – level of reliability required for a
software product should be specified in the SRS document.
Reliability Metrics
• A good reliability measure should be observer independent, – so that different people can agree on
the reliability.
Rate Of Occurrence of Failure
• ROCOF measures: – frequency of occurrence failures.– observe the behavior of a software
product in operation over a specified time interval and calculate the total number of failures during the interval.
Mean Time To Failure
• MTTF is average time between two successive failures: – Determine by observing over a large
number of failures.
Mean Time To Failure
• MTTF is not as appropriate for software as for hardware:– Hardware fails due to a component’s
wear and tear thus indicates how frequently the component fails
– When a software error is detected and repaired the same error never appears.
Mean Time To Failure
• We can record failure data for n failures:– let these be t1, t2, …, tn– calculate (ti+1-ti)– the average value is MTTF
(ti+1-ti)/(n-1)
Mean Time to Repair
• Once failure occurs additional time is lost to fix faults
• MTTR measures average time it takes to fix faults.
Mean Time Between Failures
• We can combine MTTF and MTTR:– to get an availability metric:
MTBF = MTTF + MTTR
• MTBF of 100 hours would indicate– Once a failure occurs, the next failure
is expected after 100 hours of clock time (not running time).
Availability
• Measures how likely the system shall be available for use over a period of time: – considers the number of failures
occurring during a time interval, – also takes into account the repair
time (down time) of a system.
Availability
• Important for Systems having real time requirement:– telecommunication systems, – operating systems, etc. which are
supposed to be never down– where repair and restart time are
significant and loss of service during that time is important.
Failure Classes/ Category
• Transient: – Transient failures occur only for certain
inputs.
• Permanent: – Permanent failures occur for all input
values.
• Recoverable: – When recoverable failures occur the system
recovers with or without operator intervention.
Failure Classes
• Unrecoverable: – the system may have to be restarted.
• Cosmetic: – May cause minor irritations. Do not
lead to incorrect results. • Eg. mouse button has to be clicked twice
instead of once to invoke a GUI function.
Reliability Growth Modelling
• A reliability growth model:– a model of how software reliability
grows as errors are detected and repaired.
• A reliability growth model can be used to predict when (or if at all) a particular level of reliability is likely to be attained.– i.e. how long to test the system?
Reliability Growth Modelling
• There are two main types of uncertainty which render any reliability measurement inaccurate:
• Type 1 uncertainty:– our lack of knowledge about how the
system will be used, i.e. • its operational profile
Reliability Growth Modelling
• Type 2 uncertainty:– reflects our lack of knowledge about
the effect of fault removal.• When we fix a fault we are not sure if the
corrections are complete and successful and no other faults are introduced
• Even if the faults are fixed properly we do not know how much will be the improvement to interfailure time.
Step Function Model
• The simplest reliability growth model:– a step function model
• The basic assumption:– reliability increases by a constant
amount each time an error is detected and repaired.
Step Function Model
ROCOF
Time
Step Function Model
• Assumes:– all errors contribute equally to
reliability growth– highly unrealistic:
• we already know that different errors contribute differently to reliability growth.
Jelinski and Moranda Model
• Realizes each time an error is repaired reliability does not increase by a constant amount.
• Reliability improvement due to fixing of an error is assumed to be proportional to the number of errors present in the system at that time.
Jelinski and Moranda Model• Realistic for many applications,• Shortcomings
– Most probable failures (failure types which occur frequently) discovered early during the testing process.
– Repairing these faults contribute maximum to the reliability growth initially.
– This implies rate of reliability growth should be large initially slow down later on
Littlewood and Verall’s Model
• Assumes different fault have different sizes, thereby contributing unequally to failures.
• Allows for negative reliability growth
Littlewood and Verall’s Model
• Large sized faults tends to be detected and fixed earlier
• As number of errors is driven down with the progress in test, so is the average error size, causing a law of diminishing return in debugging
Other Reliability growth models
• Variations exists– LNHPP (Littlewood non homogeneous
Poisson process) model
• Goel – Okumoto (G-O) Imperfect debugging model– GONHPP
• Musa – Okumoto (M-O) Logarithmic Poisson Execution Time model
Applicability of Reliability Growth Models
• There is no universally applicable reliability growth model.
• Reliability growth is not independent of application.
Applicability of Reliability Growth Models
• Fit observed data to several growth models.– Take the one that best fits the data.
Statistical Testing
• A testing process:– the objective is to determine
reliability rather than discover errors.– uses data different from defect
testing.
Statistical Testing
• Different users have different operational profile:– i.e. they use the system in different
ways– formally, operational profile:
• probability distribution of input
Operational profile: Example
• An expert user might give advanced commands:– use command language interface,
compose commands
• A novice user might issue simple commands:– using iconic or menu-based interface.
How to define operational profile?
• Divide the input data into a number of input classes:– e.g. create, edit, print, file operations,
etc.
• Assign a probability value to each input class:– a probability for an input value from
that class to be selected.
Steps involved in Statistical testing (Step-I)
• Determine the operational profile of the software:– This can be determined by analyzing
the usage pattern.
Step 2 in Statistical testing
• Manually select or automatically generate a set of test data: – corresponding to the operational
profile.
Step 3 in Statistical testing
• Apply test cases to the program:– record execution time between each
failure– it may not be appropriate to use raw
execution time
Step 4 in Statistical testing
• After a statistically significant number of failures have been observed:– reliability can be computed.
Statistical Testing
• Relies on using large test data set. • Assumes that only a small
percentage of test inputs:– likely to cause system failure.
Statistical Testing
• It is straight forward to generate tests corresponding to the most common inputs:– but a statistically significant
percentage of unlikely inputs should also be included.
• Creating these may be difficult:– especially if test generators are used.
Advantages of Statistical Testing
• Concentrate on testing parts of the system most likely to be used:– results in a system that the users find
more reliable (than actually it is!).
Advantages of Statistical Testing
• Reliability predictions based on test results:– gives an accurate estimation of
reliability (as perceived by the average user) compared to other types of measurements.
Disadvantages of Statistical Testing
• It is not easy to do statistical testing properly:– there is no simple or repeatable way
to accurately define operational profiles.
• Statistical uncertainty.
Software Quality
a measure of usefulness and
practicality
Quality
a degree or level of innate excellence
extent of departure from the standard,
degree of conformity
a measure of the skill of the creator and value to the person for whom it was created
comparative-dictionary
fitness-for-purpose-Juran
quantitative -manufacturing
subjective -artistic...
Introduction
• Consider a software product:– functionally correct– but unusable user interface.
• Functional correctness alone does not determine quality
Modern view of quality• Associates several quality factors with a
software product :– Correctness– Reliability– Efficiency (includes efficiency of resource
utilization)– Portability– Usability– Reusability– Maintainability
Correctness
• A software product is correct, – if different requirements as specified
in the SRS document have been correctly implemented.
– Accuracy of results.
Portability
• A software product is said to be portable, – Work on different operating systems – Or on different machines– Or with other software products
Reusability
• Different modules of the product can easily be reused to develop new products.
Usability
• A software product has good usability, if different categories of users (i.e. both expert and novice users) can easily invoke the functions of the product.
• HCI aspect
Maintainability
• A software product is maintainable, – If faults can be corrected– Functionalities can be added or
modified (customized)
Software Quality
• the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs .
– ISO 8402
Difficulties in achieving software quality
• its ethereal nature• its discreteness (no tolerances)• informal/semiformal discipline• volatility of requirements• inherent complexity of large
software systems
Cont…
• intellectual (read, semi-disciplined) input
• complex intra-team communication• deadline pressures• rapid advances in technology• high (often unreasonable)
expectations of clients
McCall’s Model of Software Quality
• Operation
• Revision• Transition
Operation Attributes
usability
correctnessintegrityreliability
efficiency
Revision Attributes
maintainability
testability
flexibility
Transition Attributes
portability reusability
interoperability
Crosby’s Maturity Level
Level 1: Uncertainty
Level 2: Awakening
Level 5: Certainty
Level 3: Enlightenment
Level 4: Wisdom
based on management
attitudes
SQA Environment
• Quality Policy• Quality Management• Quality Assurance• Quality Control• Quality Management System• Total Quality Management
Evolution of Quality Systems
• Initial product inspection method :– gave way to quality control (QC).
• Quality control:– not only detect the defective products
and eliminate them – but also determine the causes behind
the defects.
Quality control (QC)
• Quality control aims at correcting the causes of errors: – not just rejecting defective products.
• Statistical quality control– quality of the output of the process is
inferred using statistical methods – in stead of inspection or testing of all
products
Quality control (QC)
• The next breakthrough, – development of quality assurance
principles
Quality assurance
• Basic premise of modern quality assurance: – if an organization's processes are
good and are followed rigorously, • the products are bound to be of good
quality.
Quality assurance
• All modern quality paradigms include:– guidance for recognizing, defining,
analyzing, and improving the production process.
Total quality management (TQM)
• Advocates: – continuous process improvements
through process measurements.
Quality Policy
the overall quality intentions and direction of an organization as regards quality, as formally expressed by top management.
- ISO 8402
Quality Management
that aspect of the overall management function that determines and implements the quality policy.
- ISO 8402
Quality Assurance
a planned and systematic pattern of all actions necessary to provide adequate confidence that the item or product conforms to established technical requirement.
- IEEE 729
Quality Control
the operational techniques and activities that are used to satisfy the quality requirements.
- ISO 8402
Quality Management System
the organizational structure, responsibilities, procedures, processes and resources for implementing quality management.
- ISO 8402
Total Quality Management
Quality is to satisfy customer's requirements continuallyTotal quality is to achieve quality at low costTotal quality management is to obtain total quality by involving everyone's daily commitment
- Kanji
SQA Techniques - Error detection & removal
• inspections• walkthroughs• reviews• testing• demonstrations
SQA Techniques - Error avoidance• systematic and error-proof
determination of user needs• systematic and error-proof translation of
user needs into software products• methods and tools• rigorous specifications• configuration management• adherence to standards• root cause analysis
Role of SQA
• evolve and approve a Software Quality Assurance Plan (SQAP) for every project
• Audit the development process for proper execution in conformance with SQAP and Software Development Plan (SDP)
• Participate in the assessment of internal deliverables and of overall software quality
Role of SQA
• Perform, witness or audit tests; and attend selected life cycle reviews to assure appropriate Verification & Validation
• Participate in configuration management and QIP
• Provide appropriate visibility in situation of non-conformance
Quality Guidelines
• start with management commitment to ensure proper attention
• specify quality goals to promote understanding
• establish strategy to achieve quality to ensure proper resources for quality
• apply predictive measures to evaluate quality and increase insight to identify deficiencies and problem areas
Quality Guidelines
• assess quality achieved in final product to determine how well project satisfied goals
• analyze results to identify areas to improve
• provide feedback to exploit lessons learnt
Quality System Activities:• Auditing of projects• Development of:
– standards, procedures, and guidelines, etc.
• Production of reports for the top management – summarizing the effectiveness of the
quality system in the organization.• Review of the quality system itself.
Business Process reengineering
• A term related to TQM. • Process reengineering goes a step
further than quality assurance:– aims at continuous process
improvement.
Business Process reengineering
• BPR aims at reengineering the way business is carried out in any organization – not just software development
organizations.
Total quality management (TQM)
• TQM goes beyond documenting processes – optimizes them through redesign.
• Over the years the quality paradigm has shifted:– from product assurance to process
assurance.
ISO 9000
• ISO (international Standards Organization): – Consortium established to formulate
and foster standardization.
• ISO published its 9000 series of standards in 1987.
What is ISO 9000 Certification?
• ISO 9000 certification: – serves as a reference for contract
between independent parties.
• The ISO 9000 standard: – specifies guidelines for maintaining a
quality system.
What is ISO 9000 Certification?
• ISO 9000 specifies: – guidelines for repeatable and high
quality product development.– Also addresses organizational aspects
• responsibilities, reporting, procedures, processes, and resources for implementing quality management.
ISO 9000
• A set of guidelines for the production process.– not directly concerned about the
product it self.– a series of three standards:
• ISO 9001, ISO 9002, and ISO 9003.
ISO 9000
• Based on the premise:– if a proper process is followed for
production: • good quality products are bound to
follow.
ISO 9001:
• Applies to:– organizations engaged in design,
development, production, and servicing of goods.
– applicable to most software development organizations.
ISO 9002:• ISO 9002 applies to:
– organizations who do not design products: • but are only involved in production.
• Examples of this category of industries:– steel or car manufacturing industries– buy the product and plant designs from
external sources:• only manufacture products.
– not applicable to software development organizations.
ISO 9003
• Applies to organizations involved only in installation and testing of the products.
ISO 9000 for Software Industry
• ISO 9000 is a generic standard:– applicable to any industry
• Many clauses of ISO 9000 documents:– use generic terminologies– very difficult to interpret them in the
context of software organizations.
Software vs. other industries
• No direct interpretation exist in context of software industry
Software vs. other industries
• Software is intangible. Therefore difficult to control. – In contrast, in a car manufacturing unit:
• we can see a product being developed through stages such as fitting engine, fitting doors, etc.
• one can accurately tell about the status of the product at any time.
– Software project management is an altogether different ball game.
Software vs. other industries• During software development:
– the only raw material consumed is data.• For any other product development:
– Lot of raw materials consumed• e.g. Steel industry consumes large volumes
of iron ore, coal, limestone, etc.
• ISO 9000 standards have many clauses corresponding to raw material control .
• not relevant to software organizations.
Software vs. other industries
• Radical differences exist between software and other product development, – difficult to interpret various clauses of
the original ISO standard in the context of software industry.
ISO 9000 Part-3
• ISO released a separate document called ISO 9000 part-3 in 1991 – to help interpret the ISO standard for
software industry.
• At present official guidance is inadequate
Benefits of ISO 9000 Certification?
• Gain confidence of customers on organization
• Government contracts. This is especially true in the international market.
How to Get ISO 9000 Certification?
• An organization intending to obtain ISO 9000 certification: – applies to a ISO 9000 registrar for
registration.
• ISO 9000 registration process consists of several stages.
How to Get ISO 9000 Certification?
• Application stage:– Applies to a registrar for registration.
• Pre-assessment:– the registrar makes a rough
assessment of the organization.
How to Get ISO 9000 Certification?
• Document review and adequacy audit:– process and quality-related
documents.– the registrar reviews the documents – makes suggestions for improvements.
How to Get ISO 9000 Certification?
• Compliance audit: the registrar checks – whether the suggestions made by it
during review have been complied.
How to Get ISO 9000 Certification?
• Registration:– The registrar awards ISO 9000
certificate after successful completions of all previous phases.
• Continued surveillance:– The registrar continues monitoring
the organization periodically.
ISO 9000 Certification
• An ISO certified organization – can use the certificate for corporate
advertizements– cannot use the certificate to advertize
products.• ISO 9000 certifies organization's process • not any product of the organization.
– An organization using ISO certificate for product advertizements:
• risks withdrawal of the certificate.
Summary of ISO 9001 Requirements
• Management responsibility(4.1):– Management must have an effective
quality policy. – The responsibility and authority of all
those whose work affects quality:• must be defined and documented.
Management responsibility(4.1)
• Responsibility of the quality system. – independent of the development
process, – can work in an unbiased manner.
• The effectiveness of the quality system: – must be periodically by audited.
Quality system (4.2) and contract reviews (4.3):
• A quality system must be maintained and documented.
• Contract reviews (4.3):– Before entering into a contract, an
organization must review the contract • ensure that it is understood,• organization has the capability for
carrying out its obligations.
Design control (4.4):
• The design process must be properly controlled, – this includes controlling coding also.
• A good configuration control system must be in place.
Design control (4.4):
• Design inputs must be verified as adequate.
• Design must be verified.• Design output must be of required
quality.• Design changes must be
controlled.
Document control (4.5):
• Proper procedures for – document approval, issue and
removal.
• Document changes must be controlled. – use of some configuration
management tools is necessary.
Purchasing (4.6):
• Purchased material, including bought-in software:– must be checked for conforming to
requirements.
Purchaser Supplied Products (4.7):
• Material supplied by a purchaser,– for example,
• client-provided software must be properly managed and checked.
Product Identification (4.8):
• The product must be identifiable at all stages of the process. – In software development context this
means configuration management.
Process Control (4.9) :
• The development must be properly managed.
• Quality requirements must be identified in a quality plan.
Inspection and Testing (4.10) :
• In software terms this requires effective testing i.e., – unit testing, integration testing and
system testing.
• Test records must be maintained.
Inspection, measuring and test equipment(4.11):
• If integration, measuring, and test equipments are used, – must be properly maintained and
calibrated.
Control of nonconforming product (4.13) :
• In software terms, – keeping untested or faulty software
out of released product, • or other places whether it might cause
damage.
Corrective Action (4.14) :
• This is both about correcting errors when found, – investigating why they occurred– improving the process to prevent
further occurrences.
• If an error reoccurs despite the quality system, – the system needs improvement.
Handling (4.15) and Quality audits (4.17):
• Handling (4.15) Deals with: – storage, packing, and delivery of the
software product.
• Quality Audits (4.17) :– quality system audit must be carried
out to ensure its effectiveness.
Training (4.18) :
• Training needs must be identified and met.
• Most items of ISO standard– are largely common sense.
Salient features of ISO 9001 requirements:
• All documents concerned with the development of a software product – should be properly managed,
authorized, and controlled.
• Proper plans should be prepared– progress against these plans should
be monitored.
Salient features of ISO 9001 requirements:
• Important documents independently checked and reviewed: – for effectiveness and correctness.
• The product should be tested :– against specification.
• Several organizational aspects: – e.g., management reporting of the
quality team.
Shortcomings of ISO 9001 Certification (1)
• ISO 9000 requires a production process to be adhered to:– but does not guarantee the process to
be of high quality.– Does not give any guideline for
defining an appropriate process.
Shortcomings of ISO 9001 Certification (2)
• ISO 9000 certification process– not fool-proof– no international accredition agency
exists. – likely variations in the norms of
awarding certificates: • among different accredition agencies and
among the registrars.
Shortcomings of ISO 9001 Certification (3)
• Organizations qualifying for ISO 9001 certification:– tend to downplay domain expertise. – tend to believe that since a good
process is in place, • any engineer is as effective as any other
engineer in doing any particular activity relating to software development.
Shortcomings of ISO 9001 Certification (4)
• In manufacturing industry– clear link between process quality and
product quality– once a process is calibrated:
• can be run again and again producing quality goods
• Software development is a creative process:– individual skills and experience is significant
Shortcomings of ISO 9001 Certification (5)
• Many areas of software development are very specialized: – special expertize and experience
(domain expertize) required.
• ISO 9001– does not automatically lead to
continuous process improvement, – does not automatically lead to TQM.
Shortcomings of ISO 9001 Certification (6)
• ISO 9001 addresses mostly management aspects.
• Techniques specific to software development have been ignored– Configuration management– Reviews– Release builds– Problem Notification system– Intranets
SEI Capability Maturity Model
• Developed by Software Engineering Institute (SEI) of the Carnegie Mellon University, USA:– to assist the U.S. Department of
Defense (DoD) in software acquisition.
– The rationale was to include:• likely contractor performance as a factor
in contract awards.
SEI Capability Maturity Model
• Major DoD contractors began CMM-based process improvement initiatives: – as they vied for DoD contracts.
• SEI CMM helped organizations: – Improve quality of software they developed– Realize adoption of SEI CMM model had
significant business benefits.
• Other organizations adopted CMM.
SEI Capability Maturity Model
• In simple words,– CMM is a model for apprising the
software process maturity of a contractor into different levels.
– Can be used to predict the most likely outcome to be expected from the next project that the organization undertakes.
SEI Capability Maturity Model
• Can be used in two ways:– Capability evaluation– Software process assessment.
Capability Evaluation
• Provides a way to assess the software process capability of an organization– Helps in selecting a contractor– Indicates the likely contractor
performance
Software Process Assessment
• Used by an organization to assess its current process:– Suggests ways to improve the
process capability.– This type of assessment is for purely
internal use.
SEI Capability Maturity Model
• The SEI CMM classifies software development industries into: – Five maturity levels.– Stages are ordered so that
improvements at one stage provide foundations for the next
– Based on the pioneering work of Philip Crosby
SEI Capability Maturity Model
Initial (1)
Repeatable (2)
Defined (3)
Managed (4)
Optimizing (5)
Level 1: (Initial)
• Organization operates– without any formalized process or
project plans
• An organization at this level is characterized by – ad hoc and often chaotic activities.
Level 1: (Initial)
• Software production processes are not defined, – different engineers follow their own
process – development efforts become chaotic. – The success of projects depend on
individual efforts and heroics.
Level 2: (Repeatable)
• Basic project management practices – tracking cost, schedule, and functionality
are followed.
• Size and cost estimation techniques – function point analysis, COCOMO, etc. used.
• Production process is ad hoc – not formally defined– also not documented.
Level 2: (Repeatable)
• Process used for different projects might vary between projects: – earlier success on projects with
similar applications can be repeated.– Opportunity to repeat process exist
when a company produces a family of products.
Level 3: (Defined)
• Management and development activities: – defined and documented.– Common organization-wide
understanding of activities, roles, and responsibilities.
Level 3: (Defined)
• The process though defined, – process and product qualities are not
measured.
• ISO 9001 aims at achieving this level.
Level 4: (Managed)
• Quantitative quality goals for products are set.
• Software process and product quality are measured: – The measured values are used to
control the product quality.– Results of measurement used to
evaluate project performance • rather than improve process.
Level 4: (Managed)
• Organization sets quantitative quality goals
• World-wide about 100 organizations assessed at this level.
Level 5: (Optimizing)
• Statistics collected from process and product measurements are analyzed:– continuous process improvement
based on the measurements.• Known types of defects are prevented
from recurring by tuning the process• lessons learned from specific projects
incorporated into the process
Level 5: (Optimizing)
• Identify best software engineering practices and innovations:– tools, methods, or process are
identified– transferred throughout the
organization
• World-wide about 50 organizations have been assessed at this level.
Key Process Areas
• Each level is associated with a key process area (KPA) identifies– where an organization at the previous
level must focus to reach this level
Level 2 KPAs
• Software project planning– Size, cost, schedule.– project monitoring
• Configuration management• Subcontract management
Level 3 KPAs
• Process definition and documentation
• Reviews• Training program
Level 4 KPAs
• Quantitative measurements• Process management
Level 5 KPAs
• Defect prevention• Technology change management• Process change management
Comparison between ISO 9001 and SEI CMM
• ISO 9001 awarded by an international standards body– can be quoted in official documents
and communications
• SEI CMM assessment is purely for internal use.
Comparison between ISO 9001 and SEI CMM
• SEI CMM was developed specifically for software industry:– addresses many issues specific to
software industry.
• SEI goes beyond quality assurance– aims for TQM– ISO 9001 correspond to SEI level 3.
Comparison between ISO 9001 and SEI CMM
• SEI CMM provides a list of key areas– on which to focus to take an organization
from one level to the other
• Provides a way for gradual quality improvements over several stages.– e.g trying to implement a defined process
before a repeatable process:• counterproductive as managers are
overwhelmed by schedule and budget pressure.
Remarks on Quality Model Usage• Highly systematic and measured approach
to software development process suits certain circumstances– negotiated software, safety-critical software, etc
• What about small organizations?• Typically handle applications such as internet, e-comm. • without an established product range,• without revenue base, experience on past projects, etc.• CMM may be incompatible
Small Organizations
• Small organizations tend to believe:– We are all competent people hired to
do a job, we can’t afford training– We all communicate with one another
• Osmosis works because we are so close
– We are all heroes• We do what needs to be done• Therefore rules do not apply to us
Small Organizations
• Often have problems:– Undocumented requirements– Inexperienced managers– Documenting the product– Resource allocation– Training– Peer reviews
Small Organizations
• A two week CMM-based appraisal is probably excessive:
• Small organizations need to operate more efficiently at lower levels of maturity– Must first fluorish if eventually they
are to mature
Personal Software Process (PSP)
• Based on the work of Humphrey• PSP is a scaled down version of
industrial software process– suitable for individual use
• Even CMM assumes that engineers use effective personal practices
Personal Software Process (PSP)
• A process is the set of steps for doing a job
• The quality and productivity of an engineer – largely determined by his process
• PSP is framework that– helps software engineers to measure
and improve the way they work.
Personal Software Process (PSP)
• Helps developing personal skills and methods– Estimating and planning method– Shows how to track performance against
plans– Provides a defined process
• can be fine tuned by individuals• Recognizes that a process for individual use is
different from that necessary for a team project.
Time Management
• Track the way you spend time– Boring activities seem longer then
actual– Interesting activities seem short
• Record time for– Designing – Writing code– Compiling– Testing
Personal Software Process (PSP)
Planning
Design
Code
Compile
TestPostmortem
Logs
Project plan summary
PSP-Planning
• Problem definition• Estimate max, min, and total LOC• Determine minutes/LOC• Calculate max,min, and total
development times• Enter the plan data in project plan
summary form• record the planned time in Log
PSP-Design
• Design the program• Record the design in specified
format• Record the Design time in time
recording log
PSP-Code
• Implement the design• Use a standard format for code
text• Record the coding time in time
recording log
PSP-Compile
• Compile the program• Fix all the defects• Record compile time in time
recording log
PSP-Test/Postmortem
• Test– Test the program– Fix all the defects found– Record testing time in time recording log
• Postmortem– Complete project plan summary form
with actual time and size data– Record postmortem time in time record
Personal Software Process (PSP)
PSP 0
PSP 1
PSP 2
PSP 3
Personal measurement
Basic size measures
Personal planning
Time and schedule
Personal quality management
Design and code reviews
Personal process
evolution
Fitting pieces
Building the CMM Structure
Capability Maturity Models• CMMI SM – CMM IntegrationSM • SW-CMM – Capability Maturity Model® for
Software • P-CMM – People Capability Maturity Model • SA-CMM – Software Acquisition Capability
Maturity Model • SE-CMM – Systems Engineering Capability
Maturity Model • IPD-CMM – Integrated Product
Development Capability Maturity Model
SW - CMM
• Describes principles and practices underlying software process maturity
• Intended to help software organizations improve the maturity of their software processes
SE - CMM
• Describes essential elements of an organization's systems engineering process
• Provides a reference for comparing actual systems engineering practices against these essential elements.
IPD - CMM• Guide organizations in IPD design,
development, appraisal, and improvement.
• Involves a teaming of the functional disciplines to integrate and concurrently apply all necessary processes to produce an effective and efficient product that satisfies the customer's needs
CMMI
• The content and characteristics of SW-CMM [draft version 2(c)], SE-CMM [EIA/IS 731], and IPD-CMM [Version 0.98] models provide the basis for the CMMI product suite
• CMMI models eventually replace SW-CMM Version 1.1
Motivation for Integrations
• Models created within an environment of evolving national and international standards and frameworks
• Maintaining harmonization between them and improvement models become a continuing challenge, particularly across discipline
Frameworks Quagmire
CMM Integration
• Objective is to develop a product with a set of integrated products to support process and product improvement.
• Preserve investment in process improvement and enhance the use of multiple models.
• Output will be integrated model(s), assessment method(s), and training material.
CMMI Objectives• Eliminating inconsistencies• Reducing duplications• Increasing clarity and understanding• Assuring consistency with ISO 15504
Longer-term objective is to lay foundation for later addition (acquisition, security)
CMM Based Appraisals• CMM-Based Appraisal for Internal Process
Improvement (CBA IPI): To determine the state of one's own processes
• Software Capability Evaluation (SCE):To determine the state of someone else's processes.
• To ensure that these methods are both rigorous and consistent, the Process Program developed and maintains the CMM Appraisal Framework (CAF), which defines the standards that an appraisal method must meet.
When to apply CBA IPI
• For investigating your organization, using mostly your own team members, and making principal use of the results yourself, then a CBA IPI assessment is the recommended tool. This method assumes a collaborative environment and a collegial approach to gathering data.
When to apply SCE• If, on the other hand, you are investigating
somebody else's organization, using your own team members (not theirs), and you will use the results for your purposes (even if you plan to share the results with the organization), then an SCE is the recommended tool. This method assumes a potentially audit-like environment and a more intrusive approach to gathering data
SEI’s Role In CMM-Based Appraisals• Authorized Lead Assessors for CBA
IPI are required to return a report to the SEI for each completed assessment. The data is entered into the Process Appraisal Information System. All assessment data that is returned to the SEI is kept confidential.
• The SEI trains and authorizes qualified persons to lead appraisals
Cont…
• The SEI does not validate or certify appraisal results.
• An assessment's findings and maturity level are owned by the assessment sponsor. The sponsor may publicize this information at their discretion.
• The Maturity Profile Report is published by the SEI twice a year
That’s all?
• Not exactly• SQA Activities
– Cleanroom methodology– Six Sigma
• Assessment models– SPR Assessment– Malcolm Baldrige National Quality
Award
Cleanroom Software Engineering
• Metaphor coming from integrated circuit manufacturing– Free from dust, flecks of skin etc
• Whenever defect occurs, they are considered as defect in process than product– Invoke process inspection, review and
improvement cycle
Six Sigma
• Motorola’s baby that won them MBNQA award for quality in 1988
• Six sigma is a quantitative approach to eliminate defects
• Applicable to any industry - manufacturing, product development, service
Six Sigma• To achieve six sigma
– a process must not produce more than 3.4 defects per million opportunities.
– 5 Sigma -> 230 defects per million– 4 Sigma -> 6210 defects per million
• Six sigma methodologies– DMAIC (Define, Measure, Analyze, Improve,
Control)– DMADV: (Define, Measure, Analyze, Design,
Verify)
Six Sigma Methodologies
• The methodologies are implemented by Green belt and Black belt workers– Supervised by Master black belt worker
• Pareto Chart:– Simple bar chart to represent defect
data– Identify the problems that occurs with
greatest frequency• or incur the highest cost
Software Productivity Research Inc.• Methodology developed parallel to SEI
Process maturity model• SPR question covers both Strategic
corporate issues and tactical project issues that affect quality, productivity and user satisfaction
• 400 questionnaire as against SEI’s 85 questions– Lickert Scale for response (Excellent, Good,
Average, Marginal, Poor)
Malcolm Baldrige National Quality Award• Established in 1988 by US Department
of Commerce• Examination criteria against seven
categories– Leadership– Information and analysis– Strategic Quality Planning– Human Resource Utilization– Quality assurance of products and services– Quality Results– Customer satisfaction
References
• CMMI Distilled– Ahern, Clouse, Turner. Addison Wesley
• CMM in Practice– Pankaj jalote. Pearson Education
• Introduction to Personal SW Process, Introduction to Team SW Process– Watts Humphrey. Addison Wesley
Web References• General Info
– http://www.sei.cmu.edu– http://www.sei.cmu.edu/sema/profile.html
• PSP, TSP– http://interactive.sei.cmu.edu
/Features/1999/June/Background/Background.jun99.pdf [Both]
– http://www.sei.cmu.edu/pub/documents/articles/pdf/psp.pdf [PSP]
– http://wuarchive.wustl.edu/languages/ada/sei/documents/00.reports/pdf/00tr023.pdf [TSP]
Alternatives/Additions• ISO Standards
– Generic Specifications in areas of Quality Requirements (9000), Environmental Management (14000)
– Latest on Information Technology (ISO/IEC TR 15504-#:1998)
, where 0<#<10
– Some client demand for ISO Certified institutes
http://www.iso.ch/iso/en/iso9000-14000/tour/magical.html
Other References
• http://www.software.org/quagmire/• Presentation by Rajib Mall