Expert System Presentation On…. Software Certification for Industry - Verification and Validation Issues in Expert Systems By Anca I. Vermesan Presented.

Expert System Presentation

On….Software Certification for

Industry - Verification and Validation Issues in Expert

SystemsBy

Anca I. Vermesan

Presented byWilford Morrison

Software Certification for Industry - Topics

• Introduction• KBS Characteristics applicable to V&V

– Task Related– Architecture Related

• Software Certification– Product vs Process Certification

• Components of Certification– Quality Characteristics of KBS– V&V Methods & Techniques– Methods of Certification

• Industrial Needs– V&V of Safety Critical Systems

• Conclusion

V&V – Verification & Validation

KBS- Knowledge-based Software

• What is Verification and Validation?• V&V techniques used• What is Certification?• Components of Certification• Why Certification for KBS?• Issues with Certification

Software Certification for Industry

Introduction


• New V&V techniques developed for KB component

• Two major characteristics:– Type of task performed – What are specific

task being carried out within a narrow domain?– Type of Internal architecture – The similarity of

architecture across all KBS

Characteristics of KBS Applicable to V&V


• The vague nature of the requirement specification

• Incremental nature of the knowledge elicitation process

• Not amenable to formal analysis


Task-Related :

Issue: In KBS what serve as input is typically not well defined at the outset and the relation of possible inputs to the output may not be clear either compared to conventional system


• Uses declarative style programming languages (usually Rule-based)

• Often cause different types of errors and anomalies (new V&V technique needed)


Architecture-Related :

Issue: While some of the errors are similar to conventional software, the effects and methods of detection are rather different


• Why software certification?– Gives confidence, quality and

dependability– Determinant factor in software

competitiveness– Assuring their conformance with

requirements and specifications

Software Certification


• Composed of three types:-– Self certification- when a producer declares

that their product conforms to specified standards

– Buyer certification- a particular buyer may require that products be submitted for certification by a specified body

– Third-party- a body independent of both the producer and the user carries out the certification process (the focus of this paper)

Software Certification ….cont’d


• Consist of three major activities:-– Software quality evaluation- involves V&V

techniques, tests, static/dynamic analysis as a measurement to determine quality of the software

– Software quality assessment- the process of comparing the actual tests and measurements of the functions of interest with the specifications

– Software quality certification- an independent third party gives written assurance that a software product or process conforms to specified functions




Product vs. Process Certification

• The idea that process-approach gave way to product-approach

• Ultimate goal of certification is to assess the quality of the software product

• Therefore, process certification is intended to analyse the software development process


• Process Oriented Certification steps include:-– Certifier and manufacturer agree on criticality

of the software– A software methodology is then agreed upon– Software is developed, validated and tested

according to the methodology– Documented results allow the certifier to

approve the process and a certificate may be granted




• Product Oriented Certification steps include :-– Analysis of the actual requirement for the

evaluation (This may refer to standards and regulations with which the product should comply)

– Specification of the evaluation that is to comply with the requirements identified by the analysis

– Design of the evaluation which shall identify evaluation techniques and plan

– Performing the evaluation (consist of inspecting and testing product parts)

– Reporting the result obtained from performing the above activities




• The three major components are:-– Quality characteristics– V&V methods and techniques– Certification methods

Components of Certification


• This list consist of:-– Competency- this deals with the quality of the

knowledge in the system relative to human skills– Completeness- this involves ensuring all the

knowledge is referenced and there is no attempts to access non-existing knowledge

– Correctness- the knowledge within the KB should be 100%

– Reliability- determines how often a system fails to arrive at the correct solution to a problem

– Testability- system should be designed so that a test plan can be carried out

– Verifiability- possible to establish logical correctness of the KB through verification

Components of Certification ….cont’d

Quality characteristics


• Clustered into two main groups:-– Non-method specific techniques- involves

human analysis of a product– Method specific techniques- involves using a

formal and structured approach


V&V Methods and Techniques


• Non-method specific techniques– Involves human analysis of product, relying on

individuals to use their experience to find errors– Involves the general examination of programs by

using reviews, walkthroughs and inspections to identify defects and discrepancies

– Inspections are aim at detecting semantically incorrect knowledge



Issue: Use of this technique usually lead to bias and omissions


• Method specific techniques– The properties of the product are well defined

and can be expressed in a structured and formal way

– This method may offer rigorous checks of the code functionality to mathematical proof of specification refinement

– Can be further classified into:-• Empirical methods• Logical methods




Empirical MethodsInvolves running a prototype system on a selection of test cases and assessing the results. (Known as “black-box” validation )

Logical MethodsConsist of identifying anomalies such as redundant, contradictory, circular or deficient knowledge. (Known as “white-box” validation )



Issue: exhaustive testing is impossible due to the many paths through the KB


• Common methods used are:-– Static Analysis– Dynamic Testing– Inspections– Modeling


Methods of Certification


Static Analysis– This involves subjecting the program texts (and its requirements

and specifications documents) to scrutiny and review in order to detect inconsistencies and omissions Static Analysis

– This is a generic form for techniques of software analysis which do not involve software execution

– Other forms include, dataflow diagrams, symbolic execution, requirements tracing and program proving



Note: Static methods are in general judged cheaper, more practical to perform, and more amenable to automatic testing than dynamic methods


Dynamic Testing– Encompasses any form of software analysis, which involves software

execution.– In KBS, the input space may be so large that it is infeasible to examine the

behavior for all inputs– Also for KBS, an expert is usually required to judge the errors detected, and

many test cases need to be generated for systems with extensive chaining between rules

– Other forms include, random testing, equivalence partitioning, boundary testing and cause-effect graphs



Note: Dynamic testing for evaluation in the view of certification would most likely involve re-running test cases prepared during the software process development and applied during the testing phase


• Software Certification has strategic importance to all industries

• Mature software industry help to integrate other industries

• Improved V&V techniques along with considerable experience and expertise has lead to a transfer to industry

Industrial Needs


• As essential components- where the KB element performs an essential function within the system and without which the system will not perform

• As advisory components- where they act in an advisory or supervisory capacity to enhance an independent system

Industrial Needs….cont’d

KBS in Safety Critical System

Note: - When a KBS fails, it usually fails catastrophically rather than safely - KBS put a burden on the overall system hazard and safety analysis


• V&V of KBS are as much potential value to certification specialists as they are to software producers

• Certification of KBS cannot be sufficiently accomplished by use of static testing means

• V&V need objective points of reference on which to guide the certification activities

Conclusion


• The “process culture” of traditional software engineering may at last lead to the “product culture” in knowledge engineering

• Increasing use of software embedded in systems that need to be certified for safety reasons will demonstrate the necessity for software certification

• Software certification has a strategic importance, as it is a way in which the software industry can become mature and integrate harmonically with other industries

Conclusion….cont’d

Software Certification for Industry - Verification and Validation Issues in Expert

Systems

Questions?

Expert System Presentation On…. Software Certification for Industry - Verification and Validation Issues in Expert Systems By Anca I. Vermesan Presented.

Documents

contdsoftware certification

industrysoftware certification

vvsoftware certification

papersoftware certification

standardsbuyer certification

self certification

software product

software quality evaluation