[EN] Club Automation presentation "Quality Model for Industrial Automation", Nov. 22nd 2011

1

Quality Model for Industrial Automation

Safe design of control applications

Tuesday, November 22nd, 2011

Thierry COQ [email protected]

System and Software Reliability

Principal Consultant

Denis CHALON [email protected]

Technical Director

mailto:[email protected]


2 All rights reserved

Content

Software - apprehension or apprehension

Software quality in traditional computing

Application to automation

Real Case study – DNV Audit

Quality model’s thresholds for automation

Conclusions


Software is everywhere

Increasingly complex applications - More variables, more I/Os, more treatments - Applications distributed over several PLCs

Replacing hardware functions by software features (more flexible, cheaper)

The development is mostly subcontracted

Re-use of already developed libraries


Apprehension of software

Where is software? How is its integrity managed over its life span?

What is the quality delivered by our suppliers? How to ensure that suppliers are qualified?

What are the causes of software errors? How can we trust software corrections later in the project? And during the operation?

How to prevent delays?


Methods Project Manager Automation Engineer

Stakeholders around software

Different jobs

Very different environments and tools

Knowledge hardly shared

Software is more difficult to grasp than mechanical and electrical controls

Client

Different levels of focus required:

400mm 120mm 14mm


Methods

Projet Manager Automation Engineer

How is software perceived ?

Client

400mm fix

Quick, quick, Done before,

copy/paste

Unreadable, no test,

late

?

PLC 1 PLC 2 PLC 3

PLC 4 PLC 5 PLC 6

PLC 7 PLC 8 PLC 9 Always down, maintenance complains,

poor performance ?

Unfinished, complicated,

important

?


Quick, quick, Done before, copy/paste...

Unfinished, complicated, Important…

Unreadable, no test,

late

Always down, maintenance complains,

average performance

Need to make software visible

Methods

Projet Manager

Automation Engineer

Client

Software must be measurable: - objectively - repeatedly

This measure must be shared by all stakeholders


Software quality in traditional computing

What do computer specialists do to make software more visible? How to define software quality? How can we measure it? Does measuring really make sense?


A brief history of software quality

1970's - Theory formalized by Mac Cabe

1980's - Available tools (eg. Logiscope)

1990's - Tools are mainly used for critical software

2000's - Democratization of the monitoring methods:

Automating data generation from source code

Simplifying the use of quality measurement tools (no need for specialist anymore)

Ergonomic user interfaces, tailored to different stakeholders

Standardization of concepts (ISO9126)

"If you can not measure it, you can not improve it."

(Lord Kelvin)


Principle of a quality model

CMMI ISO9126

code complexity

reusability

testability

reliability

scalability

efficiency

maintainability

coupling

exceptions handling

fault tolerance

comprehensibility

readability

ergonomics

performance

operational reliability

features

bug detection rate

maintenance cost

architecture INTERNAL

EXTERNAL

11

Methods

Automation engineer

1

Client

Develops Development workshop

2'

Controls

Source code

Analysis tool

Automatic operation

Analysis results Projet manager

Dashboards

Automatic operation Analysis model

Quality monitoring cycle

4

Decides

3

Follows

2 Corrects


Readability

Comprehensibility

Testability

Reliability

Scalability

Effectiveness

Maintainability

Reusability No GOTO

Comment ratio

…

Attributes of the program Sub-attributes Measure and

verification points

Methods

Project Manager

Automation Engineer

Client

Quality model


Analysis model

Attribute 1

Attribute 2

Attribute 3

Attribute 4

Attribute 5

Attribute N

Measure 1

Measure 2

Measure 3

Measure N

Verification 1

Verification N

Verification 2

Analysis model

...

...

...

The “fctn_vannes” function has 67

lines of code The program has a good testability

The “cbfe_34” variable has no comment


Characteristics of the SQALE method

The SQALE method takes into account the entire life cycle of the software, including maintenance, renovation and reuse.

The program features are hierarchical:

Who wants to reuse a non reliable program?

Who can demonstrate the reliability of a non testable program?

The result is a measurement of a technical debt:

How much does it cost to have a quality program from the

current situation?

The problems to be solved are counted once on the attribute

with the highest priority

- The quality properties are regarded as independent

The methods tells you where to start

SQALE is independent from a language and from a specific technology

- The results are directly comparable from one program to another

Contrary to ISO9126, SQALE applies directly, does not require to be interpreted

SQALE is automated and economical to implement. It is standardized.

http://www.sqale.org/




Application to industrial automation

What software analysis should be used? - cross-PLC brands - 5 languages of IEC-61131 Which quality model should be implemented? - transposition of the quality models of traditional computing - specificities of industrial automation Which tools for stakeholders? - how to cope with the diversity of stakeholders? - how to manage outsourcing?


Workshops

Software analysis tool

Quality model

Dashboard

One solution

The black box is open to all stakeholders…

5 IEC languages


Methods

Project Manager

Automation Engineer

Control engineer and software

Client

Solved problems:

Make objective the non-functional evaluation of the program

Positive feedback on the programming practices

Higher level view than just the application under development


Methods

Project Manager

Automation Engineer

Project Manager and software

Client

Solved problems:

Quality monitoring

Monitoring the progress of the project

Benchmarking

The dashboard allows navigation from overall

vision to detail

It also allows a temporal

monitoring of the project’s

progress

80-400mm


Methods

Project Manager

Automation Engineer

Methods and software

Client

Solved problems:

Taking into account existing data

Verify that specifications and code match

Formalization and sharing of development methods

Transversal software indicators

24-120mm


Methods

Project Manager

Automation Engineer

The end client and software

Client

Solved problems:

Simplify decision making on the means to assign, according to an objective view

Possible correlation with other sources of information available in the plant

PLC 1 PLC 2 PLC 3

PLC 4 PLC 5 PLC 6

PLC 7 PLC 8 PLC 9 10-24mm


Real case study – The DNV Audit

Is it usable in real life? How to implement it? How much time is saved?


PLC program audit

Need: risk management

Client: DNV, Malmö, Sweden

Function: PLC in charge of controlling the lay tower on a boat

PLC: Rockwell RSLogix 5000

Analysis tool: PLC Checker

Method: SQALE

Unrepresentative image of the boat in question


Code audit : the need

Objectives:

- Identification of key risks associated with software

Scope:

- Software for control systems of the lay tower

- Reliability, maintainability and dependability of the tower

Actions: manual and automated code review

- Functional analysis of the software application

- Analysis of the development process

- Specifications, design, coding, unit testing, integration testing and acceptance testing

- Analysis of the internal quality of the software application: SQALE


Code Audit : SQALE analysis

LADDER code, about 7000 code locations

Normalized index figures

Most common problems:

- Testability: variables written in several places, dead code, important complexity, code in comments

- Reliability: variables are read before being written

- Maintainability: uncommented code


Code Audit : the results

Consistent with other SQALE results for other languages (traditional computing)

The results are better than what is usually observed

Consistent with manual code reviews and “top ten” verifications

Some persistent difficulties with the tools have to be solved

- Interaction between the program and the HMI may not be identified automatically

- Copy / paste of code not yet detected

Final comment of the Swedish client:

« The SQALE analysis provided a very valuable complement to the manual part of the software review »

« While the manual review focused on thoroughly checking selects parts of the code, the SQALE analysis measured defined quality characteristics of the complete code »


How to validate that the thresholds of the quality model are suitable for automation?

Why would a traditional computing quality model suit the IEC languages? How to tune the model to ensure a good match between the ratings and the actual quality?


A study based on real life programs

Step 1: Formalization of measurements to be made on the programs

Step 2: Creation of a client program database

- No test program

- Multi-PLC (Schneider Electric PL7 Pro and Unity Pro, Siemens Step7, Rockwell RSLogix5000)

Step 3: Running the analyzer (PLC Checker) on each program with formalized measures

Step 4: Analysis of results

- Results per PLC

- Results of all PLCs combined

- Definition of thresholds


A few figures

~25 measures

~300 PLC codes Step7, Unity Pro, PL7 Pro and RSLogix5000

~180 000 Program Organisation Units (POUs)

~2 500 000 instructions

~2 000 000 variables

55 hours of calculation

Results: 112MB of raw data to analyse


The quality model is not elitist, it must correspond to the actual use:

50%: A

75%: A or B

90%: A, B or C

95%: A, B, C or D

97,5%: A, B, C, D or E

99%: A, B, C, D, E or F

Definition of thresholds

Acceptance criterion


Number of lines of code

APPLICATIONS

No quality criteria on the size of the application, just an information

Analysed applications are up to 60,000 lines of code

50% <10 000 lines

PROGRAM ORGANIZATION UNITS

Ensure that each POU has a reasonable size

90% of POUs have less than 100 lines of code

The threshold is comparable to what is recommended in traditional computing


Complexity of the codes

Are programs easy to understand?

Two different complexities:

- cyclomatic complexity, essential complexity

- in both cases, the thresholds are in line with the levels seen in traditional computing:

eV(G) < 5

V(G) < 15

Most automation engineers already program correctly Beware! The cyclomatic complexity is not available as such on all languages (limitations on graphical languages : SFC, FBD and LD).




Level of comments of codes


Are applications well commented?

Elements within the program:

50% of all applications have a comment ratio greater than 85%

75% have a ratio greater than 70%

90% have a ratio greater than 60%

Check on size of comments

Density of comments in the code:

50% of all applications have a density of comments greater than 67%

75% have a density greater than 57%

90% have a density greater than 52%


Conclusion

Low dispersion on very general measurements

PLC programs are comparable with one another regardless of their functionality

The quality model used in the experiment is sound

The complexity thresholds used in traditional computing can also be used in automation, with the following restrictions: - have to be tuned for graphical languages (SFC, FBD and LD) - detection limits on copy/pasted codes - has to take into account typical malpractices


Conclusion

How to participate?

How to use?

I have a use case, what to do?

Can I adapt all of this to my needs?

For more information


Key Takeaways

The late discovery of bugs and low quality is costly. The monitoring of quality during the life cycle prevents it:

Thanks to the democratization of quality control, with the following parts…

- Dashboards that allow navigation between high and low level view - Analysis tools automatically generating data - Quality models implementing ISO 9126

…That are applicable and suitable for automation:

- Cross-PLC brands support - Support of all languages of the IEC-61131

Calling all end users and

integrators

Invitation to participate in the development of a

Quality Model adapted to PLCs

Please contact DNV or IAS

To go further, we are looking for: motivated industrial end

users and system integrators, willing to participate in the

improvement of a quality model suitable for automation


Want to know more?

Software quality on Wikipédia -http://en.wikipedia.org/wiki/Software_quality

SQALE website - http://www.sqale.org/

Der Norske Veritas - http://www.dnv.com/

Itris Automation Square – http://www.automationsquare.com/plc-checker.html

SQUORING - http://www.squoring.com/en

Inspearit - http://www.inspearit.com/en/

Thierry COQ [email protected]

Principal consultant

Denis CHALON [email protected]

Technical Director

http://en.wikipedia.org/wiki/Software_quality





http://www.dnv.com/

http://www.dnv.com/

http://www.automationsquare.com/plc-checker.html




http://www.squoring.com/




http://www.inspearit.com/en/

http://www.inspearit.com/en/



[EN] Club Automation presentation "Quality Model for Industrial Automation", Nov. 22nd 2011

Engineering