Lockheed Martin Aeronautics Company Copyright ©2007 Lockheed Martin Corporation. All Rights Reserved. Empirical Study of Software Quality and Reliability.

Lockheed Martin Aeronautics Company

Copyright ©2007 Lockheed Martin Corporation.

All Rights Reserved.

Empirical Study of Software Quality and Reliability

Michael F. Siok, PELockheed Martin Aeronautics CompanyP.O. Box 748, MZ 8604Fort Worth, TX 76101Tel: (817) [email protected]

High Assurance Systems Engineering ConferenceHigh Assurance Systems Engineering ConferenceIEEE HASE 2007IEEE HASE 2007

14 November 2007

Jeff Tian, Ph.D., PESouthern Methodist UniversityDallas, TX USA 75275Tel: (214) 768-2861FAX: (817) [email protected]

Lockheed Martin Aeronautics Company 2© 2007 Lockheed Martin Corporation

Avionics Software Development

How much software process is enough ?

How much software is enough ?

Benchmarks ?

In-house Best Practice ?

Cost too Much

Takes too long

Metrics, metrics, metrics

Good Software = f (productivity, reliability, quality)Good Software = f (productivity, reliability, quality)


Aircraft Software at LM Aero . . .

Radar

MissionComputer

Gyros

Accelerometers

Flight Control

Rudder(s)

Electro-Optical

GlobalPositioning

Inertial

Stabilizers

Ailerons

Controls Displays

WeaponsExternalStores

Stick andThrottle

Engine(s)

CharacteristicsCharacteristics

• Large Complex Systems

• Decades Lifespan

• Frequent Software Updates

• Mix of Computation Types

• Computational

• Displays

• Logic/State Machine

• Signal Processing

• Feedback Control

• Hard & Soft Real-Time

• Severe Computing Resource Constraints

• COTS requirements

• Legacy reuse


Aircraft Software at LM Aero . . .

Radar

MissionComputer

Gyros

Accelerometers

Flight Control

Rudder(s)

Electro-Optical

GlobalPositioning

Inertial

Stabilators

Ailerons

Controls Displays

WeaponsExternalStores

Stick andThrottle

Engine(s)

Object AObject AObject AObject A

Object A Object A Object A

Object A

Object A Object A Object A

Object A Object A

Object A Object A

Object A Object A

• Embedded Code size ~2 MSLOC & climbing fast

• Hundreds of object instances

• Some COTS

• Products: F-16, C-130, U-2, F-22, JSF, support equipment, others . . . .


Background (Cont’d)

Avionics Software Managers want to know . . . How are projects performing, individually and collectively? Is OO better than SA/SD? Does programming language make a difference? Productivity, reliability, quality . . . How are we doing? Issues . . .

Measuring success Benchmarking How to improve

Use statistics to provide some answers


Topics

Background

Avionics Software Project Data

Data Analysis

Wrap up and Next Steps

Empirical Study of Embedded Software Quality and ProductivityBy Michael F. Siok and Jeff Tian

IEEE HASE 2007 Paper:


Avionics Software Project Data

39 software projects carefully chosen for study Project Application Domain (a.k.a. project complexity) Project Size Software Development Methodology Programming Language used

Metrics Framework Size Cost Schedule TP & Quality

Project Size OOA/OOD SA/SD Ada C J73

Small

Medium

Large 9 4 9 4

Small

Medium 4 1 3

Large 7 7

Small 4 3 1

Medium 7 5 2

Large 4 4

# of Projects 9 30 9 9 21Total 39 39

Methodology Language

Pro

ject

Co

mp

lexi

ty

Co

mp

ute

Inte

nsi

veD

isp

lay

Inte

nsi

veL

og

icIn

ten

sive


Avionics Software Project Data (Cont’d)

Metrics Framework provides reference to capture metric and scope

27 metrics available from all projects and normalized, where appropriate Comparable

Individually In Aggregate


Data Analysis -- Approach

Study metrics data using Descriptive Statistics Individual Metrics Metrics in aggregate

Perform Hypothesis Tests to answer management questions Domain Separation OO –vs- SA/SD Size –vs- Reliability Size –vs- Productivity Cost –vs- Reliability Cost –vs- Productivity Language –vs- Productivity

Summarize & Report Results


Data Analysis (Cont’d) -- Descriptive Statistics

Used to discover interesting characteristics about each metric in dataset


Data Analysis (Cont’d) -- Hypothesis Tests

Hypothesis Testing Since data not normally distributed, used non-parametric tests

to accept or reject hypotheses H0 – Data compared is from same population

H1 – Data compared is from different populations

Wilcoxon Rank Sum Test


Data Analysis (Cont’d) -- Reporting

Hypothesis Testing (Continued) Conducted on metrics, results collected, summarized

Majority rules policy on Accept/Reject test results Reject = difference at .05 level of significance Accept = no difference


Wrap Up and Next Steps

Avionics software organization needed methodology to Assess project performance Assess project performance relative to other similar projects Identify and act on opportunities for improvement

Software project data was difficult to acquire Data actually very easy to get Projects had to demonstrate selected process controls

Managed variability in metrics Projects had to submit data to company metrics repository and use it Projects had to validate data in company repository

Analysis method fairly simple, straight-forward Descriptive Statistics to study metrics behaviors Hypothesis Testing Summary reporting to capture analysis results for action


Wrap Up and Next Steps (Cont’d)


Wrap Up and Next Steps (Cont’d)

Statistical Testing did not uncover a clear ‘best all around project’ Want to identify well-rounded best-in-class project(s) Project demonstrated best in cost, schedule, performance, and quality

Use Data Envelopment Analysis (DEA) as benchmarking method to identify best in class software projects Non-parametric analysis method Establishes Multivariate production efficiency frontier

Statistical Analysis coupled with DEA will provide repeatable methodology to study & assess company software project data To understand software project and organizational performance To identify best performing software projects To clearly identify practical software process & product improvement

opportunities

10110001010101100010

10110001010101100010

10100010100100010010

10100010100100010010

00101001010111001010

00101001010111001010

11010101010111111001

11010101010111111001

01011111100000101010

01011111100000101010

101010100010111101010100010111

. . . to better the business practice of software development. . . to better the business practice of software development


Points of Contact

Michael F. Siok, PELockheed Martin Aeronautics CompanyP.O. Box 748, MZ 8604Fort Worth, TX 76101Tel: (817) [email protected]

Jeff Tian, Ph.D., PESouthern Methodist UniversityDallas, TX USA 76Tel: (214) 768-2861FAX: (817) [email protected]

Lockheed Martin Aeronautics Company Copyright ©2007 Lockheed Martin Corporation. All Rights Reserved. Empirical Study of Software Quality and Reliability.

Documents