Assessing Requirements Quality through Requirements Coverage

Assessing Requirements Quality through Requirements Coverage

Ajitha Rajan University of Minnesota

Mats Heimdahl University of Minnesota

Kurt Woodham L3 Communications

2

http://w

ww

.umsec.um

n.edu

Model-Based Development

SpecificationModel

Visualization PrototypingTesting

Code

Analysis

Properties

3

http://w

ww

.umsec.um

n.edu

Model Validation problem

1. Are the requirements sufficiently defined for the system?

2. How well does the model implement the behaviors specified by the requirements?

We propose a testing approach that explores the relationship between requirements-based structural coverage and model-based structural coverage

4

http://w

ww

.umsec.um

n.edu

Are the Requirements Sufficient?

Run tests on the model and measure coverage achieved

Informal Requirements

Model / Design Specification

(MUT)

Requirements-Based Tests

Generate to provide requirements coverage

Poor coverage of model implies one or more of the following

1. Missing or implicit requirements

2. Behavior in the model not derived from requirements

3. Requirements-based tests are inadequate

Measure

5

http://w

ww

.umsec.um

n.edu

Model Implements Requirements?

Run tests on the requirements and measure coverage achieved

Model/DesignSpecification

Requirements Model-Based Tests

Generate to provide model coverage

Measure

Poor coverage of requirements implies one or both of the following

1. Model does not adequately implement behaviors specified in the requirements

2. Model is correct and requirements are poorly written

6

http://w

ww

.umsec.um

n.edu

Experimental Setup

• Requirements Coverage Metric – Unique First Cause (UFC) coverage defined over formal LTL requirements

• Model Coverage Metric – Modified Condition/Decision Coverage (MC/DC)

• Four industrial case examples from the avionics domain

• Two kinds of assessments Generate test suites to provide requirements UFC

coverage and measure MC/DC over the model Generate test suites to provide MC/DC over the model

and measure UFC achieved over formal requirements.

7

http://w

ww

.umsec.um

n.edu

Experiment Results

MC/DC Achieved by Requirements-Based Tests

Model MC/DC Achieved by UFC Suites

0

20

40

60

80

100

DWM1 DWM2 FGS1 FGS2

Perc

ent

MC/DC Achieved

Max. MC/DC

9

http://w

ww

.umsec.um

n.edu

Experiment Results

• UFC metric “cheated” by the structure of requirements

LTLSPEC G( var_a > (

case

foo : 0 ;

bar: 1 ;

esac +

case

baz : 2 ;

bpr : 3 ;

esac

));

LTLSPEC G( var_a > (

case

foo & baz : 0 + 2 ;

foo & bpr : 0 + 3 ;

bar & baz : 1 + 2 ;

bar & bpr : 1 + 3 ;

esac

));

10

http://w

ww

.umsec.um

n.edu

Experiment Results

Requirements UFC Coverage Achieved by Model-Based Tests

Reqs. UFC Achieved by MC/DC suites

0

20

40

60

80

100

DWM1 DWM2 FGS1 FGS2

Per

cent

UFC

UFC Achieved

Max. UFC

12

http://w

ww

.umsec.um

n.edu

Conclusions

• Analyzing requirements coverage Vs model coverage provides a promising means of assessing requirements quality.

• Effectiveness is dependent on the rigor and robustness of the coverage metrics used UFC sensitive to the structure of the

requirements