Assessing Requirements Quality through Requirements Coverage
Ajitha Rajan University of Minnesota
Mats Heimdahl University of Minnesota
Kurt Woodham L3 Communications
2
http://w
ww
.umsec.um
n.edu
Model-Based Development
SpecificationModel
Visualization PrototypingTesting
Code
Analysis
Properties
3
http://w
ww
.umsec.um
n.edu
Model Validation problem
1. Are the requirements sufficiently defined for the system?
2. How well does the model implement the behaviors specified by the requirements?
We propose a testing approach that explores the relationship between requirements-based structural coverage and model-based structural coverage
4
http://w
ww
.umsec.um
n.edu
Are the Requirements Sufficient?
Run tests on the model and measure coverage achieved
Informal Requirements
Model / Design Specification
(MUT)
Requirements-Based Tests
Generate to provide requirements coverage
Poor coverage of model implies one or more of the following
1. Missing or implicit requirements
2. Behavior in the model not derived from requirements
3. Requirements-based tests are inadequate
Measure
5
http://w
ww
.umsec.um
n.edu
Model Implements Requirements?
Run tests on the requirements and measure coverage achieved
Model/DesignSpecification
Requirements Model-Based Tests
Generate to provide model coverage
Measure
Poor coverage of requirements implies one or both of the following
1. Model does not adequately implement behaviors specified in the requirements
2. Model is correct and requirements are poorly written
6
http://w
ww
.umsec.um
n.edu
Experimental Setup
• Requirements Coverage Metric – Unique First Cause (UFC) coverage defined over formal LTL requirements
• Model Coverage Metric – Modified Condition/Decision Coverage (MC/DC)
• Four industrial case examples from the avionics domain
• Two kinds of assessments Generate test suites to provide requirements UFC
coverage and measure MC/DC over the model Generate test suites to provide MC/DC over the model
and measure UFC achieved over formal requirements.
7
http://w
ww
.umsec.um
n.edu
Experiment Results
MC/DC Achieved by Requirements-Based Tests
Model MC/DC Achieved by UFC Suites
0
20
40
60
80
100
DWM1 DWM2 FGS1 FGS2
Perc
ent
MC/DC Achieved
Max. MC/DC
9
http://w
ww
.umsec.um
n.edu
Experiment Results
• UFC metric “cheated” by the structure of requirements
LTLSPEC G( var_a > (
case
foo : 0 ;
bar: 1 ;
esac +
case
baz : 2 ;
bpr : 3 ;
esac
));
LTLSPEC G( var_a > (
case
foo & baz : 0 + 2 ;
foo & bpr : 0 + 3 ;
bar & baz : 1 + 2 ;
bar & bpr : 1 + 3 ;
esac
));
10
http://w
ww
.umsec.um
n.edu
Experiment Results
Requirements UFC Coverage Achieved by Model-Based Tests
Reqs. UFC Achieved by MC/DC suites
0
20
40
60
80
100
DWM1 DWM2 FGS1 FGS2
Per
cent
UFC
UFC Achieved
Max. UFC
12
http://w
ww
.umsec.um
n.edu
Conclusions
• Analyzing requirements coverage Vs model coverage provides a promising means of assessing requirements quality.
• Effectiveness is dependent on the rigor and robustness of the coverage metrics used UFC sensitive to the structure of the
requirements