Measuring and Improving the Completeness of Natural Language Requirements Alessio Ferrari 1 Felice dell’Orletta 2 Giorgio O. Spagnolo 1 Stefania Gnesi 1 1 ISTI-CNR (Formal Methods and Tools Lab), Pisa, Italy http://fmt.isti.cnr.it 2 ILC-CNR (ItaliaNLP Lab), Pisa, Italy http://www.italianlp.it April 8, 2014 A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 1 / 27
27
Embed
Measuring and Improving the Completeness of Natural ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Measuring and Improving the Completenessof Natural Language Requirements
Alessio Ferrari1 Felice dell’Orletta2
Giorgio O. Spagnolo1 Stefania Gnesi1
1ISTI-CNR (Formal Methods and Tools Lab), Pisa, Italyhttp://fmt.isti.cnr.it
CompletenessAn SRS is complete if nothing is missing from the specification
Nothing is missing with respect to WHAT?Different perspectives shall be considered...
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 2 / 27
Internal and Forward Completeness
Internal completenessNothing is missing with respect to the specification itself
No To-be-determined (TBD) itemsNo nonexistent references
Requires guidelines or templates
Forward completenessNothing is missing with respect to the system-to-be
No missing specification items (e.g., missing interfacespecifications)No missing functionsNo missing products (i.e., part of the actual software that are notmentioned in the specification)
Requires a model of the system
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 3 / 27
Backward Completeness
Backward CompletenessNothing is missing with respect to what comes before the SRS
What comes before the SRS?Transcripts of meeting with customersPreliminary specification documentsReference standardsLegacy documents
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 4 / 27
Research Questions
RQ1. How to measure the backward completeness of arequirements specification document?RQ2. How to improve the backward completeness of arequirements specification document?RQ3. Does the backward completeness help in improving theforward completeness of the specification?
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 5 / 27
Measuring Completeness
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 6 / 27
Defining complete requirements for a rock band
Rock’n Roll
In the earliest rock and roll styles of the late 1940s and early 1950s, either the PIANO or saxophone was o;en the lead instrument, but these were generally replaced or supplemented by GUITAR in the middle to late 1950s. The BEAT is essenFally a BLUES RITHM with an accentuated BACKBEAT, the laGer almost always provided by a snare drum. Classic rock and roll is usually played with one or two ELECTRIC GUITARS (one lead, one rhythm), a string bass or (a;er the mid-‐1950s) an ELECTRIC BASS GUITAR, and a DRUM KIT.
Requirements for My Band • My Band shall have ELECTRIC GUITARS • The BEAT of my band shall be a BLUES RITHM • My band shall have an ELECTRIC BASS GUITAR • My band shall have a DRUM KIT • The ELECTRIC GUITARS shall play with an
ELECTRIC BASS GUITAR and a DRUM KIT
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 7 / 27
Backward Completeness
Backward CompletenessAll the relevant concepts and interactions among conceptsexpressed in the input documents are also treated in the requirements
We distinguish between:Concept CompletenessInteraction Completeness
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 8 / 27
Concept Completeness
Relevant TermsRelevant concepts can be approximated with the relevant terms ofthe input documentsA term is a single word (e.g., “guitar”) or a multi-word term (“rockand roll”)
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 9 / 27
Concept Completeness
Rock’n Roll
In the earliest rock and roll styles of the late 1940s and early 1950s, either the PIANO or saxophone was o;en the lead instrument, but these were generally replaced or supplemented by GUITAR in the middle to late 1950s. The BEAT is essenFally a BLUES RITHM with an accentuated BACKBEAT, the laGer almost always provided by a snare drum. Classic rock and roll is usually played with one or two ELECTRIC GUITARS (one lead, one rhythm), a string bass or (a;er the mid-‐1950s) an ELECTRIC BASS GUITAR, and a DRUM KIT.
Requirements for My Band • My Band shall have ELECTRIC GUITARS • The BEAT of my band shall be a BLUES RITHM • My band shall have an ELECTRIC BASS GUITAR • My band shall have a DRUM KIT • The ELECTRIC GUITARS shall play with an
ELECTRIC BASS GUITAR and a DRUM KIT
degCC = TERMS in Requirements
TERMS in Input Docs =
5
8 = 0.6
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 10 / 27
Concept Completeness
Rock’n Roll
In the earliest rock and roll styles of the late 1940s and early 1950s, either the PIANO or saxophone was o;en the lead instrument, but these were generally replaced or supplemented by GUITAR in the middle to late 1950s. The BEAT is essenFally a BLUES RITHM with an accentuated BACKBEAT, the laGer almost always provided by a snare drum. Classic rock and roll is usually played with one or two ELECTRIC GUITARS (one lead, one rhythm), a string bass or (a;er the mid-‐1950s) an ELECTRIC BASS GUITAR, and a DRUM KIT.
Requirements for My Band • My Band shall have ELECTRIC GUITARS • The BEAT of my band shall be a BLUES RITHM • My band shall have an ELECTRIC BASS GUITAR • My band shall have a DRUM KIT • The ELECTRIC GUITARS shall play with an
ELECTRIC BASS GUITAR and a DRUM KIT
degCC = TERMS in Requirements
TERMS in Input Docs =
5
8 = 0.6
The PIANO!
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 11 / 27
Iteraction Completeness
Relevant Relations among TermsRelevant interactions can be approximated with relevantrelations among termsPairs of relevant terms that occur in the same sentence of theinput documents form a relevant relationGUITAR is normally a lead instrument in ROCK AND ROLL
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 12 / 27
Interaction Completeness
Rock’n Roll
In the earliest rock and roll styles of the late 1940s and early 1950s, either the PIANO or saxophone was o;en the lead instrument, but these were generally replaced or supplemented by GUITAR in the middle to late 1950s. The BEAT is essenFally a BLUES RITHM with an accentuated BACKBEAT, the laGer almost always provided by a snare drum. Classic rock and roll is usually played with one or two ELECTRIC GUITARS (one lead, one rhythm), a string bass or (a;er the mid-‐1950s) an ELECTRIC BASS GUITAR, and a DRUM KIT.
Requirements for My Band • My Band shall have ELECTRIC GUITARS
• The BEAT of my band shall be a BLUES RITHM
• My band shall have an ELECTRIC BASS GUITAR • My band shall have a DRUM KIT • The ELECTRIC GUITARS shall play with an
ELECTRIC BASS GUITAR and a DRUM KIT
degIC = RELATIONS in Requirements
RELATIONS in Input Docs =
4
6 = 0.66
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 13 / 27
Interaction Completeness
Rock’n Roll
In the earliest rock and roll styles of the late 1940s and early 1950s, either the PIANO or saxophone was o;en the lead instrument, but these were generally replaced or supplemented by GUITAR in the middle to late 1950s. The BEAT is essenFally a BLUES RITHM with an accentuated BACKBEAT, the laGer almost always provided by a snare drum. Classic rock and roll is usually played with one or two ELECTRIC GUITARS (one lead, one rhythm), a string bass or (a;er the mid-‐1950s) an ELECTRIC BASS GUITAR, and a DRUM KIT.
Requirements for My Band • My Band shall have ELECTRIC GUITARS
• The BEAT of my band shall be a BLUES RITHM
• My band shall have an ELECTRIC BASS GUITAR • My band shall have a DRUM KIT • The ELECTRIC GUITARS shall play with an
ELECTRIC BASS GUITAR and a DRUM KIT
degIC = RELATIONS in Requirements
RELATIONS in Input Docs =
4
6 = 0.66
The BACKBEAT in the RITHM!
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 14 / 27
A Word Game to Improve Completeness
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 15 / 27
Completeness Assistant for Requirements
Automatic Concept
Extraction
Automatic Relation
Extraction
Input Documents
RelevantTerms
RelevantRelations
Write a requirement with the following terms:
BEATBLUES RITHM
BACKBEAT
Completeness Assistant for
Requirements (CAR)
The BEAT shall be a BLUES RITHM with BACKBEAT
Concept completeness
Interactioncompleteness
80% 90%
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 16 / 27
Completeness Assistant for Requirements
Automatic Concept
Extraction
Automatic Relation
Extraction
Input Documents
RelevantTerms
RelevantRelations
Write a requirement with the following terms:
BEATBLUES RITHM
BACKBEAT
Completeness Assistant for
Requirements (CAR)
The BEAT shall be a BLUES RITHM with BACKBEAT
Concept completeness
Interactioncompleteness
80% 90%
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 17 / 27
Completeness Assistant for Requirements
Automatic Concept
Extraction
Automatic Relation
Extraction
Input Documents
RelevantTerms
RelevantRelations
Write a requirement with the following terms:
BEATBLUES RITHM
BACKBEAT
Completeness Assistant for
Requirements (CAR)
The BEAT shall be a BLUES RITHM with BACKBEAT
Concept completeness
Interactioncompleteness
80% 90%
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 18 / 27
Completeness Assistant for Requirements
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 19 / 27
Experiment
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 20 / 27
Automatic Train Supervision (ATS) System
ATS SystemCentralized system that monitors and regulates the movement ofthe trainsAutomatically routes trains, and sends them speed profiles thatshall be followed while moving through the railway networkNormally equipped with a user interface where the ATS operatorcan view the position of all the trains, their schedule, and otherinformation
Input DocumentThe IEEE Std 1474.1-2004 includes the basic requirementsof an ATS system
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 21 / 27
Experiment: Phase 1
Completeness Assistant for
Requirements (CAR)
IEEE Std 1474.1-2004
(5 pages)
67 Relevant Terms
316 Relevant Relations
Requirements Document Subject A
Requirements DocumentSubject B
Com
pute
Bac
kwar
d C
ompl
eten
ess
Com
pute
For
war
d C
ompl
eten
ess
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 22 / 27
Experiment: Phase 2
Completeness Assistant for
Requirements (CAR)
IEEE Std 1474.1-2004
(5 pages)
67 Relevant Terms
316 Relevant Relations
Requirements Document Subject B
Requirements DocumentSubject A
Com
pute
Bac
kwar
d C
ompl
eten
ess
Com
pute
For
war
d C
ompl
eten
ess
A. Ferrari, et al. (ISTI-CNR) Requirements Documents Completeness 23 / 27
Results
Subject Num.Reqs Tool degCC degIC degFEC degLCC
A 36Yes 68.7% 17.1% 47.6% 40%No 52.3% 12.8% 61.9% 50%