How DOORS Helps JPL Get to Mars and Beyond Best Practices in Metrics, V&V and Traceability Margaret H. Smith Jim M. Grimes Ross M. Jones Trisha Jansma Jet Propulsion Laboratory California Institute of Technology
Nov 18, 2014
How DOORS Helps JPL Get to Mars and Beyond
Best Practices in Metrics,
V&V and Traceability
Margaret H. Smith
Jim M. Grimes
Ross M. Jones
Trisha Jansma
Jet Propulsion Laboratory
California Institute of Technology
Outline
Overview of DOORS Use at JPL
Metrics
Verification and Validation (V&V)
Traceability
5/24/2012 M. Smith - 2 Caltech – Jet Propulsion Laboratory
Topic 1
Topic 2
Topic 3
JPL use of DOORS at JPL Statistics
– 800+ users – 600+ users have taken our in-house DOORS training. – 50 DOORS licenses – 50 projects have used DOORS, 22 are active – 15 years using DOORS, preceded by 8 years using in-
house TRACER tool. – 5 years of a standard process with in-house training – 2 System Analysts for daily project support – 1 Full-time Equivalent (FTE) Systems Engineer for
defining the method, teaching and project consultation
5/24/2012 M. Smith - 3 Caltech – Jet Propulsion Laboratory
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 4
JPL
TRACER
(1988)
QSS
DOORS
(1996)
IBM Rational
DOORS
(2008)
DOORS Provenance – According to JPL
Source code and User Guide were
distributed from NASA COSMIC site from
1990 to ~1995. Developed by the Cassini
Project and used by JPL and ESA
Telelogic
DOORS
(2000)
JPL TRACER
mothballed
(1995)
DOORS
adopted at
JPL(1996)
Sample Project Sizes
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 5
Large legacy infrastructure project
Medium complexity earth orbiting mission
Smaller complexity lunar mission (JPL managed)
Large complexity deep space mission (JPL managed)
Measurements were generated by
the JPL Trace Tree Tool (TTT)
Flagship mission
Project # objects # links # Modules installed
TTT website size (GB)
DSN 98995 5709 125 1.9
SMAP 17278 13311 91 1.9
GRAIL 36031 30650 123 2.7
SES_JUNO 84844 43526 227 3.6
MSL 25641 5808 106 1.2
Note: Number of objects includes all objects in formal modules: table cells, headers, regular objects with or without text or OLE content. The number of
requirements is about one-third the number of objects.
0 50 100 150 200 250
0 20 40 60 80 100 120
DSN
SMAP
GRAIL
SES_JUNO
MSL
# Modules
# Links, Objects (Thousands)
# links
# objects
# Modules
Why Have a Standard Process?
• DOORS is a general purpose tool and there are many possible ways to use it.
• For the first 15 years we used DOORS, each new project reinvented the wheel. – Big projects could afford to invent their own method and have internal support.
• We learned a lot from what the big projects did wrong and did right.
– We now have more small projects and fewer big projects.
• Small projects cannot afford to reinvent. Need a ready-made solution.
• The most common mistake: trying to run the old document-centric process in DOORS and not taking advantage of the tool’s capabilities.
• Projects were making mistakes, blaming DOORS, and delaying import of requirements into tool as late as possible. – Managing requirements in spreadsheets and not getting the benefits of early
sharing and collaboration.
5/24/2012 M. Smith - 6 Caltech – Jet Propulsion Laboratory
What Has and Hasn’t Been Standardized? Has
• Ownership, allocation and negotiation mechanism
• Attributes and their definitions – for requirements and V&V planning/tracking
• Views – requirements traceability, allocation status, requirements to V&V traceability
• High-level requirements template, boilerplate material
• Scripts – Requirement Maturity Metrics, and V&V Burndown and Statistics, and numerous housekeeping scripts used by DOORS System Analysts (SAs).
• Tools – Trace Tree Tool for graphically displaying individual requirements traceability and project requirements flow-down.
• Practices – the JPL Systems Engineering practices that culminate in requirements.
• Licensing and Support – the JPL institution purchases DOORS licenses for use by all projects and funds two Systems Analysts (support).
• Training/Deployment – Engineers who are involved in requirements work attend a JPL developed course that teaches basic DOORS skills and the JPL standard method for development requirements and tracking verification in DOORS.
Hasn’t
• Requirements levels and element decomposition
• Detailed requirements templates
• DOORS collaboration method with partners and subcontractors
We owe the success of the JPL Institutional Requirements and V&V Tracking process and training program to Richard Stoller (retired), and Karen Boyle (who sadly passed away in February, 2011).
5/24/2012 M. Smith - 7 Caltech – Jet Propulsion Laboratory
Metrics
• Maturity Definition: + V&V Method/Approach defined + Rationale defined + Linked to parent(s) + Links exist to this requirement from Satisfiers (including V&V activities) + Satisfier (lower level owner) accepts allocated requirement – TBD, TBR, TBCs
• Requirement Maturity Metric script:
– Implemented in DOORS Extension Language (DXL) – Generates per module statistics across all modules in a Project, one module per
row – Produces .csv file of statistics. – Project can customize output to suit their Project Management/reporting needs. – Provides a quick summary of requirement progress and facilitates comparison
between modules.
5/24/2012 M. Smith - 8 Caltech – Jet Propulsion Laboratory
Topic 1
Example
5/24/2012 M. Smith - 9 Caltech – Jet Propulsion Laboratory
Gold
plating?
Allocations
undecided or under
negotiation
Unknown
capability?
219
3
What Has Changed Today?
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 10
Assessing Requirements activity and churn
Type of change
Module name
Requirement number
Daily email to Project’s requirements owner
Requirements are marked Modified if any
attribute, with the setting Affect change dates
turned on, has been changed since the last time
this report was generated.
• Initial benefit -- requirements changes were immediately visible to engineers responsible for verifying requirements because verification activity artifacts and verification events were now linked directly to requirements
• Projects can use as-is or customize a generic DXL script to generate verification statistics and graphics.
• JPL Projects have been recording Verification information in DOORS for the last 10 years.
Verification Planning and Tracking
Other benefits:
• Lower tooling cost– use existing institutionally supported tool and support
• Enables useful views of requirements, verification planning & status, and verification events.
• Saves projects money through reuse of tooling and method.
5/24/2012 M. Smith - 11 Caltech – Jet Propulsion Laboratory
Topic 2
Verification Process Project
Verification Planning
Verification Activity Planning
Verification Activity
Execution
Verification Status
Reporting
Requirements Development
Get DOORS Training / consulting
Reqmts to VA mapping captured
Plans and procedures developed outside DOORS and hyper-
linked from the DOORS VA
Verification Activity Groups
defined
Execute VA procedure, analyze
results and determine status
Record status of the VA (procedure)
run and create problem/failure
reports (if needed)
Cognizant person approves VA
execution result
Cognizant person approves that the requirement has
been verified
Generate V&V burndown charts and statistics and
present at management
meetings
Take corrective action (if
necessary)
Schedule VA activity
completion (need by) date
Develop V&V method, venue and approach
5/24/2012 M. Smith - 12 Caltech – Jet Propulsion Laboratory
Verification Event (VE) Verification Item (VI) 1..M
1..M
Verification Activity Group (VAG)
1
1..M
1
0..M
1 1
0..1
1
VE data elements originate in the AR and are pushed to VE to complete the status picture.
1
Verification Activity (VA)
Activity Report (AR)
Test Data Directory
1
Test Plan Test
Procedure 1..M 0..1
1
In DOORS
In Other Tools
VI – Verification Item, is anything that needs to be verified. All requirements become Vis when they are being verified. VA – Verification Activity. The testing activities described in a single Test Plan, Procedure or Analysis. A group of associated VAs are a VA Group (VAG) VE – Verification Event. A run or execution of a VA.
Definitions
Verification Elements
5/24/2012 M. Smith - 13 Caltech – Jet Propulsion Laboratory
VE • VA id (a reference) • …. data imported from AR • url of Activity Report • url of test data directory
VI or ITL item •object identifier (ID) •statement of requirement •rationale •…. •method •verification approach •audit status •audit completed on (date) •audited by
1..M 1..M 1
0..M
1
0..M
1 1
0..1
1
1
VA • activity name or activity
group name • VA priority • VA status • VA owner • venue • date scheduled (for
completion) • date completed • plans and procedures • PFRs
Activity Report (AR)
Test Data Directory
0..1
1..M
Test Plan • version
Test Procedure • version
1
V&V Activity Planning V&V Activity Execution
V&V Status Reporting
Requirements Development
Verification Attributes
5/24/2012 M. Smith - 14 Caltech – Jet Propulsion Laboratory
Project Module Configurations VIM – Verification Item (Requirements) Module VAM – Verification Activity Module VEM – Verification Event Module
VEM VAM
VIM
VIM VIM
VIM VIM VIM
A single VAM for the Project’s complete requirement tree • partition VAM to give owners
granular write access
VEM
VIM
VIM VIM
VIM VIM VIM
VAM
VAM VAM
VAM VAM VAM
Multiple VAMs reflecting the Project’s requirements structure or V&V organization.
VAM / VEM
VIM
VIM VIM
VIM VIM VIM
Combined VAM and VEM for projects that only need information on the latest run/execution.
5/24/2012 M. Smith - 15 Caltech – Jet Propulsion Laboratory
Verification Item (VI) Burndown
0
50
100
150
200
250
1/8
/2008
1/2
2/2
008
2/5
/2008
2/1
9/2
008
3/4
/2008
3/1
8/2
008
4/1
/2008
4/1
5/2
008
4/2
9/2
008
5/1
3/2
008
5/2
7/2
008
6/1
0/2
008
6/2
4/2
008
7/8
/2008
7/2
2/2
008
8/5
/2008
8/1
9/2
008
9/2
/2008
9/1
6/2
008
9/3
0/2
008
10/1
4/2
008
10/2
8/2
008
11/1
1/2
008
11/2
5/2
008
12/9
/2008
12/2
3/2
008
1/6
/2009
Nu
mb
er
of
Item
s t
o b
e V
eri
fied
Interval End Date
Cu
rre
nt d
ate
Scheduled
Not Confirmed
Count of VIs (modules included selected from pick list): Scheduled = total number of VIs remaining to be completed per schedule Not Completed = total number of VIs remaining to be completed actually Not Confirmed = total number of VIs remaining to be Confirmed-Final = total number of VIs remaining to be completed but not scheduled
Not Scheduled
Scheduled Actual
Not Confirmed
Not Scheduled
Actual (auto status)
A pick list allows selection of any group of VIMs for this chart.
5/24/2012 R. Stoller, K. Boyle - 16 Caltech – Jet Propulsion Laboratory
Verification Activity Burndown
0
2
4
6
8
10
12
14
16
Nu
mb
er
of
Act
ivit
ies
to b
e V
eri
fie
d
Scheduled - Count of not completed linked VAs scheduled to be completed
Not Completed - Count of not completed linked VAs remaining to be completed Not Scheduled - completed-NF or not scheduled
Scheduled
Actual
Not Scheduled
(Actual)
A pick list allows selection of any group of VAs for this chart.
Scheduled = total number of VAs remaining to be completed per schedule
Actual = total number of VAs remaining to be completed actually
Not Scheduled = total number of VAs that are not completed and are not
scheduled
5/24/2012 R. Stoller, K. Boyle - 17 Caltech – Jet Propulsion Laboratory
Verification Item Statistics from V&V Burndown Script that generates data to Excel
"Bad" if > 0
"Neutral"
"Good"
"Normal"
Count of VIs in VIMs (shall statements only)
A: B: C: D: E: F: G: H: I: J: K: L M: N:
VIM NAME Total VIs
VIs Linked to
VAM
VIs Linked to
VAGs
VIs Linked to
VAs VIs Auto
Verif VIs Conf
Verif
VIs Not Linked to
VAM
VIs Linked to
Reject
VIs Not Linked to
a VA
VIs Not Auto Verif
VIs Not Conf Verif
VIs Not Verif and
Late
V&V Engr Responsible for this VIM
VIM1 625 600 15 575 200 150 25 10 50 425 450 5 John Jones
VIM2 500 500 10 485 100 75 0 5 15 400 etc 5 Susie Jones
VIM3 1000 1000 0 1000 0 0 0 0 0 1000 etc 0 Mary Smith
… … … etc etc etc etc etc etc etc etc etc etc
VIM n 200 200 5 190 25 5 0 5 10 175 etc 0
Total 2325 2300 30 2250 325 230 25 20 75 1975 etc 10
Relationships: B=C+H; I is not included in D; E=C-I-D; J=B-E=H+I+D; K=B-F; L=B-G
M = VIs not auto verified AND scheduled completion date is missed or has not been scheduled or is not
linked. Any object containing the word “shall” enclosed in quotes (“shall”) is not included in any statistic.
5/24/2012 R. Stoller, K. Boyle - 18 Caltech – Jet Propulsion Laboratory
* VAs are organized in VAM under VAGs and sub-header VAGs, but tabulated separately here.
** Identifies if there is an http entry in the Plan and Procedure attribute.
VA NAME*
# of VI Parent Links
Are VAs Fully Linked to VIs
? VA Status VA Sched-
uled?
** Is there a VA Plan &/or a Procedure?
Are there any Suspicious VI
Links?
Schedule Days Remaining (if not
completed) V&V Engineer /
Owner of VA
VA-x 25 Yes Blank 08/10/09 No Yes 10 Jack Jones
VA-y 20 Yes Completed-NF 07/26/09 Yes No -5 Mary Jones
VA-z 100 Yes Completed No Yes No Susie Que
… 350 … Blank No No No Not Sched Tom Smith
VA-n 55 Yes Blank 08/20/09 Yes Yes 20 Susie Smith
Verification Activity Statistics from V&V Burndown Script that generates data to Excel
"Bad" if > 0
"Neutral"
"Good"
"Normal"
5/24/2012 R. Stoller, K. Boyle -19 Caltech – Jet Propulsion Laboratory
DOORS Trace Tree Tool (T3)
• Primary Motivations – Provide JPL users with an alternative graphical presentation of
requirement flow (links) – Allows navigation of requirements based on hierarchy or flow – Enables seamless navigation between modules following flow-
down/links – Provides easy access via Web-based, platform independent interface
• Secondary Motivation - evolved with implementation – Provide an indication of timeliness of requirements flow-down – Provide an efficient way to checkout verification planning
• History – Conceived and developed by a JPL Software Systems Engineer (James
Grimes) in 2006 – Enthusiastically used by 8 JPL Flight Projects
5/24/2012 J. Grimes - 20 Caltech – Jet Propulsion Laboratory
Topic 3
JPL has parent-to-child requirements relationships where a parent has 1 to tens of children, and links between levels can be many-to-many. We have trouble seeing the forest for the trees and navigating requirements flow within DOORS.
Trace Tree Tool View – Example Index Page
• • •
ProjX
ProjX
ProjX
ProjX
ProjX
5/24/2012 J. Grimes - 21 Caltech – Jet Propulsion Laboratory
Project Map
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 22
Number of links/flowed-down requirements appear on edges
Module Page
• • •
…….
ProjX
5/24/2012 J. Grimes - 23 Caltech – Jet Propulsion Laboratory
Requirement Page
Attributes continue below
…….. ………………….
5/24/2012 J. Grimes - 24 Caltech – Jet Propulsion Laboratory
Object Text Graphic
Attributes continue below
………….. ……………………… ……………….. …………..
………….. ……………….. ……………………….
………………………
……………………… ………………………
5/24/2012 J. Grimes - 25 Caltech – Jet Propulsion Laboratory
Verification Approach Graphic
Attributes continue below
………………………………………………………………
…………………………………………………………
……………
…………………………………………………………
…………… …………………………………………………………
5/24/2012 J. Grimes - 26 Caltech – Jet Propulsion Laboratory
Typical DOORS T3 User Scenarios • Quickly obtain the contents of a requirement with a known ID
– Main page Module page Requirement number
• Scan a particular Project module for a specific requirement
– Main page Module page, then
• Scroll up and down at web browser speeds
• Use browser’s find capability (not as flexible as DOORS, but faster)
• Trace requirements flow-down starting at any requirement
– Click on boxes in graph going up- or downstream in requirements flow
– Go from there to text, attributes
• Obtain immediate visual indication of timeliness of flow-down
– E.g., has upstream requirement X been updated more recently than my requirement? If yes, then it’s red. Actual date is shown on X’s page.
– Is flow-down complete? Does the requirement have a parent?
• Provide supportive evidence during Project Reviews
– Quickly show existence of flow-down
– Quickly show V&V attribute, implementation status (if in attribute) 5/24/2012 J. Grimes - 27 Caltech – Jet Propulsion Laboratory
DOORS Trace Tree Tool (T3) High-Level Process/Execution Flow
5/24/2012 Caltech – Jet Propulsion Laboratory T. Jansma - 28
Development Tools Used • DXL – DOORS eXtension Language, a DOORS proprietary C-like, object-
oriented language available within the DOORS client
• Perl – a scripting language widely available on UNIX, Mac OS X, Linux, Windows
• Graphviz – open source Graph Visualization software developed at AT&T
that includes the Dot language and dot program for drawing directed graphs
• XML and Perl/C-based XML parsing software – open source software
• Java for processing images
• Standard UNIX utilities such as date, rm, ls, rsync, etc.
• SVN – Subversion open-source version control tool
5/24/2012 J. Grimes - 29 Caltech – Jet Propulsion Laboratory
JPL’s Direction with DOORS • Systems Engineers on JPL Projects are starting to infuse Model Based
Systems Engineer / SysML.
– Modeling includes architecture, behavior and resources.
– Many pockets of discipline specific modeling tools exist in the Spacecraft subsystem organizations
– Vision is to use a central, standards based repository to interconnect and exchange model data.
– Requirements will still be important but there is now an opportunity to augment requirements with more extensive and formal designs.
• Small and technology development Projects may need a lighter weight DOORS-based process.
• JPL is working on more partnership projects. Module exchange needs to be negotiated and piloted early.
• We are piloting a Requirements Architecture, table-top review that will occur before Project and System requirements are developed.
• We have begun teaching just-in-time DOORS and process classes directly to and with Flight Projects.
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 30
Aimed at
influencing,
sharing
Lessons
Learned and
emerging best
practices.