Implications of Capability-based Planning on Requirements Engineering Leonard Sadauskas Presented at NDIA SE Conference Requirements Development & Management Session 22 October 2008
Implications of Capability-based Planning on Requirements Engineering
Leonard SadauskasPresented at
NDIA SE Conference Requirements Development & Management Session
22 October 2008
22 Oct 2008 Leonard Sadauskas 2
Disclaimer
The views and opinions expressed in this presentation are those of the author and do not reflect the policy of the Department of Defense
22 Oct 2008 Leonard Sadauskas 3
Scope of this Presentation
• Capability-based planning• The problem and solution space interface• The dual roles of measures of effectiveness (MOEs)• Capability feedback process• Issues, challenges and trends
22 Oct 2008 Leonard Sadauskas 4
Definitions• Capability-based planning (CBP):
– An overarching framework for planning under uncertainty that provides capabilities suitable for a wide range of modern-day challenges and circumstances while working within an economic framework that necessitates choice
• Capabilities-Based Assessment (CBA)– Study that identifies the capabilities (and operational
performance criteria) required to successfully execute missions
• Capability:– The ability to execute a specified course of action
• Move troops rapidlyCandidate Solutions:• Truck• Ship• Aircraft
22 Oct 2008 Leonard Sadauskas 5
Capabilities-based Planning Framework(work in progress since 2003)
FieldedCapabilities
CBP Analysis
Acquisition PPBE
Feedback
Non-materielSolutions
CBA
22 Oct 2008 Leonard Sadauskas 6
Focus of this Presentation
Problem SpaceCBA
Solution SpaceAcquisition
Fielded CapabilityO&S
Feedback Process
22 Oct 2008 Leonard Sadauskas 7
Draft CJCSI 3170.01G JCIDS Processand Acquisition Decisions
ProblemSpace
SolutionSpace
ICD
EMDDICD
DCR
CBA
AoA TechnologyDevelopment LRIP
DraftCDD CDD CPD
ICD
MDD MS APDR
MS B MS C
22 Oct 2008 Leonard Sadauskas 8
Capabilities-Based Assessment (CBA)
Existing Guidance
FAAWhat arewe talkingAbout?
FNAHow goodAre we atDoing it?
FSAWhatShould weDo about it?
From:CBA User’s GuideV2 December 2006
22 Oct 2008 Leonard Sadauskas 9
Draft 5000.02 The Defense Acquisition Management Framework.
IOCBA
Technology Development
Engineering and Manufacturing Development & Demonstration
Production & Deployment
Systems Acquisition
Operations & Support
C
Sustainment
The Materiel Development Decision precedes entry into any phase of the acquisition framework
Entrance criteria met before entering phase
Evolutionary Acquisition or Single Step to Full Capability
FRP DecisionReview
FOC
LRIP/IOT&EPost-CDRAssessment
Pre-Systems Acquisition
(ProgramInitiation)
MaterielSolutionAnalysisMateriel Development Decision
User Needs
Technology Opportunities & Resources
= Decision Point = Milestone Review
22 Oct 2008 Leonard Sadauskas 10
Problem / Solution Space Interface
Business Analyst Requirements / SystemsEngineer
22 Oct 2008 Leonard Sadauskas 11
Information Transfer at the InterfaceIf your goal in software development is to "make the business case come true" (and by 'business case' I mean the initial justification for spending time, money, and effort on the development in the firstplace), then the most important thing to understand is: why are we building this? That is, what are the needs of the customers (or business)? If you don't know, or clearly understand, the customer needs, then you cannot know if you are building the right system - which then makes the technical correctness of the functional spec (what we intend to build) or the design spec (how we think it should work) a moot point.
Richard Zultner 30 Sep 2008 Requirements-Engineering Group
22 Oct 2008 Leonard Sadauskas 12
AoA and Effectiveness Analysis Process
Task 1 Task 2 Task 3
Alternative1
Alternative2
Alternative3
MOE1-1
MOE1-2
MOE1-3
MOE2-1
MOE2-2
MOE2-3
MOE3-1
MOE3-2
MOE3-3
MOEsFunctional
Needs
Analysis GuidancePlanning &
Methodology
Determine Alternative Solutions
Select ModelsAnd Data
PerformAnalysis
Study Operational Effectiveness Analysis
Initial Capabilities Document(ICD)
Materiel Approaches
Mission Tasks
22 Oct 2008 Leonard Sadauskas 13
Definitions and Attributes of MOEs
• MOEs are standards against which the capability of a solution to meet the needs of a problem may be judged. The standards are specific properties that any potential solution must exhibit to some extent.
• Therefore, MOEs are independent of any solution.
• A meaningful MOE must be quantifiable and a measure to what degree the real objective is achieved.
The MOE is part of both the AoA andthe CBP feedback process
22 Oct 2008 Leonard Sadauskas 14
Feedback Process
• Multiple sources of capability information• Separate JS, COCOM and Service Processes• Not part of JCIDS or AMS• Statutory for fielded capability as
Post Implementation Review (PIR)
PIR
Fielded Capability
Feedback Path
22 Oct 2008 Leonard Sadauskas 15
Post Implementation Review (PIR)Defined
An analysis of an investment or acquired system that is part of a capability portfolio, operating in its intended environment, using data collected from various sources to answer the question:
Did we get what we needed, and if not what to do about it?
22 Oct 2008 Leonard Sadauskas 16
PIR Information Path in Feedback Process
Capability Based AnalysisEstablishes Need and MOEs that Define Need Fulfillment
Doctrine, Organization, Training, Leadership and Education, Personnel and Facilities (DOTLPF)
Plan and Execute Process and Cultural Change Management
MaterielDevelop System Requirements, Acquire System and conduct OT&E on Key Performance Parameters (KPPs)
Post Implementation Review (PIR) Assesses Outcome Of Investment By Measuring MOEs
A B CMDD FRPD IOC FOCPIRCBA
22 Oct 2008 Leonard Sadauskas 17
ICD
MS A MS B
CPD
DT&E
OT&E
MS C
TEMP
TEMPFOT&E
PIR Plan
•Platform Readiness Assessments•CC Exercise results•User Satisfaction Surveys•Annual CFO Report Input•Mission Readiness Assessments•ROI Computation•War Games
PIR
SEPIntegration& Test
Contract
IOC FOC
FCB/Sponsor
PIR
Build
CDD
FCB: Functional Capabilities BoardICD: Initial Capabilities DocumentCDD: Capability Development DocumentCPD: Capability Production DocumentMOE: Measure of Effectiveness
MOEs
PIR in the Feedback Process
22 Oct 2008 Leonard Sadauskas 19
Model Compatibility & Sharing Opportunity at the Problem-Solution Interface
Task 1 Task 2 Task 3
Alternative1
Alternative2
Alternative3
MOE1-1
MOE1-2
MOE1-3
MOE2-1
MOE2-2
MOE2-3
MOE3-1
MOE3-2
MOE3-3
MOEsFunctional
Needs
Analysis GuidancePlanning &
Methodology
Determine Alternative Solutions
Select ModelsAnd Data
PerformAnalysis
Study Operational Effectiveness Analysis
Initial Capabilities Document(ICD)
Materiel Approaches
Mission Tasks
Models
22 Oct 2008 Leonard Sadauskas 20
MOE Deficiencies in CBA
CBA Document MOE DeficienciesDec 2005 through Jul 2008
0
5
10
15
20
25
A B C D E F G H I
Functional Capability Board# CBA Docs Reviewed# MOE Deficiencies
22 Oct 2008 Leonard Sadauskas 21
Potential Impact of MOE Deficiencies
• Likely Scenario: – 43% ICDs submitted to the JS for review during past 30
month period contained no MOEs• Assumptions (conservatively stated)
– Requirements volatility accounts for 10% of Program of Record cost overruns.
– Lack of MOEs accounts for 10% of requirements volatility– The 2008 DoD Major Program cumulative expenditure is
$800B + $800B less than major = $1,600B– Cost overrun is 5% or $80B
• Cost of not providing MOEs to the SE process:– .1 x .1 x .43 x $80B = $344M
22 Oct 2008 Leonard Sadauskas 22
Recent Trends
• Publication of CBA Guide v2 by JS-J8 in Dec 2006– Describes CBA process– Guidance for study plan and planning– Discusses analytic approaches
• Development of MOEs• Implementation of requirements manager training and
certification – USD(AT&L) Memoranda of 2 September 2008, Requirements
Management Certification Training Program Policy, John Young– Includes training and certification of requirements authors,
reviewers and validators • Joint Staff considering shortening the CBA cycle to a month or
two instead of a year or two.– Impact on development of MOEs not clear– May be signal that Problem-Solution interface boundary is shifting
22 Oct 2008 Leonard Sadauskas 24
Some Remaining MOE Issues
MOEDesiredCapability = MOE Existing Capability + MOE Gap Capability
MOEGap Capability = MOEDOTLPF + MOEICDs
where DOTLPF = ƒ(Existing processes + changes needed
to maximize benefit of materiel investment)��
• How could MOEs be allocated?
• How could MOEs be traced?Could MOEs be traced through the DOTLPF and materiel acquisitionprocesses in a manner analogous to requirements tracing by the systems engineers?
22 Oct 2008 Leonard Sadauskas 26
Models Bridge Layers of Requirementsand Provide Verification Criteria
INCOSE Work Shop 08
Functionalmodeling
Non-Functionalmodeling
Functionalmodeling
e.g Goal / Usage modeling
e.g. Functionalmodeling
Sponsor Capability
Requirements
DesignSpecification
SystemRequirements
Statementof need
e.g. Performancemodeling
After Jeremy Dick’s Sandwich Requirements & Modeling Concept
Measures of Effectiveness
VerificationRequirements
Integration Requirements
Capability Assessment
Results
System Test Results
IntegrationTest Results
22 Oct 2008 Leonard Sadauskas 27
Typical MOE Situations
• Outcome metrics presented but measures deferred for CDD
• Study team not adequately staffed
• Study team neither tasked nor funded to undertake analytic approach needed to develop MOEs
• Outcome measures stated in narrative but solution performance parameters KPPs presented as MOEs
• CJCSM 3170.01 does not explicitly require MOEs for the ICD, Draft CJCSI 3170.01G has eliminated the term MOE– Uses the term desired effects
• Developed MOEs do not address desired outcomes
22 Oct 2008 Leonard Sadauskas 28
Cause - Effect Candidates
• Lack of capability analyst training– Analyst jumps into solution mode comfort zone
• Capability lexicon confusion– Miscommunication amongst analysts and reviewers
• Regulatory MOE requirement inconsistencies– Analyst takes path of least work
– ICD approval available without MOEs
• Inadequate study team guidance– Analyst not steered to analytic approaches needed to develop MOEs