Joan WeszkaLockheed Martin Mission Systems
Systems & Software Resource Center
© Copyright Lockheed Martin Corporation 2003
CMMI® Technology Conference 2003November 19, 2003
.
Transition from SW-CMM to CMMI: The Benefits Continue!
® CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 2
Agenda
Context Background on Lockheed Martin’s (LM) CMMI®
transition approach Benefits in the Software CMM® and CMMI® eras
LM Systems Integration (Owego, NY)LM Maritime Systems & Sensors – Tactical Systems (Eagan, MN)LM Maritime Systems & Sensors – Radar Systems (Syracuse, NY)LM Maritime Systems & Sensors – Undersea Systems (Manassas, VA)
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 3
Context
A number of Lockheed Martin organizations that tracked quantitative process improvement benefits during their SW-CMM high maturity journey have now transitioned to CMMI
Experience to date indicates that these benefits have continued with CMMI implementation
Benefits derived are not attributable only to CMMI
Many initiatives are underway concurrently with CMMI (and SW-CMM) deployment
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 4
Lockheed Martin’s CMMI® Deployment Approach
Many Lockheed Martin (LM) companies have institutionalized best-of-breed integrated processes (e.g., IPPD)
Multiple process models and standards are in use LM identified industry and internal best practices
as sources for corporate-wide process requirements The LM Integrated Engineering Process (LM-IEP)
standard synthesizes these requirements
Lockheed Martin’s Integrated Engineering Process Standard is being deployed under corporate policy.
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 5
Key Tenets of Lockheed Martin’s CMMI® Transition
Address CMMI® in the context of your organization’s business requirementsLockheed Martin’s Integrated Engineering Process
(LM-IEP) standard includes CMMI®, in addition to other standards and requirements (e.g., ISO/IEC 15288, ISO 9001:2000)
Adopt an incremental appraisal approach Lockheed Martin Continuous Appraisal Method (CAM)
has been successfully deployed with CMMI® and is being extended for use with LM-IEP
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 6
LM-IEP, CAM and CMMI® Relationship
CMMI® provides a set of integrated process and appraisal method requirements
LM-IEPLM-IEPLM-IEPLM-IEP
CMMICMMI®®
ProcessProcessRequirementsRequirements
CMMICMMI®®
ProcessProcessRequirementsRequirements
ARC* Class AARC* Class AAppraisalAppraisal
RequirementsRequirements**
ARC* Class AARC* Class AAppraisalAppraisal
RequirementsRequirements** Lockheed Martin Integrated Engineering Process
* Appraisal Requirements for CMMI®® (ARC)
CAM
CAM meets ARC Class A requirements.
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 7
LM CPSs
ANSI/EIA-632
ISO 9001:2000
IEEE 1220
LM HardwareProcess Standard
ISO/IEC-12207
CMMI®® V1.1
ISO/IEC-15288
LM-IEP Context Diagram
Common Source Standards
OrganizationalStandard
Process
LM-IEPStandard
Industry Standards
GovernmentStandards
Domain Specific Standards
Industry Stds
Government Stds
Project Specific Standards
ProjectDefinedProcess
In LM-IEP V2 issued 1/03
Reference only
Integrated Processes
Integrated Work Products
Integrated Methods
Tailored OSP
EPI Process & Methods
Informative only
CPS – Corporate Policy StatementEPI – Engineering Process Improvement
Company or Business Unit
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 8
Continuous Appraisal Method (CAM) Design Goals
Minimize appraisal preparation and reduce cost Integrate process improvement with process
appraisal activities Facilitate appraisal scheduling and minimize
disruption for participants Provide an appraisal environment conducive to
process improvement Promote institutionalization
CAM is being used with CMMI® and being extended for appraisal of LM-IEP requirements.
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 9
Project Appraisals
Overview of Incremental Appraisal Using CAM
Maintenance Review
Institutionalization focus with minimal project disruption
OSP = Organizational Standard ProcessPA = Process Area
OSPAppraisal
Appraise OSP Fix Weaknesses Re-appraise
Fix Weaknesses Re-appraise Appraise PA 1
...
Planning
Fix Weaknesses Re-appraise Appraise PA 1
Fix Weaknesses Re-appraise Appraise PA 1
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 10
Feedback on CAM usage CAM has been or is being deployed at 12 Lockheed Martin
operating units using CMMI®
6 prior CAMs have been completed using EIA/IS 731
Experience with CAM has been positive:More focus on process improvementLess invasive to programsLess stressful to the organizationMore value-add, in-depth findingsMore cost effective
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 11
Lockheed Martin Systems Integration - Owego Process CredentialsLockheed Martin Systems Integration - Owego Process Credentials
Q-100 Award
19901990
19931993 IBM Baldrige Gold Award
New York State Excelsior Award
19941994
19951995 RIT / USA TODAYQuality Cup Finalist
19961996 ISO 9001:1994 /SW CMM Level 3
19971997 SW CMM Level 5
19981998 LM21 Best Practices
20012001 ISO 9001:2000 /SEI CMMI Pilot Site
20032003 CMMI Level 5 /AS 9100A
20002000 Lean / Six Sigma
TQM
20022002
20032003
20022002
20012001
20002000
19981998
19971997
19961996
19951995
19941994
19931993
19901990
Joan Weszka - 12 © Copyright Lockheed Martin Corporation 2003
11/19/03
Lockheed Martin Systems Integration Owego - QualityLockheed Martin Systems Integration Owego - Quality
Software Defects per Million Delivered Source Lines of Code
-80%
-60%
-40%
-20%
0%
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002
% R
edu
ctio
n S
TR
s /
MS
LO
C Improvements Since ‘92
80.5% Overall
Inte
grat
ed P
roce
sses
SW
CM
M L
evel
5
CM
MI
Leve
l 3
Cap
able
CM
MI
Leve
l 5
* STR / MSLOC = Software Trouble Report/Million Source Lines of Code
SW
CM
M L
evel
3
2003
® ® CMMI and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon UniversityCMMI and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University..
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 1311/19/03
© Copyright Lockheed Martin Corporation 2003Joan Weszka - 13
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
140.0%
160.0%
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002
CM
MI
Leve
l 5in
200
3
SW
CM
M L
evel
3
SW
CM
M L
evel
5
Improvements Since 1992:
9.8% Average per Year
ISD
Pro
cess
es
SW
CM
M L
evel
3 C
apab
le
Lockheed Martin Systems Integration Owego Lockheed Martin Systems Integration Owego -- Software ProductivitySoftware Productivity
Software Productivity (All Software including Reuse)
Contributors:•Increased Reuse (Domain Specific)•Process Maturity and Compliance•Process Consistency•Increased use of High Order Language / 4th Generation / Object Oriented•Use of development and test tools
CM
MI
Leve
l 3
Cap
able
2003
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 14
LM Maritime Systems & Sensors Tactical SystemsProcess Improvement Credentials
Oct. 1999 – Attained SW-CMM® level 4 Dec. 2000 – Attained Systems Engineering Capability
Model (EIA 731) level 3 Jan. 2001
Began focus on integrated process improvementBegan transition to CMMI®
June 2002 – May 2003 CAM AppraisalOSP: Target profile 5 for CMMI®-SE/SW/IPPD/SS Projects: Target profile 3 for CMMI®-SE/SW
August 2003 – SCAMPISM AppraisalAchieved target profile 3 for CMMI®-SE/SW
® SW-CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 15
LM Maritime Systems & Sensors Tactical SystemsInitiatives
Streamlining of references and guidance documents which supplement the OSP
Value stream mapping of engineering process/business model
LM-IEP gap analysisSelf audit process complianceAirworthiness manual Risk Management methodology best practiceUpgrade of Process Asset Library (PAL)Mechanical engineering guidebook Change management pilotDefect prevention pilot
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 1611 /19 /03
© C op yrigh t L ockh eed M artin C orp ora tion 2003Joa n W es zk a - 16
L M M aritim e System s & Sensors T actical System s Softw are P roductiv ity
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
1979 1984 1989 1994 1999 2004
Year
So
urc
e S
tate
men
ts p
er H
ou
r
Level 3C M M ®
Level 4 C M M ®
Level 3 C M M I®
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 17
LM Maritime Systems & Sensors Radar SystemsProcess Improvement Credentials
SW-CMM® level 5 (CBA IPISM) in Dec. 1999
Systems Engineering Capability Model (EIA/IS 731) level 3+ (CAM) in Dec. 1999
Focus on integrated process improvement including hardware began in 2000
Transition to CMMI® began in 2000 CMMI®-SE/SW/IPPD target profile 4
(CAM) in Nov. 2002
® SW-CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 18
LM Maritime Systems & Sensors Radar Systems Functional Excellence Objectives - 2003
LM 21 business excellence Enhance program performance Improve productivity Continue to manage the business with quantitative data Improve quality of the product and process Provide a supportive infrastructure for process
improvement Demonstrate continued process maturity leadership Manage technology innovations to support program
performance Document processes and procedures core to our success Develop and train employees for current and future
assignments
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 19
LM Maritime Systems & Sensors Radar SystemsSoftware Productivity and Quality
Lockheed Martin: NE&SS-Syracuse - Software Productivity & Quality Performance History
0%
100%
200%
300%
400%
500%
600%
700%
800%
1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003
SoftwareProductivityGrowth (%)
-100%
-90%
-80%
-70%
-60%
-50%
-40%
-30%
-20%
-10%
0%
Software Error RateReduction (%)
Software Productivity and Quality Performance Application of Best Practices and Investment Has Resulted in Significant Improvements in Quality and Cost. As error rates declined,
productivity increased by 80+%.
CM
M
L4
CM
M
L5
CM
M
L2
CM
M
L3
Error Rate
Productivity Growth
CM
Mi
L4
L
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 20
LM Maritime Systems & Sensors – Undersea Systems Process Credentials
Systems & Software Engineering* Software CMM® Level 4 (CBA IPISM) - June 1995 Software CMM® Level 5 (CBA IPISM ) - February 1999 CMMI® & EIA-731 Level 3 (CAM) - October 2001 CMMI® Level 5 (CAM) - October 2002
Quality Management ISO 9001:1994 - September 1995 AS9000 - November 1997 Defense Contract Management Agency (DCMA) ISO 9001
Qualified - December 1997 ISO 9001:2000 - December 2000 AS9100A – December 2002
Certificate No.
10061Certificate No.
7092Certificate No.
9227
Certificate No.
E00015Certificate No.
EN10002
*Assessed programs comprise over 80% of the Undersea Systems development programs, and all parts of the development cycle.
*Assessed programs comprise over 80% of the Undersea Systems development programs, and all parts of the development cycle.
SM CBA IPI is a service mark of Carnegie Mellon University
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 21
LM Maritime Systems & Sensors – Undersea SystemsProcess Chronology
1970s Top-down Structured Programming Design & Code Inspections 1980-2 Functional Decomposition SW Engineering Workshop Advanced Design Workshop 1983 SW Management Workshop 1984 Ada Workshop 1985 Requirements Inspections 1986 FSC Practices & Measurements 1988 SW Technology Steering Group Organizational Operating Procedures
1990 SW Engineering Process Group Formed First SW-CMM® Assessment (Level 3) Formal Estimation Procedures
1991Market Driven QualityReuse Focus 1992 Defect Prevention Process
1993Integrated TeamsStandard Development EnvironmentIntegrated Process Group 1994 Automated Metrics (MAX) Process Coordination Group (PCG) 1995-6 Integrated Process Library ISO 9001 Registration Software CMM® Level 4) 1997-9 ISO 14001 Registration, AS9000 and DCMA ISO 9001 qualification Software CMM® Level 5
2000-2 ISO 9001: 2000
Certification EIA-731 Level 3 CMMI® SE/SW/IPPD/SS
Level 5
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 22
1 1 / 1 9 / 0 3© C o p y r i g h t L o c k h e e d M a r t i n C o r p o r a t i o n 2 0 0 3
J o a n W e s z k a - 2 2
L M M a r i t i m e S y s t e m s & S e n s o r s – U n d e r s e a S y s t e m sS o f t w a r e P r o d u c t Q u a l i t y ( D e f e c t R a t e )
0
5 0 0
1 0 0 0
1 5 0 0
2 0 0 0
2 5 0 0
3 0 0 0
1 9 8 0 1 9 8 4 1 9 8 8 1 9 9 2 1 9 9 6 2 0 0 0
Maj
or D
efec
ts P
er M
illio
n D
eliv
ered
Sou
rce
Sta
tem
ents
C M M ®
L e v e l 3
C M M ®
L e v e l 4
C M M ®
L e v e l 5
C M M I ®
L e v e l 5
O u r q u a l i t y im p r o ve d a s o u r m a t u r i t y i n c r e a s e d .
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 23
1 1 / 1 9 / 0 3© C o p y r i g h t L o c k h e e d M a r t i n C o r p o r a t i o n 2 0 0 3
J o a n W e s z k a - 2 3
L M M a r i t i m e S y s t e m s & S e n s o r s – U n d e r s e a S y s t e m sP r o c e s s I m p r o v e m e n t R e t u r n - o n - I n v e s t m e n t S u m m a r y
N o t e : O t h e r i n i t i a t i v e s u n d e r w a y d u r i n g t h i s p e r i o d i n c l u d e
• I S O 9 0 0 1 r e g i s t r a t i o n , f o l l o w e d b y A S 9 0 0 0• I n t e g r a t e d T e a m i n g , a n d c r e a t i o n o f a n I n t e g r a t e d P r o c e s s L i b r a r y• I n t e g r a t i o n o f S y s t e m s E n g i n e e r i n g a n d S W E n g i n e e r i n g
S W - C M M ® L e v e l 3
S W - C M M ® L e v e l 4
S W - C M M ® L e v e l 5
C M M I ® L e v e l 5
I m p r o v e m e n t M e t r i c 1 9 9 0 1 9 9 5 1 9 9 9 2 0 0 2Q u a l i t y D e f e c t s / M D S S 6 0 0 3 0 0 1 5 0 5 1P r o d u c t i v i t y E S S / L a b o r M o n t h 2 2 0 2 8 0 3 4 0 3 7 9C o s t & S c h e d u l e + - V a r i a n c e 1 5 % 1 0 % 8 % 8 %R e w o r k e x p r e s s e d a s a %
o f i n d u s t r y a v g 6 % 3 % 2 % 2 %R e u s e P e r c e n t 6 8 % 7 5 % 8 2 % 8 2 %
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 24
0.5373
1.7010
0.4617
2.7931
4.4119
2.5100
0.00
1.00
2.00
3.00
4.00
5.00
6.00
HDHD DDDD CDCD UTUT CTCT SWITSWIT T&IT&I ATAT Life-cycle phasesMaj
or
Def
ects
/ K
-Eq
uiv
alen
t S
ou
rce
Sta
tem
ents
In this case, representing an actual program, Latent Defect Rate = 0.51 Defects / Thousand Equivalent Source Statements
We use defect data from formal inspections to project product quality.
We take corrective actionsif projections show a deviation from the goals.
Latent Defect Projection Model(best fit to Exponential, Inflected-S, Rayleigh)
We produce high quality products by monitoring the quality level throughout the program’s performance.
LM Maritime Systems & Sensors – Undersea SystemsProduct Quality
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 25
11/19/03© Copyright Lockheed Martin Corporation 2003
Joan Weszka -25
SigmaDefects/MSS1 690,000.02 308,537.03 66,807.04 6,210.05 233.06 3.4
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
1.00 2.00 3.00 4.00 5.00 6.00
Sigma Number
Def/M
illion P
arts
5 Sigma
Our quality rate is 20 times better than the average industry rates.
LM Maritime Systems & Sensors –Undersea Systems Software Quality
MS = Million Source Statements
Product Quality Level is in Five Sigma Range
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 26
1 1 /1 9 /0 3© C op yr ig h t L o ck h e ed M a rtin C orp o ra tion 2 0 0 3
J oa n W e s zk a - 2 6
Product Productivity Percent ImprovementDelivered Source Lines of Code per Labor Month
All Languages
0%
50%
100%
150%
200%
250%
300%
350%
400%
1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
L e v e l 1
L e v e l 2
L e v e l 3
L e v e l 4 ,5
L M M a ritim e S y ste m s & S e n so rs – U n d e rse a S y ste m s S o ftw a re P ro d u c tiv ity v s . S W -C M M ® M a tu rity L e v e l
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 27
Reference: “A Correlational Study of the CMM® and Software Development Performance”Lawlis, Flowe & Thordahl, CROSSTALK, September 1995
SCATTER PLOTS BETWEEN “1” AND “3” RATINGS ARE INDUSTRY SAMPLES.
LOCKHEED MARTIN DATA HAS BEEN ADDED IN RED.
LM Maritime Systems & Sensors – Undersea SystemsCost & Schedule Estimates vs. Actuals at Maturity Level 5
DATA FROM 8 PROGRAMS IN 2Q 2003
Cost Performance Index (CPI) & Schedule Performance Index (SPI)
0
1
2
3
1 2 3 4,50
1
2
3
1 2 3 4,5
+
++++
+
+
++
++
+++
+++=+++= =
==++
+++
++++++
++++
+ ==
====
CPI
SPI
CMM® Rating
+ +++
CMM® Rating
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 28
Summary
At Lockheed Martin, benefits derived during SW-CMM® implementation continue to be realized as CMMI® maturity evolves
Allocating benefits to their sources is difficult when implementing multiple models/standards
SW-CMM® and CMMI® are viewed as significant (but not sole) contributors to process improvement benefits to date
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 29
Contact Information
Joan WeszkaLockheed Martin Mission SystemsSystems & Software Resource Center(301) 240-7013 / Fax [email protected]
11/19/03 © Copyright Lockheed Martin Corporation 2003
Joan Weszka - 30
Acronyms ARC – Appraisal Requirements for CMMI®
CAM - Continuous Appraisal Method CMMI® - Capability Maturity Model Integration ESLOC or ESS – Equivalent SLOC/SS; a normalized value derived from
new development, plus SLOC/SS that are modified, retained, ported, etc. IPPD - Integrated Product and Process Development LM - Lockheed Martin LM-IEP - Lockheed Martin Integrated Engineering Process OSP – Organizational Standard Process PA - Process Area SLOC – Source Line of Code SS – Source Statement (sometimes called a “Logical SLOC”)