© Copyright 2011. Richard W. Selby. All rights reserved. 0 Rick Selby Director of Software Products Northrop Grumman Aerospace Systems 310-813-5570, [email protected].

© Copyright 2011. Richard W. Selby. All rights reserved.1

Rick Selby

Director of Software ProductsNorthrop Grumman Aerospace Systems310-813-5570, [email protected]

Adjunct Professor of Computer ScienceUniversity of Southern California

Lessons Learned for Development and Management of Large-Scale

Software Systems


Embedded software for Advanced robotic

spacecraft platforms High-bandwidth satellite

payloads High-power laser

systems Emphasis on both system

management and payload software

Reusable, reconfigurable software architectures and components

Languages: O-O to C to assembly

CMMI Level 5 for Software in February 2004; ISO/AS9100; Six Sigma

High-reliability, long-life, real-time embedded software systems

Organizational Charter Focuses on Embedded Software Products

Prometheus / JIMO

Software Peer ReviewSoftware Development Lab

Restricted

JWST NPOESS EOS Aqua/Aura Chandra

Airborne Laser

Software Analysis

Software Process Flow for Each Build, with 3-15 Builds per Program

GeoLITE AEHF MTHEL


Lessons Learned for Development and Management of Large-Scale Software Systems Early planning

People are the largest lever Engage your stakeholders and set expectations Embrace change because change is value

Lifecycle and architecture strategy Prioritize features and align resources for high-payoff Develop products incrementally Iteration facilitates efficient learning Reuse drives favorable effort and quality economics

Execution Organize to enable parallel activities Invest resources in high return-on-investment activities Automate testing for early and frequent defect detection Create schedule margin by delivering early

Decision making Measurement enables visibility Modeling and estimation improve decision making Apply risk management to mitigate risks early


Lessons Learned for Development and Management of Large-Scale Software Systems

Early planning

Lifecycle and architecture strategy

Execution

Decision making


People are the Largest Lever

Barry Boehm’s comparisons of actual productivity rates across projects substantiated the COCOMO productivity multiplier factors of 4.18 due to personnel/team capability, etc.

Source: B. Boehm, “Improving Software Productivity,” IEEE Computer, Vol. 20, Issue 9, September 1987, pp. 43-57.


Engage Your Stakeholders and Set Expectations

Source: B. Boehm, “Critical Success Factors for Schedule Estimation and Improvement,” 26 th International Forum on Systems, Software, and COCOMO Cost Modeling, November 2, 2011.


Embrace Change because Change is Value

Criteria for high adoption rate for innovations: Relative advantage – The innovation is technically superior

(in terms of cost, functionality, “image”, etc.) than the technology it supersedes.

Compatibility – The innovation is compatible with existing values, skills, and work practices of potential adopters.

Lack of complexity – The innovation is not relatively difficult to understand and use.

Trialability – The innovation can be experimented with on a trial basis without undue effort and expense; it can be implemented incrementally and still provide a net positive benefit.

Observability – The results and benefits of the innovation’s use can be easily observed and communicated to others.

Source: E. M. Rogers, Diffusion of Innovations, Free Press, New York, 1983.



Early planning


Execution

Decision making


Prioritize Features and Align Resources for High-Payoff Synchronize-and-stabilize lifecycle has planning,

development, and stabilization phases Planning phase

Vision statement – Product and program management use extensive customer input to identify and prioritize product features

Specification document – Based on vision statement, program management and development group define feature functionality, architectural issues, and component interdependencies

Schedule and feature team formation – Based on specification document, program management coordinates schedule and arranges feature teams that each contain approximately 1 program manager, 3-8 developers, and 3-8 testers (who work in parallel 1:1 with developers)

Development phase Program managers coordinate evolution of specification. Developers design, code, and debug. Testers

pair up with developers for continuous testing. Subproject I – First 1/3 of features: Most critical features and shared components. Subproject II – Second 1/3 of features. Subproject III – Final 1/3 of features: Least critical features.

Stabilization phase Program managers coordinate OEMs and ISVs and monitor customer feedback. Developers perform

final debugging and code stabilization. Testers recreate and isolate errors. Internal testing – Thorough testing of complete product within the company. External testing – Thorough testing of complete product outside the company by “beta” sites such as

OEMs, ISVs, and end-users. Release preparation – Prepare final release of “golden master” version and documentation for

manufacturing.


Develop Products Incrementally

Documents and Intermediate Activities

Milestone I release

Milestone II release

Project plan approval

Milestone 0

Schedule complete

Milestone III release

Code complete

Zero bug release

Release to manufacturing

Visual freeze

Pla

nn

ing

3-1

2 m

on

ths

Sta

bil

iza

tion

3-8

month

s

Feature complete

Vision statement

Specification document

Prototypes

Testing strategySchedule

Project review

Implementation plan

Specification review

Postmortem document

Internal testing

Beta testingBuffer time

Buffer time

Major Reviews

MilestonesTimeline

Su

bp

roje

ct

IIISu

bp

roje

ct

II Su

bp

roje

ct

I

Design feasibility studies

OptimizationsTesting and debugging

OptimizationsTesting and debugging

(Ship date)

6-1

2 m

on

ths

Develo

pm

en

t• Integration • Testing and debugging

2-5 weeks

• Buffer time2-5 weeks

6-10 weeks • Code and optimizations • Testing and debugging • Feature stabilization

Development Subproject 2-4 months

(1/3 of all features)

Phases

Dev

elop

men

t

6-16

mon

ths

Synchronize-and-stabilize lifecycle timeline and milestones enable frequent incremental deliveries


Iteration Facilitates Efficient Learning

Figure 4.3-4. JIMO Incremental Software BuildsWe provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05

Data Server Unit (DSU) Builds

Science Computer Unit (SCU) Builds

Flight Computer Unit (FCU) Builds

Note: Science Computer builds for common software only (no instrument software included)

FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13

Prelim Exec and C&DH Software


FCU2

FCU3

FCU4

FCU5

FCU6

FCU7

Final Exec and C&DH Software

Science Computer Interface

Reactor Controller Interface

AACS (includes autonomous navigation)

Thermal and Power Control

Configuration and Fault Protection

SCU1

Final Exec and C&DH SoftwareSCU2

DSU1

DSU3


DSU2 Final Exec and C&DH Software

Data Server Unique Software

Ground Analysis Software (GAS) Computer Builds

GAS1Preliminary Ground Analysis Software

GAS2Final Ground Analysis Software

A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154

JPL/NGC, Prelim. Hardware/Software IntegrationJPL/NGC, Final Hardware/Software IntegrationJPL, Mission Module Integration

JPL, Prelim. Hardware/Software IntegrationJPL, Final Hardware/Software Integration

NR, Reactor Controller IntegrationNGC, AACS Validation on SMTBNGC, TCS/EPS Validation on SSTB

NGC, Fault Protection S/WValidation on SSTB

NGC, Prelim. Hardware/Software Integration

NGC, Final Hardware/Software IntegrationNGC, HCR Integration on SMTBJPL, Prelim. Integration into Ground System

JPL, Final Integration into Ground System

1 Requirements2 Preliminary Design3 Detailed Design4 Code and Unit Test/Software

Integration5 Verification and Validation

Legend: N is defined as follows:541 32=

2 53 41=

542 31=

PrototypeActivity

NGC

N Performer of Activity NJPL

Role/activity shared by JPL and NGC

Design Agent

P

Figure 4.3-4. JIMO Incremental Software BuildsWe provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1


DSU1

DSU3







A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154











2 53 41=

542 31=

PrototypeActivity

NGC



Design Agent

P

We provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1


DSU1

DSU3







A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154











2 53 41=

542 31=

PrototypeActivity

NGC



Design Agent

P

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1

Final Exec and C&DH SoftwareSCU2 Final Exec and C&DH SoftwareSCU2

DSU1

DSU3


DSU2 Final Exec and C&DH SoftwareDSU2 Final Exec and C&DH Software





A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154











2 53 41=

542 31=

PrototypeActivity

NGC



Design Agent

P

Power Power

Incremental software builds deliver early capabilities and accelerate integration and test

Iteration helps refine problem statements, create potential solutions, and elicit feedback


Reuse Drives Favorable Effort and Quality Economics

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Module origin

Fau

lts

per

mo

du

le (

ave.

)

Mean 1.28 1.18 0.58 0.02 0.85

Std. dev. 2.88 1.81 1.20 0.17 2.29

New development

Major revision Slight revision Complete reuse All

Analyses of component-based software reuse shows favorable trends for decreasing faults

Data from 25 NASA systems Overall difference is statistically significant ( < .0001). Number of

components (or modules) in each category is: 1629, 205, 300, 820, and 2954, respectively.



Early planning


Execution

Decision making

© Copyright 2011. Richard W. Selby. All rights reserved.

Organize to Enable Parallel Activities

14 16

Legend

Auto-matable activity

Source: “Flowchart,” www.wikipedia.org, May 2010.

=

Track cycletime between activities


Total Ave. Ave. / EKSLOC

Reviews 257 29 N/A

Prevention cycles

15 1.7 N/A

Defects 2621 291 7.3

Defects per review

N/A 15 N/A

Defects out-of-phase

N/A 8.1% 1.3

Invest Resources in High Return-on-Investment Activities

Return-on-investment (ROI) for software peer reviews ranges from 9:1 to 3800:1 per project

Return-on-investment (ROI) = Net cost avoidance divided by non-recurring cost 2621 defects, 257 reviews, 9 systems, 1.5 years High ROI drivers

Mature and effective processes already in place Significant new scope under development Early lifecycle peer reviews (e.g., requirements phase) Four of the five programs with >80% requirements and design defects had

relatively higher ROI

9

3326

37823867

292143

16

3638

1877

0

100

200

300

400

500

600

700

800

# D

efec

ts

0

500

1000

1500

2000

2500

3000

3500

4000

4500

RO

I

# Defects ROI

Projects


Software Defect Injection Phase

0.0%2.3%

49.1%

9.0%12.1% 11.7%

2.5% 1.9%

10.6%

0.7% 0.1% 0.0%0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

System Development Phase

De

fec

ts (

%)

Automate Testing for Early and Frequent Defect Detection

Distribution of software defect injection phases based on using peer reviews across 12 system development phases

3418 defects, 731 peer reviews, 14 systems, 2.67 years 49% of defects injected during requirements phase


Create Schedule Margin by Delivering Early

Source: “The Network Diagram and Critical Path,” www.slideshare.net, May 2010.

LegendCritical path

Critical path defines the path through the network containing activities with zero slack



Early planning


Execution

Decision making


Measurement Enables Visibility

Requirements

0

5

10

15

20

25

30

Jun-04 Jul-04 Aug-04 Sep-04

Plan Actual

LCL UCL

Reuse

0%10%20%30%40%50%60%70%

Proposal SSR PDR CDR

Plan Actual

LCL UCL

Technology Infusion

01234567


Plan Actual

LCL UCL

Progress

0

50

100

150

200

250


Plan Actual

LCL UCL

Cycletime

0

5

10

15

20

25


Plan Actual

LCL UCL

Deliveries

0

5

10

15

20

25


Plan Actual

LCL UCL

Pre-Delivery Defects

0

50

100

150

200

250


Plan Actual

LCL UCL

Post-Delivery Defects

02468

101214


Plan Actual

LCL UCL

Contact Contact

ContactContactContact

ContactData Data

DataDataData

DataOutliers Outliers

OutliersOutliersOutliers

Outliers

Help Help

HelpHelpHelpContactDataOutliers

Help

HelpContactDataOutliers

Help

Up Down Level 1 AllUp Down Level 1 All

Up Down Level 1 All

Up Down Level 1 All

Up Down Level 1 All Up Down Level 1 All Up Down Level 1 All

Up Down Level 1 All

DASHBOARD Metrics: Organization: Project: Manager: Contact: Status: Development ABC Products Division XYZ System FirstName LastName [email protected] x12345 10/1/2004

Interactive metric dashboards provide framework for visibility, flexibility, integration, and automation

Interactive metric dashboards incorporate a variety of information and features to help developers and managers characterize progress, identify outliers, compare alternatives, evaluate risks, and predict outcomes


Modeling and Estimation Improve Decision Making

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Completeness ( = 100% less % of False Negatives)

Co

nsi

sten

cy (

= 10

0% le

ss %

of F

alse

P

osi

tives

)

Overall average without optimizations Predicting all to be "+"

Optimizations for Consistency Predicting none to be "+"

Optimizations for Completeness

Target: Identify error-prone (top 25%) and effort-prone (top 25%) components

16 large NASA systems

960 configurations Models use metric-

driven decision trees and networks

Analyses tradeoff prediction consistency versus completeness


WBS: 4.0 Spacecraft IPTRisk Owner: McWhorter, Larry Printed: 14 Dec 2005Risk: CEV-252 - Flight Software Requirements Management

1 : Conduct BM1 2 : Estimate softw are requirements scope (preliminary)3 : Establish softw are control board (preliminary) 4 : Release SDP (w ith spec tree, change process)5 : Complete RTOS lab evaluation; Validate capabilities using sim 6 : Estimate softw are requirements scope (f inal)7 : Implement system development process f low models 8 : Spacecraft/etc users define/val/sim use-cases (I/F, funct, off-nom, ...)9 : Finalize/val/sim IFC1 requirements: Infrastructure SW 10 : Baseline allocation of SW requirements to IFCs w ith grow th/correction11 : Establish softw are control board (f inal) 12 : Conduct Sw RR13 : Finalize/val/sim IFC2 requirements: Inter-module & inter-subsystem I/F 14 : Initial end-to-end architecture model15 : Finalize/val/sim IFC3 requirements: Subsystems major functions 16 : Finalize/val/sim IFC4 requirements: Nominal operations17 : Deliver IFC3: Subsystems major functions 18 : Finalize/val/sim IFC5 requirements: Subsystems off-nominal operations19 : Finalize/val/sim IFC6 requirements: System off-nominal operations 20 : Deliver IFC7: No new capabilities; Only corrections; 1st mission ready

Ris

k S

core

26

25

2423

22

21

20

1918

17

16

15

1413

12

11

10

9

87

6

5

4

32

1

0

1

2

3

4 5 6

7

8

9 10

11

12

13

14

15 16

17

18 19 20

2005 2006 2007 2008 2009 2010

SDR

Moderate

High

Low

Apply Risk Management to Mitigate Risks Early

Last Updated: 03-October-05

CSRR PDR

Last Updated: 14-Dec-05

Events

CDR

TRL4

TRL5 TRL

6

TRL7

WBS: 4.0 Spacecraft IPTRisk Owner: Wood, Doug Printed: 15 Nov 2005Risk: CEV-252 - Flight Software Requirements Management

Planned Risk Level Planned (Solid=Linked, Hollow =Unlinked) Control PointsActual Risk Level Completed Completed

Ris

k S

core

26252423222120191817161514131211109876543210

Spacecraft/etc users define/val/sim use-cases (I/F, funct, off-nom, ...)

Conduct Sw RR

Finalize/val/sim IFC4 requirements: Nominal operations

Deliver IFC3: Subsystems major functions

2005 2006 2007 2008 2009 2010

Exit/Success Criteria:1. BM1 complete; customer concurs with approach2. Software requirements scope estimated (preliminary).3. Software control board established (preliminary); change control process established.4. SDP released. Spec tree defined.5. RTOS lab evaluation completed. Capabilities validated using sim.6. Software requirements scope estimated (final)7. System development process flow models implemented.8. Spacecraft/subsystems/etc. users define use cases (for I/Fs, functions, nominal ops, off-nominal ops, etc.)

completed. Validated using models/sim.

9. Finalize IFC1 requirements: Infrastructure SW completed. Validated requirements using models/sim.10. Baseline allocation of SW requirements to IFCs with growth/correction/deficiency completed.11. Software control board (final) established12. SwRR conducted. NASA customer agrees with software requirements.13. Finalize IFC2 requirements: Inter-module & inter-subsystem I/Fs completed. Validated requirements using

models/sim.14. Initial end-to-end architecture model completed.15. Finalize IFC3 requirements: Subsystems major functions completed. Validated requirements using models/sim.16. Finalize IFC4 requirements: Nominal operations completed. Validated requirements using models/sim.17. Deliver IFC3: Subsystems major functions completed. Validated capabilities using sim.18. Finalize IFC5 requirements: Subsystems off-nominal operations completed. Validated requirements using

models/sim.19. Finalize IFC5 requirements: Subsystems off-nominal operations completed. Validated requirements using

models/sim.20. Deliver IFC7: No new capabilities; Only system I&T corrections completed. SW complete for 1st mission.

Projects define risk mitigation “burn down” charts with specific tasks and exit criteria


Lessons Learned for Development and Management of Large-Scale Software Systems Early planning

People are the largest lever Engage your stakeholders and set expectations Embrace change because change is value

Lifecycle and architecture strategy Prioritize features and align resources for high-payoff Develop products incrementally Iteration facilitates efficient learning Reuse drives favorable effort and quality economics

Execution Organize to enable parallel activities Invest resources in high return-on-investment activities Automate testing for early and frequent defect detection Create schedule margin by delivering early

Decision making Measurement enables visibility Modeling and estimation improve decision making Apply risk management to mitigate risks early

© Copyright 2011. Richard W. Selby. All rights reserved. 0 Rick Selby Director of Software Products Northrop Grumman Aerospace Systems 310-813-5570, [email protected].

Documents

software productivity

payload software reusable

early decision

laser systems emphasis

system management

risk management

high return

decision making