Joint Confidence Level for Small Projects Yunjin, Kim, Alberto Ortega, and Yolanda Cuevas Jet Propulsion Laboratory California Institute of Technology Presented at the PM Challenge 2011Conference, February 9-10, 2011, Long Beach, California
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Joint Confidence Level for Small Projects
Yunjin, Kim, Alberto Ortega, andYolanda Cuevas
Jet Propulsion Laboratory
California Institute of Technology
Presented at the PM Challenge 2011Conference, February 9-10, 2011, Long Beach, California
2
Agenda
• Fundamentals of Joint Confidence Level (JCL)– Cost-schedule relationship– Importance of the probability density function for schedule activities– Simple examples
• NuSTAR example– NuSTAR JCL process– NuSTAR JCL results– Lessons learned
• Use of JCL for subsystems• Conclusions
02/09/2011
3
What is Joint Confidence Level (JCL)?
• Official definition of JCL (from JCL FAQ from NASA HQ)– The probability that cost will be equal or less than the targeted cost and
schedule will be equal or less than the targeted schedule date.– A process and product that helps inform management the likelihood of
a projects’ programmatic success.– A process that combines a projects’ cost, schedule, and risk into a
complete picture.
• NASA policy for JCL (NPD 1000.5)– Joint cost and schedule confidence levels are to be developed and
maintained for the life cycle cost and schedule associated with the initial lifecycle baselines (such as project baselines at KDP-C).
02/09/2011
4
Joint Confidence Level (JCL) Process Used for NuSTAR Mission
• Inputs– Resource loaded schedule
• Summary project schedule• Burn rate for each cost element
– Probability density functions for schedule activities– Probability density functions for cost elements
• Monte Carlo simulation to produce a two dimensional (cost and schedule) probability density function for cost/schedule success
• Outputs
02/09/2011
( , ) where cost, schedulecsP c s c s
0
0
Probability of cost less than Pr( , ) ( , )
Probability of schedule less than Pr( , ) ( , )
R
R
C
R R cs
S
R R cs
C c C s P c s dc
S c s S P c s ds
5
A Simple Example (1)
• Consider a single schedule activity– Duration: 100 working days
• Schedule risk probability density function = uniform probability over [-10 working days, +30 working days]
– Burn rate = $5k/ working day
02/09/2011
400
450
500
550
600
650
700
80 90 100 110 120 130 140
Schedule Duration (in working days)
Cost (in
$k)
70% schedule = 118 working days
70% cost = $590k
100 days & $5k/working days
Scheduleuncertainty
6
A Simple Example (2)• Consider a single schedule activity
– Duration: 100 working days• Schedule risk probability density function = uniform probability
over [-10 working days, +30 working days]– Burn rate = $5k/ working day
• Cost probability density function = uniform probability over [0%, +30%]
02/09/2011
400
450
500
550
600
650
700
750
800
850
900
80 90 100 110 120 130 140
Schedule Duration (in working days)
Cost (in
$k)
70% schedule = 117 working days
70% cost = $676k
100 days & $5k/working days
Costuncertainty
Scheduleuncertainty
7
A Simple Example (3)
• Consider a slightly more complex case shown below.– Total schedule: 220 working days– Burn rate
• Task 1: $15k/day• Task 2: $20k/day• Task 3: $10k/day• Task 4: $5k/day
– For all four schedule activities• Schedule PDF = uniform over [-10 working days, +30 working days] • Cost PDF = uniform over [0%, 30%]
02/09/2011
Task 1
Task 2 (100 working days)
Task 3 (80 working days)
Task 4 (90 working days)
20
40
30
100 working days
8
A Simple Example (4)
• Correlation coefficient– Cost
• Task 1: 0.537, Task 2: 0.755, Task 3: 0.349, Task 4: 0.162– Schedule
• Task 1: 0.766, Task 2: 0.547, Task 3: 0.099, Task 4: 0.130
02/09/2011
4500
5000
5500
6000
6500
7000
7500
8000
150 170 190 210 230 250 270 290
Schedule Duration (in working days)
Cost (in
$k)
70% schedule = 232 working days
70% cost = $6371k
9
PDF Selection for JCL
• It is obvious that the most important information for JCL is the probability density function for each schedule activity.
• Two probability density functions that we considered are– Uniform PDF = h
– Triangle PDF = (h1+ h2)/2
• A truncated Gaussian PDF can also be used based on the central limit theorem.
02/09/2011
MaximumMinimum
MaximumMinimum
10
Determination of Maximum and Minimum Values of PDF
• To specify uniform PDF or triangular PDF, we have to specify both the maximum and the minimum values.
• To be conservative, we can use the baseline schedule and budget to specify the minimum value.
• The maximum value can be specified based on the project risk list.
02/09/2011
1102/09/2011
NuSTAR Mission Overview
Science• NuSTAR will open a new window on the Universe by making maps of the high-energy X-ray sky (6 keV to 79
keV ) that are more than 100 times deeper than from any previous mission• Objective 1: Determine how massive black holes are distributed through the cosmos, and how they influence the
formation of galaxies like our own• Objective 2: Understand how stars explode and forge the elements that compose the Earth• Objective 3: Determine what powers the most extreme active black holes
Salient Features• PI-led (PI: Fiona Harrison, Caltech) SMEX mission• NuSTAR will carry the first high-energy X-ray
focusing telescope• NuSTAR partners include Caltech, JPL, GSFC,
Orbital, ATK, UCB, DTU, KSC, Columbia University and ASI
• JPL managed project• Category 3, Class D (enhanced) mission• Launch readiness date: November 15, 2011
• Launch date: February 3, 2012• Science operations: 2 years
Goddard Space Flight Center
Kennedy Space Flight Center
12
NuSTAR JCL Process (1)
• The NuSTAR JCL was completed in November 2009.• From the project integrated master schedule (about 3500 lines), a
summary schedule (about 162 lines) was developed.– This step is critical to the efficient implementation of JCL.– The schedule includes the actual performance and costs incurred through
August 2009.– The summary schedule must maintain the work flow and the schedule
network accurately.– The summary schedule was reviewed several times to validate the accuracy.
• The cost information was included in MS project based on the burn rates of schedule activities.
• The schedule/cost probability density functions were determined by reviewing the project risk list.– These probability density functions were also reviewed with the system
managers.
02/09/2011
13
NuSTAR JCL Process (2)
• A “penalty” task was created in the schedule in the form of a hammock task to capture the “marching army” costs associated with supporting a launch past its planned date .
• Monte Carlo simulations were performed using @risk add-on tool to Microsoft Project.
02/09/2011
14
NuSTAR JCL Results (1)
02/09/2011
70% confidence for Launch = 11/9/201170% confidence for the LCC = $110,839,400
$90,000,000
$95,000,000
$100,000,000
$105,000,000
$110,000,000
$115,000,000
$120,000,000
$125,000,000
8/5/11 9/24/11 11/13/11 1/2/12 2/21/12
Project Cost
15
NuSTAR JCL Results (2)
02/09/2011
110110
95 105 115 125
25.78% 44.22% 30% 105.2 110.8394
Distribution for NuSTAR JCL Schedule/Cost
Values in Millions
0.000
0.200
0.400
0.600
0.800
1.000
95 105 115 125
$105.2M has a confidence level of 25%.
$110.8 has a confidence level of 70%
16
NuSTAR JCL Results (3)
02/09/2011
0.141
0.305
0.592
0.748
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
Flight Optics Assembly
OSC OBS Integration and Env Testing
Spacecraft Cost Uncertainty
Instrument Cost Uncertainty
Key Cost Drivers
17
NuSTAR JCL Results (4)
02/09/2011
Mean=10/20/2011 Mean=10/20/2011 Mean=10/20/2011
11/9/201111/9/2011
8/5/2011 10/11/2011 12/17/2011 2/22/2012
5% 65% 30%8/17/2011 11/9/2011
Mean=10/20/2011 Mean=10/20/2011
Distribution for Launch (C/D)/Finish
0.000
0.200
0.400
0.600
0.800
1.000
Mean=10/20/2011
8/5/2011 10/11/2011 12/17/2011 2/22/2012
Current launch date of 8/15/11 is slightly less than 5%.
70% confidence of 11/9/11 projects close to a 3 month slip.
18
NuSTAR JCL Results (5)
02/09/2011
0.099
0.237
0.531
0.669
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
Structure Fabrication, Assy and test
Istrument I&T Schedule
Flight Optics Assembly
OSC OBS Integration and Env Testing
Key Schedule Drivers
19
Lessons Learned for Determining Schedule/Cost PDF
• The PDFs used for the NuSTAR JCL were too optimistic for heritage hardware.
• The PDFs did not include typical test failures and engineering mistakes accurately.
• When we derived the PDFs, we should have considered the historical data.
• Overall, the schedule PDFs were optimistic.
02/09/2011
20
JCL Concept Applied to Subsystem Schedule/Cost Assessment (1)
• The JCL concept can be used to estimate the subsystem delivery date and the cost.
• An example is shown below.– This assessment was done in early September 2010.– The acceptance test started in late September 2010.– Schedule probability density functions are shown in the diagram.– The cost probability density function is [0%, 30%] uniform. The burn rate is
$15k/day .
02/09/2011
Acceptance test(10 working days
PDF =[0,10] uniform)
Integration(10 working days
PDF =[0,10] uniform)
Vibration test(8 working days
PDF =[0,3] uniform)
Function check-out & alignment
(15 working daysPDF=[0,10] uniform)
Preparation for shipping(3 working days
PDF =[0,1] uniform)
21
JCL Concept Applied to Subsystem Schedule/Cost Assessment (2)
• Subsystem JCL in September 2010
02/09/2011
800
900
1000
1100
1200
1300
1400
45 50 55 60 65 70 75 80 85
Schedule Duration (in working days)
Cost (in
$k)
70% schedule = 65.8 working days
70% cost = $1140.5k
22
JCL Concept Applied to Subsystem Schedule/Cost Assessment (3)
• The subsystem JCL was repeated in October 2009 based on the acceptance test progress.– The baseline schedule and the schedule probability density functions
have been revised as shown in the diagram.– The cost probability density function is [0%, 20%] uniform. The burn
rate is $15k/day .
02/09/2011
Acceptance test(25 working days
PDF =[0,5] uniform)
Integration(15 working days
PDF =[0,5] uniform)
Vibration test(8 working days
PDF =[0,3] uniform)
Function check-out & alignment
(15 working daysPDF=[0,15] uniform)
Preparation for shipping(3 working days
PDF =[0,1] uniform)
23
JCL Concept Applied to Subsystem Schedule/Cost Assessment (4)
• Subsystem JCL repeated in October 2010 based on the acceptance test progress
02/09/2011
1000
1100
1200
1300
1400
1500
1600
60 65 70 75 80 85 90 95 100
Schedule Duration (in working days)
Cost (in
$k)
70% schedule = 83.5 working days
70% cost = $1340.3k
24
JCL Concept Applied to Subsystem Schedule/Cost Assessment (5)
• Comparison between the September JCL and the October JCL
02/09/2011
800
900
1000
1100
1200
1300
1400
45 50 55 60 65 70 75 80 85
1000
1100
1200
1300
1400
1500
1600
60 65 70 75 80 85 90 95 100
Schedule Duration (in working days)
Cost (in
$k)
Schedule Duration (in working days)
Cost (in
$k)
October
70% schedule = 83.5 working days70% cost = $1340.3k
September
70% schedule = 65.8 working days70% cost = $1140.5k
25
Conclusions
• The NuSTAR JCL demonstrated that JCL can be performed efficiently using a summary schedule derived from the project integrated master schedule.
• The most important step in JCL is driving the probability density function for each schedule activity based on the project risk list.– In addition to the project risk list, historical data should be
considered if available.
• The NuSTAR JCL accurately predicted key schedule/cost drivers.
• The JCL concept can be used to estimate the subsystem delivery date and cost.
02/09/2011