Armstrong

Orion CDR Process:A Streamlined Approach

Charles ArmstrongNational Aeronautics and Space AdministrationOrion VIO SE&I Plans & Processes [email protected]

Tony ByersLockheed Martin Space Systems CompanySystems Engineering – Program Integration [email protected]

NASA PM Challenge 2011

mailto:[email protected]

mailto:[email protected]

Project OrionProject Orion

Agenda

2

• Introduction

• Preliminary Design Review

• CDR Streamlining Focus

• DRDs / Documentation

• RIDs / Actions

• Reviews

• Critical Design Review Process

• Management Challenges

• Keys to Success

• Objectives & Criteria

• Process

• Closing Remarks


3

Introduction

• Presentation Focus:• Historically, NASA Human Spaceflight major milestone reviews have been

conducted by allowing participation by a large number of agency personnel. • Review a wide range of materials • Write a large number of candidate issues in the form of RIDs

• Very expensive • Significant infrastructure and logistics • Large effort to review, disposition, and close

• Using the lessons learned from the Orion PDR, the Orion Project has pursued a more streamlined approach in conducting its CDR.

• Expands the use of subsystem design reviews• More fully utilizes technical reviews of design products / documentation

• Uses highly qualified personnel• Changes the way issues are identified and vetted• Limits CDR participation to individuals with an identified ability to

make a value-added contribution

If successful, this methodology will reduce the cost of conducting CDR to about one-third that of PDR and set the new standard for the conduct of major milestone reviews within the agency.


4

Current Context

• August 2009 – PDR successfully concluded

• Block 1 (ISS) and Block 2 (Lunar) configurations

• February 2010 – President’s 2011 Budget request removed CxP

• Included the cancellation of Ares & Orion

• 2010 Appropriations Bill prevented CxP cancellation during FY10

• February 2010 – Completed optimization to align with FY10 funding profile

• Refocused vehicle configuration on Block 0 (Orion 1 & Orion 2)

• Set 2013 target launch date for Orion 1

• August 2010 – Senate passes NASA Authorization Bill (includes MPCV)

• September 2010 – Congress passes NASA Authorization Bill (includes MPCV)

• September 2010 – Completed optimization needed for 2013 OFT-1

• October 2010 – President signs NASA Authorization Act (includes MPCV)

Throughout 2010, the Orion Project continued to stay focused on design and development. The integration team quickly adapted to the changing landscape and routinely revisited CDR expectations.


5

Preliminary Design Review


6

PDR Process Refresher

• PDR Process– The PDR process included a series reviews which provide a solid foundation for

determining if the current design is at PDR level of maturity. • 184 Technical Reviews• 104 Peer Reviews• 10 Combined Tech/Peer Reviews• 18 Subsystem Design Reviews (59 days)

– Each SSDR assessed against a common set of criteria• 1 System and Module Review (7 days)• 8 RID Teams responsible for RID Screening & Disposition (8 days)

– 56 days of total DRD review• 1 PDR Pre-Board (7 days)

– Each module assessed against a common set of criteria– ITAs assessment integrated

• 1 PDR Board (1 day)

• PDR Data Package– 170 DRDs delivered and reviewed– 53 “Drafts Available” DRDs provided to NASA for comment

500+ Review Participants

1500+ Review Participants

Approximately 100,000 pages of design, process, and plan documentation reviewed.Included requirements, interface definition, verification details, drawings, general plans, etc.


June July August Sep Oct

DRD Update & Redeliver(Type 1 + Final Type 2)

DRD Waterfall / Early Review(Tech Reviews & Peer Reviews)

Orion PDR ProcessProject Orion

SMR

PDR KO7/13 7/21

8/10 NASA ScreeningTeam Concludes RID Screening

All RIDsDispositioned8/20

4 wk DRD Review(RID Creation &

Sponsor)

System &Module

Presentations

Decision to Proceed to DAC-4

RIDs Captured Against Design

DRDs Delivered7/7 PDR -45

NASAScreening

TeamNASA Lvl III Review & Filter RIDs

(Work disconnects with external stakeholders)(LM SEIT led SME team available to answer

questions)

RID Tool OpenTBD

RID Tool Close (all RIDs Sponsored)7/31

DRDsRedeliveredNLT 2/1/10Pre

Board5 1/2 Day Pre-Board to work issues in parallel

with completion of RID Disposition

Orion PDR

All RIDs receive product owner or lead recommendedDisposition8/17

Starts 8/3

Starts8/12

Product OwnerRecommendDisposition

Starts 8/17

8/4 BoardDecision Gate

8/10 ForwardDaily Board Decision Gates

S

IntegTeam

PIT Lead with Module/IPT SupportStarts 8/27

PIT War Room

8/31-9/1 PDR Board

Review TeamDisposition

Dry-Runs

Subsystem DesignReviews (SSDRs)

2-WkSSDRs

SMR Prep

7


8

PDR Key Elements

• Technical Reviews (Tabletops)– Provided participants an opportunity to identify and resolve design, integration, document and/or

process issues prior to formal product/document review– Working meetings that provide participants the opportunity to work design issues real time

– Provided a joint NASA/LM forum for horizontal integration – Technical Reviews can be iterative, as required

• Peer Reviews– Focused primarily on detecting and eliminating defects/findings in critical work products or DRDs.

– More formal than a Technical Review – Facilitated by a trained moderator– Subject matter experts discuss issues identified within the work products or DRDs and strive

for concurrence on the identified defects and recommended solutions – Includes a formal closure of all actions/findings

– Provided a joint NASA/LM forum for horizontal integration


Subsystem Design Reviews

• Subsystem Design Reviews (SSDRs) – Provided the Orion stakeholders the opportunity to review each subsystem’s preliminary design – Provided stakeholders the opportunity to review each subsystem’s preliminary design, including

GFE and ADP as appropriate. – Products reviewed at SSDRs provided objective evidence that the preliminary subsystem design

meets requirements with acceptable risk – Additional Objectives included:

– Reviewing the preliminary design as documented in the technical design products for each subsystem

– Participating in a face-to-face open forum, raising issues/concerns, answering questions or clarifying design details

– Providing forward plans for significant issues from preceding reviews– Demonstrating horizontal integration with other subsystems– Reviewing the design against functional and performance specifications and ground and

mission operability characteristics – Evaluating the technical adequacy and maturity of design– Ensuring correct design options have been selected– Ensuring interfaces have been identified and defined– Reviewing the verification approach– Reviewing associated risks and risk management strategies– Evaluating the Subsystem readiness to proceed to PDR

9


System & Module Review (SMR)

10

• System & Module Review (SMR)– Provided an overview of the DAC-3 close-out (606G) vehicle configuration and system architecture– Provided an opportunity to ensure all 1500+ PDR reviewers developed a common understanding of

the following:– System architecture and PDR technical baseline– Ground systems and operations requirements and plans– Flight operations by mission phase

– Provided a status on major open system, module, and subsystem issues– Provided clarification of what products were officially delivered for PDR data package review– Provided the RID criteria and ground rules– Served as the kick-off for the official DRD review and acceptance process

• RID Review Process culminating in PDR Board– DRDs waterfalled over 3 months leading up to the start of the official DRD review– Provided NASA four weeks of dedicated DRD review– Eight joint NASA/LM review teams reviewed, integrated, and dispositioned 3000+ initial RIDs

– Teams operated in parallel for 8 days resulting in 1061 RIDs (after consolidation of like issues)

– Conducted a 7 day Pre-board chaired by the NASA Vehicle Integration Manager– Reviewed the final criteria assessment and worked to disposition remaining RIDs

– Conducted a 1 day Board chaired by the NASA Orion Project Manager


PDR RID Process Flow

11


12

Orion PDR Data Package Review

• 170 DRDs Delivered & Reviewed– 100,000+ pages of design, process,

and plan documentation

– Requirements, Interface Definition, Drawings, Verification Details, General Plans, etc.

• 1470 RID Initiators (DRD Reviewers)– 520 unique initiators wrote a RID

(35.4%)

– 399 unique initiators with an approved RID (76.7%)

• 3625 Sponsored RIDs– 2049 were approved (56.5%)

Orion PDR Sponsored RIDs by Discipline

CM, 452

LAS, 267

Mech, 115

SM, 126

ECLSS, 124

EPS, 103

Propulsion, 76

Pyrotechnics, 88

ICDs, 107

Av: C&DH, 30

Av: Comm & Track, 22

Con Ops, 204

LRS, 57

EGSE, 43

Vehicle, 142

Wiring, 40

Other, 272

Av & Sw , 605

Crew Systems, 88

Hazard Analysis, 45

T&V, 204

GN&C, 65

ICCS, 120

A key CDR streamlining goal is to improve the overall quality of RIDs while reducing the number of RIDs against known, out of scope or duplicate issues.


13

PDR Lessons Learned

• PDR Lessons Learned were solicited from all participants– Orion team and all reviewers

• Significant Lessons Learned Include:– SSDRs were very successful; however, several could have used more time and

fewer could have been conducted in parallel– Modify SMR to a forum similar to SSDRs– Merge the Screening and Review Team steps in the RID Process– More time was needed between Review Teams and the Board– Initiators rewrote Tech Review actions as RIDs – due to lack of TR feedback– Minimize duplication of Pre-declared RIDs, SSDR actions, and Errata

• 597 RIDs were duplicate to Pre-Declared RIDs during PDR– RIDs/issues were discussed several times during the PDR Process and decisions

on issues were not made effectively– RIDs/issues were not Sponsored effectively - out of scope issues clogged process– Reviewers attending by Webex and telecon were unable to support effectively– Improve how contractual/cost issues are addressed during CDR

Executing a Lessons Learned process made streamlining CDR much easier. All stakeholders quickly aligned to ensure an efficient thorough CDR process was developed.


14

CDR Streamlining Focus

Crew Exploration Vehicle

Incremental DRD Delivery

• The DRDs are divided into four groups• CDR DRDs (Design Data Books, ICDs, Drawings, etc.) ~104 DRDs

• Post-CDR DRDs (Reports, Plans, etc.) ~ 21 DRDs

• Pre-CDR DRDs ~ 73 DRDs

• Software CDR DRDs ~ 125 DRDs

Focus 1: Improve the effectiveness of reviewing and accepting DRDs at CDR

Action Item / RID Process

Project DRD Review & Approval Process

Process TBD

Nearly half of all DRDs originally linked to CDR for delivery will be adjusted to allow delivery based on need (design, procurement, etc.). This provides increased project flexibility in prioritizing work and

increases overall CDR efficiency and review quality.

* Component Specs & Supporting documents

are developed & delivered based on procurement need.

Reduced core set of DRDs to be reviewed and accepted as part of the SSDRs & CDR

DRDs delivered per PIT managed schedule.

Reviewed and accepted through NASA’s normal

C&DM process.

15

Project Orion

16

Pre-CDR DRDs

Tech/PeerReview

InputFeedback

LMCM/DM

PostPortal

DRDReview

ConsolidateComments

CommentDisposition

VICBSummary

Post FinalComments

ContractLetter

1-2 15 5 <1 10 1 5 1 2 3

NASA & LM Tech

POCs

LM Book Owners

LM CM/DM

NASA CM/DM

NASA Review

ers

NASA CM/DM

DRD POC (2), TCM (2), & IPT

(1)

VICB Outbrief

NASA CM/DM& DRD POC

NASACOTR & PIT

CDRStart

CDR + 90Redelivery

DRDDelivery

Draft DRDPosted on Portal

day

sw

ho

Note: Days are in business days.

20 business days (4 weeks)

Pre-CDR DRDs Include


• Con Ops• Ground Sys Reqt’s• Safety Docs• Fault Tree Analysis• Lunar Upgrade Plan

• Workmanship Stds• Model & Sims Plans• Sw Dev Plan• IRDs

Pre-CDR DRD Review & Delivery Flow

• DRD Quantity: 73• Time to Review: 2 Weeks• Review Team: IPTs• Re-delivery: CDR +90 days • Comment/Action Mechanism: Excel or ConCERT

Review & Delivery Details

Project Orion

17

CDR DRDs

Tech/PeerReview

InputFeedback

LMCM/DM

Post onPortal

DRDReview

SSDR Caucus Reclama Loaded as IDE Action

ContractLetter

1-2 15 5 <1 5+ 5 5 1 2 3

NASA & LM Tech

POCs

LM Book Owners

LM CM/DM

NASA CM/DM

NASA Tech POCs

SSDR Co-Chairs & Joint PIT

SSDR Co-Chairs & Tech POCs

MRT & Board,

As Required

LM PIT & MRI

NASACOTR & PIT

CDRBoard

CDR + 90Redelivery

DRDDelivery

Draft DRDPosted on PL

day

sw

ho

CDRStart


CDR DRDs Include


• CAD Models• Drawings• MVP• ICDs• ADD

• Mass Properties• Data & Command Dictionary• Avionics DDB

CDR DRD Review & Delivery Flow

• DRD Quantity: 104• Time to Review: 2 Weeks• Review Team: SSDR Caucus• Re-delivery: CDR +90 days • Comment/Action Mechanism: IDE Action Tool


Project Orion

18

Post-CDR DRDs

Tech/PeerReview

InputFeedback

LMCM/DM

PostPortal

DRDReview

ConsolidateComments

CommentDisposition

VICBSummary

Post FinalComments

ContractLetter

1-2 15 5 <1 10 1 5 1 2 3

NASA & LM Tech

POCs

LM Book Owners

LM CM/DM

NASA CM/DM

NASA Review

ers

NASA CM/DM

DRD POC (2), TCM (2), & IPT

(1)

VICB Outbrief

NASA CM/DM& DRD POC

NASACOTR & PIT

CDRBoard

DRDDelivery

Draft DRDPosted on Portal

day

sw

ho


20 business days (4 weeks)

Post-CDR DRDs Include


• Ops Data Book• Test Procedures

Post-CDR DRD Review & Delivery Flow

• DRD Quantity: 21• Time to Review: 2 Weeks• Review Team: IPTs• Re-delivery: TBD• Comment/Action Mechanism: Excel or ConCERT


Project Orion

19

RID Quality & Quantity

• Increase education of external initiators with Orion CDR Kick-off– Model after LM’s Orion Orientation Training

• Enhance sponsor selection and training

• Increase the quality of Pre-Declared RIDs– No automatic conversion of SSDR Actions not complete by PDR

• Increase training of RID Initiators– Create 1-Page Quick Reference Guide emphasizing Good/Bad RID Criteria

• Re-examine initiator list– 35.4% of Initiators actually submitted a RID (520 of 1470)

– 76.7% of Initiators who submitted a RID had an approved RID (399 of 520)

Focus 2: Improve the quality of RIDs and reduce overall quantity


Action & RID Creation & Disposition

• Tech Reviews were not effective in communicating issue resolution– Resolution: Improved closed-loop process will communicate dispositions of

issues raised at Tech Reviews to avoid repeated Action Items/RIDs during CDR

• RIDs/issues were discussed several times during the PDR Process and decisions on issues were not made effectively– Resolution: All Action Items/RIDs will come through SSDRs, decreasing the

amount of times issues are discussed

– Resolution: Management Review Team created to make decisions on Action Items/RIDs while SSDRs are being held

• RIDs/issues were not Sponsored effectively, causing out of scope issues to clog the PDR Process– Resolution: Improved process of raising all issues at SSDRs will greatly

improve Action Item/RID Sponsoring and increase the efficiency of dispositioning during CDR

Focus 2: Improve the quality of RIDs and reduce overall quantity

20

Project Orion

21

CDR SSDR Action Item/RID Process

Process Details

- Initiators review DRDs prior to SSDR and note issues

- Initiators required to raise issues during the SSDR

- Co-Chair ruling officially “sponsors” the Action/RID

- SSDR Caucus makes initial disposition on Actions/RIDs

- Recommended dispositions posted on the CDR Portal

- Management Review Team (MRT) handles Reclamas, issues with Cost/Schedule/Risk, and “Big Red” Issues

- CDR Board formally approves Action/RID disposition

- Addresses Reclamas

- All Actions/RIDs w/ their dispositions are placed on the CDR Portal for project historical record

- Approved Action/RIDs are input to IDE for LM action tracking

- Updates placed on the CDR Wiki for initiator tracking

PROPOSED FLOW

Focus 3: Improve the efficiency of the Action/RID Process & lifecycle in CDR

Path Legend1. Reject 2. Reclama 3. Accept 4. Accept with Mod 5. Merge 6. Elevate for Discussion 21

Project Orion

22

SSDR Action Item/RID Process Flow

1. Before an Action Item/RID is written, a question must be asked at the SSDR– If not resolved, an Action Item/RID is documented– Question/Answer session is held at the end of the day to give reviewers an

opportunity to ask a question that may have been missed earlier – Additional time is provided at the end of the week for questions/answers

2. Action Items/RIDs is dispositioned in the Caucus

3. Disposition of Action Item/RID is provided to reviewers the following day– If reviewer disagrees with the disposition of the Action/RID, they bring up that

they disagree with the disposition with the Co-Chairs (5-10 minutes given after lunch for people to let Co-Chairs know if the disagree with the disposition)

– If not resolved prior to the caucus, the reviewer is invited to the caucus to discuss issue

4. Once all Action Items/RIDs are dispositioned, the Co-Chairs identify the major design issues for the subsystem– Detailed Forward Plans are developed for the major issues


Project Orion

23

SSDR Action Item/RID Process Flow

5. All major design issues, any reclamas/disagreements, cross-team integration issues, and cost issues (> $500,000 (TBR)) for each subsystem are discussed at the Management Review Team (IPT leads appropriate to SSDR, Chief Engineers, Crew Rep, MOD rep, chaired by Mark Geyer and Cleon Lacefield) the week following the SSDR– Detailed Forward Plans are discussed and are modified/approved for each major

issue– Any reclama or disagreement is resolved in the Management Review Team– Pre-Board topics are identified by the Management Review Team

6. Major issues will be presented at the Pre-Board after the forward work plan has been reviewed and approved by the Management Review Team


Project Orion

24

Disposition Definition

• Issues identified that are outside of the scope of CDR - Normal forward work items will not be accepted• Issues that are not accepted by SSDR/IVDR/MRT/TMR/Board

Design issue that needs to be resolved to ensure an appropriate level of CDR maturity is achieved along with resolution EXACTLY AS WRITTEN BY INTIATOR

Same as accept; however, additional tailoring or editing to the action or resolution is needed

Action Item/RID/Comment Dispositions

RejectReject

AcceptWith Mod

AcceptWith Mod

AcceptAccept

Dispositions will be used in the Caucus and Review Teams


All Approved Action Items/RIDs are expected to be incorporated in the CDR redelivery. This ensures a consistent and well documented technical baseline.


Overall CDR Process

• Enhance Technical Reviews of the design work products with the correct NASA Orion Level 3 participation

– Design Data Books will receive significant review support and serve as the NASA Orion Level 3 chance to review the design at the subsystem level

– DRDs are provided to TR participants 5 to 10 days prior to the review• Review dates are documented in a easily accessible integrated schedule

• Schedule will be deconflicted where possible to allow for Project support

– Minimum required stakeholders will be identified for all DRDs• Each DRD has an assigned NASA POC to ensure stakeholder selection is correct and the

work product receives adequate review

– TRs must be given high priority by each appropriate Orion organization• Appropriate SMEs are expected to support DRD TRs

– Note: Official delivery to NASA occurs after the TRs per the J-2

• Software CDR and Component CDRs will occur separately from the System CDR

– The combination of System CDR and Software CDR will satisfy compliance to NPR 7123

Focus 4: Improve the efficiency and effectiveness of the reviews.

25


SSDRs

• Additional time is given during each SSDR to display/discuss the design in the Interactive Reading Rooms (IRRs)

– SMEs are available in the IRRs to answer/discuss design questions

• General Improvements to Process– Caucus extended into the following week

• Reduce overload on Co-Chairs

– Stacked/Overlapping Schedule• 3-4 SSDRs per week for ~5 weeks

– SSDRs deconflicted so Co-Chairs can participate in review and Caucuses

26

Focus 4: Improve the efficiency and effectiveness of the reviews.

SSDR Chairs never chair a review on

consecutive weeks. Remain focused on an

SSDR until Caucus completion.

Project Management available throughout process to resolve issues quickly.


27

Critical Design Review Process


CDR Management Challenges

28

• Planning Challenges

– Develop and execute a Critical Design Review (PDR) process that ensures NASA’s next generation spacecraft can meet all mission objectives and satisfy requirements with acceptable risk

– Develop a multi-tiered approach that ensures the integrated subsystems, modules and vehicle are reviewed to ensure the design approach is at a sufficient level of maturity to production, assembly, integration, and test.

– Balance thoroughness and schedule. The desire to review every aspect of the design down to the lowest possible level must be balanced against the available time to execute the review as to not delay the development activities unnecessarily.

– Balance inclusion and efficiency. Develop a process that ensures all stakeholders have an adequate time to participate and provide input helping to shape the design, ensure mission success, and ultimately provide a safe crew exploration vehicle.


CDR Management Challenges

29

• Execution Challenges– Communicate the expectations to a diverse 2000+ personnel organization

consisting of NASA, prime contractor (LM), and many support contractors

– Balance the desire to improve the design with the need to efficiently transition to production.

– From the time the CDR baseline is struck, a lot of potential energy develops to revisit the requirements (did we miss something?) and improve the design (could it be better?).

– This energy must be managed to ensure that identified issues and design refinement are necessary to ensure mission success and crew safety and not simply to resolve an “I would have did it differently” comment

– CDR products must align as close as possible to the CDR baseline.

– Monitor and control process execution throughout the duration of the effort


Keys to CDR Success(Operational Excellence)

• Teamwork– Created a joint NASA/LM CDR Program Integration Team (PIT) to ensure common expectations are

communicated across the NASA, LM, and support contractor teams• Key personnel from each Module and IPT are included in team• Planning and C&DM personnel incorporated into team

• Communication– Focus on communicating early and often– Conduct a series of TIMs and Training Sessions with each of the subsystem leads to develop

common expectations and identify issues needing resolution prior to CDR– Institute multiple statuses and forums targeting all stakeholders

• Commitment– NASA and LM each assigned a manager responsible for ensuring CDR success– With the goal of “Get it right the 1st time”, the Orion Management team routinely stresses the

importance of attaining the CDR level of design maturity– Management must routinely communicate the CDR expectations and monitor status weekly

• Accountability– Each Subsystem Lead must take ownership of their respective Subsystem Design Review

preparation and execution– A detailed schedule will be developed to manage the data package review and delivery– Detailed metrics will be developed for tracking all deliverables by System and Subsystem

30


CDR Objectives & Criteria

31

• CDR Objectives– Demonstrate that the maturity of the design is appropriate to support

proceeding with full scale fabrication, assembly, integration, and test.– Determine that the technical effort is on track to complete the flight and

ground system development and mission operations, and to meet mission performance requirements within the Project’s cost and schedule constraints.

– Enable the Project to baseline the component specifications.

• CDR Criteria– NASA plans to authorizing contractor to use their own Command Media

criteria for a successful CDR (Significantly greater detail than NPR 7123.1A)– PIT developing a detailed map of NPR 7123.1A criteria to LM criteria

– NASA/LM IPTs and Subsystems will develop a detailed self assessment against the entry and success criteria

– Subsystem Criteria will be evaluated at the SSDRs– System & Module CDR Criteria will be evaluated first at the Pre-Board and

final at the CDR Board.

PIT will mature CDR Objectives and Criteria, coordinate with all impacted stakeholders, and document in the CxP 72422 Orion CDR Plan.


CxP 72422 Orion CDR Plan

32

• A comprehensive CDR plan will be documented in CxP 72422 and placed under configuration control

• All significant stakeholders will participate in review of document prior to release

• Formalizes the overall CDR process

• Documents the requirements and associated documentation baseline for CDR

• Incorporates NPR 7123.1A criteria into plan (includes any tailoring)

• Serves as the primary method of documenting all agreements regarding method to meet the SSDR and CDR criteria

• Used as a reference throughout the execution to ensure the process is adhered to


Orion CDR Process

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Week 11

CDR Kickoff

CDR DRDs Review

Interactive Reading Room (DRDs, Drawings, Etc.)

SSDRs

SSDR Caucuses

System Analysis Review / T&V / S&MA (1 to 2 days)

Reviews Begin (Interactive Reading Rooms available)

Pre-Board

Board

Integrated Vehicle Design ReviewIVDR

SSDR Material Dropped Tool Closed for all SSDR DRDs

Only Major Issues and Disagreements are brought

Post-CDR DRDs

Component reviews Executed or a Interim Design Review conducted for those reviews not scheduled prior to CDR

Management Review Team (as needed)

IVDR Caucus

Product Owners/SS Leads provide action plan and feedback to Caucuses

Pre-CDR

DRDs

Tool Closes for SSDR Products as SSDR concludes

Tool Closed for all DRDs

33


Orion CDR Process


CDR Kickoff

CDR DRDs Review


SSDRs

SSDR Caucuses



Pre-Board

Board




Post-CDR DRDs



IVDR Caucus


Pre-CDR

DRDs



• Majority of Component CDRs will be executed prior to CDR – For component CDRs which occur post

CDR an Interim Design Review should be conducted prior to CDR

• Majority of Component CDRs will be executed prior to CDR – For component CDRs which occur post

CDR an Interim Design Review should be conducted prior to CDR

34


Orion CDR Process


CDR Kickoff

CDR DRDs Review


SSDRs

SSDR Caucuses



Pre-Board

Board




Post-CDR DRDs



IVDR Caucus


Pre-CDR

DRDs



• CDR Kickoff– Purpose: kick off the Orion Critical Design Review and to provide an

overview of the DAC-4 close-out (607B) vehicle configuration and system architecture

– Goals:• Common understanding of the system architecture and CDR technical baseline

(Requirements, Environments, etc.)• Common understanding of the Design Reference Missions• Common understanding of flight operations by mission phase• Status on significant design issues and their LM/NASA agreed to

mitigations/closure plans• Clarification of what design documents are available at CDR and what design

reviews the DRDs trace to• Action Item/RID criteria and ground rules

– 1 Day Review• Available to all CDR participants (e.g. NESC, SRB, NASA Crew Office, NASA

MOD, NASA Engineering, NASA S&MA, NASA Health & Medical, Constellation, Orion Project, & LM Central Engineering) but tailored to external stakeholders

– A better educated reviewer will provide improved feedback and support to the review

• CDR Kickoff– Purpose: kick off the Orion Critical Design Review and to provide an

overview of the DAC-4 close-out (607B) vehicle configuration and system architecture

– Goals:• Common understanding of the system architecture and CDR technical baseline

(Requirements, Environments, etc.)• Common understanding of the Design Reference Missions• Common understanding of flight operations by mission phase• Status on significant design issues and their LM/NASA agreed to

mitigations/closure plans• Clarification of what design documents are available at CDR and what design

reviews the DRDs trace to• Action Item/RID criteria and ground rules

– 1 Day Review• Available to all CDR participants (e.g. NESC, SRB, NASA Crew Office, NASA

MOD, NASA Engineering, NASA S&MA, NASA Health & Medical, Constellation, Orion Project, & LM Central Engineering) but tailored to external stakeholders

– A better educated reviewer will provide improved feedback and support to the review

35


Orion CDR Process


CDR Kickoff

CDR DRDs Review


SSDRs

SSDR Caucuses



Pre-Board

Board




Post-CDR DRDs



IVDR Caucus


Pre-CDR

DRDs



• Systems Analysis Review– Prior to the SSDRs, a System Analysis Review

(1 to 2 days) will be held to educate the reviewers on the integrated analysis products that apply to multiple subsystems (Loads, Environments, Mass, etc.)• Begin taking Action Items/RIDs at the beginning of the

System Analysis Review

– Topics:• Environments

– Aerosciences• Loads and Dynamics

– Thermal Analysis– MMOD

• Specialty– Radiation– EMI/EMC

• Special SSDR-like reviews will address T&V and S&MA during this period– T&V Review will cover the integrated T&V activities

(module-level, external interfaces, facilities, and common Subsystem T&V)

• Systems Analysis Review– Prior to the SSDRs, a System Analysis Review

(1 to 2 days) will be held to educate the reviewers on the integrated analysis products that apply to multiple subsystems (Loads, Environments, Mass, etc.)• Begin taking Action Items/RIDs at the beginning of the

System Analysis Review

– Topics:• Environments

– Aerosciences• Loads and Dynamics

– Thermal Analysis– MMOD

• Specialty– Radiation– EMI/EMC

• Special SSDR-like reviews will address T&V and S&MA during this period– T&V Review will cover the integrated T&V activities

(module-level, external interfaces, facilities, and common Subsystem T&V)

36


Orion CDR Process


CDR Kickoff

CDR DRDs Review


SSDRs

SSDR Caucuses



Pre-Board

Board




Post-CDR DRDs



IVDR Caucus


Pre-CDR

DRDs



• SSDRs and Interactive Reading Rooms (IRRs)– CDR SSDRs will be conducted similarly to PDR SSDRs– SSDR data packages are dropped 5 business days prior to the beginning of

the SSDRs– Action Items/RIDs will be taken during the review– A dedicated IRR for each subsystem is available to all reviewers during each

SSDR and contain the following• Hard copies of significant design products and/or computers with all pertinent review data

(Requirements, Compliance Data, Drawings, Analyses, Test Results)• IRRs are staffed at all times to answer questions or show the design work products; runners

available to get SMEs• Test or prototype HW/SW if available

• SSDRs and Interactive Reading Rooms (IRRs)– CDR SSDRs will be conducted similarly to PDR SSDRs– SSDR data packages are dropped 5 business days prior to the beginning of

the SSDRs– Action Items/RIDs will be taken during the review– A dedicated IRR for each subsystem is available to all reviewers during each

SSDR and contain the following• Hard copies of significant design products and/or computers with all pertinent review data

(Requirements, Compliance Data, Drawings, Analyses, Test Results)• IRRs are staffed at all times to answer questions or show the design work products; runners

available to get SMEs• Test or prototype HW/SW if available

37


Orion CDR Process


CDR Kickoff

CDR DRDs Review


SSDRs

SSDR Caucuses



Pre-Board

Board




Post-CDR DRDs



IVDR Caucus


Pre-CDR

DRDs



• SSDR Caucuses– Purpose: disposition Action Items/RIDs identified during the SSDRs,

review forward action plans, and complete the Subsystem criteria assessment

– Membership: SSDR Co-Chairs, SS Leads, Module Rep, SEIT rep, ITA reps as appropriate (8-10 people in Caucus)

– Action Items/RIDs are dispositioned daily – Dispositions are provided to reviewers the next day– Reviewers reclama the disposition with the Co-Chairs at the SSDR if they

disagree (5-10 minutes given after lunch for people to let co-chairs know if the disagree with the disposition)


• SSDR Caucuses– Purpose: disposition Action Items/RIDs identified during the SSDRs,

review forward action plans, and complete the Subsystem criteria assessment

– Membership: SSDR Co-Chairs, SS Leads, Module Rep, SEIT rep, ITA reps as appropriate (8-10 people in Caucus)

– Action Items/RIDs are dispositioned daily – Dispositions are provided to reviewers the next day– Reviewers reclama the disposition with the Co-Chairs at the SSDR if they

disagree (5-10 minutes given after lunch for people to let co-chairs know if the disagree with the disposition)


38


Orion CDR Process


CDR Kickoff

CDR DRDs Review


SSDRs

SSDR Caucuses



Pre-Board

Board




Post-CDR DRDs



IVDR Caucus


Pre-CDR

DRDs



• Management Review Team (MRT)– Purpose: to review all major design issues, any reclamas/disagreements, cross-team

integration issues, and cost and schedule issues (> $500,000 (TBR))– Membership:

– Chairs: Mark Geyer and Cleon Lacefield– IPT, Chief Engineers, Crew, MOD, and other ITA rep as needed

– “Hot” issues will be elevated to the MRT after the caucuses to discuss the issues and provide integration at the IPT level– Detailed Forward Plans are discussed and modified/approved for each major issue– Any reclama or disagreement is resolved in the MRT– Pre-Board topics are identified by the MRT

– Allows SSDR caucuses to disposition all Action Items/RIDs and the IPTs to view all the major issues and provide feedback

• Management Review Team (MRT)– Purpose: to review all major design issues, any reclamas/disagreements, cross-team

integration issues, and cost and schedule issues (> $500,000 (TBR))– Membership:

– Chairs: Mark Geyer and Cleon Lacefield– IPT, Chief Engineers, Crew, MOD, and other ITA rep as needed

– “Hot” issues will be elevated to the MRT after the caucuses to discuss the issues and provide integration at the IPT level– Detailed Forward Plans are discussed and modified/approved for each major issue– Any reclama or disagreement is resolved in the MRT– Pre-Board topics are identified by the MRT

– Allows SSDR caucuses to disposition all Action Items/RIDs and the IPTs to view all the major issues and provide feedback

39


Orion CDR Process


CDR Kickoff

CDR DRDs Review


SSDRs

SSDR Caucuses



Pre-Board

Board




Post-CDR DRDs



IVDR Caucus


Pre-CDR

DRDs



• Integrated Vehicle Design Review (IVDR)– A 5-day IVDR follows the SSDRs by one week– The IVDR is treated similar to the SSDRs– All design DRDs that did not trace to an SSDR will trace to

the IVDR

– Candidate Topics:– Ground/Flight Operations– Integrated Vehicle Performance and Margins

(operational mitigations to hazards)– Integrated Aborts Performance and Margins– Critical Events as a function of Orion 1 and Orion 2– External Interfaces– Logistics (Either covered here or in AI&P Review)– Module Interfaces– Integrated Module and System Test and Verification

Summary– Module and System Assembly, Integration, and

Production– Vehicle level hazards and reliability

– Chairs: Chuck Dingell and Bill Johns– The Management Review Team (MRT) will convene to

handle issues from the IVDR Caucus

• Integrated Vehicle Design Review (IVDR)– A 5-day IVDR follows the SSDRs by one week– The IVDR is treated similar to the SSDRs– All design DRDs that did not trace to an SSDR will trace to

the IVDR

– Candidate Topics:– Ground/Flight Operations– Integrated Vehicle Performance and Margins

(operational mitigations to hazards)– Integrated Aborts Performance and Margins– Critical Events as a function of Orion 1 and Orion 2– External Interfaces– Logistics (Either covered here or in AI&P Review)– Module Interfaces– Integrated Module and System Test and Verification

Summary– Module and System Assembly, Integration, and

Production– Vehicle level hazards and reliability

– Chairs: Chuck Dingell and Bill Johns– The Management Review Team (MRT) will convene to

handle issues from the IVDR Caucus40

Project Orion

41

LM IDE

Assignee Works Issue & Submits Result to Initiator

VerifierRejectsVerifierRejects

VerifierClosesVerifierCloses

Process Detail

- Action Item/RIDs are dispositioned during the SSDR Caucus

- Approved Actions/RIDs/Comments are entered into the LM IDE Action tracking tool

- Estimated closure dates are established with rationale

- Assignee resolves issue and documents in IDE- IDE notifies initiator of “issue answered”- Initiator either accepts or rejects closure

- The verifier is identified by the NASA SSDR Co-Chair- IDE notifies verifier of issue status- Verifier reviews and either closes issue or rejects

closure

- If initiator, verifier, and assignee don’t agree that Action/RID/Comment is closed, the issue will be elevated to VICB/CPCB as required

- Verifier dispositions as “Reject” or “Close” in the LM IDE

Action Item/RID/Comment Closure Process

Approved Action/RID/Comments

Verifier Reviews Closure Rationale

VICB (if necessary)

Rew

ork

/ U

pd

ate

Rew

ork

/ U

pd

ate

Project Orion

42

CDR Action/RID ProcessProject Orion

Initiator

Action/RID Flow

SSDR

CDR Board

Caucus

MRT

Withdrawn Actions/RIDs

Approved Actions/RIDs

CDR DispositionProcess

Reduce Number of DRDs

Structured

Improvem

entSt

ruct

ured

Impr

ovem

ent

PDR DispositionProcess

PDRProcess

ElevatedRIDs


CDR Streamlining Benefits

• SSDR sponsoring of Action Items/RIDs will greatly improve the Sponsoring step and decrease the amount of instances issues are discussed throughout the process

• Having all pertinent reviewers in the room will increase the effectiveness of the review and ensure all stakeholders are aware of issues that are discussed

• Reviewing only design DRDs during the CDR Process will decrease the amount of Action Items/RIDs and spread out the time and effort needed to review and accept all the documents

• The design will become the focus of the reviews during CDR

43


44

Closing Remarks


CDR Supporting Documentation

• CDR Plan– CxP 72422 CxP Orion PDR Plan will be available on the CDR Wiki and CDR

Portal/Dashboard

• PDR Resources– CDR Wiki

• Used as a resource to all Orion personnel and CDR stakeholders to communicate the current planning status

• Once created, users can find the link from the Orion Vehicle Integration Wiki– https://ice.exploration.nasa.gov/confluence/pages/viewpage.action?pageId=12793039

– CDR Portal• Used to formally provide all SSDR and CDR details, data package, and review results• Federal record of review• Once created, users can find the link from the Orion Vehicle Integration Wiki

– https://ice.exploration.nasa.gov/confluence/pages/viewpage.action?pageId=12793039– Or the ICE Portal CDR Top-level Site– https://ice.exploration.nasa.gov/ice/site/cx/review_cdr/

45

Throughout CDR planning and execution activities, frequent communication and easy access to CDR related data is essential to ensure success.


CDR Integration Team

As the program nears the more active CDR planning phase, additional CDR Primary Integration Team (PIT) members will be identified.

46


Questions?

47


48

Backup Charts•OFT1 PTRs•Orion CDR DRDs Comparison•PDR Results Summary


OFT-1 Periodic Technical Review (PTR)Scope & Objectives

• PTR Series (1, 2, & 3) Scope & Assumptions– The purpose of PTR process is to:

• Establish the OFT-1 requirements baseline and the associated DRM

• Disclose the complete system design in full detail

• Show that the analysis validates the current design

• Ensure the system design meets the requirements and DRM

• Ascertain that technical problems and design anomalies have been resolved

• Ensure that the design maturity justifies the decision to initiate fabrication/manufacturing, integration, and verification of hardware and software

– As a result of successful completion of the three PTRs, the OFT-1 “build-to” baseline, production, and verification plans will be approved.

– PTRs will establish compliance to LM’s CDR tailored criteria for OFT-1• Where possible the PTRs will establish compliance to the CDR criteria for Orion-2

• The Orion-2 CDR and associate SSDRs will complete the Orion CDR process

– PTRs can be viewed as analogous to traditional program reviews with reduced data-package expectations

• Data-packages consist of design and analysis products (not DRDs)

• DRD deliveries are not aligned to PTRs– DRDs aligned to need dates

49


Orion PTR Board Members

• PTR Chairs are responsible for: – Presiding over PTR and for making decisions based

on inputs from Board Members– Judging adequacy of the PTR– Making final decisions on Action Item dispositions– Presenting results of the PTR to next controlling

auth.PTR Board Members PTRs

Project Management NASA & LM

Tech Auth – Chief Engineer NASA & LM (Chair)

Tech Auth – S&MA NASA

Tech Auth – AI&P NASA

VIO/SEI NASA & LM

CSM NASA & LM

LAS NASA & LM

T&V NASA & LM

S&MA NASA & LM

Production Ops NASA & LM

COTR NASA

Mission Ops NASA

PTR Board Members are responsible for: Participating in technical interchange Supporting the Chairs on issues or actions Supporting the reporting of findings and

recommendations made by the Chairs Voting following the completion of the PTR

50


OFT-1 Data Flow

• DRDs delivered to NASA and reviewed by NASA

• TCM held with LM SMEs to discuss comments

• Comments are provided back to LM contractually

• Type 2 documents require NASA approval

OFT-1 DRDs

OFT-1 Specification

OFT-1 Verification Plan

OFT-1 Post-Flight Data Package

OFT-1 DataOFT-1 Integrated Data

Plan

OFT-1 DataPer the Integrated Data Plan

Delivered to NASA

Uploaded to ProjectLink

Managed by LM

Data is uploaded to ProjectLink per the Integrated Data Plan

Data is managed on ProjectLink by LM

NASA has access to all of the OFT-1 Data once uploaded (per the Integrated Data Plan)

Technical Baseline data is accessible per the Integrated Data Plan

Data Type 2

Data Type 2

Data Type 3

OFT-1 Integrated DataSchedule

Optimization team will develop the plan• SEIT owned after release

Planning will develop the schedule• SEIT & C&DM will support• Planning owns integrated schedule

IPTs develop data/documentation• C&DM responsible for maintaining & populating data repository 51

DRAFT


OFT-1 Data/Work Product Scheduling

CSM

Scheduling Managing

LAS

AVP

Systems Analysis

SEIT

ATLO

IPT/Lower Level Schedules

Integrated Data

Schedule

Data Products not captured in

Lower Level Schedules

Lower Level Schedules are networked with the Integrated

Data Schedule

IDS Weekly Report:- Soon to come due- Changes to dates- Metrics - Status of data/work products (G/Y/R)

PIT Review- SEIT, C&DM, Planning d - IPTs as needed

Issue?

Yes

No

Morning Tagup

Briefing

Review the IDS status and highlight any overdue data/work products

Ensures changes to schedules align with

project needs

Developed based on the most current IDS

Requires additional coordination with the lower-level scheduling team

Stop

DRAFT

52


Incremental DRD Sets for Orion CDR

Timing Pre-CDR CDR Post-CDR

Quantity 73 104 21

Types of DRDs Con OpsGround Sys Reqt’sSafety DocsFault Tree AnalysisLunar Upgrade PlanWorkmanship StdsModel & Sims PlansSw Dev PlanIRDs

CAD ModelsDrawingsMVPICDsADDMass PropertiesData & Cmd DictionaryAvionics DDB

Ops Data BookTest Procedures

Time to Review 2 weeks 2 weeks 2 weeks

Review Team IPTs SSDR Caucus IPTs

Re-delivery CDR + 90 days(Delivered DRD + Approved Comments are reference)

CDR + 90 days TBD

Comment Mechanism

Spreadsheet or ConCERT*

IDE Action Tool Spreadsheet or ConCERT*

53

Project Orion

PDR Assessment

• Design Issues– A detailed pre-assessment was performed by each of the module prior to entering PDR– All known issues were presented at each SSDR and the SMR

• No new significant issues surfaced during the reviews

– Significant focus was applied to ensure known issues had agreed-to plans to proceed

– Top issues contributing to non-green assessments• Mechanisms – FBC R&R – FBC Recontact, CM/SM R&R Maturity• LRS – FBC Recontact / Parachute Architecture & Testability• TPS & Structures – Parachute Architecture Impacts to Baseline• EPS – 120 V Incorporation• AV&SW – C&T Incorporation, BFCS Maturity• ECLS – ARS Suit Loop, Procurement Schedule• CM Prop – Procurement Schedule• Pyro – Devices Dependent on Mechanism Maturity• LAS ACM – Production Motor Maturity

• 18 Subsystem Design Reviews (SSDRs) executed successfully– Each SSDR assessed against a common set of entry and success criteria plus a joint set of NASA/LM agreed to objective

evidence customized for each Subsystem

– 59 Total Review Days & approximately 500 review participants

• PDR executed successfully– Each module assessed against a common set of entry and success criteria; ITAs assessment integrated

– 56 days of DRD review, 7 day System & Module review, 1 week pre-Board and a Board

– Approximately 1200 reviewers

Yellow / Green

• Project Management assessed PDR a success and authorized the program to proceed to the detailed design phase

• Assessment based on a Red, Yellow, Green scale

Parachute Architecture Study was underway prior to entering PDR. Assessment concluded in October 2009.

Driven by significant post SDR requirements changes

Procurement schedules were being updated to align with project funding profile

54

Project Orion

PDR Data Package• NASA/LM jointly assessed PDR

deliverables to determine which DRDs were necessary to meet PDR expectations and satisfy criteria

• PDR –• 146 Deliverable Documents

• SSDR & PDR –• 26 Design & Data Books

• Data Package Total• 172 Total PDR Documents• 53 “Drafts Available” DRDs

provided to NASA for comment

Waterfalled Review and Delivery• Managed via the PDR Integration

Team• Captured in a 2000+ line MS

Project Schedule• Metrics generated & reported on

a weekly basis• Deliveries culminated on 7/7/09,

PDR-45 Days

PDR Data Package Delivery

As of July 8, 2009

PDR -45 days

55

Project Orion

Action Item & RID Closure

PDR RIDs

LAS 150

CM 241

SM 90

Av&Pwr&Wire 444

Software 38

T&V, Fac, GSE 411

AI&P 18

SR&QA 91

Systems 566

Total 2049

SSDR Actions

AV&SW 140

Crew Systems 39

ECLSS 49

EPS 97

GN&C 26

LAS AM 60

LAS ACM 96

LAS JM 60

LAS SE&I AVP 12

LRS 42

Mechanisms 56

Propulsion 150

PTC 33

Pyro 26

Structures 141

TPS 20

Wiring 14

Total 1061

• Detailed Action and RID screening occurred throughout the SSDR & PDR process

• All actions and RIDs dispositioned and closure aligned with project post PDR work plan

• All RIDs approved by PDR Board

• Project approved closure plan implemented

• Approximately 100,000 pages of design, process, and plan documentation reviewed

• Included requirements, interface definition, verification details, general plans, etc.

• Approximately 100,000 pages of design, process, and plan documentation reviewed

• Included requirements, interface definition, verification details, general plans, etc.

56

Armstrong

Technology

pdr process refresher

design drds

review participants

module review

series reviews

orion cdr process

review filter

current design