Problem Reporting - TaxonomyBob Beil
6/12/07
Standards Working Group, NASA STD-0006
Bob Beil
NESC Systems Engineering Office
June 12, 2007
This briefing is for status only and does not represent complete engineering data analysis 2
Bob Beil
6/12/07Agenda
• Background
• Team Listing
• Status
• Current Activities
• Backup
This briefing is for status only and does not represent complete engineering data analysis 3
Bob Beil
6/12/07Information Overload!
SSP, ISSPRACAdatabasesExploration….
Robotic Missiondatabases
GSFCSOARSdatabase
JPL, ARC,Stennis etc
We are drowning in data, but starving for knowledge!
This briefing is for status only and does not represent complete engineering data analysis 4
Bob Beil
6/12/07
NESC led Data Mining & Trending Working Group (DMTWG)
• NESC tasked to form DMTWG to assist the Agency in the formulation and implementation of a plan to strengthen trending of NASA programs and projects, and to ensure appropriate visibility of data mining and trending within the Agency
– Function as an advisory group to the NASA Office of the Chief Engineer and the EMB.
– Assist the Office of the Chief Engineer in the development, coordination, and implementation of best practices for data mining and trending of technical data.
This briefing is for status only and does not represent complete engineering data analysis 5
Bob Beil
6/12/07NESC Data Mining/Trending Activities To Date
• NESC received data mining/trending task
• Conducted initial DMTWG workshop– Recommendations and lessons learned (including identification of
need for taxonomy/data dictionary)
• Initiated Recurring Anomalies effort– Completed SSP recurring anomalies effort
– ISS report to be presented to NRB June ‘07
• NESC funded taxonomy working group produced a proposed taxonomy standard
• Proposed Standard formed backbone of CxP requirements document (CxP 70068)
• Response team formed to perform requested data mining activities within the Agency
This briefing is for status only and does not represent complete engineering data analysis 6
Bob Beil
6/12/07Team Listing
Team ListingBob Beil Systems Engineering Office (SEO) - Co-LeadVickie Parsons NESC SEO – Co-LeadTina Panontin Ames Research CenterRoxana Wales Ames Research CenterMichael Rackley Goddard Space Flight CenterJames Milne Goddard Space Flight CenterTim Barth Kennedy Space CenterJohn McPherson Marshall Space Flight CenterJayne Dutra Jet Propulsion LaboratoryLarry Shaw Johnson Space Center
This briefing is for status only and does not represent complete engineering data analysis 7
Bob Beil
6/12/07Background cont., why taxonomy
• There is a frequently quoted statistic that more than 80% of information housed in corporate repositories is unstructured data, for NASA it is at least equivalent
• The reality is that until computers can consistently and accurately recognize concepts, using a taxonomy as a framework for categorizing documents will aid in navigation and retrieval.
– Natural language searching and keyword searching yield high retrieval but can miss essential pieces of content that do not contain the specific terms that are being searched.
– A taxonomy can be counted on to improve search precision and facilitate discovery when drilling down into a subject hierarchy
This briefing is for status only and does not represent complete engineering data analysis 8
Bob Beil
6/12/07Background, Taxonomy
• The Office of Chief Engineer requested the NESC lead a small group to develop a proposal for a common taxonomy to be used by all NASA projects in the classifying of nonconformance's, anomalies and problems
– The intent was to determine what information is required to be collected and maintained in order to facilitate future trending and root cause analysis
• Taxonomy:
The science of classification
Today, our electronic technology allows us the opportunity to present information from multiple viewpoints maximizing the probability of discovery of relevant information by the end user
•A taxonomy provides an important piece of the puzzle to performing efficient data mining and trending
This briefing is for status only and does not represent complete engineering data analysis 9
Bob Beil
6/12/07Background cont.
• NASA personnel with experience in both human spaceflight and robotic missions were recruited
– Expertise in knowledge management systems, anomaly resolution, trending, current problem reporting systems and taxonomy development
• Developed use cases across various missions
• Using use cases, developed taxonomy(ies) for a Problem Reporting System
• Product provides suggestions for inclusion in a problem reporting system and may serve as a partial guide to system developers; the intent was not to design a problem reporting system
This briefing is for status only and does not represent complete engineering data analysis 10
Bob Beil
6/12/07Status
• In Review
ACTION STAGE DUE DATE OPEN ACTION
COMPLETED/DATE
NASA-STD-0006Common NASA Taxonomy for Problem Reporting, Analysis, and Resolution Standard
Agency Review OVERDUE
(Note: Preparing for EMB Concurrence)
ARC
JSC
LaRC
NESC
DFRC/08-23-06
GRC/10-17-06
GSFC/c/11-08-06
JPL/10-02-06
KSC/c/09-01-06
MSFC/c/No Date
SSC/11-08-06
WSTF/08-17-06
OSMA/08-09-06
This briefing is for status only and does not represent complete engineering data analysis 11
Bob Beil
6/12/07Resolution Matrix Summary
73 comments
• Concur
– 47
• Reject
– 24, have discussed all but 3 with authors
• Require further consideration
– 2, forward work
This briefing is for status only and does not represent complete engineering data analysis 12
Bob Beil
6/12/07Current Activities
• CxP PRACA, CxP 70068 (Constellation Program Problem Reporting, Analysis, and Corrective Action (PRACA) Requirements)
– Proposed Standard provide backbone for this document
– Noted weakness of Standard was software inclusion
• Working to incorporate lessons learned into proposed Standard
• NESC funded effort planned to benchmark the data dictionary/taxonomy
This briefing is for status only and does not represent complete engineering data analysis 13
Bob Beil
6/12/07
Back-upBack-up
This briefing is for status only and does not represent complete engineering data analysis 14
Bob Beil
6/12/07Report Summary
• Data elements marked with an asterisk in the share column represent the minimum set of data elements that all projects should make available through a common user interface to organizations such as the NESC, tasked with performing trending across projects
– This set of data elements should provide enough information to facilitate the root cause and trend analysis required of the individual projects by NPR 7120.5C
– Given differences between human spaceflight and robotic missions life cycles and post launch activities, some data elements may not be applicable for both.
This briefing is for status only and does not represent complete engineering data analysis 15
Bob Beil
6/12/07
Recommended Data Elements and Taxonomies for Problem Reporting
Field Share Source Definition EXPECTED PROBLEM REPORT PROCESSING PHASE(S) FOR DATA FIELD POPULATION: 1=Initiation, 2=Analysis, 3=CloseOut
Criticality Code * Pick list Assessment of the severity of the problem based on FMEA and CIL for the assembly level … this is the functional criticality level
1-2
Problem Description
* Large text string
Detailed description of the problem - "prescription" for what would be information to be included is provided
1
Mishap Report? Yes/No Did this problem result in a formal mishap report due to damage of equipment or personal injury?
1-2-3
Last Update Field * Formatted String
Automatically filled by software when record saved
1-2-3 (auto)
Program Taxonomy Program name (program attributes defined in NPR 7120.5C)
1
Defect Characteristics
* Example list
A fault/flaw/discrepancy/nonconformance in a component or process that causes discrepant performance of the component or assembly involved
2
This briefing is for status only and does not represent complete engineering data analysis 16
Bob Beil
6/12/07Pick List
Field Potential Values Definitions of Values
Criticality Code 1 Single failure that could result in death or loss of vehicle
1R Redundant hardware items that could cause a criticality 1 event if all items fail
1S Safety or hazard monitoring hardware items that could cause the system to fail to detect, combat, or operate when needed during a hazardous condition, potentially resulting in a criticality 1 event
2 Single failure that could result in severe and/or permanent injury, major property damage, or a loss of mission
2R Redundant hardware items that could cause a criticality 2 event if all items fail
3 Single failure that could result in minor injury, minor property damage, a significant mission delay, or a mission degradation in which some mission goals not achieved
4 All other failures that result in unscheduled maintenance or repair
Where pick lists or taxonomies are provided for individual data elements, these are meant to be a starting point and not all inclusive
This briefing is for status only and does not represent complete engineering data analysis 17
Bob Beil
6/12/07Example List
Field Potential Values Definitions of Values
Defect Characteristics
MiswiredAbradedDingedPart omittedWornShort-circuitedDelaminated
These are definitely not an exhaustive list.
NOTE: These are examples, individual projects should create pick lists or links wherever possible to represent their situation. These data fields were included here because the team did not believe that even a high level set of
values would be consistent between projects.
This briefing is for status only and does not represent complete engineering data analysis 18
Bob Beil
6/12/07Taxonomy
Field Taxonomy Comments
Program NASA taxonomy http://lurch.hq.nasa.gov/2005/11/21/pops.owl
These fields should be linked into NASA's formal taxonomies to auto-fill where possible.
This briefing is for status only and does not represent complete engineering data analysis 19
Bob Beil
6/12/07Report Summary cont.
• Critical need for projects to capture information on aberrant events in order to determine the causes and prevent future occurrences
• Relevant data needs to be translated into shared data elements and provided to a common source so trending across projects may be accomplished by independent organizations
• Report includes suggested definitions for key terms
• Report also includes a white paper on the characteristics of a good taxonomy
• Additional recommendations were provided for consideration when developing an actual system
This briefing is for status only and does not represent complete engineering data analysis 20
Bob Beil
6/12/07Background, Why Data Mining?
• Data mining to find patterns
• Analysis to identify correlations or clusters
• Search for explanations
A B C A D C D B A C B B B D D D A D C B B D B A C B B D A D C B B D D
B B B B B B B B B
D D D D D D
AC AC AC AC AC AC
B B B
D D D D D
A B C A D C D B A C B B B D D D A D C B B D B A C B B D A D C B B D D
“The nontrivial extraction of implicit, previously unknown, and potentially useful information from data.”
Trending: the search for patterns and subtle relationships in data to infer relative data behavior.
This briefing is for status only and does not represent complete engineering data analysis 21
Bob Beil
6/12/07NESC Data Mining Activities
• Data Mining Trending Working Group (DMTWG) meeting monthly
– Comprised of representatives from all centers as well as NTSB, FAA, DHS, & INPO
– Workshop scheduled for June 20/21 to discuss successes in data mining
• Purchased and Installing SAS (statistical analysis software) on a LaRC server to make available for DMTWG
• ARC developing custom data mining tool (clustering based) to aid search for recurring anomalies
• Successful data mining and trending involves a variety of methods and tools as well as subject matter expertise