The NASA Standards Process for Earth Science Data Systems Richard Ullman, NASA Yonsook Enloe, SGT Inc IGARSS 2010.

The NASA Standards Process for Earth Science Data Systems

Richard Ullman, NASAYonsook Enloe, SGT IncIGARSS 2010

Technology Diffusion

2NASA Earth Science Data Systems Standards - IGARSS - August, 2010

100%

0%

Adop

tion

Time

Source: Geoffrey Moore, Crossing the Chasm, 1999modified after Everett Rodgers, 1962

Innovative PragmaticTechnology Demonstration Mission Reliability/Stability

Innovators Early Early Late Laggards Adoptors Majority Majority

NA

SA

’s

Ch

asm

Standards Diffusion

Chasm between:• early adopter - technology for strategic advantage• early majority - pragmatic focus.

A practical path across chasm … “Community-led”• successful practice in specific community• broader community adoption• community-recognized “standards”

Community-led relies on:• trusted endorsements• strong leadership• “whole product”


ESDS Standards Process – Seeking Pragmatic Recommendations

A specification or practice is recommended as a standard … Only after practices have been shown to:• (1) have demonstrated implementation and • (2) benefit to operation will they be endorsed for preferential use.

Ideas come from innovators or mission planners and are tempered by the significant demands of writing an RFC.Review process permits adoption only after “significant” community endorsement.

THE RFC PROCESS PROVIDES PRAGMATIC REFERENCES


Crossing our Chasm – ESDS and the Pragmatic Decadal Survey

SPG has been successful as a pragmatic voice: NASA HQ has endorsed SPG recommendations for Decadal Survey Mission use. Mission planners have embraced SPG as subject matter experts.NASA HQ recognize the benefit of choosing standards that are already recommended by the community• The frustrating thing about standards from a HQ perspective

is that everyone seems to want a different one.• Program-focused HQ planners want agency investments to

have broad return. i.e, for data: effective use for the primary mission and multiple secondary uses

Decadal survey mission planners recognize the benefit of choosing standards that are already recommended by NASA.• The frustrating thing about standards from a mission is that

there are so many to choose from.• Mission-focused pragmatic planners want to make the right

choice. Standards that work and standards that won’t get them in hot water with their stakeholders.


Decadal Survey [2]

SPG process has engendered:• Greater confidence that mission products will be interoperable

with heritage and current NASA data holdings• Stakeholder community benefits because their preferences

and opinions are available to the mission planner• Lower barriers to entry and use of NASA data by external

discipline communities within NASA and outside NASA because community standards are used.

NASA Earth Science Data Systems Program is actively evolving the Earth Science data “system of systems” architecture for the Decadal Survey missions and beyond


Tier 1 Decadal Survey Missions

Tier 1 Decadal Survey missions are the first group of missions to be formulated since NASA chartered the SPGSoil Moisture Active Passive (SMAP) MissionIce, Cloud, and Land Elevation Satellite 2 (ICESat 2)Climate Absolute Radiance and Refractivity Observatory (CLARREO)Deformation, Ecosystem Structure, and Dynamics of Ice (DESDynI)

The Tier 1 missions are actively working with the SPG in order to explicitly assure that the mission data systems solutions draw on the consensus-built lessons of agency experience. Tier 1 missions have a level 1 requirement to use SPG-endorsed standards


Crossing our Chasm – ESDS and Innovative Technologists

It is a challenge to encourage technology demonstration projects to use RFC process.• Technologist are often more interested in the technology

puzzle rather than in practical dissemination.• Early adopters are often more interested in strategic

advantage rather than wide use.• System Engineering focused on mission requirements makes

“outreach” of writing RFC for community good “out of scope”“Whole Product” may be missing.• Considerable systems engineering is still required when

abstract standard is to be embedded into a mission implementation.


Process Basics

3 types of reviews and all reviews can be done simultaneously – some reviewers perform multiple reviews. • Technical specification review• Usefulness for purpose review• Operational readiness

If the proposed standard has been adopted by an another standards organization, then do not need a technical specification reviewFor a mature defacto standard, we will perform a subset of the three types of reviews. May not need an operational readiness review or usefulness for purpose reviewThe SPG will determine which reviews will be conducted for each proposed candidate standard. Can be a subset.Encourage strong community leader – essential to the process


Responsibilities

Community Leader• Identify someone in their community who will document

standard according to SPG guidelines.• Work with the community to get an extended review of the

proposed standard.SPG• Assign “RFC editor” to advise on RFC document.• Publish and publicize RFC• Assign “TWG”, technical working group to organize

community review and evaluate responses.• Recommend action to NASA HQ.


Endorsed Standards and Tech Notes

StandardsData Access Protocol (DAP) V2.0 OGC Web Map Service V 1.1.1HDF 5HDF EOS 5NetCDF ClassicGCMD DIFICARTT

Technical NotesOGC Web Map Service V1.3AURA GuidelinesBacktrack Orbit Search AlgorithmInteroperability Between OGC CS/W and WCS ProtocolsLessons Learned Regarding WCS Server Deisgn and ImplementationCreating File Format Guidelines: The Aura Experience


Near Term & Long Term Future RFCs

RFCs Under ReviewCF Metadata ConventionsECHO Metadata ModelMapping HDF5 to DAP2NetCDF-4/HDF5 File Format

Tech Areas for Future RFCsWeb Services (e.g. REST, SOAP, ..)Catalog services (e.g. OGC CSW,…)Metadata standards (e.g. ISO 19115,…)Visualization standards (e.g. KML,..)File Formats (e.g. NetCDF4,..)Interface standards (e.g. DAP v4, …)Data modeling Data Fusion


BACK-UP


Decadal Survey Background

• At NASA’s request, the National Research Council (NRC) looks out ten or more years into the future and prioritize research areas, observations, and notional missions to make those observations

• NRC completed its first decadal survey for Earth science in Jan 2007 at the request of NASA, NOAA, and USGS.

• For the next decade, the decadal survey identified 15 missions for NASA and 3 for NOAA including 1 joint NASA/NOAA

• The 17 total (NASA & NOAA) missions are presented as the result of prioritization methodology designed to achieve a robust, integrated program.

• The 15 missions for NASA are presented in 3 time-phased blocks.

• The 17 missions form a minimal, yet robust, observation component of an Earth information system that is capable of addressing a broad range of societal need.


Process Diagram

15

Proposed Standard

Community Review

Recommendation

SPGEvaluate Proposed

Standard Implementations and Community Response

TWGEvaluate Proposed Standard

Stakeholders

Initial Screening Initial review of the RFCProvide RFC submission support Form TWG; set schedule

Recommended Standard

RFC

Review Questions:• Technical Specification• Operational Readiness• Suitability for Use

TWG

Technical Note

Technical Note

Reject

Reject

NASA Earth Science Data Systems Standards - IGARSS - August, 2010

SPG

NASA ESDS RFC List

ESDS-RFC Title Rev Class Status

ESDS-RFC-001 Charter of the ESDS Standards Process Group 2 Note Final

ESDS-RFC-002 The ESDS Standards Process 2 Note Final

ESDS-RFC-003 Instructions to Authors 2 Note Final

ESDS-RFC-004 The Data Access Protocol -- DAP 2.0 1.1 Standard Final

ESDS-RFC-005 OpenGIS ® Web Map Service Version 1.3 1 Note Final

ESDS-RFC-006 OpenGIS ® Web Map Service Version 1.1.1 1 Standard Final

ESDS-RFC-007 HDF 5 1 Standard Final

ESDS-RFC-008 HDF EOS 5 1 Standard Final

ESDS-RFC-009 Aura Guidelines Technical Note 1 Note Final

ESDS-RFC-010 Backtrack Orbit Search 1 Note Final

ESDS-RFC-011 NetCDF Classic 1 Standard Final

ESDS-RFC-012 GCMD Directory Interchange Format (DIF) 1 Standard Final


NASA ESDS RFC List

ESDS-RFC Title Rev Class Status

ESDS-RFC-013 GCMD Service Entry Resource Format (SERF) 0.1 Standard Proposed

ESDS-RFC-014 Interoperability between OGC CS/W and WCS Protocols 0.1 Note Final

ESDS-RFC-015 Provenance within Data Interoperability Standards .01 Note Proposed

ESDS-RFC-016 Lessons Learned Regarding WCS Design and Implementation 0.1 Standard Final

ESDS-RFC-017 Mapping HDF5 to DAP2 0.1 Note Proposed

ESDS-RFC-018 Creating File Format Guidelines - The Aura Experience 0.1 Note Final

ESDS-RFC-019 ICARTT 0.1 Standard Proposed

ESDS-RFC-020

ECHO Metadata Standard 0.1 Standard Proposed

ESDS-RFC-021 CF Metadata Conventions 0.1 Standard Proposed

ESDS-RFC-022 NetCDF-4/HDF-5 File Format RFC 0.1 Standard Proposed


The NASA Standards Process for Earth Science Data Systems Richard Ullman, NASA Yonsook Enloe, SGT Inc IGARSS 2010.

Documents

community standards

nasa standards process

nasa hq

outside nasa

standards diffusion

esds standards process

chasm community

decadal survey mission