1 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Addressing NOAA’s Data Management Challenges Presentation to the World Data Center for.

1 WDC for Glaciology 30th Anniversary Workshop

25 October 2006

Addressing NOAA’s Data Management Challenges

Presentation to the World Data Center for Glaciology 30th Anniversary Workshop

Dr. Christopher G. FoxDirector

NOAA National Geophysical Data Center


25 October 2006

NOAA Encompasses a Challenging Diversity

NOAA currently manages >90 environmental observing systems, some with hundreds of stations: including land-, sea-, air, and space-based observing platforms

These systems gather >300 diverse environmental parameters (e.g. marine biological health, economic fisheries data, physical and chemical state of the atmosphere and ocean, paleoclimate proxy data, geodetic survey points, etc.)

NOAA also requires other national, international and commercial data in its operations (some in real-time)

NOAA data management infrastructure includes numerous significant stovepipe systems

Future observing systems will produce vastly increased data volumes that will need to be archived and efficiently accessed by an expanding number of users

NOAA is migrating from this current stovepipe environment to an information enterprise


25 October 2006

NOAA’ Top Ten Challenges1

10) Alphabet Soup

9) Stove Pipes

8) Integration

7) Architecture

6) Data Sharing

5) User Needs

4) Maximizing Benefits

3) Communication

2) Data Management

1) Execution

1. Vice Admiral Lautenbacher in address to American Meteorological Society Annual Meeting, January 30, 2006


25 October 2006

Data Management - a top priority

Improving data management is among the highest priority challenges facing NOAA


25 October 2006

Response - NOAA’s GEO-IDEGlobal Earth Observation Integrated Data Environment

Scope – NOAA-wide architecture development to integrate legacy systems and guide development of future NOAA environmental data management systems

Vision – NOAA’s GEO-IDE is envisioned as a “system of systems” – a framework that provides effective and efficient integration of NOAA’s many quasi-independent systems

Approach – evolution of existing systems into a service-oriented architecture built upon agreed standards, principles and guidelines

Result – a single system of systems (user perspective) to access the data sets needed to address significant societal questions


25 October 2006

Project Management

Undersecretary for Oceans and Atmosphere

DMITData Management Integration Team

NOSCNOAA Observing System

Council

DMCNOAA Data Management

Committee

All

NO

AA

Go

als

All

NO

AA

Lin

e O

ffic

es


25 October 2006

Scope

Concerned with environmental and geospatial data and information obtained or generated from worldwide sources to support NOAA's mission

Does not consider administrative support systems such as finance, personnel, acquisition or facilities management

Includes all aspects of data management, including data acquisition, ingest, data processing, archive and access (CLASS)


25 October 2006

Vision

“System of systems” – a framework to effectively and efficiently integrate NOAA’s many systems

Minimize impact on legacy systems Utilize standards Work towards a service-oriented architecture


25 October 2006

Approach

Each NOAA LO/program/project continues to manage its data independently

Standards– Adopt, adapt and only as a last resort, create– Open, inclusive process for adoption– Inclusive not exclusive use of standards

Service Oriented ArchitectureReference: Federal CIO Council, Jan ’06 "Services and Components Based Architectures: A Strategic Guide for Implementing Distributed and Reusable Components and Services in the Federal Government"


25 October 2006

Standards

Standard names and terminology Metadata standards

– e.g. FGDC and ISO 19115 w/ remote sensing extensions

Standard formats for delivery of data/products– WMO, NetCDF, HDF, GeoTIF, JPEG, etc.

Web Services Standards– World Wide Web Consortium– OGC (Maps, Features, Coverage, GML)– Community Standards: OPeNDAP, Unidata’s

Common Data Model (CDM),COARDS, CF– REST /SOAP / UDDI / WSDL where appropriate


25 October 2006

NOAA GEO-IDE Standards Process

Submitted Standard – Standards can be submitted by anyone in NOAA for evaluation to see if they address needs and could be applicable to NOAA

Proposed NOAA Standard – Can be provisionally used within NOAA for technical evaluation

Recommended NOAA Standard – NOAA data systems should consider supporting the standard wherever applicable for evaluation in real-world NOAA systems

NOAA Standard – Approved and mandated where appropriate

Phased approach of submission, evaluation and adoption


25 October 2006

Fast-track submissionAn initial set of well known and/or widely used standards are being considered as an initial set of “submitted” standards

• Discovery Metadata: FGDC, ISO 19115 (w/ extensions), OBIS

• Keyword Lexicon: Start with Global Change Master Directory

• Metadata Exchange: XML compliant with FGDC CSDGM and OBIS

• Catalog Search: Compatible with Geospatial One Stop specifications (currently Z39.50 or OAI-PMH)

• File Transfer: FTP and HTTP

• Data Base Access: ODBC and JDBC

• API and Web Services: OpenDAP and OGC service specifications

• Data and product formats: The usual suspects…


25 October 2006

Service-Oriented Architecture

Under an SOA, capabilities are built one at a time to create “Web Services”

The fabric of the SOA is built upon standards for:– discovery (e.g. FGDC, ISO, DIF,CSW)– transport (e.g. HTTP, FTP, OPeNDAP)– use (e.g. netCDF, HTML, etc.)

Can be SOAP, REST, or some other standard

Define core common services and build Service layer agreements across NOAA


25 October 2006

SOA (continued)

Operational Public Access Services: for public access to data, products and information services

Operational Services: where security, timeliness, and reliability are paramount

Scientific Services: where efficient and flexible discovery and access to data sets are required

Commercial value-added services

Four general classes of web services are anticipated for NOAA:


25 October 2006

Key Development StrategiesCoordinate activities through Communities of

Interest organized by “Data Types”– Grids, time-series, moving-sensor multi-dimensional, profiles,

trajectories, geospatial framework, point data and metadata

Evolutionary development through pilot projectsMinimize impact on legacy systemsDevelop Services and Component based

Architecture (SCBA) using a top-down, bottom-up iterative software development process (Federal CIO Council)


25 October 2006

Communities of interest based upon data typesAn initial list

Grids (e.g. model output, gridded data products) Moving-sensor multidimensional fields (e.g. satellite

swaths, side-scan sonar, weather radar) Time series (e.g. fish landings, sun spot activity, climate data,

paleo-records) Profiles (e.g. atmospheric soundings, ocean casts, profiling floats) Trajectories (e.g. underway ship measurements, aircraft track

data, ocean surface drifters) Geospatial Framework Data (e.g. shorelines, fault lines,

marine boundaries, map annotations) Point data (e.g. tsunami or seismic occurrences, geodetic

control) Metadata - information needed for the use and interpretation of

data


25 October 2006

Future Direction - Priorities FY07 Work with scientists/data system managers to assess

requirements and systems– Develop enterprise architecture and GEO-IDE Implementation

Plan– Implement standards process– Active out-reach activities

FY08/09 Incrementally execute work packages– Adopt/Adapt/Develop data standards and interoperability

mechanisms, e.g., translators and directory services– Direct, test and evaluate changes being made to data

management systems

FY10/11 Re-evaluate architecture related to new data systems (across NOAA & with national & international partners)


25 October 2006

Happy Anniversary NSIDC

We at NGDC and NESDIS look forward to working with you to integrate your snow and ice data archives into NOAA’s GEO-IDE framework…

Questions?

1 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Addressing NOAA’s Data Management Challenges Presentation to the World Data Center for.

Documents