NASA CEOP Status & Demo CEOS WGISS-24 Oberpfaffenhofen, Germany October 15, 2007

1

NASA CEOP Status & Demo

CEOS WGISS-24

Oberpfaffenhofen, GermanyOctober 15, 2007

Yonsook Enloe

2

CEOP/NASA Collaboration• CEOP Scientists

– need efficient access to satellite, model output, and field data – need to compare values of the different data types

• emphasis on reference sites, specific monsoonal regions, specific observation periods

– lack advanced tools to access the satellite data with the services needed to support data integration

• CEOP Program initiated a discussion with CEOS WGISS to ask for assistance – CEOP science community identified difficulty of accessing and using

satellite data to compare with model output and field data.

• NASA prototype will improve improve CEOP science community access to satellite data

3

Approach: Leverage OPeNDAP

• OPeNDAP– Protocol for data access

• Includes subsetting• Endorsed as a NASA Earth Science Standard

– Server Implementation: Hyrax

• OPeNDAP Clients– Especially popular among modelers

• Significant time investment in tools of choice

– Provide OPeNDAP Client access to OGC Web Coverage Servers, to facilitate access to satellite data

4

CEOP Data Access Prototype

WCS Server

Grid Data

georectified NetCDF

Web Coverage request

OPeNDAP Server

WCS Handler

GrADS

Live Access Server

Ferret

SciFlow

Giovanni

IDL

matlab

OPeNDAP request

georectified spatial subset

Analysis / Display Tools

Approach: Enable existing tools to access data from Web Coverage Servers

JAXA

Bosilovich

Swath Data

5

Project Goals and Accomplishments • Developed design for the CEOP Satellite Data Server

– Implement as “handler” module for type “wcs” within OPeNDAP Hyrax architecture (Change from original plan)

– Enables additional OPeNDAP clients accessing WCS servers

• Developed converter for HDF-EOS to NetCDF (CF-1) in WCS Server• Developed the WCS handler to request and obtain data from WCS

server• Beta Release expected on Oct 31, 2007 with AIRS Level 2 and some

MODIS data online

Any WCS Server that can return a NetCDF CF or HDF-EOS CF response can be linked to the Hyrax OPeNDAP Server and accessed by OPeNDAP enabled clients

Any OPeNDAP enabled client may access any of the WCS Servers whose holdings are registered in the Hyrax Server via the THREDDS catalog configuration.

6

Current Development Plans• CEOP Satellite Data Server Enhancements

– More data types– Time dimension– Mosaic-ing for single orbit using time dimension– Complexities (multi-parameter & full time periods) needed

by CEOP science community– Satellite data access for the full CEOP time periods– Additional interpolation methods– Quality screening of data– Performance improvements– Start Attribution, lineage and data documentation

7

DEMO: CEOP Satellite Data Server

• GrADS (Grid Analysis and Display System) Client– MODIS & AIRS– AIRS & MERRA

• Ferret Client– GoCart Model & MODIS

• Live Access Server

8

NASA CEOP Prototype Demo

WCS Serverat GSFC DAAC

Grid Data

georectified NetCDF

Web Coverage request

OPeNDAP Hyrax Server

in Rhode Island

WCS Handler

GrADSon

User’s PC

Live Access Server in RI

Ferret

georectified spatial subset

Analysis / Display Tools

Approach: Enable existing tools to access data from Web Coverage Servers

Swath Data

WCS Serverat DataFed at

Washington University in St Louis

Grid Data Swath Data

Web Coverage request

georectified NetCDF

OPeNDAP request

OPeNDAP request

georectified spatial subset

USER in Germany

http

9

GrADS w/MODIS & AIRS'sdfopen http://test.opendap.org:8080/opendap/wcs/GSFC/MODIS/2004/NetCDF/MOD05_L2.A2004210.0430.005.2007025074420.wcs''set lat 34 38''set lon -102 -94''set gxout shaded''set mpdset hires''d water_vapor_inf''sdfopen http://test.opendap.org:8080/opendap/wcs/GSFC/AIRS/2004/NetCDF/AIRS.2004.07.28.087.L2.RetStd.v4.0.9.0.G05260231438_totH2OStd.wcs''set gxout contour''set clevs 0 5 10 15 20 25 30 35 40 45 50''d toth2ostd.2'

10

GrADS: AIRS & MERRA'sdfopen http://test.opendap.org:8080/opendap/wcs/GSFC/AIRS/2004/NetCDF/AIRS.2004.07.28.087.L2.RetStd.v4.0.9.0.G05260231438_totH2OStd.wcs''set lat 34 38''set lon -102 -94''set gxout shaded''set clevs 0 5 10 15 20 25 30 35 40 45 50''set mpdset hires''d toth2ostd''sdfopen http://test.opendap.org:8080/opendap/wcs/GSFC/MERRA/NetCDF/MERRA00.prod.assim.tavg1_2d_int_Nx.20060127.wcs''set gxout contour''d tpw.2(time=00:00Z27jan2006)'

11

Some Current Issues

• Data Lineage: GoCart Model is generated at GSFC but is served by the DataFed WCS Server at Washington University in St. Louis. However, no lineage information is shown to the user who may not know where this data originated.

• Data Quality: need to screen data for quality before putting online and give this information to the user

• Get feedback from the CEOP science community after the Oct 31 release and enhance prototype with response to feedback

12

OGC-Geoscience Gateway Project Background

• THREDDS– Provides catalogs of datasets served via a variety of protocols

• Geoscience community protocols: http/netCDF, OPeNDAP, ADDE (Abstract Data Distribution Environment - I.e. McIDAS)

• OGC protocols: WCS (Web Coverage Service)

– Depends on other organizations and protocols for discovery and search of the data

– Phase 1 of the OGC GALEON1 Interoperability experiment showed that WCS getCapabilities is not rich enough for discovering contents of THREDDS servers

• OGC Geoscience gateway complements THREDDS by:– harvesting the information in THREDDS catalogs – providing a standards-based (i.e., OGC) discovery interface that enables

users to search for data available on distributed THREDDS servers

• One remaining gap is a standards-based interface to allow clients to browse the hierarchy of catalogs on THREDDS servers.

1 Geo-interface for Atmosphere, Land, Earth, and Ocean netCDF

13

OGC-Geoscience Gateway

DataCatalog

SemanticCatalog

THREDDSCatalog

Catalog Ingestor

GeoscienceServers

Tools

WC

S

WF

S

CS/W protocol WCS protocol WFS protocol

Geoscience Protocols

DataCatalog

SemanticCatalog

THREDDSCatalog

Catalog Ingestor

GeoscienceServers

Tools

WC

S

WF

S

CS/W protocol WCS protocol WFS protocol

Geoscience Protocols

14

Approach: CS/W1 access to THREDDS

• Ingest THREDDS metadata into database

• Modify GMU CS/W server to support search of the THREDDS metadata

1 Catalog Services for the Web

15

Project Milestones and Accomplishments (Geoscience)

• Designed THREDDS to ISO 19115/19115-2 mapping document

• Implemented THREDDS to CS/W metadata ingestor – Ingested the Unidata THREDDS catalog (weather forecast model) and

the NCDC THREDDS catalog– THREDDS metadata ingestor installed at GMU to ingest metadata

remotely

• Modified existing GMU CS/W server to include THREDDS mapped metadata

• Developed a rudimentary web client for searching THREDDS metadata through the CS/W server

16

Current Issue

Issue: CS/W client/server interoperability is difficult because of the wide variety of metadata profiles and variations of the profiles that can be adopted.

Need to facilitate GMU CS/W access from a variety of available clients from different organizations (CS/W clients with ebRIM Profile, WCS clients, OPeNDAP enabled clients, etc) that do not currently have this capability. – More than 10 commercial CSW clients have been

developed by various vendors and registered at OGC website.

– Have identified two existing clients (ESRI, GI-GO) to facilitate access to the GMU CS/W Server

17

Community Outreach

– Talks at • AGU Dec 2006• IGARSS 2006 & 2007, • CEOS WGISS meetings in 2006 & 2007• OPeNDAP Developers Conference, 2007• CEOP Science Meeting in March 2007• Unidata User and Policy Committees

– CEOP Satellite Data Server proof of concept demo at eGY booth at 2006 Dec AGU meeting

– Abstracts submitted to AGU 2007 for CEOP and Geoscience work