ESIP Federation Meeting 15-18 July 2008 Data and Metadata Standards for GHRSST: present to future ESIP Federation Meeting 15-18 July 2008 Univ. of New.

ESIP Federation Meeting 15-18 July 2008

Data and Metadata Standards for GHRSST: present to future


Univ. of New Hampshire, Durham, NH

Ed Armstrong

Jet Propulsion Lab/CalTech

Physical Oceanography DAAC

Pasadena, California


Data throughput…G

b D

ay-1


User Base

Power Users:• NAVO

• UK Met Office

• NASA JPL

• Canada Meteorological Agency

• Danish Meteorological Institute

2021371 621184

62141

147

12327112327

1391184

5

894818

2741

234

2

4

3

2

25

1

159

4

Australia Belgium Business Organizations

Brazil Canada Chile

Commercial Czech Republic Germany

Denmark Educational Estonia

Spain Finland France

US Government Greece Croatia/Hrvatska

Indonesia India International Organizations

Italy Japan Jet Propulsion Laboratory

Korea, Republic of Morocco US Department of Defense

Mexico National Aeronautics and Space Administration (US) Networks

Netherlands Norway New Zealand

(Non-Profit) Organizations Peru Poland

Portugal Romania Russian Federation

Sweden Singapore Turkey

Taiwan United Kingdom United States

Unknown (numeric host id) South Africa


Product Specifications - 2003

• netCDF 3.6 • Self describing format

– Popular in the ocean modeling and NWP community

• Direct access– Improves usability: L2P products can have over 20 arrays

• Portable– GHRSST is an international project with data produced on a variety of platforms

• “bundled” with CF/COARDS compliant metadata• Limitations:

– No internal compression. Externally compressed (bzip2 or gzip)– No chunking


Metadata Specifications - 2003

• NASA Directory Interchange (DIF) format • Compact, well defined user history, simple to implement• A granule level XML metadata (File Record [FR]) for each file made by producer

– typical entries are dynamic such as time and position – e.g., <Southernmost_Latitude></Southermost_Latitude>

• A product level XML metadata (Dataset Description [DSD])– typical entries are more static contacts, document links, and summary statements

– e.g., <Summary></Summary>

• Some “custom” tags added. For example, <file_compression>• These are stored as separate databases at the JPL GDAC Master Metadata Repository (MMR)

– mySQL relational database for search, query, and assembly


Metadata Specifications - 2003

• FGDC conversion• All GHRSST products are transferred to NOAA Longterm Stewardship and Reanalysis Facility (LTSRF) after a period of 30 day• NOAA has requirement for FGDC compliant metadata for its products. No data ingested without metadata.• A conversion of the FR and DSD records to FGDC is performed on file-by-file basis

– Mappings from DIF to FGDC done using XSLT style sheets– For L2P data multiple files of same product type in same FGDC record. Up to 288 files per FGDC record.


Mapping Anomaly

• Some mappings are not clean. FR record:<Personnel>

<Role>Technical Contact</Role>

<First_Name>Ed</First_Name>

<Last_Name>Armstrong</Last_Name>

<Email>[email protected]</Email>

<Address>

Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA

</Address>

</Personnel>

• mapped FGDC record:<address>


</address>

<city>


</city>

<state>


</state>

<postal>


</postal>

<country>


</country>


Measurements of SST (netCDF CF1.3)

Dep

th

10m

1 mm

1 m

10 m

SSTskin – SST10m (K)

1. Night-time (or strong winds) profile in red2. Day time situation, strong solar radiation and light winds

sea_surface_skin_temperature

sea_surface_subskin_temperature

sea_water_temperature(Attribute:depth)

Sea_surface_foundation_temperature

-0.17 0 0.5 1.0 1.5 2.0 2.5 3.0

Nig

ht-

tim

e (

or

str

on

g w

ind

s)

pro

file

in

red

surface_temperature


CF 1.3 Specification for SST standard names

• surface_temperature (SSTint)

• sea_water_temperature (SSTdepth)

• sea_surface_skin_temperature; K

• sea_surface_subskin_temperature; K

• sea_surface_foundation_temperature: K


Future product development

• NetCDF version 4• Benefits

– Internal compression and chunking (from HDF storage model)– Support for Parallel I/O– Backward compatibility with v3.6– The lure of “new stuff”

• Advantages must be weighted against a format that has proven metrics and use

– Operationally it is difficult to implement a new file format– The user may (will) complain. New software and libraries to install etc.

• Evaluated this summer, with a decision by next GHRSST Science Team Meeting June 2009 for adoption


Future metadata requirements

• European Space Agency will require metadata for next generation Sentinel satellite derived products to be ISO 19115-2 compliant. This directly impacts GHRSST.

• Also required for using GHRSST products in WMO Integrated Global Observing Systems (WIGOS) systems (a GEOSS component)

• What to change in our metadata production system ?• We have an investment in something that works. We can’t break everything and start

from scratch.• The way forward is likely to generate mappings to ISO 19115 similar to FGDC

conversion.– Currently reviewing our metadata architecture (FR and DSD XML records) to see what needs to

be added/modified– Review must be completed in the next months. A written protocol by the end of the year.– What part of ISO 19115 to conform to?? North American profile…Remote Sensing extensions.– ISO metadata embedded in netCDF v4 GHRSST file ??


Summary

• GHRSST has demonstrated significant achievements with regard to data and metadata standards on a relatively modest budget

• Although focused on SST discipline, is serves as a model for a working GEOSS• Supporting Science and Applications through well thought out and implemented best practices

• One of the challenges will be to evolve with new technologies wrt to format and metadata standards

• ISO 19115– Relatively straightforward once the mappings are determined

• netCDF v4 – More challenging. More investigation needed.

• Others: Search, Discovery, Imaging– THREEDS: data cataloguing– JPL Datacasting: RSS-like data subscription– Swath granule subset– ECHO metadata inventory– OPeNDAP– WMS POET


Backups

•


(A)ATSR reduces uncertainty in global average SST (John Kennedy, Nick Rayner)


GDAC overview


• Polar Orbiting infrared has high accuracy & spatial resolution • Geostationary infrared has high temporal resolution• Microwave Polar orbiting has all-weather capability

GHRSST-PP L4 products

(A. Harris)


SST system quality control and uncertainty estimation

AnalysisProduct

s net

CD

F f

orm

at

Input SST

data of Varied format& with

no errors

Datamerging

andanalysis

SST Obs.

Products net

CD

F f

orm

at

Add ancillary

QC SSES& dynamic

flags

The GHRSST-PP Strategy

SSTObservations

ApplicationsL3

L1P Radiance data


GDS-v2.0Rev 0.1June 2008Generate L2P

data products (L2Pc -> L2P)

Generate L4 analysed data products (e.g.,

SSTfnd) based on L2P data input.

Satellite SST data

Ingest & QC input data streams

Auxiliary data

Reference data fields

Generate L3Pdata products


Analysed L4 SST data products

Rolling and long-term

archive system

Derive Single Sensor error

statistics (SSES)

Satellite and in situ MDB

Single Sensor Error

Statistics

Generate MDB records

MDB/SSES system

MDB recor

ds

Diagnostics, metrics and GDS data product

validation

HR-DDS archive GDAC &RDAC

L4 SST HR-DDS data granules

L2P HR-DDS data

granules

Extract HR-DDS L2P granules

User data access through LAS/DODS HR-

DDS server

Extract HR-DDS L4 SST granules

HR-DDS system

In situ data

CDR SST data products

Re-Analysis f L2P/L3P/L4

L3P SST data products


L2P/L2Pc SST data products

Metricsdefinition

s

GMPE SST ensembleproducts

Generate GMPE

products


Main data flow

HR-DDS data flow

MDB/SSES data flow

RAN data flow

GMPE data flow


Auxiliary data

GMPE systemRAN system

ESIP Federation Meeting 15-18 July 2008 Data and Metadata Standards for GHRSST: present to future ESIP Federation Meeting 15-18 July 2008 Univ. of New.

Documents

usa slide

future esip federation

temperature slide

usa jet propulsion laboratory

oak grove dr

fgdc compliant metadata

metadata specifications

california slide