Www.hdfgroup.org The HDF Group November 3-5, 2009 HDF-OPeNDAP Project Update HDF/HDF-EOS Workshop XIII1 Joe Lee and Kent Yang The HDF Group James Gallagher.

www.hdfgroup.org

The HDF Group

November 3-5, 2009

HDF-OPeNDAP Project Update

HDF/HDF-EOS Workshop XIII 1

Joe Lee and Kent Yang The HDF Group

James Gallagher OPeNDAP, Inc.

The 13th HDF and HDF-EOS Workshop November 3-5, 2009

www.hdfgroup.orgNovember 3-5, 2009

Outline

• Introduction to OPeNDAP

• What’s New?

oHDF5-OPeNDAP Handler (minor)

oHDF4-OPeNDAP Handler (major)



Access HDF via FTP


Users

FTP Clients

IDL

FTP Servers

HDF Files

AIRS/OMI

FTP

Local HDF Files

Aqua/Aura


Client Library(libnc_dap)

Data Access Protocol (DAP) via Internet

Server(Hyrax)

Handler(hdf4/5_handler)

Remote Data(HDF4/5)

View Data (User)

OPeNDAP Concept

4

Visualization Tools

(gradsdap)

Typical Remote Access

FTP/HTTP

Visualization tools

(e.g., grads)

HDF/HDF-EOS Workshop XIII


Access HDF via OPeNDAP


Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF Files

Hyrax AIRS/OMI

DAP

libnc-dap

Visualization Tools

HDF HANDLER

Aqua/Aura


What’s New?

HDF5-OPeNDAP Handler Work



HDF5 Handler Update


Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF5 Files

Hyrax OMI Grid

DAP

libnc-dap

Aura

OMI Swath

HDF5 HANDLER Update

Visualization Tools


HDF5-OPeNDAP Handler Update

• Support Aura HDF-EOS5 OMI Swath

• Access some files that follow HDF5

Dimension Scale Specification

• Change in Short Name Option


RadiativeCloudFraction

A3RadiativeClou


Summary: Updated HDF5 Handler

• Official Release: October 6, 2009

• http://opendap.org/hdf5_handler



What’s New?





• Problems with the old HDF4 Handler

• Our Work

• Limitations

• Future Work



Old HDF4 Handler Problem

• OPeNDAP Visualization Tools CAN

NOT display most NASA HDF-EOS2

and HDF4 data



Old HDF4 Handler Problem


Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax AIRS Grid

DAP

libnc-dap

Aqua

Old HDF4 HANDLER

Visualization Tools


Our Work

• Let Visualization Clients Access HDF-EOS2 and HDF4 files



What’s Missing?

• My house costs 1. • Data – Computer Science

• int my_house = 1; // Use boolean to save?


• My house in Hawaii costs $1M. • Information – Earth Science

• Geo-Location, Geo-Location, Geo-Location

• Unit, Scale factor


What’s Missing?

• Clients expect certain data structureso DAP Grids and Shared dimensions

• Clients expect some key attributeso CF conventions



Solution: Metadata Parser


• Goal: Meet the Client’s Requirements

• Parse the structMetadata too generate the DAP Grids and Shared Dims

o generate the CF-compliant attributes


New HDF4 Handler (parser)


Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax AIRS Grid

DAP

libnc-dap

Aqua

Old HDF4 HANDLER

Visualization Tools

New HDF4 HANDLER (parser)


Demo: Parser Solution


Problem: Only a few HDF-EOS2 files work!

AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdfhttp://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10MODIS GridMOD08_D3.A2008284.005.2008289105354.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html

AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/

AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdfMODIS Swath

MOD06_L2.A2001222.0025.004.2003127111841.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html


New Challenges for HDF4 Handler

• Hundreds of HDF-EOS2 and HDF4 products

oGrid Projections

o Swath Dimension Map


www.hdfgroup.orgNovember 3-5, 2009 HDF/HDF-EOS Workshop XIII 21

HDF-EOS2 Grid projections• Geographic• Sinusoidal• Polar Stereographic• etc.

New Challenges

HDF-EOS2 Swath Dimension Map


Solution: Use HDF-EOS2 Library


• Goal: Support more HDF-EOS2 products

• Use HDF-EOS2 Library too handle different Grid projections

o handle Swath dimension map


New HDF4 Handler (EOS2 library)


Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax MODIS Swath

DAP

libnc-dap

Aura

MODIS Swathw/ Dim. Map

New HDF4 HANDLER (parser)New HDF4 HANDLER (EOS2 library)

Visualization Tools

New HDF4 Handler (Parser)


Demo: HDF-EOS2 Library Solution


Swath with Dimension Map


Two Solutions

Metadata Parser

• Pro: Simple installation

• Con: Limited products

HDF-EOS2 Library

• Pro: More products

• Con: HE2 library installation



Two Solutions - Implementation

Metadata Parser

• hdf5_handler

HDF-EOS2 Library

• h4toh5 tool


• Adapted the implementation of previous software packages


Solution Comparison



Live Demo

• Tutorial Exampleo http://hdfeos.org/software/idv.php

• Demo Serverso http://hdfgroup.org/projects/opendapo http://hdfgroup.org/projects/opendap4



Limitations

• 7 Reasons for Visualization Failures

• Sample Files for Visualization Failures

o http://hdfgroup.org/projects/opendap4/limits.html



The Tip of the Iceberg


The current HDF4 handler

CAN NOT visualize most of NASA HDF-EOS2 / HDF4 files


More Challenges for HDF4 Handler

• Tackle limitations and other challenges.

• Handle NASA HDF4 (e.g.,CERES) files.

• In 2010



Summary: Enhanced HDF4 Handler

• Official Release on Oct. 6, 2009.

• http://opendap.org/hdf4_handler

• Support some HDF-EOS2 Grid / Swath files

• Use HDF-EOS2 library

• Future work: support more NASA HDF-EOS2

/ HDF4 files


www.hdfgroup.org

The HDF Group

November 3-5, 2009

Thank You!



Acknowledgements

This work was supported by cooperative agreement number NNX06AC83A, NNX08AO77A and

NNX06AG75A from the National Aeronautics and Space Administration (NASA).

Any opinions, findings, conclusions, or recommendations expressed in this material are

those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space

Administration.


www.hdfgroup.org

The HDF Group

November 3-5, 2009

Questions/comments?


Www.hdfgroup.org The HDF Group November 3-5, 2009 HDF-OPeNDAP Project Update HDF/HDF-EOS Workshop XIII1 Joe Lee and Kent Yang The HDF Group James Gallagher.

Documents