ESDIS 2008 tasks status

04/21/23 1

ESDIS 2008 tasks status

Kent Yang, Mike Folk

The HDF Group

March 31, 2009

Annual briefing to ESDIS

04/21/23 2

Summary of tasks

• Total 28 tasks: Most tasks received from the annual briefing and

the workshop 4 NPP/NPOESS tasks transferred to NPOESS

project 10 tasks have been finished 6 tasks are on-going activities 8 tasks are in progress

Annual briefing to ESDIS

04/21/23 3

Finished tasks highlights

Annual briefing to ESDIS

HDF-EOS automatic configuration

• Both HDF-EOS2 and HDF-EOS5• Added the szip support

Both encoder and decoder Decoder only Without szip

• Added Fortran wrapper support• Updated source code under Fortran testdrivers

directory • Updated installation files for automatic

configuration

04/21/23 4Annual briefing to ESDIS

04/21/23 5

Investigation of HDF Support in Some Open

Source Software Packages

Annual briefing to ESDIS

04/21/23 6

Five open source packages

• PyHDF Python interface to HDF4 http://pysclint.sourceforge.net/pyhdf/

• Geospatial Data Abstraction Library (GDAL) Translator library for Raster Geospatial Data Formats Supports about 100 file formats http://gdal.org/

• NCAR Common Language (NCL) Interpreted Language for Data Analysis and Visualization http://ncl.ucar.edu/

• Grid Analysis and Display System (GrADS) Interpreted Language for Data Analysis and Visualization http://iges.org/grads/

• GNU Data Language (GDL) Interpreted Language for Data Analysis and Visualization Data Analysis and Visualization http://gnudatalanguage.sourceforge.net/

Annual briefing to ESDIS

04/21/23 7

Evaluation - File Formats

HDF4 HDF5

NCL SDSNetCDF4 classic

GrADS SDS

PyHDFSDS, Vdata, Vgroup

GDL SDS, VdataGeneral HDF5 objects

GDAL SDS, GR, EOS2General HDF5 objects

04/21/23 7Annual briefing to ESDIS

04/21/23 8

Evaluation - Documentation

Document Example

NCL sufficientMany examples with screenshots

GrADS sufficient Several screenshots

PyHDFsimple, but sufficient

Several simple examples

GDL IDL documents IDL documents

GDAL SufficientA few examples in TutorialWorking applications included

04/21/23 8Annual briefing to ESDIS

04/21/23 9

Evaluation - Installation

binaries source build

NCLAIX, IRIX, Linux, MacOS X, Solaris, Cygwin

not tested

GrADSLinux, MacOS X, SunOS

not tested

PyHDF Requires few settings

GDLRequires several packagesAssumes HDF4 and HDF5 without SZIP

GDAL Requires one patch

04/21/23 9Annual briefing to ESDIS

Usage example - GrADS

04/21/23 10

$ gradsga> xdfopen AE_RnGd.xdfga> set lon -180 180ga> set gxout shadedga> display RrLandRainga> draw title Total Rain Rate over Land in July 2007

Annual briefing to ESDIS

04/21/23 11

Performance evaluation of using SSHFS-FUSE to

access HDF5 files

Annual briefing to ESDIS

• NFS requires a special permission. General users are not allowed.

remote systemremote system local systemlocal system

Using NFS

04/21/23 12

NFSNFS

Annual briefing to ESDIS

• What if only a small part is necessary from a huge file?

local systemlocal system

1. Downloading a whole file

04/21/23 13

localfilesystem

localfilesystem

remote systemremote system

ftp, sftp, …

Annual briefing to ESDIS

• Given that permission to access FUSE is granted, general users can mount remote filesystems.

remote systemremote system local systemlocal system

2. Using FUSE+SSHFS

SSHFSFUSE

SSHFSFUSE

04/21/23 14Annual briefing to ESDIS

Accessing HDF5 through SSHFS-FUSE

• Experiment:• For 4 files,

Copy a whole file Copy one dataset in a file Copy one small hyperslab in a dataset

04/21/23 15Annual briefing to ESDIS

• elapsed time ratio

- SSHFS / downloading a whole file and subsetting

04/21/23 16

Results of the experiment

sshfs/download&local File 1

Whole file 3.42

One dataset 0.23

One hyperslab 0.10

SSHFSconsumedless time

SSHFSconsumedless time

SSHFSconsumedmore time

SSHFSconsumedmore time

Annual briefing to ESDIS

Other finished tasks

• The third dimension support of HDF-EOS The documentation of HDF-EOS needed to be

updated

• HDF4.2r4 release Static library with static external libraries C shared libraries

• HDFView 2.5 release Fixed bugs of ‘switch base index’ and ‘fill value’

• Help build EOS toolkit on solaris• NetCDF4/NetCDF3/HDF5 performance study

http://www.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf

04/21/23 17Annual briefing to ESDIS

04/21/23 18

On-going tasks overview

Annual briefing to ESDIS

On-going tasks overview

• Applications/other technologies(in separate talks) Support OPeNDAP HDF4 handler for OPeNDAP Independent HDF4 mapping

• HDF5 library Support backward/forward compatibility

• Tests with other software packages Maintain HDF-EOS2 and HDF-EOS5 daily tests Regression tests with IDL and Matlab

04/21/23 19Annual briefing to ESDIS

04/21/23 20

Tasks in progress• Some tasks almost finished• Some tasks we need to know the next step• Some tasks are re-scoped into our next year’s plan

Annual briefing to ESDIS

Special value RFC

• 90% done. We have written the RFC We then sent out to NASA users and hdf-forum A decision has been made based on feedback

o Provide examples to demonstrate how to handle special values

04/21/23 21

Hi Kent,

Thank you for your message. I appreciate your taking the time to

communicate with us.

I appreciate even more that your decision was guided by the needs,

concerns, and interests expressed by hdf5 data producers.

- Paul

Annual briefing to ESDIS

High Level Language Support

• Motivation A better high-level language support for HDF5

such as Python, Perl, Ruby etc.

• Progress Investigated SWIG-HDF5 interface Held a discussion in HDF-EOS workshop Formed a High level language wiki to further

discuss the general issuehttp://wiki.hdfgroup.org/High+level+language+supports+for+HDF5

Suggestions ?

04/21/23 22Annual briefing to ESDIS

Simplifying installation procedures

• Request Investigate simplifying build installation procedures,

including 3rd party libraries

• Progress The HDF group is investigating the use of RPM to

simplify the whole process A RFC is being written

04/21/23 23Annual briefing to ESDIS

Using NetCDF4 APIs to read HDF-EOS5

• Motivation: To allow the access of HDF-EOS5 files via

NetCDF4 APIs To ensure that HDF-EOS5 files still accessed by

the HDF-EOS5 library

04/21/23 24Annual briefing to ESDIS

Using NetCDF4 APIs to read HDF-EOS5

• Progress: Wrote an investigation report on this issue Discussed with Aura, netcdf and EOS developers Finalized a solution

o Augmented the EOS5 file to support dimensionso Relaxed creation order requirements of NetCDF4 to

access HDF-EOS5 data A prototyped augmentation tool is implemented

• Work to be done: Wait for NetCDF4 developer to release a snapshot to

support the relax of creation order Write a netCDF4 fortran example to read EOS5 files

04/21/23 25Annual briefing to ESDIS

HDF5 to jpeg tool prototype

• Request from HDF-EOS5 to support the conversion in HDF-EOS2 to HDF-EOS5 tool

• h52jpeg traverses an HDF5 file and saves each HDF5 image found to a jpeg file

• h52jpeg is only a prototype Not fully tested Not fully documented Not released with HDF5 library Source code is available only at a special branch

http://svn.hdfgroup.uiuc.edu/hdf5/branches/jpeg_converter/hl/tools/h52jpeg/

• Is there any request to make this tool fully supported?

04/21/23 26Annual briefing to ESDIS

Command line format conversion tools

• Request: In order of priority - netCDF, geotiff, grib and ascii

• Progress: This is a big task. We sent out a newsletter to ask if there are any

available tools; no useful responses received. We were implementing an HDF-EOS5 to netCDF4

conversion tool.

• Questions: Is this task still necessary? If yes, what’s the priority of this task?

04/21/23 27Annual briefing to ESDIS

EOS access example codes and tools

• Requests: Tools such as HEG, IDL , Matlab etc. Well-commented example codes

• Progress: Some experimental work has been done

• Plan: The work is re-scoped to be the focus of the new

year

04/21/23 28Annual briefing to ESDIS

Review dimension scale model

• Requests: Review dimension scale model and identify use

cases

• Plan: We will start investigating this after the briefing

• Any suggestions?

04/21/23 29Annual briefing to ESDIS

Thank You!

04/21/23 30Annual briefing to ESDIS

Acknowledgements

This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A.

Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.

04/21/23 31Annual briefing to ESDIS

Questions/comments?

04/21/23 32Annual briefing to ESDIS