Www.hdfgroup.org The HDF Group Update on NASA Wish List Mike Folk, Elena Pourmal, Kent Yang, Joe Lee The HDF Group Briefing to ESDIS April 4, 2013 4/4/2013HDF.

www.hdfgroup.org

The HDF Group

Update on NASA Wish List

Mike Folk, Elena Pourmal, Kent Yang, Joe Lee The HDF Group

Briefing to ESDISApril 4, 2013

4/4/2013 HDF Briefing to NASA 1

www.hdfgroup.org

Overview

• NASA Wish List is a set of tasks requested during last briefing.

• Total 48 items in the list.• Focus: 36 High Priority Items

• Two Divisions• General Maintenance, QA, and Support

(GMQS)• ESDIS Support, Operation Users Support, and

Studies (EOS)

4/4/2013 HDF Briefing to NASA 2

www.hdfgroup.org

GMQS Division

Accountable: Elena Pourmal

Categories:• HDF4 and HDF5 Libraries• General Tools (HDFView, h5dump, etc.)• General Support (Releases)

4/4/2013 HDF Briefing to NASA 3

www.hdfgroup.org

EOS Division

Accountable: Kent Yang

Categories:• ESDIS Software (including OPeNDAP)• Promotion• Studies

4/4/2013 HDF Briefing to NASA 4

www.hdfgroup.org

GMQS > Libraries

HDF4 Library• Item 12: Modify a compressed HDF4 data array.

HDF5 Library• Item 34: Add “parallel” gzip.

4/4/2013 HDF Briefing to NASA 5

www.hdfgroup.org

GMQS > Libraries > Item 12

Item Modify a compressed HDF4 data array

Action Reviewed and confirmed that HDF4 library cannot overwrite a part of the dataset (all or nothing). Created an example code that HDFView can use to resolve user’s request.

Original comments

HDF Java user reported this problem for his quality screen application.

Status In Progress

Comments • Update HDFView (version 2.10). • Do more investigation in HDF4. Fix bugs.• Provide recommendations to HDF4 library

users on how modify compressed datasets.

4/4/2013 HDF Briefing to NASA 6

www.hdfgroup.org

GMQS > Libraries > Item 34

Item Add parallel gzip

Action Reviewed parallel gzip code. It cannot be used with HDF5 as is. Talked to other stakeholders interested in the feature.

Original comments

GES DISC (?) looked at parallel gzip to improve performance.

Status In Progress

Comments Work on the design document for the feature.Come up with the estimate to do the work.Talk to all stakeholders to raise funds.

4/4/2013 HDF Briefing to NASA 7

www.hdfgroup.org

GMQS >Tools

h5dump• Item 9: Clean ASCII output from h5dump

HDFView• Item15: HDFView and netCDF-4 files with VL attributes• Item 4,18, 35: HDF-EOS plug-in and HDFView• Item 17, 22-27: New features in HDFView

4/4/2013 HDF Briefing to NASA 8

www.hdfgroup.org

GMQS > Tools > Item 9

Item Clean ASCII output for h5dump

Action RFC is sent to James Johnson and Chris Lynnes. The fix is made and ready for 1.8.11 release.

Original comments

Goal is to easily provide clean ASCII output, without attribute info., etc.

Status Done

Comments

4/4/2013 HDF Briefing to NASA 9

www.hdfgroup.org

GMQS > Tools > Item 15

Item NetCDF-4 files with string type attributes give variable length of size of one.

Action Fixed and changes committed to SVN repository.

Original comments

There doesn’t seem to be a way to create variable length strings.

Status Done

Comments It’ll be in HDFView 2.10, which is available upon request.

4/4/2013 HDF Briefing to NASA 10

www.hdfgroup.org

GMQS > Tools > Item 4

Item Obtain permission to redistribute HDF-EOS plugin with HDFView

Action Sent a letter to Dan and confirmed with Lou.

Original comments

Lou will look into getting permission. Lou may need to ask Dan for a letter of approval.

Status Done

Comments We can release HDF-EOS plug-in with HDFView in our website.

4/4/2013 HDF Briefing to NASA 11

www.hdfgroup.org

GMQS > Tools > Item 18

Item Synchronize EOS plugin with HDFView

Action We plan to come up with the RFC document that discusses the 4 solutions and we will solicit recommendations.

Original comments

Status Planned

Comments See the next slide for 4 solutions.

4/4/2013 HDF Briefing to NASA 12

www.hdfgroup.org

GMQS > Tools > Item 18

4/4/2013 HDF Briefing to NASA 13

1. Add the plug-in to the HDFView installer but have an option to install the plug-in or no.

2. Have a separate HDFView installer that packed with the plug-in and will install the plug-in automatically.

3. Have a Add plug-in in HDFView so that users can add the plug-in anytime.

4. Ask Abe to distribute HDFView with the plug-in (like #2) and we will provide a link to his release of HDFView.

www.hdfgroup.org

GMQS > Tools > Item 35

Item HDF-EOS Plug-in Functionality

Action Discuss the identified features with James Johnson and other ESDIS stakeholders (RFC)

Original comments

James asked if HDFView can have some functionality like Panoply.

Status Gathered requirements and identified list of features

Comments The common feature between HDF-EOS Plug-in and Panoply is the ability to display data on map.

4/4/2013 HDF Briefing to NASA 14

www.hdfgroup.org

GMQS > Tools > Item 25

Item New Features in HDFView - OPeNDAP

Action We’re working on RFC.

Original comments

You’ll probably get more requests for this in the future as far as MODIS goes.

Status In Progress

Comments Joe had discussion with Peter on OPeNDAP.

4/4/2013 HDF Briefing to NASA 15

www.hdfgroup.org

GMQS > Tools > Item 17, 24, 27, 23

Item New Features in HDFView:Dimension Scales, DOI, Graphics, GeoTIFF

Action Dimension Scales and DOI: AprilGraphics (fill values, etc.): MayGeoTIFF: June

Original comments

Status Scheduled RFC

Comments GeoTIFF work depends on Item 8, which will be discussed in EOS division later.

4/4/2013 HDF Briefing to NASA 16

www.hdfgroup.org

GMQS > Tools > Item 22, 26

Item New Features in HDFView:CF Conventions, New web service queries

Action The survey report was completed and published.

Original comments

Peter will do this as part of writing up results of survey.

DAACS are starting to provide services for WCS, OpenSearch, etc. but there are very few clients that support them.

Status No plan

Comments http://hdfgroup.org/hdf-java-html/hdfview/hdfview_surevey_report_2013.pdf

4/4/2013 HDF Briefing to NASA 17

www.hdfgroup.org

GMQS: General Support Issues

• Item 1: Supported OS and compilers• Item 3: Cygwin support is needed by HDF-EOS• Item 20: Tools of value to users

4/4/2013 HDF Briefing to NASA 18

www.hdfgroup.org

GMQS > General Support > Item 1

Item Need to know what OS and compilers are supported.

Action • OS and compilers are planned for each maintenance release.

• An HDF developer is assigned at the time of release planning for support issues on a particular platform.

Original comments

Status In Progress

Comments See our wiki page in the next slide.

4/4/2013 HDF Briefing to NASA 19

www.hdfgroup.org

GMQS > General Support > Item 1

4/4/2013 HDF Briefing to NASA 20

www.hdfgroup.org

GMQS > General Support > Item 3

Item Cygwin support is needed by HDF-EOS.

Action Cygwin is tested daily for HDF4 and HDF5.

Original comments

Status Done

Comments

4/4/2013 HDF Briefing to NASA 21

www.hdfgroup.org

GMQS > General Support > Item 20

Item Tools of value to users

Action • We updated and reorganized tools pages* to help users to find the right tool.

• We’ve been working with DAACs to raise awareness of tools through telecons and direct contacts.

Original comments

Earthdata.nasa.gov describes tools available to DAACs. What we do here should be integrated with that and inform them.

Status In Progress

Comments *http://www.hdfgroup.org/products/hdf5_tools/toolsbycat.html

4/4/2013 HDF Briefing to NASA 22

www.hdfgroup.org

10 Minute Break

4/4/2013 HDF Briefing to NASA 23

www.hdfgroup.org

EOS > ESDIS Software

OPeNDAP HDF5 Handler• Item 19:Support NetCDF-4 model in HDF5 handler (Done)• Item 32:HDF5 handler maintenance

OPeNDAP HDF4 Handler• Item 28:HDF4 handler maintenance

OPeNDAP fileout_netcdf Handler• Item 5: Fix fileout_netcdf Handler. (Done) • Item 42:Dump as NetCDF-4 for fileout_netcdf

4/4/2013 HDF Briefing to NASA 24

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 25

EOS > ESDIS Software > Item 19

Item Make the HDF5 handler support the netCDF-4 data model

Action Kent added the support to HDF5 OPeNDAP handler in December

Original comments

GES DISC had netCDF-4 files that would like to be served by Hyrax

Status Done

Comments A RPM was created and was sent to GES DISC in Jan. Everything works fine.

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 26

EOS > ESDIS Software > Item 32

Item HDF5 OPeNDAP handler maintenance

Action Improve code quality and tests for HDF5 OPeNDAP handler (default option only)

Original comments

Status Will get started on June 1st, 2013

Comments

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 27

EOS > ESDIS Software > Item 28

Item HDF4 OPeNDAP handler maintenance

Action Improve the code quality and other maintenance

Original comments

Need to improve the code readability, error handling etc.

Status In Progress

Comments • We replaced the new[ ] with vector according to the suggestions of OPeNDAP team.

• We added support in handling MEaSUREs VIP product and the version 6 of MODIS geo-location files.

www.hdfgroup.org

EOS > ESDIS Software > Item 5

Item OPeNDAP’s file netCDF out module is not working in Hyrax 1.8

Action Investigate the problem: Then fix the bug or make recommendation.

Original commenats

Chris Lynnes and James Johnson: This is the first priority.

Status Done

Comments GES DISC, LP DAAC and MODAPS all installed this version and they reported the file netCDF dumper works fine.

4/4/2013 HDF Briefing to NASA 28

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 29

EOS > ESDIS Software > Item 42

Item netCDF-4 (as opposed to netCDF-3) as a file dumper output possibility for OPeNDAP.

Action Make this item a high priority based on the conversation with GES DISC (Previous priority: medium high)

Original comments

Status Will work on this in July

Comments Using netCDF-4 classic(with the compression) as the file dumper output becomes more important.

www.hdfgroup.org

EOS > ESDIS Software

H4CF Conversion Toolkit• Item 21:H4CF Conversion Toolkit

HDF-EOS5 to NetCDF-4 Tool• Item 29:Retire HDF-EOS5 to NetCDF-4 conversion tool (Done)

4/4/2013 HDF Briefing to NASA 30

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 31

EOS > ESDIS Software > Item 21Item Improve the H4CF conversion toolkit

Action Upgrade the H4CF conversion toolkit to make it consistent with the way how the HDF4 OPeNDAP handler handled the NASA HDF-EOS2 and HDF4 data.

Original comments

• Will update naming conventions, fix critical bugs, improve features for popular products such as MODIS.

• Will implement a test suite and update the configuration process.

Status In Progress. ( Release plan: June 15)

Comments • Good progress upgrading the toolkit to follow CF conventions as the HDF4 OPeNDAP handler.

• Implemented comprehensive test suite• Updated configuration and conversion to netCDF3

and netCDF4-classic. • MISR data & coding style still need to be improved.

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 32

EOS > ESDIS Software > Item 29

Item Retire "HDF-EOS 5 to netCDF-4" conversion tool

Action Made an announcement and waited for replies

Original comments

HDF-EOS5 augmentation tool is a superset of this tool.

Status Done

Comments • Announced on Dec. 5th, 2012. • No response by Feb. 15th, 2013, so we

announced tool’s official retirement on Feb. 15th, 2013.

www.hdfgroup.org

EOS > Operation Users Support

Guide Data Producers with HDF Best Practices• Item 11: Document issues for NASA HDF4/HDF-EOS2 products

Make HDF-EOS Easy to Build• Item 30:CMake for HDF-EOS• Item 31:Autoconfiguration for the EOS Toolkit

4/4/2013 HDF Briefing to NASA 33

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 34

EOS > Op. Users Support > Item 11

Item Publish the document describing issues for NASA HDF4 (EOS2) products

Action Write up the issues in a white paper. Make available on website. Push to all relevant stakeholders.

Original comments

Needs to be a clear and concise document. "Guide to good use of HDF." "Best practices," something like that. Perhaps create a "best practices" section on the web site.

Status In Progress

Comments We are working on the outline of the white paper now.

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 35

EOS > Op. Users Support > Item 30

Item CMake for HDF-EOS

Action

Original comments

Clarify with Abe what it means.

Status Not started yet, assigned the person to work on this

Comments Communicated with Abe several times.

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 36

EOS > Op. Users Support > Item 31

Item Autoconfiguration for the ECS toolkit.

Action Meet with Abe and look at all of his tools in the toolkit (including libraries) and create a roadmap for implementing autoconfiguration.

Original comments

Status Not started yet, assigned the person to work on this

Comments Communicated with Abe several times.

www.hdfgroup.org

EOS > Promotion

HDF-EOS website• Item 16: Hard to find zoo link (Done)• Item 2: Promote example pages on HDF-EOS web site• Item13: Publicize HDF-EOS website

4/4/2013 HDF Briefing to NASA 37

www.hdfgroup.org4/4/2013 HDF Briefing to NASA 38

EOS > Promotion > Item 16

Item Hard to find zoo link at hdfeos.org

Action Improve the ability to find example pages

Original comments

Status Done

Comments Put the zoo link at the top section in examples page with a description.

Put the zoo link in the front page’s DID YOU KNOW tip section.

www.hdfgroup.org

EOS > Promotion > Item 2

4/4/2013 HDF Briefing to NASA 39

Item Promote example pages on HDF-EOS web site

Action With Lou's help, contact Earth data pages about  putting pointers to these pages. Contact DAAC user services working groups. Work with ESDIS to submit an article to Earth Observer.

Original comments

Status In Progress

Comments 1) A link* to the comprehensive page at hdfeos.org has been added to earthdata.nasa.gov.

2) We worked with ESDIS to submit an article about the comprehensive example to Earth Observer.

*http://earthdata.nasa.gov/user-resources/tutorials-and-examples

www.hdfgroup.org

EOS > Promotion > Item 13

4/4/2013 HDF Briefing to NASA 40

Item Publicizing HDF-EOS website

Action Write brochure about website. Get help distributing brochure at conferences such as AGU. Contact ESDIS staff about this.

Original comments

Status In Progress

Comments We contacted with ESDIS staff and we are planning to create a NASA brochure about the HDF-EOS comprehensive example page.

www.hdfgroup.org

EOS > Studies

Joe• Item 6: Open Source Apache Incubator pilot project

Kent• Item 8: GeoTIFF investigation• Item 33: A specification for CF metadata for HDF5 and HDF5 is needed

Mike• Item10: DOI support in HDF4 and HDF5

4/4/2013 HDF Briefing to NASA 41

www.hdfgroup.org

EOS > Studies > Item 6

4/4/2013 HDF Briefing to NASA 42

Item Open Source Apache Incubator Pilot Project

Action Legal issues like NOSA license were reviewed.A new project proposal* for HDF to GeoTIFF converter is being written.

Original comments

Status In Progress

Comments • We are recruiting team members and sponsors!• Champion: Paul Ramirez (NASA JPL)• Mentor: Chris Mattman (NASA JPL)

*http://wiki.apache.org/incubator/HotdoGProposal

www.hdfgroup.org

EOS > Studies > Item 8

4/4/2013 HDF Briefing to NASA 43

Item GeoTIFF investigations

Action Write white paper, jointly with Abe, evaluating GeoTIFF support needs and opportunities.

Original comments

Need to determine which products are important for GeoTIFF support.

Status In Progress

Comments • Collected GeoTiFF information from several data centers and Abe.

• Held a phone discussion with Abe about the paper.

• Started on outline of white paper.

www.hdfgroup.org

EOS > Studies > Item 33

4/4/2013 HDF Briefing to NASA 44

Item A specification for CF metadata for HDF4 and HDF5

Action Write a paper on the current status of work on CF conventions, then decide what is best.

Original comments

• We need to better understand who is doing what on this topic. Ted Habermann is working on it. There will be a discussion on it at the ESDSWG meeting.

• Would like the spec to be made available. Would be good for ESDIS to publish as a tech note/standard.

Status In Progress

Comments We participated in the telecon of ESDS HDF5 and provided material and suggestions about CF and interoperability to the WG chairs.

www.hdfgroup.org

EOS > Studies > Item 10

4/4/2013 HDF Briefing to NASA 45

Item Write paper on DOI support in HDF5

Action White paper completed, ready for review.

Original comments

Status Done

Comments Highlights:• In EOS, DOIs assigned at

program/mission/collection/product level• DOI API and BNF grammar proposed• Implementation estimated at 100 hoursNext steps?

www.hdfgroup.org

Thank You!

• Questions? • Comments?

4/4/2013 HDF Briefing to NASA 46

www.hdfgroup.org

Acknowledgements

4/4/2013 HDF Briefing to NASA

This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

47