Towards Long-Term Archiving of NASA HDF-EOS and HDF Data - Data Maps and the Use of Mark-Up Language

Towards Long-Term Archiving of NASA HDF-EOS and HDF Data

Data Maps and the Use of Mark-Up Language

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Ruth Duerr, Mike Folk, Muqun Yang, Chris Lynnes, Peter Cao

http://www.hdfgroup.org/images/hdf_logo.jpg

Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland



Outline

• Background

• Data Mapping Project Description

• Plans and Early Results




Outline

• Background






A Concern

• The majority of the data from NASA’s Earth Observing System (EOS) have been archived in HDF Version 4 (HDF4) or HDF-EOS 2 format.

• HDF files have a complex internal byte layout, requiring one to use the API to access HDF data

• Long-term readability of HDF data depends on long-term allocation of resources to support the API




A Proposal from the Workshop Last Year

• Chris Lynnes noted that What was needed was a map to the

contents of an HDF file The output of the HDF4 tools (e.g., hdfls,

hdp, etc.) already provide much of the information needed

Extending these tools to create a map to the contents of the file might be feasible




Outline

• Background






Data Mapping Project Description

• Assess and categorize NASA holdings of HDF4 data

• Investigate methods of mapping HDF4 files• Develop requirements for tools to create

maps of HDF4 files• Create a prototype tool to create maps• Test the utility of these maps by developing 2

independent tools that use the maps to read real data




Data Mapping Project Description (continued)

• Assess the utility of this approach

• Document our findings

• Present results and options for proceeding to the user community

• Evaluate the effort required for a full solution that meets community needs

• Submit a proposal for that effort




Outline

• Background






Assess and Categorize NASA Holdings

While the volume of NASA data stored in HDF4/HDF-EOS2 format is measured in PB; the fraction of the total number of NASA data sets archived in HDF4/ HDF-EOS2 is “small”

• NASA provided a starter list of data sets held

• NASA data centers were requested to provide a list at a project briefing

• Results from each DAAC being compared to ECHO assessment of data sets using a .hdf extension




Assess and Categorize NASA Holdings (continued)

• Examples of each of the hdf4 data sets have been obtained and examined*

• Information kept summarized below:

• Product id/name• Data Center• Product Version• Multi-file product?• HDF/EOS info (if any)

HDF/EOS version Point info Swath info Grid info

• HDF info Version Raster image info Palette SDS info V data info Annotation

* For the most part




Assess and Categorize NASA Holdings (continued)

• Very preliminary findings Roughly 50/50 split between HDF-EOS

and plain HDF Point data is relatively rare and when found

is not accompanied by swath or grid data No indexes yet While a few products use the image types,

there are no palettes yet




Investigate Methods of Mapping HDF4 Files

• NSIDC and GES-DISC have provided THG sample data files• Preliminary priorities for capabilities to tackle:

Contiguous SDS Contiguous SDS with unlimited dimension Chunked SDS Compressed SDS Chunked and compressed SDS SDS and attributes Vdata and attributes Annotation Vgroup Raster image and attributes




Investigate Methods of Mapping HDF4 Files

• NSIDC and GES-DISC have provided THG sample data files• Preliminary priorities for capabilities to tackle:

Contiguous SDS Contiguous SDS with unlimited dimension Chunked SDS Compressed SDS Chunked and compressed SDS SDS and attributes Vdata and attributes Annotation Vgroup Raster image and attributes




Develop Requirements for Tools to Create Maps

• Maps will be XML-based

• A draft of a map format specification has been started




Create a Prototype Tool to Create Maps

• An iterative process is being used to create the prototype

• Each iteration adds the next capability from the prioritized list shown earlier

• At this point, the tool just creates a text description




Communications Plan

• Bi-weekly telecons with our sponsors (may move to monthly)

• Briefing to NASA Data Center managers held, expect to provide periodic updates

• Brief community at the HDF-Workshop and other relevant meetings (e.g., AGU)

• Submit a paper to the special issue of IEEE Transactions of Geoscience and Remote Sensing devoted to Data Archiving and Distribution

• Public wiki established but not yet populated




Summary

• We’ve started a project to assess and prototype the ability to create maps to the contents of HDF4 files that allow programmers to develop code to read data without using the HDF APIs

• We welcome community involvement

Towards Long-Term Archiving of NASA HDF-EOS and HDF Data - Data Maps and the Use of Mark-Up Language

Technology