HDF Update

- 1 -

University of Illinois at Urbana-Champaign

HDFHDF

Mike Folk

HDF-EOS Workshop IV

Sept. 19-21, 2000

HDF UpdateHDF Update

HDFHDF

- 2 -


HDFHDF

TopicsTopics

• What is HDF?

• HDF community

• HDF4– past and future work

• HDF5– HDF5-HDF5 interoperability– Activities and growth– Tools and utilities– Focus for 2001

- 3 -


HDFHDF

NCSA HDF MissionNCSA HDF Mission

To develop, promote, deploy, and support To develop, promote, deploy, and support open and free technologies that facilitate open and free technologies that facilitate scientific data storage, exchange, access, scientific data storage, exchange, access,

analysis and discovery. analysis and discovery.

- 4 -


HDFHDF

What is HDF?What is HDF?

• Format and software for scientific data

• Stores images, arrays, tables, etc.

• Emphasis on storage and I/O efficiency

• Free and commercial software support

• Emphasis on standards

• Users from many engineering & scientific fields

- 5 -


HDFHDF

HDF PeopleHDF People

• NCSA– QA, maintanance & support– Library development & performance– Tools development

• EOS & ASCI– Users, applications and tools– Performance

• Volunteers from all over– Users, applications & tools

- 6 -


HDFHDF

Who is supporting HDF?Who is supporting HDF?

• NASA/ESDIS– Earth science applications, instrument data– All aspects of data management

• DOE/ASCI (Accelerated Strategic Computing Init.)– Simulations on massively parallel machines– Emphasis on parallel I/O performance, functionality

• NCSA– Grid, Vis, other R&D, user support

• Others– Applications, support, some R&D

- 7 -


HDFHDF

HDF4HDF4

• User support, tools, maintenance, QA

• JHV (Java viewer) - two update releases

• HDF4 dumper (hdp) - many improvements

• Next HDF4 Release (Oct. 2000)– Bug fixes– Chunking for General Raster (GR) images– Unlimited dimensions for SDS– GIF <--> HDF4 conversion tools

- 8 -


HDFHDF

HDF4 ArchivingHDF4 Archiving

• New File Format and Specification Manual– Facilitate long term maintenance of HDF4– Necessary for bug fixing, adding new features– Decreases learning curve for new developers– Available soon in

• PDF, HTML and PS formats

- 9 -


HDFHDF

HDF5HDF5

- 10 -


HDFHDF

The growth of HDF5The growth of HDF5

• Users in 27 countries

• Users include– Government agencies– National labs– Companies– Universities

- 11 -


HDFHDF

The growth of HDF5The growth of HDF5

• Scientific fields– Astronomy, astrophysics, aerospace engineering

– Geophysics, remote sensing, meteorology,

– Oceanography, environmental Science

– Informational Science

– Medical Research (brain, cancer, biotech)

– Product model data

- 12 -


HDFHDF

Facilitating interoperability Facilitating interoperability between HDF4 and HDF5 between HDF4 and HDF5

• HDF4 to HDF5 mapping specification– “Mapping HDF4 Objects to HDF5 Objects”

• http://hdf.ncsa.uiuc.edu/HDF5/papers

– Rules for mapping high level HDF4 objects to HDF5• How to describe HDF4 objects in HDF5• How to interpret HDF5 objects as HDF4 objects

• HDF4-to-HDF5 conversion software (later)• Space Research, Inc. Explorer 1.1

– reads both HDF4 and HDF5

- 13 -


HDFHDF

HDF5 Activities in 2000HDF5 Activities in 2000

• HDF5 1.2.2 library release

• Fortran 90 & C++ API

• HDF5 Abstract Data Model

• XML Document Type Definition (DTD)

• HDF5 tools

• Support for users and application developers

• HDF5 Tutorial

• Searchable, printable documentation

- 15 -


HDFHDF

Tools & UtilitiesTools & Utilities• NCSA

– Java wrapper for HDF5– H5View: Java browser/Editor for HDF5– H5gen: XML-to-HDF5 file generator (Java)– H5dump & H5ls

• Others– VisAD data adapter for HDF5 (Java toolkit)– HDF Inspector/Explorer– Open Data Explorer (IBM)– Ensight

- 16 -


HDFHDF

H5ViewH5View

- 17 -


HDFHDF

H5ViewH5View

• Java-based tool for browsing and editingJava-based tool for browsing and editing

• Display structure of fileDisplay structure of file

• Display content of objectsDisplay content of objects

• Create and delete objectsCreate and delete objects

• Modify values of objects and attributesModify values of objects and attributes

- 18 -


HDFHDF

HDF5 XML DTDHDF5 XML DTD

• A flexible standard language that can describe an HDF5 file in precise detail– Datasets, dataspaces, datatypes – Groups and links, structure of the file– Values of the attributes and data

• http://hdf.ncsa.uiuc.edu/HDF5/XML

- 19 -


HDFHDF

H5Gen: XMLH5Gen: XMLHDF5HDF5

• Reads an XML description of an HDF5 file

• Generates the corresponding HDF5 file

• Validates XML description vs. HDF5 DTD

- 20 -


HDFHDF

Some uses of XMLSome uses of XMLView View

HDF5 files HDF5 files using a web using a web

browserbrowserXML XML

to to HTMLHTML

Java Java viewers,editors,viewers,editors,

other toolsother tools

XML to JavaXML to JavaJava to XMLJava to XML

Description Description in XML using in XML using HDF5 DTDHDF5 DTD

HDF5 HDF5 filefile

HDF5 HDF5 filefile

XML to HDF5 (H5gen)XML to HDF5 (H5gen)HDF5 to XMLHDF5 to XML

Generate, validate,Generate, validate,reconstruct HDF5 filesreconstruct HDF5 files

Data Data location location servicesservices

Catalog Catalog records records in XMLin XML

ArchiveArchiveArchiveArchiveXML XML

documentationdocumentation of metadata of metadata & structure& structure

- 21 -


HDFHDF

Focus areas for 2001Focus areas for 2001

• Support Terra & Aqua• Get ready for Aura• Support HDF-EOS 3 • Enhance HDF5View• Refine XML DTD & design tools around it• HDF5 converters

– HDF5XML– HDF4-to-HDF5– Others (e.g. GIFHDF5)

- 22 -


HDFHDF

Focus areas for 2001Focus areas for 2001

• Expand list of applications and users• Facilitate access to other tools & software• Get vendors on board

• Performance testing and tuning• Extend API with ease-of-use functions• Clusters and other new environments• Implement a thread-safe version• HDF5 advanced tutorial

- 23 -


HDFHDF

Thank you!Thank you!

• HDF website– http://hdf.ncsa.uiuc.edu/

• HDF5 Information Center– http://hdf.ncsa.uiuc.edu/HDF5/

• HDF Helpdesk– [email protected]

• HDF users mailing list– [email protected]

HDFHDF

55

HDF Update

Technology

hdf5 hdf4

hdf5 h5gen hdf5

xml hdf5

hdf5 converters hdf5

hdf5 h5view

hdf5 activities

hdf5 dtduniversity of

hdf5 dtdxml