File Formats, Conventions, and Data Level Interoperability ESDSWG New Orleans, Oct 20, 2010 Joe Glassy, Chris Lynnes ESDSWG Tech Infusion.

File Formats, Conventions,and

Data Level Interoperability

ESDSWG New Orleans, Oct 20, 2010Joe Glassy, Chris Lynnes ESDSWG Tech Infusion

Introduction & overview

• Outline of objectives:– Discuss role of standard, self-describing “File

formats” in data level interoperability– Summarize common file formats in use, their

properties, & benefits --“data life cycle economics”– Discuss criteria for choosing a file format, matching

it to needs of consumer/producers.– Discuss critical role of Conventions – any file format

needs good recipes to make them interoperable!– Examples: NASA Measures F/T, SMAP, AIRs, Aura

Role(s) Of File Formats in Interoperability

• File formats represent versatile “packages” for multi-dimensional science data and metadata.

• Offer self-describing “well-known structures” to codify desired, common conventions and practices.

• Offer well-documented reference cases to encapsulate specific data models.

• Standard file formats dock with format-aware tools to offer users a seamless end-to-end experience and platform portability

• Enhance Mission-to-Mission continuity

…investment life-cycle economics…

Why (and how) are file formats important?

• Standard formats– Come with thorough documentation– Provide good Reference implementations

• Common formats– More datasets in a format more tools that read

that format• Canonical structures and names

general purpose handlers for coordinates, etc. smarter tools

A generic work flow…

• Consider user community needs and culture, fit within architecture, institutional policies & preferences

• Choose a standard file format (or sub-variant)• Design a convention-enabled, specific internal layout

with metadata interfaces• Prototype: Implement in prototype, evaluate• Implement in production context• Integrate within discovery and catalog environments

(Catalog interoperability…)

Examples of standard file formats

• HDF5 – a file format on its own, as well as a broad foundation for others

• netCDF v4 (stable at v4.1.1, newest : v4.1.2-beta1)– v4 Classic (widespread adoption, some limitations…)– v4 Enhanced (support Groups, User-defined, variable length

types, and more)• netCDF v3 Classic (legacy+ , tools+, but limited)• HDFEOS2, HDFEOS5 – EOS Terra, Aqua, Aura…• HDF4 – legacy, extensive use by MODIS Terra, Aqua• Many other domain-specific, less generic formats abound…

(need transform tools to/from HDF?)

Some selection criteria…• Do file-format’s capabilities support required

functionality?• What is breadth of acceptance, adoption within larger

community? (and/or, does institutional policy dictate a specific format?)

• Presence and quality of documentation (reference, examples and especially tutorials), API software, and community support?

• Contribution to investment, data life-cycle economics?• What is the level of standardization?• Adaptability of format to widely used conventions like

CF 1.x, or other accepted convention(s)?

Internal Layout / Design(once format is chosen & adopted…)

• Define &refine High level organization /structure• /DATA• /METADATA

• Distinguish ‘data’ from ‘metadata’, core structure vs. ‘attributes’– Dimensions, Coordinate Variables, projection attributes– Missing_data, _Fillvalue vs. internal fill value– Units, Gain, offset, min, max, range, etc.

• Prototype it!– Leverage script environments (Python H5Py, PyTables, etc)– Panoply, HDFView also quick, useful for prototyping, feedback

Using “Groups”

• HDF5 (and NetCDF v4-Enhanced) support full use of groups e.g. /DATA vs. /METADATA, etc.

• Groups useful in partitioning out functionally related sets of data or attributes; Hierarchical view mimics file-system

• Facilitates appropriate information-hiding, highlights needed info, shield other (principle of least privilege…)

• Well supported by modern tools (Panoply, HDFViews, PyTables, H5Py) and low-lev APIs.

Example(s) of File Formats In Action

• HDF5 – NASA Measures – NASA Measures Freeze/Thaw (soon available at NSIDC)– http://measures.ntsg.umt.edu/sample_2007_day180.zip

• AQUA AIRS Level 2 (from earlier talk):– http://airspar1u.ecs.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.005/201

0/285/AIRS.2010.10.12.090.L2.RetStd.v5.2.2.0.G10286064818.hdf

• Aura TES (TES-Aura_L3-CH4_r0000002135_F01_05.he5)

Example: NASA Measures Freeze/Thaw, Daily in HDF5Metadata Block: Attributes

Example: NASA Measures Daily Freeze/Thaw in HDF5Data Variable (FT_SSMI) and Attributes

Example: NASA Level 2 AIRS (Swath) in HDF4

Example: NetCDF, (tos) Sea surface temperatures collected by PCMDI for use by the IPCC, illustrating CF v1.0 layout

Example: TES (HDFEOS5) illustrating CF v1.0 layout

CF Conventions & file formats:--how they contribute to interoperability.

• CF v1.4.x -- the term “CF” is now broader than just climate-forecasting!

• Standard Name Table -- a step towards wider adoption of names, controlled vocabularies, units terminology

• CF v1.4.x provides tool-makers with helpful “lingua-franca” guidance.

• Within a file-format, adopting conventions like CF promotes common layout, names, semantics, for dataset-to-dataset compatibility -- a key to wider data level interoperability.

Attributes vs. Metadata?one man’s ceiling is another man’s floor…

• Collection level vs. Data Set vs. Granule level• Structural vs. science-content• Swath vs. grid vs. point• Commonly used attributes:

– CONVENTIONS attrib, communicates which convention was used

– Basic globals: title, history, institution, source, references– Coordinate variables, axis, formula_terms– Units, _Fillvalue, missing_data, valid_range– Short_name, long_name, other provenance– (gain,offset /scale_factor,addOffset), etc.

Challenges? (just a few remain…)

• Evolution, bifurcation, asymmetric support can result in occasional user confusion:– HDF v1.8.x vs. v1.6.x families?– NetCDF v4 Enhanced vs. NetCDF v4 Classic vs. v3?– HDFEOS5 vs. HDFEOS2?

• Both GUI tool and API support tends to vary by platform (Linux, Mac, Win7) and sub-flavor…

• Multi-library dependency stacks beg for fully bundled, version-matched end-to-end install pkg!

• Conventions community (CF v1.4.x) and metadata standards communities also in motion (but that’s good too…)

Resources : URLs• Climate Forecast (CF) Conventions (now at 1.4.x):

– http://cf-pcmdi.llnl.gov/– http://cf-pcmdi.llnl.gov/documents/cf-conventions

• HDF: – http://www.hdfgroup.org/HDF5/doc/index.html

• HDFEOS– http://www.hdfgroup.org/hdfeos.html– http://hdfeos.org/software/aug_hdfeos5.php

• NetCDF: – http://www.unidata.ucar.edu/software/netcdf/– http://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.ht

ml• General:

– http://www.oceanteacher.org/OTMediawiki/index.php/Self-Describing_Formats

– http://en.wikipedia.org/wiki/List_of_file_formats

Resources: File format related Tools

• Panoply: http://www.giss.nasa.gov/tools/panoply/

• HDFView: http://www.hdfgroup.org/hdf-java-html/hdfview/

• OpenDAP: http://opendap.org

• IDV: http://www.unidata.ucar.edu/software/idv/

• McIDAS: http://www.unidata.ucar.edu/software/mcidas/

• Python: – h5py : http://code.google.com/p/h5py/, http://h5py.alfven.org/, – PyTables: http://www.pytables.org/moin

• Perl: PDL-IO-HDF5, and Biohdf?

• Many others: HEG, MTD, HDFEOS plug-in for HDFview, HDFLook, (ncdump, h5dump, and cousins), GRADS, Matlab, binary APIs

A provisional DOI, UUID Strategy

• What we used for NASA Measures Freeze/Thaw, daily (v2) just delivered:– DOI: assigned to our reference paper, by IEEE

Transactions in Geoscience and Remote Sensing– UUID recipe, seedString =

www.our.url/GranuleName/Datetime8601StampImport uuiduuid= uuid.uuid5(seedString)