Hdf5 intro

www.hdfgroup.org

The HDF Group

Introduction to HDF5

Barbara Jones

The HDF Group

The 13th HDF & HDF-EOS Workshop

November 3-5, 2009

November 3-5, 2009 1HDF/HDF-EOS Workshop XIII

www.hdfgroup.org

Before We Begin …

HDF-EOS Home Page: http://hdfeos.org/

Workshop Info:

http://hdfeos.org/workshops/ws13/workshop_thirteen.php

The HDF Group Page: http://hdfgroup.org/

HDF5 Home Page: http://hdfgroup.org/HDF5/

HDF Helpdesk: [email protected]

HDF Mailing Lists:http://hdfgroup.org/services/support.html


www.hdfgroup.org

HDF5 is the second HDF format • Development started in 1996

• First release was in 1998

HDF4 is the first HDF format• Originally called HDF

• Development started in 1987

• Still supported by The HDF Group

HDF = Hierarchical Data Format


www.hdfgroup.org

HDF5 is like…

5


www.hdfgroup.org

HDF5 is designed …

• for high volume and/or complex data

• for every size and type of system (portable)

• for flexible, efficient storage and I/O

• to enable applications to evolve in their use of HDF5 and to accommodate new models

• to support long-term data preservation


www.hdfgroup.org

HDF5 Technology

HDF5 is a data model, library and file format for managing data.

November 3-5, 2009 HDF/HDF-EOS Workshop XIII 6

www.hdfgroup.org

HDF5 Technology

• HDF5 (Abstract) Data Model• Defines the “building blocks” for data organization and

specification• Files, Groups, Datasets, Attributes, Datatypes, Dataspaces, …

• HDF5 Library (C, Fortran 90, C++ APIs)• Also Java Language Interface and High Level Libraries

• HDF5 Binary File Format• Bit-level organization of HDF5 file• Defined by HDF5 File Format Specification

• Tools For Accessing Data in HDF5 Format• h5dump, h5repack, HDFView, …


www.hdfgroup.org

The HDF Group

HDF5 Abstract Data Model

a.k.a. HDF5 Logical Data Model

a.k.a. HDF5 Data Model


www.hdfgroup.org

HDF5 File

lat | lon | temp----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6An HDF5 file is a

container that holds data objects.

Experiment Notes:

Serial Number: 99378920

Date: 3/13/09

Configuration: Standard 3


www.hdfgroup.org

HDF5 Groups and Links

lat | lon | templat | lon | temp----|-----|---------|-----|----- 12 | 23 | 3.112 | 23 | 3.1 15 | 24 | 4.215 | 24 | 4.2 17 | 21 | 3.617 | 21 | 3.6

Experiment Notes:Serial Number: 99378920Date: 3/13/09Configuration: Standard 3

/

SimOutViz

HDF5 groups and links organize data objects.


www.hdfgroup.org

HDF5 Objects

• HDF5 Group: A grouping structure containing zero or more HDF5 objects

• HDF5 Dataset: Raw data elements, together with information that describes them

(There are other HDF5 objects that help support Groups and Datasets.)

The two primary HDF5 objects are:


www.hdfgroup.org

HDF5 Groups

“/”A B

C

k ltemp

• Used to organize collections• Every file starts with a root group• Similar to UNIX directories• Path to object defines it• Objects can be shared: /A/k and /B/l are the same

• Used to organize collections• Every file starts with a root group• Similar to UNIX directories• Path to object defines it• Objects can be shared: /A/k and /B/l are the same

= Group

= Dataset


temp

www.hdfgroup.org

HDF5 Datasets

HDF5 Datasets organize and contain your “raw data values”. They consist of:• Your raw data• Metadata describing the data:

- The information to interpret the data (Datatype)

- The information to describe the logical layout of the data elements (Dataspace)

- Characteristics of the data (Properties)

- Additional optional information that describes the data (Attributes)


www.hdfgroup.org

HDF5 Dataset

DataMetadataDataspace

3

RankRank

Dim_2 = 5Dim_1 = 4

DimensionsDimensions

Time = 32.4

Pressure = 987

Temp = 56

(optional)(optional)AttributesAttributes

Chunked

Compressed

Dim_3 = 7

PropertiesProperties

Integer DatatypeDatatype


www.hdfgroup.org

HDF5 Dataspaces

An HDF5 Dataspace describes the logical layout for the data elements:• Array

• multiple elements in dataset organized in a multi-dimensional (rectangular) array

• maximum number of elements in each dimension may be fixed or unlimited

• NULL• no elements in dataset

• Scalar • single element in dataset

• November 3-5, 2009 15HDF/HDF-EOS Workshop XIII

www.hdfgroup.org

HDF5 Dataspaces

Two roles:Dataspace contains spatial information (logical

layout) about a dataset stored in a file

• Rank and dimensions• Permanent part of dataset

definition

Partial I/0: Dataspace describes application’s data buffer and data elements participating in I/O

Rank = 2Rank = 2

Dimensions = 4x6Dimensions = 4x6

Rank = 1Rank = 1

Dimension = 10Dimension = 10


www.hdfgroup.org

HDF5 Datatypes

The HDF5 datatype describes how to interpret individual data elements.

HDF5 datatypes include:− integer, float, unsigned, bitfield, …− user-definable (e.g., 13-bit integer)− variable length types (e.g., strings)− references to objects/dataset regions− enumerations - names mapped to integers− opaque− compound (similar to C structs)


www.hdfgroup.org

HDF5 Dataset

Dataspace: Rank = 2 Dimensions = 5 x 3


Datatype: 16-byte integer

3

5

V

www.hdfgroup.org

HDF5 Properties

• Properties (also known as Property Lists) are characteristics of HDF5 objects that can be modified

• Default properties handle most needs

• By changing properties one can take advantage of the more powerful features in HDF5


www.hdfgroup.org

Storage Properties


ChunkedChunked

Chunked & Chunked & CompressedCompressed

Better access time for subsets; extensible

Improves storage efficiency, transmission speed

ContiguousContiguous(default)(default)ContiguousContiguous(default)(default)

Data elements stored physically adjacent to each other

www.hdfgroup.org

HDF5 Attributes (optional)

• An HDF5 attribute has a name and a value

• Attributes typically contain user metadata

• Attributes may be associated with

- HDF5 groups

- HDF5 datasets

- HDF5 named datatypes

• An attribute’s value is described by a datatype and a dataspace

• Attributes are analogous to datasets except…

- they are NOT extensible

- they do NOT support compression or partial I/ONovember 3-5, 2009 22HDF/HDF-EOS Workshop XIII

www.hdfgroup.org

HDF5 Abstract Data Model Summary

• The Objects in the Data Model are the “building blocks” for data organization and specification

• Files, Groups, Links, Datasets, Datatypes, Dataspaces, Attributes, …

• Projects using HDF5 “map” their data concepts to these HDF5 Objects


www.hdfgroup.org

The HDF Group

HDF5 Software


www.hdfgroup.org

HDF5 Software Layers & Storage

HDF5 File Format File Split

Files

File on Parallel Filesystem

Other

I/O Drivers

Virtual File Layer Posix

I/OSplit Files MPI I/O Custom

Internals Memory Mgmt

Datatype Conversion

FiltersChunked Storage

Version Compatibility

and so on…

LanguageInterfaces

C, Fortran, C++

HDF5 Data Model Objects

Groups, Datasets, Attributes, …

Tunable PropertiesChunk Size, I/O Driver, …

h5dumptoolHigh Level

APIs

HDFview tool

h5repack tool

Java Interface…

API


www.hdfgroup.org

HDF5 API and Applications

…

Storage

Domain DataObjects

EOSlibrary

ApplicationsaClimate

ModelMATLAB


HDF5 Library

www.hdfgroup.org

HDF5 Home Page

HDF5 home page: http://hdfgroup.org/HDF5/• Two releases: HDF5 1.8 and HDF5 1.6

HDF5 source code:• Written in C, and includes optional C++, Fortran 90 APIs, and High

Level APIs

• Contains command-line utilities (h5dump, h5repack, h5diff, ..) and compile scripts

HDF pre-built binaries:• When possible, include C, C++, F90, and High Level libraries. Check

./lib/libhdf5.settings file.

• Built with and require the SZIP and ZLIB external libraries


www.hdfgroup.org

Useful Tools For New Users

h5dump:Tool to “dump” or display contents of HDF5 files

h5cc, h5c++, h5fc:Scripts to compile applications

HDFView: Java browser to view HDF4 and HDF5 files http://www.hdfgroup.org/hdf-java-html/hdfview/

h5dump:Tool to “dump” or display contents of HDF5 files

h5cc, h5c++, h5fc:Scripts to compile applications

HDFView: Java browser to view HDF4 and HDF5 files http://www.hdfgroup.org/hdf-java-html/hdfview/


www.hdfgroup.org

h5dump Utility

h5dump [options] [file]

-H, --header Display header only – no data -d <names> Display the specified dataset(s). -g <names> Display the specified group(s) and

all members. -p Display properties. <names> is one or more appropriate object names.


www.hdfgroup.org

Example of h5dump Output

HDF5 "dset.h5" {GROUP "/" { DATASET "dset" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) } DATA { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 } }}}

“/”‘dset’‘dset’


www.hdfgroup.org

HDF5 Compile Scripts

• h5cc – HDF5 C compiler command

• h5fc – HDF5 F90 compiler command

• h5c++ – HDF5 C++ compiler command

To compile:% h5cc h5prog.c

% h5fc h5prog.f90


www.hdfgroup.org

Compile option: -show

-show: displays the compiler commands and options without executing them

% h5cc –show Sample_c.c

Will show the correct paths and libraries used by the installed HDF5 library.

Will show the correct flags to specify when building an application with that HDF5 library.


www.hdfgroup.org

The HDF Group

Browsing HDF5 Files with HDFView


www.hdfgroup.org

HDFView

Structure of FileStructure of File Contents of DatasetContents of Dataset


www.hdfgroup.org

HDFView File Menu


www.hdfgroup.org

HDF-EOS5 File in HDFView


www.hdfgroup.org

Introduction to HDF5 Programming Model

and APIs


www.hdfgroup.org

Operations Supported by the API

• Create objects (groups, datasets, attributes, complex data types, …)

• Assign storage and I/O properties to objects

• Perform complex subsetting during read/write

• Use variety of I/O “devices” (parallel, remote, etc.)

• Transform data during I/O

• Make inquiries on file and object structure, content, properties


www.hdfgroup.org

General Programming Paradigm

• Properties of object are optionally defined Creation propertiesAccess properties

• Object is opened or created

• Object is accessed, possibly many times

• Object is closed


www.hdfgroup.org

Order of Operations

• An order is imposed on operations by argument dependencies

For Example:

A file must be opened before a dataset -because-

the dataset open call requires a file handle

as an argument.

• Objects can be closed in any order.November 3-5, 2009 40HDF/HDF-EOS Workshop XIII

www.hdfgroup.org

The General HDF5 API

• Currently C, Fortran 90, Java, and C++ bindings.

• C routines begin with prefix H5?? is a character corresponding to the type of object the function acts on

Example Functions:

H5D : Dataset interface e.g., H5Dread H5F : File interface e.g., H5Fopen

H5S : dataSpace interface e.g., H5Sclose


www.hdfgroup.org

HDF5 Defined Types

For portability, the HDF5 library has its own defined types:

hid_t: object identifiers (native integer) hsize_t: size used for dimensions (unsigned long or

unsigned long long) herr_t: function return value

hvl_t: variable length datatype

For C, include hdf5.h in your HDF5 application.


www.hdfgroup.org

The HDF5 API

• For flexibility, the API is extensive 300+ functions

• This can be daunting… but there is hopeA few functions can do a lotStart simple Build up knowledge as more features are

needed

Victronix Swiss Army Cybertool 34


www.hdfgroup.org

Basic Functions

H5Fcreate (H5Fopen) create (open) File

H5Screate_simple/H5Screate create dataSpace

H5Dcreate (H5Dopen) create (open) Dataset

H5Dread, H5Dwrite access Dataset

H5Dclose close Dataset

H5Sclose close dataSpace

H5Fclose close File


NOTE: The order specified above is not required.

www.hdfgroup.org

Other Common Functions

DataSpaces: H5Sselect_hyperslab (Partial I/O) H5Sselect_elements (Partial I/O)

H5Dget_space

Groups: H5Gcreate, H5Gopen, H5Gclose

Attributes: H5Acreate, H5Aopen_name, H5Aclose, H5Aread, H5Awrite

Property lists: H5Pcreate, H5Pclose H5Pset_chunk, H5Pset_deflate


www.hdfgroup.org

High Level APIs

• Included along with the HDF5 library

• Simplify steps for creating, writing, and reading objects.

• Do not entirely ‘wrap’ HDF5 library


www.hdfgroup.org

The HDF Group

Example HDF5 Code


www.hdfgroup.org

Steps to Create a File

1. Decide on properties the file should have and create them if necessary:• Creation properties, like size of user block• Access properties (improve performance)• Use default properties (H5P_DEFAULT)

2. Create the file

3. Close the file and the property lists, as needed


www.hdfgroup.org

Code: Create a File

hid_t file_id; herr_t status; file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC,

H5P_DEFAULT, H5P_DEFAULT);

status = H5Fclose (file_id);

Note: Return codes not checked for errors in code samples.

“/” (root)“/” (root)


www.hdfgroup.org

Dataset Components

DataMetadataDataspace

3

RankRank

Dim_2 = 5Dim_1 = 4

DimensionsDimensions

Chunked

Compressed

Dim_3 = 7

PropertiesProperties

Integer DatatypeDatatype


www.hdfgroup.org

Steps to Create a Dataset

1. Define dataset characteristicsa) Datatype – integer

b) Dataspace - 4x6

c) Properties if needed, or use H5P_DEFAULT

2. Decide where to put it• Obtain location ID:

- Group ID puts it in a Group

- File ID puts it in Root Group

3. Create dataset in file

4. Close everything

A“/” (root)


www.hdfgroup.org

HDF5 Pre-defined Datatype Identifiers

HDF5 defines* set of Datatype Identifiers per HDF5 session.

For example:

C Type HDF5 File Type HDF5 Memory Typeint H5T_STD_I32BE H5T_NATIVE_INT

H5T_STD_I32LE

float H5T_IEEE_F32BE H5T_NATIVE_FLOAT H5T_IEEE_F32LE

double H5T_IEEE_F64BE H5T_NATIVE_DOUBLEH5T_IEEE_F64LE

* Value of datatype is NOT fixed


www.hdfgroup.org

Pre-defined File Datatype Identifiers

Examples:

H5T_IEEE_F64LE Eight-byte, little-endian, IEEE floating-pointH5T_STD_I32LE Four-byte, little-endian, signed two's

complement integer

NOTE: What you see in the file. Name is the same everywhere andexplicitly defines a datatype.

*STD= “An architecture with a semi-standard type like 2’s complement integer, unsigned integer…”

Architecture*ProgrammingType


www.hdfgroup.org

Pre-defined Native Datatypes

Examples of predefined native types in C:

H5T_NATIVE_INT (int)H5T_NATIVE_FLOAT (float )H5T_NATIVE_UINT (unsigned int)H5T_NATIVE_LONG (long )H5T_NATIVE_CHAR (char )

NOTE: Memory types. Different for each machine.Used for reading/writing.


www.hdfgroup.org

Storage Properties


ChunkedChunked

Chunked & Chunked & CompressedCompressed

Better access time for subsets; extensible

Improves storage efficiency, transmission speed

ContiguousContiguous(default)(default)ContiguousContiguous(default)(default)

Data elements stored physically adjacent to each other

www.hdfgroup.org

Code: Create a Dataset

1 hid_t file_id, dataset_id, dataspace_id; 2 hsize_t dims[2];. herr_t status; . . file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC, . H5P_DEFAULT, H5P_DEFAULT); 5 dims[0] = 4;6 dims[1] = 6;7 dataspace_id = H5Screate_simple (2, dims, NULL); 8 dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT,

H5P_DEFAULT);

9 status = H5Dclose (dataset_id); 10 status = H5Sclose (dataspace_id); 11 status = H5Fclose (file_id);

Define a dataspace

rank current dims


www.hdfgroup.org


1 hid_t file_id, dataset_id, dataspace_id; . hsize_t dims[2];. herr_t status; . . file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);. dims[0] = 4;. dims[1] = 6;. dataspace_id = H5Screate_simple (2, dims, NULL);

8 dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT,H5P_DEFAULT,

H5P_DEFAULT);

Datatype

Properties(Link Creation, Dataset Creation and Access)

Where to put it

Size & shape


www.hdfgroup.org


1 hid_t file_id, dataset_id, dataspace_id; 2 hsize_t dims[2];3 herr_t status; 4 file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); 5 dims[0] = 4;6 dims[1] = 6;7 dataspace_id = H5Screate_simple (2, dims, NULL);

8 dataset_id = H5Dcreate (file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT,

H5P_DEFAULT);

9 status = H5Dclose (dataset_id); 10 status = H5Sclose (dataspace_id); 11 status = H5Fclose (file_id);

Terminate access to dataspace, dataset, file


www.hdfgroup.org

Example Code - H5Dwrite

status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL,H5S_ALL, H5P_DEFAULT, wdata);

Dataset ID from H5Dcreate/H5Dopen

Memory Datatype


www.hdfgroup.org

Partial I/O

File Dataspace (disk)H5S_ALLH5S_ALL H5S_ALLH5S_ALL

To Modify Dataspace: H5Sselect_hyperslab H5Sselect_elements


Memory Dataspace

status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,wdata);

www.hdfgroup.org

Example Code – H5Dwrite

status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata);

Data Transfer Property List (MPI I/O, Transformations,…)


www.hdfgroup.org

Example Code – H5Dread

status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata);


www.hdfgroup.org

High Level APIs: HDF5 Lite (H5LT)

#include “hdf5_hl.h“..

file_id = H5Fcreate(“file.h5",H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

status = H5LTmake_dataset (file_id,“A",2,dims, H5T_STD_I32BE, data);

status = H5Fclose (file_id);


www.hdfgroup.org

Steps to Create a Group

1. Decide where to put it – “root group”• Obtain location ID

2. Define properties or use H5P_DEFAULT

3. Create group in file. 4. Close the group.


www.hdfgroup.org

Example: Create a Group

A B“/” (root)

4x6 array of integers

file.h5


www.hdfgroup.org

Code: Create a Group

hid_t file_id, group_id; .../* Open “file.h5” */ file_id = H5Fopen (“file.h5”, H5F_ACC_RDWR,

H5P_DEFAULT);

/* Create group "/B" in file. */ group_id = H5Gcreate (file_id,"B", H5P_DEFAULT,

H5P_DEFAULT, H5P_DEFAULT);

/* Close group and file. */ status = H5Gclose (group_id); status = H5Fclose (file_id);


www.hdfgroup.org

HDF5 Tutorial and Examples

HDF5 Tutorial:http://www.hdfgroup.org/HDF5/Tutor/

HDF5 Example Code:http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/


www.hdfgroup.org

The HDF Group

Thank You!


www.hdfgroup.org

Acknowledgements

This work was supported by cooperative agreement number NNX08AO77A from the National

Aeronautics and Space Administration (NASA).

Any opinions, findings, conclusions, or recommendations expressed in this material are

those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space

Administration.


www.hdfgroup.org

The HDF Group

Questions/comments?


Hdf5 intro

Economy & Finance

transmission

obtain location

extensibleimproves

high level

hdf5 home

high level

creating intermediate

hdf5 library