Mitglied der Helmholtz-Gemeinschaft Portable Parallel I/O Handling large datasets in heterogeneous parallel environments March 14, 2013 Michael Stephan
Mitg
lied
derH
elm
holtz
-Gem
eins
chaf
t
Portable Parallel I/OHandling large datasets inheterogeneous parallel environments
March 14, 2013 Michael Stephan
Mitg
lied
derH
elm
holtz
-Gem
eins
chaf
t
Portable Parallel I/OPart I: HDF5
March 14, 2013 Michael Stephan
Learning Objectives
At the end of this lesson, you will be able toGet an idea about the HDF5 funktionality.Create a short example for HDF5 file I/O.Discuss the advantages and disadvantages of HDF5 fileI/O.
March 14, 2013 Michael Stephan Slide 3
Outline
Introduction
Introduction to HDF5 Programming Model and APIs
General Programming Paradigm
Parallele HDF5
March 14, 2013 Michael Stephan Slide 4
Outline
IntroductionMotivationTerms and Definitions
Introduction to HDF5 Programming Model and APIs
General Programming Paradigm
Parallele HDF5
March 14, 2013 Michael Stephan Slide 5
What is HDF5? I
Unique technology suite that makes possible themanagement of extremely large and complex datacollections
March 14, 2013 Michael Stephan Slide 6
What is HDF5? IIThe HDF5 technology suite includes:
A versatile data model that can represent very complex dataobjects and a wide variety of metadata.A completely portable file format with no limit on the numberor size of data objects in the collection.A software library that runs on a range of computationalplatforms, from laptops to massively parallel systems, andimplements a high-level API with C, C++, Fortran 90, andJava interfaces.A rich set of integrated performance features that allow foraccess time and storage space optimizations.Tools and applications for managing, manipulating, viewing,and analyzing the data in the collection.The HDF5 data model, file format, API, library, and tools areopen and distributed without charge.
March 14, 2013 Michael Stephan Slide 7
What is HDF5? IIIUnlimited size, extensibility, and portability
HDF5 does not limit the size of files or the size or number ofobjects in a file.The HDF5 format and library are extensible and designed toevolve gracefully to satisfy new demands.HDF5 functionality and data is portable across virtually allcomputing platforms and is distributed with C, C++, Java, andFortran90 programming interfaces.
March 14, 2013 Michael Stephan Slide 8
What is HDF5? IVGeneral data model
HDF5 has a simple but versatile data model.The HDF5 data model supports complex data relationshipsand dependencies through its grouping and linkingmechanisms.HDF5 accommodates many common types of metadata andarbitrary user-defined metadata.
March 14, 2013 Michael Stephan Slide 9
What is HDF5? VUnlimited variety of datatypes
HDF5 supports a rich set of pre-defined datatypes as well asthe creation of an unlimited variety of complex user-defineddatatypes.Datatype definitions can be shared among objects in an HDFfile, providing a powerful and efficient mechanism fordescribing data.Datatype definitions include information such as byte order(endian), size, and floating point representation, to fullydescribe how the data is stored, insuring portability to otherplatforms.
March 14, 2013 Michael Stephan Slide 10
What is HDF5? VIFlexible, efficient I/O
HDF5, through its virtual file layer, offers extremely flexiblestorage and data transfer capabilities. Standard (Posix),Parallel, and Network I/O file drivers are provided with HDF5.Application developers can write additional file drivers toimplement customized data storage or transport capabilities.The parallel I/O driver for HDF5 reduces access times onparallel systems by reading/writing multiple data streamssimultaneously.
March 14, 2013 Michael Stephan Slide 11
What is HDF5? VIIFlexible data storage
HDF5 employs various compression, extensibility, andchunking strategies to improve access, management, andstorage efficiency.HDF5 provides for external storage of raw data, allowing rawdata to be shared among HDF5 files and/or applications, andoften saving disk space.
March 14, 2013 Michael Stephan Slide 12
What is HDF5? VIIIData transformation and complex subsetting
HDF5 enables datatype and spatial transformation during I/Ooperations.HDF5 data I/O functions can operate on selected subsets ofthe data, reducing transferred data volume and improvingaccess speed.
March 14, 2013 Michael Stephan Slide 13
Who uses HDF5?
Applications that deal with big or complex dataOver 200 different types of apps2+ million product users world-wideAcademia, government agencies, industry
March 14, 2013 Michael Stephan Slide 14
Outline
IntroductionMotivationTerms and Definitions
Introduction to HDF5 Programming Model and APIs
General Programming Paradigm
Parallele HDF5
March 14, 2013 Michael Stephan Slide 15
An HDF5 “file” is a container...
...into which you can put your data objects Structures toorganize objects
March 14, 2013 Michael Stephan Slide 16
HDF5 model
Groups – provide structure among objectsDatasets – where the primary data goes
Data arraysRich set of datatype optionsFlexible, efficient storage and I/O
Attributes, for metadataOther objects
Links (point to data in a file or in another HDF5 file)Datatypes (can be stored for complex structures and reusedby multiple datatsets)
March 14, 2013 Michael Stephan Slide 17
HDF5 Dataset
March 14, 2013 Michael Stephan Slide 18
HDF5 Dataspace
Two rolesDataspace contains spatial info about a dataset stored in afile
Rank and dimensionsPermanent part of dataset definition
Dataspace describes application’s data buffer and dataelements participating in I/O
March 14, 2013 Michael Stephan Slide 19
HDF5 Datatype I
Datatype – how to interpret a data elementPermanent part of the dataset definitionTwo classes: atomic and compoundCan be stored in a file as an HDF5 object (HDF5 committeddatatype)Can be shared among different datasets
March 14, 2013 Michael Stephan Slide 20
HDF5 Datatype IIHDF5 atomic types
normal integer and floatuser-definable (e.g., 13-bit integer)variable length types (e.g., strings)references to objects/dataset regionsenumeration - names mapped to integersarray
March 14, 2013 Michael Stephan Slide 21
HDF5 Datatype IIIHDF5 compound types
Comparable to C structs (“records”)Members can be atomic or compound types
March 14, 2013 Michael Stephan Slide 22
HDF5 dataset: array of records
March 14, 2013 Michael Stephan Slide 23
Special storage options for dataset
March 14, 2013 Michael Stephan Slide 24
HDF5 Attribute
Attribute – data of the form “name = value”, attached to anobject by applicationOperations similar to dataset operations, but
Not extendibleNo compression or partial I/O
Can be overwritten, deleted, added during the “life“ of adataset
March 14, 2013 Michael Stephan Slide 25
HDF5 Group
A mechanism for organizing collections of related objectsEvery file starts with a root groupSimilar to UNIX directories
/ (root)/X/Y/X/temp
Can have attributes
March 14, 2013 Michael Stephan Slide 26
Partial I/O
Move just part of a dataset
March 14, 2013 Michael Stephan Slide 27
Partial I/O
Move just part of a dataset
March 14, 2013 Michael Stephan Slide 28
Layers – parallel example
March 14, 2013 Michael Stephan Slide 29
Virtual I/O layer
March 14, 2013 Michael Stephan Slide 30
Virtual I/O layer
A public API for writing I/O driversAllows HDF5 to interface to disk, the network, memory, ora user-defined device
March 14, 2013 Michael Stephan Slide 31
Portability and Robustness
Runs almost anywhereLinux and UNIX workstationsWindows, Mac OS XBig ASC machines, Crays, VMS systemsTeraGrid and other clustersSource and binaries available fromhttp://www.hdfgroup.org/HDF5/release/index.html
March 14, 2013 Michael Stephan Slide 32
Other Software
The HDF GroupHDFViewJava toolsCommand-line utilitiesWeb browser plug-inRegression and performance testing softwareParallel h5diff
3rd Party (IDL, MATLAB, Mathematica, PyTables, HDFExplorer, LabView)Communities (EOS, ASC, CGNS)Integration with other software (iRODS, OPeNDAP)
March 14, 2013 Michael Stephan Slide 33
HDF5 software stack
March 14, 2013 Michael Stephan Slide 34
Structure of HDF5 Library
March 14, 2013 Michael Stephan Slide 35
Goals of HDF5 Library
Provide flexible API to support a wide range of operationson data.Support high performance access in serial and parallelcomputing environments.Be compatible with common data models andprogramming languages.
Because of these goals, the HDF5 API is rich and large
March 14, 2013 Michael Stephan Slide 36
Operations Supported by the API
Create groups, datasets, attributes, linkagesCreate complex data typesAssign storage and I/O properties to objectsPerform complex subsetting during read/writeUse variety of I/O-”devices“ (parallel, remote, etc.)Transform data during I/OQuery about file and structure and propertiesQuery about object structure, content, properties
March 14, 2013 Michael Stephan Slide 37
Characteristics of the HDF5 API
For flexibility, the API is extensive300+ functions
This can be daunting but there is hopeA few functions can do a lotStart simpleBuild up knowledge as more features are needed
Library functions are categorized by object type”H5Lite“ API supports basic capabilities
March 14, 2013 Michael Stephan Slide 38
The General HDF5 API
Currently C, Fortran 90, Java, and C++ bindings.C routines begin with prefix H5?? is a character corresponding to the type of object thefunction acts on
Example APIs:H5D : Dataset interface e.g., H5DreadH5F : File interface e.g., H5FopenH5S : dataSpace interface e.g., H5Sclose
March 14, 2013 Michael Stephan Slide 39
Compiling HDF5 Applications
HDF5 wrappers (not fully tested!)h5cc HDF5 C compiler command (similar to mpicc)h5fc HDF5 F90 compiler command (similar to mpif90)h5c++ HDF5 C++ compiler command
Makefile in /bgsys/local/hdf5/examples/jugene/
March 14, 2013 Michael Stephan Slide 40
General Programming Paradigm
Properties of object are optionally definedCreation propertiesAccess property listsDefault values used if none are defined
Object is opened or createdObject is accessed, possibly many timesObject is closed
March 14, 2013 Michael Stephan Slide 41
Order of Operations
An order is imposed on operations by argumentdependencies
A file must be opened before a dataset because the datasetopen call requires a file handle as an argument
Objects can be closed in any order
March 14, 2013 Michael Stephan Slide 42
HDF5 Defined Types
For portability, the HDF5 library has its own defined types:hid t: object identifiers (native integer)hsize t: size used for dimensions (unsigned long or unsignedlong long)hssize t: for specifying coordinates and sometimes fordimensions (signed long or signed long long)herr t: function return valuehvl t: variable length datatype
For C, include hdf5.h in your HDF5 application.
March 14, 2013 Michael Stephan Slide 43
HDF5 Information
http://www.hdfgroup.org/HDF5//usr/local/hdf5/examples/ (JuGene / JuRoPA)/bgsys/local/hdf5/examples/ (JuGene)Google ;-)
March 14, 2013 Michael Stephan Slide 44
Outline
Introduction
Introduction to HDF5 Programming Model and APIs
General Programming Paradigm
Parallele HDF5Overview of Parallel HDF5 Design
March 14, 2013 Michael Stephan Slide 45
PHDF5 Requirements
Support MPI programmingPHDF5 files compatible with serial HDF5 files
Shareable between different serial or parallel platformsSingle file image to all processes
One file per process design is undesirableExpensive post processingNot usable by different number of processes
Standard parallel I/O interfaceMust be portable to different platforms
March 14, 2013 Michael Stephan Slide 46
PHDF5 Implementation Layers
March 14, 2013 Michael Stephan Slide 47
MPI-IO vs. HDF5 I
MPI-IO is an Input/Output API.It treats the data file as a “linear byte stream” and eachMPI application needs to provide its own file view and datarepresentations to interpret those bytes.All data stored are machine dependent except the“external32” representation.External32 is defined in Big Endianness
Little-endian machines have to do the data conversion in bothread or write operations.64bit sized data types may lose information.
March 14, 2013 Michael Stephan Slide 48
MPI-IO vs. HDF5 IIHDF5 is a data management software.It stores the data and metadata according to the HDF5data format definition.
HDF5 file is self-described.Each machine can store the data in its own nativerepresentation for efficient I/O without loss of data precision.Any necessary data representation conversion is done by theHDF5 library automatically.
March 14, 2013 Michael Stephan Slide 49
Programming Restrictions
Most PHDF5 APIs are collectivePHDF5 opens a parallel file with a communicator
Returns a file-handleFuture access to the file via the file-handleAll processes must participate in collective PHDF5 APIsDifferent files can be opened via different communicators
March 14, 2013 Michael Stephan Slide 50
Examples of PHDF5 API
Examples of PHDF5 collective APIFile operations: H5Fcreate, H5Fopen, H5FcloseObjects creation: H5Dcreate, H5Dopen, H5DcloseObjects structure: H5Dextend (increase dimension sizes)
Array data transfer can be collective or independentDataset operations: H5Dwrite, H5DreadCollectiveness is indicated by function parameters, not byfunction names as in MPI API
March 14, 2013 Michael Stephan Slide 51
What Does PHDF5 Support?
After a file is opened by the processes of a communicatorAll parts of file are accessible by all processesAll objects in the file are accessible by all processesMultiple processes may write to the same data arrayEach process may write to individual data array
C and F90 language interfaces
March 14, 2013 Michael Stephan Slide 52
Programming model
HDF5 uses access template object (property list) to controlthe file access mechanismGeneral model to access HDF5 file in parallel:
Setup MPI-IO or POSIX access template(access property list)Open FileAccess DataClose File
March 14, 2013 Michael Stephan Slide 53
Setup MPI-IO access template
Each process of the MPI communicator creates an accesstemplate and sets it up with MPI parallel accessinformation
C: herr t H5Pset fapl mpio(hid t plist id,
MPI Comm comm, MPI Info info);
F90: h5pset fapl mpio f(plist id, comm, info)
plist id is a file access property list identifier
March 14, 2013 Michael Stephan Slide 54
Creating and Opening Dataset
All processes of the communicator open / close a datasetby a collective call
C: H5Dcreate or H5Dopen; H5DcloseF90: h5dcreate f or h5dopen f; h5dclose f
All processes of the communicator must extend anunlimited dimension dataset before writing to it
C: H5DextendF90: h5dextend f
March 14, 2013 Michael Stephan Slide 55
Accessing a Dataset
All processes that have opened dataset may do collectiveI/OEach process may do independent and arbitrary numberof data I/O access calls
C: H5Dwrite and H5DreadF90: h5dwrite f and h5dread f
March 14, 2013 Michael Stephan Slide 56
Dataset transfer property
Create and set dataset transfer propertyC: H5Pset dxpl mpio
H5FD MPIO COLLECTIVEH5FD MPIO INDEPENDENT (default)
F90: h5pset dxpl mpio fH5FD MPIO COLLECTIVE FH5FD MPIO INDEPENDENT F (default)
Access dataset with the defined transfer property
March 14, 2013 Michael Stephan Slide 57
Writing and Reading Hyperslabs
Distributed memory model: data is split among processesPHDF5 uses HDF5 hyperslab modelEach process defines memory and file hyperslabsEach process executes partial write/read call
Collective callsIndependent calls
H5Sselect hyperslab(filespace,H5S SELECT SET,offset,
stride, count, block)
March 14, 2013 Michael Stephan Slide 58
Writing dataset by rows
March 14, 2013 Michael Stephan Slide 59
Writing dataset by rows
March 14, 2013 Michael Stephan Slide 60
Writing dataset by columns
March 14, 2013 Michael Stephan Slide 61
Writing dataset by columns
March 14, 2013 Michael Stephan Slide 62
Writing dataset by pattern
March 14, 2013 Michael Stephan Slide 63
Writing dataset by pattern
March 14, 2013 Michael Stephan Slide 64
Writing dataset by chunks
March 14, 2013 Michael Stephan Slide 65
Writing dataset by chunks
March 14, 2013 Michael Stephan Slide 66