October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 1 HDF5-iRODS Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS Workshop XII October 16, 2008
Jan 05, 2016
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 1
HDF5-iRODSPeter Cao
The HDF Group
Mike Wan
San Diego Supercomputer Center
HDF and HDF-EOS Workshop XII
October 16, 2008
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 2
Imagine
1 GB
100 Frames x 1 GB = 100 GB
HPSSHPSS
DBDBHPCHPC
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 3
Outline
• HDF5-iRODS module• Applications• Demo (if time permits)
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 4
What is iRODS?
• Stands for i Rule Oriented Data Systems.• Developed by the Storage Resource Broker
(SRB) team at the San Diego Supercomputer Center (SDSC).
• A data grid software system that enables a customizable architecture for sharing data distributed across heterogeneous resources.
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 5
What is iRODS?
Distributed Storage Database System Rule System
For more information and download, visit
www.irods.org
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 6
Motivation
iRODS HDF5
Distributed data system
Indexing and searching
Access control, etc.
Large and diverse data
High-performance I/O
Subsetting, etc.
High-performance distributed data system
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 7
Whole File Access
clientclient
Get the file
Transfer large file – slow!
HDF5HDF5
serverserverI need to see the eye of
Hurricane Bob!
HDF5HDF5
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 8
HDF5 Object or Subset Level Access
clientclient
HDF5HDF5
Get me the eye of hurricane Bob
Small transfer – fast!
serverserverI need to see the eye of
Hurricane Bob!
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 9
HDF5-iRODS Module
Distributed Storage
Database SystemRule System
HDF5 iRODS Module
Micro-services
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 10
HDF5-iRODS Data Flow
iRODS message(pack/unpack)
clientclient serverserver
HDF5 Object or Subset
(File, Group, Dataset,
Subset of Dataset, Attribute)
HDF5 LibraryHDF5 Library
iRODS message(pack/unpack)
HDF5 Object or Subset
(File, Group, Dataset, Subset of Dataset,
Attribute)
HDF5HDF5
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 11
New iRODS Micro-services
• Five iRODS micro-services msiH5File_open msiH5File_close msiH5Dataset_read
• reads entire dataset or subset of dataset msiH5Dataset_read_attribute msiH5Group_read_attribute
File
Rule Engine msiH5Dataset_read H5Dataset.read()
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 12
HDF5-Enabled iRODS Server
• HDF5 library • Other external libraries (SZIP, ZLIB)• iRODS version 1.1 or later from
https://www.irods.org/index.php/Downloads/
Follow the README instruction at module/hdf5
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 13
Client Application Requirements
• HDF5 object header files and client handlers• iRODS client library and header files• HDF5-iRODS JNI for Java applications only• $HOME/.irods/.irodsEnv
For more information and download, visit
http://www.hdfgroup.org/projects/irods
irodsHost 'kagiso.hdfgroup.uiuc.edu'irodsPort 1247irodsUserName 'rods‘…
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 14
Example: HDFView
Client Application HDF5-EnablediRODS Server
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 15
Example: HDFView
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 16
Example: islice
For more information, visit
flash.uchicago.edu
• Command-line tool to visualize data produced by FLASH simulation runs
• Data is huge (~ 100 GB)• Interesting part is small
adaptive mesh16*16*16*47531
FLASH is an adaptive-mesh simulation code for astrophysical hydrodynamics problemsFLASH is an adaptive-mesh simulation code for astrophysical hydrodynamics problems
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 17
Example: islice
Breakout point Ash Flow
Star
A slice from a 3D simulation of The Detonation of a White Dwarf Star
Collision focus point
“./islice -t flash.pal -m rpv1 -p 2 rundir_055_8km_hdf5_plt_cnt_0424”
2048*2048*8(32MB)
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 18
This project is sponsored by CIP/NLADR, NSF PACI Project in Support of the Collaboration between the National Center for Supercomputing Applications (NCSA) and the San Diego Supercomputer Center (SDSC). The project is managed under the CyberInfrastructure Partnership (CIP), a joint effort led by NCSA and SDSC to help scientists and engineers take full advantage of the high-end CyberInfrastructure resources funded by the National Science Foundation (NSF).
Thank You!
Questions/comments?
October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 19