HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community

TheHDFGroup

www.hdfgroup.org

HDF5Update

High-DataRateMXMee9ngNSLS-IIatBNLMay26,2016

[email protected]

1

www.hdfgroup.orgHDRMXNSLS-IIatBNL

Outline

2

•  HDF5featuresrequestedandsponsoredbySynchrotronCommunity(2012-2016)• DirectchunkI/O(PSI,Dectris)• Dynamicallyloadedfilters(DESY)•  SingleWrite/Mul9pleReaderorSWMR(DLS,ESRF,DESY)

• VirtualDataset(VDS)(DLS,DESY,XFEL)•  Newrequirementsandfeatures•  HDF5roadmapfor2016


Directchunkwrite:H5DOwrite_chunk

5/25/16 3


PerformanceresultsforH5DOwrite_chunk

5/25/16 4

1 Speed in MB/s 2 Time in seconds

Test result on Lunux 2.6, x86_64 Each dataset contained 100 chunks, written by chunks


Dynamicallyloadedfilters

•  Problemswithusingcustomfilters•  “Offtheshelf”toolsdonotworkwiththethird-partyfilters• h5dump,MATLABandIDL,etc.

• HDF5toolscannotreadfilecreatedby• h5py,PyTables,etc.

•  Solu9on• ModifyHDF5sourcewithyourcodeanddistributeit

•  Andwhatwillhappenifauserwantsfiltersfromthedifferentdistribu9ons????Oh….No….

• Usea1.8.11andlater• ProvidemaintainedlibraryofHDF5compressionfilters

55/25/16


Approach

•  Therearepredefineddefaultloca9onswheretheHDF5librarysearchesthesharedlibrariesorDLLswiththeHDF5filterfunc9ons.

•  Thedefaultloca9onmaybeoverwrigenbyanenvironmentvariable.

•  Encoding:Standardprogrammingmodeltoregistercompressionfilteronwriteopera9on

•  Decoding:Transparentonread;nouserac9onrequired.

• Worksgreat,but….

65/25/16


User’sheadacheandpossiblesolu9ons

• WheretofindHDF5plugins?•  Noofficialdistribu9onsite

•  ExampleofHDF5filtersdistribu9onsonGithub! hgps://github.com/dectris/HDF5Plugin! hgps://github.com/Blosc/hdf5-blosc! hgps://github.com/nexusformat/HDF5-External-Filter-Plugins! hgps://svn.hdfgroup.org/hdf5_plugins/

•  TheHDFGroupcanhelpwith• Maintainingcommonrepository•  Tes9ngpluginwiththereleases•  Distribu9ngsourceandbinariesforLinux,MacandWindows•  Freeandlicensed(forfee)binarydistribu9ons

75/25/16


SWMR:Dataaccesstofilebeingwrigen

HDF5File

Writer Reader

…whichcanbereadbyareader…

withnoIPCnecessary.

Newdataelements…

…areaddedtoadatasetinthefile…

04/01/16


SWMR

9

•  ReleasedinHDF51.10.0•  Restrictedtoappend-dataonlyscenario•  SWMRdoesn’tworkonNFS•  Filesarenotcompa9blewithHDF51.8.*libraries

•  Useh5format_converttool•  ConvertsHDF5metadatainplace• Norawdataisrewrigen


VDS

10

•  Datastoredinmul9plefilesanddatasetscanbeaccessedviaonedataset(VDS)usingstandardHDF5read/write


VDSExample

11

Seriesofimages

A

B

C

D

EF

a.h5

c.h5

e.h5

b.h5

d.h5

f.h5

A

C

E

B

D

F

t1 t2

t3 VDS.h5

111Imageat9met2

A

B

C

D

EF

A

B

C

D

EF

AADataset A

Dataset C

Dataset E

CC

Dataset B

DatasetD

Dataset FF

F

DD

EE

BB

M

n

kn

nk

k

n k n

n k

k

M

t2

t2

t2

t2

t2

t2


VDS

12

•  VDSworkswithSWMR•  FilewithVDScannotbeaccessedbyHDF51.8.*libraries

•  Useh5repacktooltorewritedata(1.10.0-patch1)


Newrequirementsandfeatures?

13

•  Tellusyourneeds:• Mul9-threadedcompressionfilters• H5DOread_chunkfunc9on•  FullSWMRimplementa9on•  Performance•  Backward/forwardcompa9bility

•  Otherrequests?


HDF5Roadmapfor2016

14

• May31-HDF51.10.0-patch1• H5repack,Windowsbuilds,FortranissuesonHPCsystems

• Mid-summerHDF51.10.1•  Someinternalissuesfoundfor1.10.0

•  December• HPCfeaturesthatdidn’tmakeitinto1.10.0release


Ques9ons?

Thankyou!

15

HDF5 updatemedsbio.org/meetings/BNL_May16_HDRMX_Meeting_Files/HDF5... · 2016-06-20 · HDRMX NSLS-II at BNL Outline 2 • HDF5 features requested and sponsored by Synchrotron Community

Documents