Top Banner
Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath ([email protected]) NCSA May 15, 2002
26

Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath ([email protected]) NCSA May 15, 2002.

Dec 27, 2015

Download

Documents

Eric Booth
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Experiments with Remote Access to Scientific Data using

HDF5

Kun YanRobert E. McGrath([email protected])

NCSAMay 15, 2002

Page 2: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Today’s Talk

1. Background

2. Experiments with JSP, XML, HDF4, HDF5, etc.

3. Experiments with CORBA, Java, HDF5

4. Discussion

Page 3: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Acknowledgements

Kun Yan conducted these experiments as a Research Assistant at NCSA.

This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA grants NAG 5-2040 and NCC5-599. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.

Other support provided by NCSA and other sponsors and agencies.

Page 4: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Java Native

Interface

HDF Libraryand File

H5view,etc

C

Java

Java

C

Applet

Java

CORBAServer

OtherApp. Other

App.

C++

Any

Any

Client/Remote Server/Local

Java Server

Platform

Webbrowser

HTML

XML

Java

Any

Java Native

Interface

Java C

Distributed Product

Demonstrated in Research

Should work, but not demonstrated

Page 5: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

2. Experiments with JSP

• Experiments with JSP, XML, XSL, etc.– Web access to Data and services.– Demonstration of browsing HDF5 datasets

• Generate HTML or XML• Convert data from HDF4 to HDF5 on demand

http://arabica.ncsa.uiuc.edu:7070/hdf5/

Page 6: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Tomcat

• is a standalone web server that supports the Servlet and JSP specifications.

• developed under the Jakarta project at the Apache Software Foundation.

• integrated into the J2EE Reference Implementation.

• Tomcat 4.x (Catalina, Servlet 2.3, JSP 1.2)

• Tomcat 3.3

Page 7: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

JavaServer Pages

• JavaServer PagesTM (JSPTM) technology

• rapidly develop and easily maintain, dynamic web pages

• uses XML-like tags and Java scriptlets

• application logic can reside in server-based resources

Page 8: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

The Flow of a Request

Page 9: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

beans

request

response

clientJSP

DB datalib

XSL

XMLbeansbeans

Xalan.jarXerces.jar

HTML

response

How JSP and Beans Work together

Application

Page 10: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

3. Experiments with CORBA

• Created a demonstration CORBA server that reads HDF5– Written in C++, linked directly to HDF5 library.

• Created a pure-java client– No CORBA required on the client– Can be used from an applet within a standard

Web browser

Page 11: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Technology used

• Orbacus 4.1.0 (C++) (free, full source)

• Java 1.3 or above

• HDF5.1.4.3, HDF5 C++ interface

Page 12: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

HDF5 CORBA Experiment -- Design DiagramCORBA Object Hierarchy

Page 13: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

HDF5 CORBA Experiment – IDL File1. module HDF5{

2. struct ObjInfo{3. long oid;4. long fid;5. long type;6. string name;7. long numOfMembers;8. };9. 10. typedef sequence<long> Sizes;

11. struct DatasetInfo{12. string name;13. long numOfDim;14. Sizes dimSizes;15. Sizes dimMaxSizes;16. long dataTypeClass;17. long dataTypeSize;18. long long storageSize;19. };20. 21. struct DatatypeInfo{22. string name;23. long type;24. long size;25. };

26. interface H5Obj{27. void getObjInfo(out ObjInfo obji);

28. void getNumAttributes(out long attrNum);29. void openAttributes (out long attr);30. };31.

32. interface H5DatasetObj:H5Obj{33. void getDatasetInfo(out DatasetInfo dsinfo);34. long long getStorageSize();35. };36. 37. interface H5GroupObj:H5Obj{38. ObjInfo getGMember(in long fid, in string gName, in long index); 39. void openGroup(in string gName, out H5GroupObj h5gobj);40. void getDatasetInfo(in string dsName, out DatasetInfo dsinfo);41. void openDataset(in string dsName, out H5DatasetObj h5ds);42. void getDataTypeInfo(in string ttName, out DatatypeInfo dtinfo);43. };44. 45. interface H5FileObj{46. void getTOC(out ObjInfo obji);47. void openRootGroup(out H5GroupObj h5gobj); 48. };49. 50. interface H5FileAccess{51. void openH5File(in string fileName, out H5FileObj h5fobj);52. void numOfGMembers(in long fileId, in string gName, out long

gNum);53. string closeH5File(in long fileId);54. };55. 56. };

Page 14: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

HDF5 CORBA Experiment -- Design Diagram C++ Servant Object Hierarchy

H5FileAccess H5FileObjH5Obj

H5GroupObjH5DatasetObj

H5FileAccess_impl

H5::H5Filepoa_

openFile:closeFile:

H5FileObj_impl

H5::H5FileObjInfopoa_

getTOCopenRootGroupsetFileName: H5GroupObj_impl

H5::H5Grouppoa_

openGroupgetGMembersopenDataset:getDatasetInfo:getDatatypeInfo:

H5Obj_impl

ObjInfopoa_

getObjInfosetObjName:

H5DatasetObj_impl

H5::H5DarasetDatasetInfo

getDatasetInfogetStorageSize

Page 15: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

HDF5 CORBA Experiment -- How to Use the Client Applet

• Select file name from drop-down list (file resides corba server side)

• Select Flags and Access Property of the selected file

• Click on “open file” to open the file

• Click on “get file TOC” to get the file object info

• Click on “open root” to open the root group

• Click on “get group TOC” to get the root group object info

• Click on “iterate group” to get the group member’s info of the root and each subgroup

• Input dataset name including path, then click on “get dataset” to get the dataset object

– Click on “get dataset TOC” to get the dataset object info

– Click on “get storage size” to get the storage size of the opened dataset object

• Input dataset name including path, then click on “get dataset info” to get object info of the given dataset

• Input datatype name including path, then click on “get datatype info” to get object info of the given datatype

Page 16: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

URLs

• JSP experiments: http://hdf.ncsa.uiuc.edu/HDF5/XML/JSPExperiments/

• http://hdf.ncsa.uiuc.edu

• http://hdf.ncsa.uiuc.edu/h4toh5/

• http://hdf.ncsa.uiuc.edu/HDF5/XML/

• http://hdf.ncsa.uiuc.edu/hdf-java-html

Page 17: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

4. Discussion

Page 18: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Java Native

Interface

HDF Libraryand File

H5view,etc

C

Java

Java

C

Applet

Java

CORBAServer

OtherApp. Other

App.

C++

Any

Any

Client/Remote Server/Local

Java Server

Platform

Webbrowser

HTML

XML

Java

Any

Java Native

Interface

Java C

Distributed Product

Demonstrated in Research

Should work, but not demonstrated

Page 19: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

HD

F5

CO

RB

A E

xperim

ent -- U

ser Inte

rface

Page 20: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

HDF5 CORBA Experiment -- Examples of Object Info window

Page 21: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.
Page 22: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Other HDF ToolsOther HDF Tools

Page 23: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Viewer/Editor

GeneratorSource code

ASCII

HDF5 file

HDF5 file

XML

XML

XML

HDF5 file

Java

HDF4

h4toh5

HDF4

Page 24: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Dumper

HDF DDL

HDF5 file XML

Non-Java

(C)

h4toh5HDF4 file HDF5

h5toh4HDF5 file HDF4

gif2hdfGIF file HDF5

fp2hdfflat file HDF5

H4 to H5 LIbrary

Page 25: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

Silly XMLTricks

Netcdf to HDF5, via XML using XSL

netcdf ncxdump nc.xmlnetcdf.dtd

h5.xml

xalan

nctoh5.xsl

h5genhdf5

Page 26: Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath (mcgrath@ncsa.uiuc.edu) NCSA May 15, 2002.

URLs

• JSP experiments: http://hdf.ncsa.uiuc.edu/HDF5/XML/JSPExperiments/

• http://hdf.ncsa.uiuc.edu

• http://hdf.ncsa.uiuc.edu/h4toh5/

• http://hdf.ncsa.uiuc.edu/HDF5/XML/

• http://hdf.ncsa.uiuc.edu/hdf-java-html