Top Banner
NCSA/Univ of Illinois at Urbana-Champaign HDF HDF 1 Mike Folk,HDF Group http://hdf.ncsa.uiuc.edu/ National Center for Supercomputing Applications University of Illinois at Urbana-Champaign HDF HDF HDF-EOS Workshop II HDF-EOS Workshop II Sept. 22, 1998 Sept. 22, 1998
51

HDF

Jan 27, 2015

Download

Technology

Source: http://hdfeos.org/workshops/ws02/presentations/folk/folk.ppt
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 1

Mike Folk,HDF Grouphttp://hdf.ncsa.uiuc.edu/

National Center for Supercomputing Applications

University of Illinois at Urbana-Champaign

HDFHDF

HDF-EOS Workshop IIHDF-EOS Workshop IISept. 22, 1998Sept. 22, 1998

Page 2: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 2

I. OverviewI. Overview

• What is HDF?

• HDF software

• HDF objects

• Who uses HDF?

Page 3: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 3

What is HDF?What is HDF?

• A data file format for managing scientific data

in heterogeneous environments

• I/O software, utilities, search and access tools

• A standard used widely among scientists

• Enabling data interoperability since 1988

Page 4: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 4

Requirements for scientific dataRequirements for scientific data

• “Scientific” data

– A variety of data types and structures

– Large structures

– Metadata in a variety of forms

• Portability

– I/O library works on many machines

– Data easily moved from machine to machine

• Fast I/O

• Efficient storage

Page 5: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 5

University of Illinois at Urbana-Champaign

Why use HDF?Why use HDF?

• Share scientific data

– in heterogeneous computing environments

– with others of like interests

• To use software that understands HDF

• To improve I/O performance

• To improve storage efficiency

• To use an open standard

Page 6: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 6

HDF Object TypesHDF Object Types

Palette

8-bit raster

24-bit raster

Scientific Data Sets (SDS)(multidimensional arrays)

Vdata (tables)

lat | lon | temp

----|-----|-----

12 | 23 | 3.1

15 | 24 | 4.2

17 | 21 | 3.6March 15, 1990. Simulation with

k=10.0, beta=1.22e3. Calculate

the magnitude ...

Vgroup

AnnotationThis HDF file containsone example of eachobject type

Page 7: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 7

An HDF File: A Collection ofAn HDF File: A Collection of

Scientific Data ObjectsScientific Data Objects

HDF file containing four 3-D arraysHDF file containing four 3-D arrays

Page 8: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 8

Mixing HDF Objects in One FileMixing HDF Objects in One File

March 15, 1990. Simulation with

k=10.0, beta=1.22e3. Calculate

the magnitude ...

3-D array3-D arrayRaster imageRaster image

2-D array2-D array

groupgroup

RasterRaster

imageimage

palettepalette

annotationannotation

HDF fileHDF file

Page 9: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 9

The HDF File StructureThe HDF File Structure

Primitive Data Object

tag ref offset length

data element

tag ref offset length

data element

Data descriptor (DD):Data descriptor (DD):

Data:Data:

(Everything else in an HDF file is build out of these.)(Everything else in an HDF file is build out of these.)

(12 bytes)(12 bytes)

Page 10: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 10

Example of hierarchical HDF structureExample of hierarchical HDF structureRaster Image Group (original format)Raster Image Group (original format)

dim's 2 offst lenimage 3 offst lenpalette 3 offst len 4 offst lenRIG

tag ref

Blksize

Nextblk

2

400 x 600

Data elements:

tag ref

palette

image

dim's

3

3

Index:

Page 11: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 11

Other features of interestOther features of interest

• Data compression

• The data part of an object can be stored in

“special” ways

– As a series of linked-blocks

– As a portion of an external file

– In chunks, or tiles

Page 12: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 12

Utilities and applications forUtilities and applications formanipulating, viewing, andmanipulating, viewing, andanalyzing data in HDF files.analyzing data in HDF files.

A software library:A software library:–– High-level, object-specific APIs. High-level, object-specific APIs.–– A low-level interface. A low-level interface.

A physical file format for storingA physical file format for storingscientific data.scientific data.

General Applications

ApplicationProgramming

Interfaces

Low-levelInterface

HDFfile

HDF SoftwareHDF Software

Page 13: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 13

HDF Applications SoftwareHDF Applications Software

• Free software

– NCSA HDF library and utilities

– Other software

• Commercial/other software that “understands”

– all of HDF (Noesys, IDL)

– certain HDF objects (MATLAB, AVS)

– certain HDF applications (SHARP, WIM)

• http://hdf.ncsa.uiuc.edu/tools.html

Page 14: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 14

University of Illinois at Urbana-Champaign

What platforms does HDF run on?What platforms does HDF run on?

• Sun: Solaris, SunOS

• SGI: Indy, Power Challenge, Origin, Cray C90,

YMP, T3E

• HP9000, HP-Convex Exemplar

• IBM: RS6000, SP2

• DEC: Alpha/Digital UNIX, OpenVMS

VAX: OpenVMS

• Intel Pentium: Solarisx86, Linux, FreeBSD 2.2,

Windows: NT/95

• PowerPC: Mac-OS

Page 15: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 15

HDF ObjectsHDF Objects

Palette

8-bit raster

24-bit raster

Scientific Data Sets (SDS(multidimensional arrays

Vdata (tables)

lat | lon | temp

----|-----|-----

12 | 23 | 3.1

15 | 24 | 4.2

17 | 21 | 3.6March 15, 1990. Simulation with

k=10.0, beta=1.22e3. Calculate

the magnitude ...

Vgroup

AnnotationThis HDF file containsone example of eachobject type

Page 16: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 16

Array Number Type

Dimension X,Y,Z

SDS attributes

DimensionAttributes

DimensionNumber Type

Dimension Scale

DimensionAttributes

DimensionNumber Type

Dimension Scale

DimensionAttributes

DimensionNumber Type

Dimension Scale

Dimension X Dimension Y Dimension Z

Scientific Data Set

Array

SDS: Scientific Data SetSDS: Scientific Data Set

Page 17: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 17

Writing an SDSWriting an SDS

• Open a file and start SD interface SdstartSdstart

• Create or open an SDS SDcreate/Sdselect

• Write out the SDS Sdwritedata

• Terminate access to the SDS SDendaccess

• Terminate access to SD interface SDend

and close the file

Page 18: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 18

Class

Vdata Name

Class_1

General vdata

Records

Field_3Field_2 Field Names

2.6 0.00

Fields

3.5 1.22

1.5 23.50

8

5

2

5.3 6.93

Field_1

6

7 12.303.8

Vdata: HDF table structureVdata: HDF table structure

Page 19: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 19

Vgroup System OrganizationVgroup System Organization

Vgroup

Data

Vgroup Vgroup

Data

Data

Data

Data

Vgroup

UNIX File System OrganizationUNIX File System Organization

Directory

File

Directory Directory

File

File

File

File

Directory

Vgroup: HDF Grouping StructureVgroup: HDF Grouping Structure

Page 20: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 20

VgroupVgroup

March 15, 1990. Simulation with

k=10.0, beta=1.22e3. Calculate

the magnitude ... 2-D array2-D array

Vgroup

RasterRaster

imageimageannotationannotation

Page 21: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 21

Image Array

Required

Array Name

Pixel Type

Width

Height

General Raster

name = IMAGE_1

Image Attribute

Optional

Palette

GR: HDF general raster imageGR: HDF general raster image

Page 22: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 22

8-bit Raster Image Pixel

Palette

0 00000000 00000000 00000000

1 00000001 00000001 00000001

2 00000010 00000010 00000010

... . . .

192 11000000 11000000 11000000

. . .

253 11111101 11111101 11111101

254 11111110 11111110 11111110

255 11111111 11111111 11111111

Color Look-up Table (Color Components)

Entry red green blue

HDF PaletteHDF Palette

Page 23: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 23

“This is a file label.”

“This is a file description.”

“This is another file label.”

“This is another file description.”

File Annotations

"This is an SDS label." "This is RIS24 description." "This is a RIS8 description."

Object Annotations

HDF File

RIS8RIS24SDS

HDF AnnotationsHDF Annotations

Page 24: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 24

Who uses HDF?Who uses HDF?

Page 25: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 25

A Sampling of HDF UsersA Sampling of HDF Users

NCSA-affiliated Science teams NCSA-affiliated Science teams Visualization, exchange, archivingVisualization, exchange, archiving

Vendors: Mathworks, Fortner Vendors: Mathworks, Fortner Visualization, data analysis, exchange,Visualization, data analysis, exchange,

Software, RSI, etc. Software, RSI, etc. data managementdata management

TRAPPIST TRAPPIST Exchange, analysis, visualization ofExchange, analysis, visualization of

(Euro consortium) (Euro consortium) non-destructive testing data (STEP)non-destructive testing data (STEP)

NIST Reactor & Cold Neutron NIST Reactor & Cold Neutron Data exchange, archivingData exchange, archiving

Res FacilityRes Facility

Stanford Univ EE DeptStanford Univ EE Dept Persistent storage of simulation dataPersistent storage of simulation data

Johns Hopkins Appl Physics LabJohns Hopkins Appl Physics Lab Planetary data exchange, archivePlanetary data exchange, archive

Inst Adv Tech in the HumanitiesInst Adv Tech in the Humanities Image annotationImage annotation

Comp Graphics Unit ManchesterComp Graphics Unit Manchester Image processing, visualizationImage processing, visualization

Page 26: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 26

Major User #1: EOSDISMajor User #1: EOSDIS

•• User support for scientists, data producers, etc.User support for scientists, data producers, etc.

•• Library and file structure improvementsLibrary and file structure improvements

•• HDF tools, utilities, access softwareHDF tools, utilities, access software

•• Software maintenance and QASoftware maintenance and QA

Page 27: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 27

Major User #2: ASCIMajor User #2: ASCI

• ASCI Data Models and Formats (DMF) Group

– open standard exchange format and I/O library for ASCI

– DOE tri-lab ASCI applications

• HDF requirements

– large datasets (> a terabyte)

– ASCI data types, especially meshes

– good performance in massive parallel environments

– emphasis on HDF 5

Page 28: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 28

II. Major HDF ProjectsII. Major HDF Projects

• Support, maintenance, QA

• Java applications

• Remote access

• HDF5: next generation

Page 29: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 29

Support, Maintenance, QASupport, Maintenance, QA

•• User supportUser support

•• Library and file structure improvementsLibrary and file structure improvements

•• Software maintenance and QASoftware maintenance and QA

•• Documentation, tutorials, etc.Documentation, tutorials, etc.

•• HDF tools and utilitiesHDF tools and utilities

Page 30: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 30

Java applicationsJava applications

• A Java HDF API

– Basis for tools that access HDF

• A Java HDF Viewer

– HDF browser/visualizer

• Java Scientific Data Server Prototype

– Lessons learned about scientific data servers

Page 31: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 31

Java HDF API (JHI)Java HDF API (JHI)

• Java HDF Interface

– Complete Java interface to the HDF library

– Analogous to the HDF F77 interface

– To be used by Java apps to access HDF files

– http://hdf.ncsa.uiuc.edu/java-hdf-html/

Page 32: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 32

Java-based HDF Viewer (JHV)Java-based HDF Viewer (JHV)

• client-based HDF viewer

– browsing

– data viewing (“spreadsheet”)

– image viewing and plotting

– image processing

– animation

– data import/export

Page 33: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 33

JHVJHV

Page 34: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 34

Remote Data AccessRemote Data Access

• Accessing Scientific Info Using Networks

– Scientific data is large and complex,

– Need to locate objects & subsets within large files

– The “download and browse” approach is infeasible

Page 35: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 35

The SDB: Web-based Server-side DataThe SDB: Web-based Server-side Data

BrowserBrowser

• Remote interactive browser for data and

metadata

– “A conversation with the data”

– Views provided at increasing levels of detail

– Returns subsets of data

– Performs simple data translations

– Adaptable to different domains (EOS, Astronomy)

– Reads HDF, netCDF, CDF, FITs formats

Page 36: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 36

Sample SDB applicationsSample SDB applications

• DIAL (Data & Info Access Link)

– HDF-EOS data access

• Distrib. Ocean Data System (DODS) WP-ESIP

– High-speed remote subsetting earth science datasets

• Boeing

– Extract space/time info from large image collections

Page 37: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 37

Other workOther work

• Java applets for scientific data browsing

• A VRML server for Radio Astronomy data

• Interoperable search with Z39.50

Page 38: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 38

HDF5: Next generation HDFHDF5: Next generation HDF

• Features

– Large arrays and files (>2GB)

– Simple, comprehensive data model

– Emphasis on Parallel I/O

– Alternate storage structures

• Collaborations

– Mesh data standard for ASCI physics

– Integrate with commercial object store (DLI)

Page 39: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 39

Basic HDF5 data objectBasic HDF5 data object

Record

int8 int4 int16 float32

Dimensionality: 5 x 3 x 4

Number type:

Page 40: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 40

III. HDF and HDF-EOSIII. HDF and HDF-EOS

• HDF profiles and HDF-EOS

• HDF configuration record project

• Future directions?

Page 41: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 41

HDF Profiles and HDF-EOSHDF Profiles and HDF-EOS

• HDF does not support specific application areas.

• To share files, users must agree on how to

organize them.

• HDF user groups create “profiles” describing

how to organize data in their HDF files.

• Profiles standardize domain-specific HDF files

• Example: HDF-EOS

Page 42: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 42

HDF-EOS profilesHDF-EOS profiles

• Profiles for Earth remote sensing data and in-

situ measurements

• Includes

–– Standard metadata for EOS dataStandard metadata for EOS data

–– API and library that reads/writes HDF-EOS filesAPI and library that reads/writes HDF-EOS files

–– Utilities to simplify the work of analyzing andUtilities to simplify the work of analyzing and

visualizing HDF-EOS files.visualizing HDF-EOS files.

Page 43: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 43

General ApplicationsHDF-EOS APIHDF-EOS API

ApplicationProgramming

Interfaces

Low-levelInterface

HDFfile

HDF-EOS software layersHDF-EOS software layers

HDF-EOS ApplicationsHDF-EOS Applications

HDF-EOS

profiles

Page 44: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 44

“HDF Configuration Record” (HCR)“HDF Configuration Record” (HCR)

To simplify the tasks of defining,To simplify the tasks of defining,

comparing, and producingcomparing, and producing

HDF-EOS filesHDF-EOS files

Page 45: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 45

HCRHCR

• Formal descriptions of HDF objects

• Based on ODL (Object Description Language)

• Supports HDF-EOS swath, grid, point

Page 46: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 46

HCR of SwathHCR of Swath

/* Project XYZ */

/* First version defined on June 10th, 1998 */

OBJECT = SWATH

NAME = SCAN1

OBJECT = Dimension

NAME = GeoTrack

Size = 1200

END_OBJECT = Dimension

OBJECT = Dimension

NAME = GeoCrossTrack

Size = 205

END_OBJECT = Dimension

OBJECT = Dimension

NAME = DataX

Size = 2410

END_OBJECT = Dimension

END_OBJECT = SWATH

END

Page 47: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 47

HCR Utilities:HCR Utilities:

• Convert HCR ↔↔↔↔ HDF-EOS

• Edit HCR and HDF-EOS

• Compare HCR with HDF-EOS file

Page 48: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 48

HCR editHCR edit

swath pageswath page

Page 49: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 49

HCR editHCR edit

ODL pageODL page

Page 50: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 50

Future possibilitiesFuture possibilities

• Deploy HCR?

• HDF as an archive format?

• Java tools for HDF-EOS?

• HDF5?

– HDF5 ↔↔↔↔ HDF4 conversion

– HDF4 API on HDF5

– HDF-EOS on HDF5

Page 51: HDF

NCSA/Univ of Illinois at Urbana-Champaign

HDFHDF 51

HDF InformationHDF Information

• HDF Information Center

– http://hdf.ncsa.uiuc.edu/

• HDF Help email address

[email protected]

• HDF users mailing list

[email protected]