ESDIS Status

05 December, 2002 HDF & HDF-EOS Workhop VI 1

ESDIS Status

Richard UllmanESDIS Project

[email protected]


Outline

• ESDIS General Status• HDF-EOS Plans• Website http://hdfeos.gsfc.nasa.gov


ESDIS Science Data Services Today

• EOSDIS provides support for high data volumes from Aqua, Terra, and Landsat 7, and continues to support QuikSCAT, ACRIMSat, SAGE-III, JASON and pre-EOS-era data including TRMM, UARS, TOPEX/Poseidon, RADARSat, and others. – EOSDIS finalizing preparations to support ICESat and SORCE– In all, EOSDIS is providing data processing, archival, and/or distribution

for over 15 Earth science satellite missions.• EOSDIS has set a new benchmark for data management. The total

volume of the science data in our archives totals over 2 Petabytes. Since 1998, the science data volume managed by the EOSDIS has increased eight-fold, and continues to grow at a rate of over 2 Terabytes per day

• In FY02, EOSDIS provided more than 16 million data and information products to over 1.8 million individuals.


FY94/95 FY96 FY97 FY98 FY99 FY00 FY01 FY02

Science Operations Supported

Heritage Missions

T/P, UARS, TOMS, ERBE

Terra ACRIMSatTRMM

Landsat 7 QuikSCAT

Aqua GRACE

SAGE III JASON

SeaWiFS Radarsat

Products Distributed

15 Million

Archive Growth

Archive Volume has doubled each year for the past three years

2 Petabytes

Science Operations Timeline

http://seawifs.gsfc.nasa.gov/SEAWIFS/SEASTAR/seawifs.mpeg


EOSDIS DAAC Data delivery FY’02October 1, 2002 through September 30, 2002

• Over 1.8 million distinct users• 16,013,980 products delivered • Notes to charts on following slides:

– Distinct users includes users accessing DAAC web pages, including web-crawlers

– Distinct user type is based on email addresses of users or URLs– Product delivered is defined as the smallest deliverable unit of data – Product delivery breakdown is based on email addresses of users

receiving ECS and Non-ECS data– “FTP Delivery” are to URLs not mapped to specific domains– “Foreign Other” includes foreign email addresses whose country is

known but domain-type could not be determined


16,013,980 Data Products DeliveredOctober 1, 2001 - September 30, 2002


Over 1.8 Million Distinct UsersOctober 1, 2001 - September 30, 2002


ESDIS Status

• After years of development, ECS is operational and generally recognized as successful.

• Primary ECS development contract is essentially completed.

• ECS Maintenance and Development (EMD) will emphasize maintenance more than development.

• Already the majority of ESDIS budget has shifted from development to operations.

• Still two major areas of new capability (see posters at AGU for more information):– Data Pools– ECHO


HDF-EOS Plans

• HDF-EOS 5 development is nearly complete.• Continue to maintain, port to newer operating systems,

bug-fix.• Need the advise of this community workshop - we will

discuss this afternoon.– What tools or capabilities are now needed?

• HDF-EOS 2 and HDF-EOS 5– When is it the right time to press EOS science teams to migrate to

HDF-EOS 5?– What steps should NASA take to facilitate?


Data Pools Concept: The Right Data, the Right Way, Right Now!

Value-Added Providers

Data Providers

Data TailoringWorkflow

Management

End Users

User-definedViews, Presentations and Data Access Requests

Dynamic Web and FTP Data Views,User-specified Data Access

Other Data Pools

Other Data Pools

Data Producers

DataServices

DataItems

Data ServiceDevelopers

GeoPIPE

PIPE = Personalized Information Presentation Engine


General Capabilities– The Right Data: Data Location

• Groups, themes, bookmarks and views• Navigation and machine-based location• Science views (e.g., science metadata)• Applications views (e.g., OGIS coverage server)• Location aids (e.g., geopolitical overlays)• External location-support services

– The Right Way: Data Tailoring• Data reduction, manipulation and reformatting services• Virtual data products• Workflow management and execution monitoring• External tailoring services

– Right Now: Rapid Access• Low latency data transfers• Secure remote file access• On-the-fly data compression• Automated request routing and load balancing• Near real-time data


EOS ClearingHOuse (ECHO) http://eos.nasa.gov/echo

• ECHO is a metadata clearinghouse– A single Internet portal for Earth science metadata search– Index of data provider inventory-level data holdings metadata.

• .ECHO is a data order broker– Forwards orders for data discovered to the data providers to fill.– Data providers retain customer fulfillment service

• ECHO is a data service broker– Registered service are associated with registered datasets– Four kinds of service association

• Advertised, Context Passing, Brokered, Order Option

• ECHO is an open client API for custom user clients – The EOS Data Gateway (EDG) is ESDIS’ ECHO client


ECHO Data Providers

• Current ECHO Data Providers– EOSDIS Core System DAACs

(17% thus far)• EDC Land Processes DAAC • Goddard DAAC • NSIDC DAAC

– ORNL DAAC (100%)• ECHO holds the metadata for

over 3 million granules, and growing

ECHO Metadata Holdings

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000

Jan-06 Feb-06

Mar-06 Apr-06 May-06

Jun-06 Jul-06 Aug-06

Sep-06

Date

ORNL_DAACNSIDC_ECSLP_ECSGSFC_ECS


HDF-EOS Tools and Information Web Sitehttp://hdfeos.gsfc.nasa.gov

[email protected]


http://hdfeos.gsfc.nasa.gov


Website status

• A resource for discovering about hdf-eos in particular.– This workshop series’ presentation archive– Links to hdf-eos tools

• Site has been revamped according to comments received at the last workshop.– Tools download page now has opportunity for user feedback.– Workshop presentations are keyword searchable

• New features planned– Better introductory material.

• Post and organize documentation of HDF-EOS.• Better navigation to NCSA site for HDF

– Host hdf-eos “web forum” • Incorporate the [email protected] listserv


HDF-EOS Profile

Richard UllmanESDIS Project

[email protected]


History

• September 1993, HDF adopted as baseline standard for EOSDIS Core System standard data product generation, archival, ingest, and distribution capabilities

• Dec. 94 - ECS Engineering Support Directive to create HDF-EOS• June 1996, HDF-EOS v1.0 library released• Upgrades every 6 mo.,

– Current version 2.8 on HDF 4• HDF5 support (called HDF-EOS 5) beginning November 2000

– Current version 5.1.3 on HDF5


HDF-EOS Data Objects

• Point• Swath• Profile (Swath subtype)• Grid• Zonal (HDF-EOS 5 only)


HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF4 with a different interface. – First released in 2000.

• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Support same data structures– Added prefix ‘HE5_’ to HDF-EOS 2 functions. – Doesn’t preclude HDF5 functionality. – Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG


HDF-EOS 5 Functionality

• Basic File I/O• Fill Values• Compression• Chunking/Tiling• Swath Interface• Grid Interface• Point Interface• Profile Interface

• Global (File), Group & Local Attributes

• External Data Files• Subsetting• Unix/Linux Support• Threadsafe Version• FORTRAN, C, C++• General Table Interface

(proposed)


The new ADDITIONAL Group has global (file) attributesThe new functionality is added to the EH(utility) interface.

STRUC. METADATA

Global (file) Attributes

HDFEOS

SWATH GRID POINT

Root -- “/”

ADDITIONAL

HDFEOSINFORMATION

Top Level of HDF-EOS 5


SwathName

Data Fields Geolocation Fields

Attribute Dataset

DataField.1

DataField.n

Longitude Latitude

Group Attribute<DataFields>:<AttrName>

Global Attribute<SwathName>:<AttrName>

Local Attribute<FieldName>:<AttrName>

CoLatitudeTime

Shaded Objects are implemented in a fixed way. User doesn’t have

direct access via the interface

Each Data Field can have Attributes and/or Dimension Scales

Profile Fields

ProfileField.1

ProfileField.n

Group

Swath Structure


Latitude Longitude Tempera-ture oC

Dew PointoC

61.12 -149.48 15.00 5.00

45.31 -122.41 17.00 5.00

38.50 -77.00 24.00 7.00

38.39 -90.15 27.00 11.00

M M M M

HDF-EOS Point

• Intended use– Discrete points in time and/or

location. – Table of data linked to table of

geographic information.– 8-level Hierarchical, each level

may contain indices to the level below


Latitude Longitude Buoy_ID25.2645 091.2564 012622.3549 -93.4657 356423.2564 -89.2546 1256

Buoy_ID

Time WaveHeight(ft)

Temp(C)

0126 01:26 2.54 18.40126 05:56 3.58 18.23564 06:28 12.64 16.41256 08:12 7.58 17.11256 09:58 7.76 17.20126 09:59 4.23 20.13564 10:16 10.23 17.5

HDF-EOS Point

• Hierarchical links:– Every level in a Point data set

must be linked into the hierarchy.

– Before two levels can be linked, a link field must exist


HDF-EOS Swath

• Intended use– Across track scanning

instruments.– Sounding instruments– Level 1: Geolocated Sensor

Units– Level 2: Geophysical

Parameters

InstrumentPath

Scan Lines

Along Track

InstrumentPath

Along Track


“Longitude”

“Latitude”DimensionName: GeotrackSize: 21

DimensionName: ScanSize: 16

DimensionName: TrackSize: 42

Map1DataDimension: “Track”

Geodimension: “Geotrack”Offset: 1

Increment: 2

“Brightness Temperature” “Time”

HDF-EOS Swath

Data Fields Geolocation Fields


HDF-EOS Grid

• Intended use– Variables mapped on uniform

space-time grid scales– Level 3 - Gridded single

measurement parameters– Level 4 - Modeled or derived

from multiple measurements


HDF-EOS Grid

XdimSize: 2000

YdimSize: 800

Projinfo

AltitudeSize 30


HDF-EOS Grid

• Projections Supported• Geographic• Transverse Mercator• Universal Transverse Mercator• Hotine Oblique Mercator• Space Oblique Mercator• Polar Stereographic• Lambert Azimuthal Equal Area

• Lambert Conformal Conic• Polyconic• Interrupted Goode’s

Homolosine• Integerized Sinusoidal • Compression Methods• Run-Length Encoding• Adaptive Huffman• Gzip


Product Levels• Level 0 - Reconstructed,

unprocessed instrument/pay-load data at full resolution; any and all communications artifacts, e.g., synch. frames, communications headers, duplicate data removed.

• Level 1A- Reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters, e.g., platform ephemeris, computed and appended but not applied to the Level 0 data.

• Level 1B - Level 1A data that have been processed to sensor units (not all instruments will have a Level 1B equivalent).

• Level 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data.

• Level 3 - Variables mapped on uniform space-time grid scales, usually with some completeness and consistency.

• Level 4 - Model output or results from analyses of lower level data, e.g., variables derived from multiple measurements.


Resources

– HDF-EOS on the web:• http://hdfeos.gsfc.nasa.gov/• http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

– HDF and HDF5 on the web:• http://hdf.ncsa.uiuc.edu/

– HDF-EOS and HDF via email:• [email protected]• [email protected]

ESDIS Status

Documents

hdf hdfeos workhop viover

preeosera data

eos science teams

data locationgroups

data processing

data management

science data volume

data products deliveredoctober