ESA Data Integration Application Open Grid Services for Earth Observation Luigi Fusco, Pedro Gonçalves.

ESA Data Integration ApplicationOpen Grid Services for Earth Observation

Luigi Fusco, Pedro Gonçalves

Topics

• Earth observation requirements

• Ozone application

• Generic application layer (GridEngine)

• Grid portal for EO services

Requirements

• 100 Gbytes of data per day in ERS missions and 500 Gbytes in ENVISAT

• Grid Requirements :– enhance the ability to access high

level products– allow reprocessing of large historical archives– improve Earth science complex applications (data

fusion, data mining, modeling …)

Ozone Application

• Input: Wave spectra data measured by the GOME instrument on the ERS (level 1)

• Output: Calculation of satellite ozone profiles (level 2 data)

• Two algorithms: OPERA (MPI) and NOPREGO (Neural Networks)

• Objective: Data validation using LIDAR measurements

• Collaboration among different institutes: France (IPSL), Italy (ESA, ENEA, UTV), Holland (KNMI)

Ozone Application Requirements

• Reuse of Legacy data warehouses and catalogues from ESA

• Constraints:1. Access to DataGRID storage elements and execution

of remote jobs within DataGRID• GLOBUS 2.2.4

2. CEOS-GRID • GLOBUS 2.4.2 • Different certificate authorities

• Web portal interface

Ozone Application Portal• Temporal and spatial selection of data • Catalogue access and data transfer from ESA

data warehouses to the GRID storage elements• Job selection and

status information• Result retrieval and

visualization in OWS• Remote MySQL

access (SOAP)• Data validation w/

ground measurements• Demonstration of

prototype at last WGISS meeting in Toulouse

Lessons learned … Joining the services

Client Applications (JAVA / Linux / Windows)

Web Portal

Client ApplicationsSO

AP

W

rap

per

SO

AP

W

rap

per

OG

C W

eb

Serv

ices

MUIS ESA Catalogue

AMS ESA Data Archive

WCS / WFS / WMS

Catalogue (CSS)

GRID EngineDATAGRID

Computing

Elements

Storage Elements

GLOBUS 2.2

Computing

Elements

Storage Elements

… other versions

Computing

Elements

Storage Elements

GRID Engine

Status : First version for DataGrid and GLOBUS 2.2

• Connects and manages jobs sessions in multiple GRID environments - (WSDL description and SOAP encoding)

• Discovery Services• Publish jobs availability and current processing

status to external services in XML• Retrieves and translates job results GRID Engine

DATAGRID

Computing

Elements

Storage Elements

GLOBUS 2.2

Computing

Elements

Storage Elements

GLOBUS 3

Computing

Elements

Storage Elements

GRID Engine

• Services are pre-defined applications managed

by templates for each GRID that are plugged into the Grid Engine

• Templates instantiate the necessary scripts for the specific job requests (files names, parameters)

• Defined by XML schema for future implementation of discovery services

GRID Engine - Operations

• SOAP Services using HTTP(S)

• Session Manager (SM)– Session and Certificates management – SM_open, SM_authenticate, SM_quit, …

• Job Manager (JM)– Job commands and status information – JM_submit, JM_list, JM_create, …

• Storage Explorer (SE)– Storage access – SE_query, SE_move and SE_transfer methods. TBD : the use of SOAP w/ Attachments (W3C sub.)

Example of EO specialized Services

• On-demand geographical orbit projection (using ESA’s EnviProj)

• On-demand orbit projection and mosaicking (using OSSIM)

• Format Conversion Services: Envisat standard, GeoTiff, HDF-EOS … (using geoTiff, OSSIM and with the future use EnviView libraries)

• GOMOS/GOME level 1 processing and validation (using ESA data processors)

EO data Search and Storage Services

• Temporal and spatial selection of data

• Data transfer from data warehouses (AMS)

• Data transfer to and from the GRID storage elements

Auxiliary Grid Services

• Job selection, launching and status information• Result retrieval and

visualization in WMS• Join personal data

folder as a WMS

Conclusion

• The Web Portal Integrates several components of the EO Infrastructure

• The EO GRID Engine provides a standard way for Web Clients to request GRID operations

• The Web Portal and the EO Grid Engine allow the complexity of the underlying components to be hidden from the user

• This Framework can be adapted to provide new tools for the satellite end-users (e.g. integration of ENVISAT specific tools, such as BEAT, BEST, BEAM …)

More information

• [email protected]

• http://giserver.esrin.esa.int/