INFSO-RI-508833 User Forum 1-3 March 2006 Enabling Grids for E-sciencE Worldwide ozone distribution by using Grid infrastructure ESA: L.

INFSO-RI-508833 User Forum 1-3 March 2006

Enabling Grids for E-sciencE

www.eu-egee.org

Worldwide ozone distribution by using Grid infrastructure

ESA: L. Fusco, J. LinfordIPSL: C. Boonne, S. Godin-Beekmann, M. Petitdidier, D. Weissenbach

KNMI: W. Som de Cerff

SCAI: J. Kraus, H. Schwichtenberg

UTV: F. DelFrate, M; Iapaolo

User Forum 1-3 March 2006 CERN 2

Enabling Grids for E-sciencE

INFSO-RI-508833

INTRODUCTION

Earth Observation by satellite: large community– Space Agencies– Scientists in many fields– Organisation in charge of natural risk alerts– SMEs for elaborated products

GOME total ozone assimilation

•Grid Technology seems very well adapted to a lot of applications related to Earth Observations

•ESA/ESRIN, IPSL and KNMI have started to port applications in DataGrid

•Figure from ESRIN

User Forum 1-3 March 2006 CERN 3

Enabling Grids for E-sciencE

INFSO-RI-508833

EARTH OBSERVATION BY SATELLITE [1]

Typical satellite case: large number of files

7 years of Ozone from GOME/ERS2 experiment

CHALLENGE

PRODUCTION: Complex Algorithms: neural algorithm and inversion

VALIDATION: For a given day find 10

profiles among 26000.

User Forum 1-3 March 2006 CERN 4

Enabling Grids for E-sciencE

INFSO-RI-508833

ESA(IT) – KNMI(NL)Processing of raw GOMEdata to ozone profiles.

2 alternative algorithms~28000 profiles/day

IPSL(FR)Validate some of the

GOME ozone profiles (~106/y)Coincident in space and time

with Ground-Based measurements

Visualization & Analyze

EGEEenvironment

Level 2

(example of 1 day total O3)Level 1

Raw satellite data from the GOME instrument(~75 GB - ~5000 orbits/y)

The EGEE Ozone processing and validation test-case [2]

Additional GRID demonstration: GOMOS, OMI, CEOS, …

User Forum 1-3 March 2006 CERN 5

Enabling Grids for E-sciencE

INFSO-RI-508833

Ozone by SATELLITE [3]

Production of Ozone profiles from satellite data GOME/ESR by UTV (Italie), ESRIN (Italie), KNMI(Hollande)

Validation of the profiles with lidar data by IPSL, UTV

NNO: Algorithm « Neural network »

2 versions (F. del Frate, M. Iapaolo, S. Casadio)

8 years (1995-2003) on EGEE:

38489 files/algorithm with ~ 1800 profiles d’ozone/file

Database: metadata taking into accountthe orbits footprint

OPERA: Algorithm – Inversion (R. van derA et al., 2002)

Several months onEGEE: 76000 Files

1 profil/file (W. Som de Cerff)

•Capability to run complex algorithms and to handle large amount of files

User Forum 1-3 March 2006 CERN 6

Enabling Grids for E-sciencE

INFSO-RI-508833

Metadata

• In ES, Metadata and data bases are very common as large data sets are handled by different teams. The RDBMS generally used are MySQL, PostgreSQL or Oracle

• Many bases already exist and the aim is at least to make a copy of them.

• If they are developed to be used on EGEE they need to be useful outside Grid.

• Another specific point is the satellite data that are stored by orbit. The orbits are described by their footprint, expressed in geographical coordinates.

• Many queries concern collocation in time and/or space• Some metadata and data are only accessible to authorized persons• Others available on web site have rules for

publications(acknowledgement, co-author)

User Forum 1-3 March 2006 CERN 7

Enabling Grids for E-sciencE

INFSO-RI-508833 http://www.scai.fraunhofer.de

ES Metadata Management (SCAI; IPSL)

Enabling Grids for E-sciencE

ES meta data clients

OGSA-DAIService

SE

EGEE UI

ES meta data client

query

lfns

lfns

dataX 509 User Proxy

use

ES meta data client

EGEE Job on WN

X 509 User Proxy

submits

lfns

use

User Forum 1-3 March 2006 CERN 8

Enabling Grids for E-sciencE

INFSO-RI-508833 http://www.scai.fraunhofer.de

ES Metadata Management

Enabling Grids for E-sciencE

• What do we need?- RDBMS support OpenGIS conform data types

• What is PostGIS?- PostGIS adds support for geographic objects to

Postgres: http://postgis.refractions.net/

• Why PostGIS?- MySQL spatial extensions only support convex

polygons

Database Aspects (Spatial Ext.)•Relational DataBase Management System

User Forum 1-3 March 2006 CERN 9

Enabling Grids for E-sciencE

INFSO-RI-508833

CONCLUSION

• In EGEE : Feasibility studies carried out on relatively simple applications

• Needs to pass from simple to complex applications and operational chain – for near-real time like daily previson of ozone.– Cascade of jobs– Inclusion of web services– ……………….

• Some solutions in gLite• Others investigations via SSA DEGREE(Dissemination

and Exploitation of GRids in Earth SciencE)