EO GRID Processing on Demand ESA grid activity report on 2007 luigi.fusco, guoqing li ESRIN, European Space Agency (ESA) Presented for WGISS-23, 2007-5-23.

EO GRID Processing on DemandESA grid activity report on 2007

luigi.fusco, guoqing liESRIN, European Space Agency (ESA)

Presented for WGISS-23, 2007-5-23

Summary

Brief introduction to ESA EO satellites

EO community and GRID requirements

EO GRID Processing on Demand (G-POD)–Operational experience–Future vision and plans

ENVISAT product’s examples

The International Charter on Space and Major Disasters

Initiated by CNES and ESA, joined by CSA, NOAA, ISRO, CONAE, JAXA, USGS and DMC

Unified system of space data acquisition & delivery in case of natural or human-made disasters

Data delivery to civil protection agencies, emergency & rescue services; UN cooperating body since 2003

Operational since 2000: 24 hrs on-duty-operator, data resources from all Charter members

Recent Examples of Charter Activation:Bam Earthquake 2003 Darfur Crisis 2004 Tsunami Catastrophe 2004/2005 Hurricane Katrina 2005 Cyclone Kyrill 2007

Key international EO programmes

GLOBAL MONITORING FOR ENVIRONMENT AND SECURITY (GMES)

European independence in critical data sources for

environmental monitoring and security

and

the European contribution to the Global Earth

Observation System of Systems (GEOSS)

Key international EO programmes

The Earth Science world

Atm

osph

ere

Oce

an

Biosphere

Cryosphere

Noisy observations

Optimal Trajectory

First guess

time

Field

Coupled and inter-disciplinary processes

Complex web of sensor

Complex data analysis

Some Earth Science digital data requirements

Global, regional, local applications– Alternative use of the data at different resolution

Large historical distributed archives– Long term data and knowledge preservation issues

Near real-time access to data– For processing, value adding and dissemination

Integration with models to provide long term trends and forecast

– Data assimilation Integrate different data sources

– Standardisation, Virtual Organisation, …Need to link data to technical information and

scientific results– Need to keep/transfer the historical Knowledge

Why GRID at ESA for EO ?

Operational issues– Envisat generate > 500GBytes of data per day…– EO archives are “scattered” – Algorithm evolutions require recurrent reprocessing– More efficient use of existing infrastructures

Scientists / user – Need near-real-time access, historical and multi-sources– have the processing algorithms– require large investments to handle the data on their site

GRID can solve the equations…– move processors close to the data – reduce dissemination costs and effort– evolutions benefit to all at once

GRID as a common shared platform for collaborations

EO GRID Processing on Demand (G-POD) Objectives

Provide a “user-segment” environment– Put data & processors together– Allow “on-demand” processing of the data

Offer scientists a “production” lab– Focus on algorithms– Reuse housekeeping functions (e.g. catalogue, software tools)– Bridge gap from “prototype” to “production” processor

Offer scientists a “collaboration” environment– Share tools and functions– reuse output of other processors– IPR is kept by the scientist

Open and scalable– host “any” processor– bring more resources (other grids) and data (other archives)

together

The ESA EO GRID environment

SecurityUser certification via ESACERT

underway

Software resources on-line– IDL, Matlab, BEAT, BEAM, BEST,

CQFD, Compilers, public domain image processing utilities

– Catalogue queries and data provision functions

– Data viewers

Computing and Storage Elements

– Over 200 Working Nodes– Over 100 TB on-line store– Middleware: LCG 2.6, GLOBUS 4,

gLite3– Link to external CE and SE (e.g.

campus, EGEE…)

Data Interfaces – GS products Rolling Archives

(ENVISAT, MSG)– MODIS NRT products over Europe– Access to AMS SatStore– Some accesses to NASA and other

external data providers

Network – Gbit LAN– 64-192 Mbps to GARR– HiSEEN WAN (e.g. to PACs/PDHS)– MEGALAB (soon 20-40 Gbps in

Frascati area)

Grid-On-Demand Core Elements

Local GRID Resources

J ob ManagerGrid Gateway ESA Cluster Nodes

Temporary Environment

Rolling TeraStorage

External Clients 3rd-Party Elements

AMS Service

SOAP Service

Auxiliary FTP Server

Cache System

EOLI Servlets

EO GridEngine(Web Service)

Authentication Services and Session Manager

J obs/User References

J ob Factory

Temporary Environment

Web ClientHTTP-based request

Muis Gateway(Web Service)

MUIS Query and Rendering Service

Browse Cache System

Grid-On-Demand Core components

Application InterfaceComponents

MyProxyServer

Proxy Certificates

HTTP

AMS/API

FTP

SFTP

SOAP

SOAPSOAP

MyP

roxy

HTTP SOAP

GIP/CIP

HTTP

HTTP

GLOBUS/LCG

SOAP

(Archive Listing)

MyProxy

Earth Observation Grid on-Demand Portal(Web Service)

User Session Manager and Security

Users Data Folder

Task Factory(I /O Interface)

Task Flow Definition

OpenGIS Web Services

Auxiliary GIS Data

Task Manager (Submission and Status)

Task Database

Task Scheduler

Task Database

Operational Procedure

EPIS

FTP/SFTP

AM

S/A

PI

Envisat PDS

FTP

Product Input Flow Product Output Flow

MUIS-C

EnvisatCatalogue

AMS

Storage

DVD

Data Reprocessing Service Operator•Login

•Define J ob

•Submit J ob

•Monitor

SO

AP

SOAP Clients

C#, Delphi, PHP, J AVA

SOAP

(ESTEC) Certificate Authority

CRLs

CR

L dow

nlo

ad

ApplicationPrograms

J ob Templates

To C

NR

, oth

er

GR

IDs

To other GRIDs(e.g. CNR)

Grid-On-Demand Core Elements

Local GRID Resources

J ob ManagerGrid Gateway ESA Cluster Nodes

Temporary EnvironmentTemporary

EnvironmentRolling Tera

StorageRolling Tera

Storage

External Clients 3rd-Party Elements

AMS Service

SOAP Service

Auxiliary FTP Server

Cache System

AMS Service

SOAP Service

Auxiliary FTP Server

Cache SystemCache System

EOLI Servlets

EO GridEngine(Web Service)

Authentication Services and Session Manager

J obs/User References

Authentication Services and Session Manager

J obs/User ReferencesJ obs/User References

J ob Factory

Temporary EnvironmentTemporary

Environment

Web ClientHTTP-based request

Muis Gateway(Web Service)

MUIS Query and Rendering Service

Browse Cache System

Muis Gateway(Web Service)

MUIS Query and Rendering Service

Browse Cache System

MUIS Query and Rendering Service

Browse Cache System

Browse Cache System

Grid-On-Demand Core components

Application InterfaceComponents

MyProxyServer

Proxy Certificates

MyProxyServer

Proxy Certificates

Proxy Certificates

HTTP

AMS/API

FTP

SFTP

SOAP

SOAPSOAP

MyP

roxy

HTTP SOAP

GIP/CIP

HTTP

HTTP

GLOBUS/LCG

SOAP

(Archive Listing)

MyProxy

Earth Observation Grid on-Demand Portal(Web Service)

User Session Manager and Security

Users Data Folder

User Session Manager and Security

Users Data Folder

Users Data Folder

Task Factory(I /O Interface)

Task Flow Definition

OpenGIS Web Services

Auxiliary GIS Data

OpenGIS Web Services

Auxiliary GIS DataAuxiliary GIS Data

Task Manager (Submission and Status)

Task Database

Task Manager (Submission and Status)

Task Database

Task Database

Task Scheduler

Task Database

Task Scheduler

Task Database

Task Database

Operational Procedure

EPIS

FTP/SFTP

AM

S/A

PI

Envisat PDS

FTP

Product Input Flow Product Input Flow Product Output Flow Product Output Flow

MUIS-C

EnvisatCatalogue

EnvisatCatalogue

AMS

Storage

AMS

StorageStorage

DVDDVD

Data Reprocessing Service Operator•Login

•Define J ob

•Submit J ob

•Monitor

SO

AP

SOAP Clients

C#, Delphi, PHP, J AVA

SOAP

(ESTEC) Certificate Authority

CRLs

(ESTEC) Certificate Authority

CRLsCRLs

CR

L dow

nlo

ad

ApplicationPrograms

ApplicationPrograms

J ob Templates

J ob Templates

To C

NR

, oth

er

GR

IDs

To other GRIDs(e.g. CNR)

Grid on Demand Infrastructure

I/F to external GRID resources via research networks

ESA GS Operat’s

Use

r ac

cess

to

ols

Only one environment for operations and new dev

EO Data on-line in GRID storage

ASAR – ASAR GM: Feb 2004 onwards– ASAR Medium Resolution: June 2005 onwards– Occasional High Resolution products

MERIS– Level 1,2 RR: full mission– Samples of FR Level1

AATSR - Level 1: full mission- Level 2: since mid 2005

Atmospheric– GOME Level 1,2: 1996 onwards– GOMOS, MIPAS, SCIAMACHY full archives from D-PAC

Third-party data – MSG SEVIRI: last 12 months– MODIS NRT over Europe

More to come …

Web Portal Interface

Temporal/spatial selection of products

Job definition, submission and live status monitoring

Customisable result visualization interfaces

Access to output products and documentation

Experience so far

Wide ranging applications supported–Systematic productions of high-level

products (e.g. level-3)–On-demand Data access and visualisation–Algorithm and data validation activities

• software test-beds (ESA toolboxes e.g. BEAT, BEAM, BEST)

• Processor test-beds

–Internal Research projects

Over 40 projects supported, more to come!

Routine Production MERIS Level-3 Products NRT generation

– Joint ESA collaboration with ACRI (France),JRC/Ispra (European Commission) and Brockmann Consult (BEAM)

– 11 monthly products published on-line http://earth.esa.int/meris/level3

Daily ASAR GM mapping of Antarctica– Internal Development in operations since 2005– Daily Generation of 400-m resolution mosaics

and publish to ESA Web Map Server http://ssems1.esrin.esa.int:8080/map/context.php

EarthCare Simulator– Collaboration with Estec and Univ. of Valencia (Spain)– Preparatory activity of the mission EarthCARE– 2.5 105 simulations carried out using ESA and CNR

GRIDs

River and Lake Processor– ESA/Montfort University (UK) collaboration– Accurate River and Lake heights measurements in

NRT from satellite altimetry (RA2)– products published online http://earth.esa.int/riverandlake

Meris True-Color Mosaics– 9km resolution global Monthly mosaics of MERIS data

On demand processing ASAR on Demand

– Integrated environment for SAR processing

– Binds separate functionality into higher-level applications (flood monitoring, co-registration, mosaicing, etc)

Imager– Multi-mission imaging tools– Data selection, user-defined processing,

image rendering, geo-coding, visualisation

MIRAVI Geo-toolboxGeocoding of MERIS full resolution images

produced by MIRAVI real-time service

Aeromeris– Fast extraction over user-area of pixels

and statistics from the complete MERIS level-2 product archive

– Output to Excel, Google Earth, XML

Algorithm Validation

Volcanoes Monitoring by Infrared

– Fast Extraction of AATSR thermal anomalies over > 200 volcanoes

– Under validation, Operational service planned for early 2007

GRIMI-2– EADS/Astrium MIPAS Level-2 reference

processor used to validate the operational PDS processor

– Test data generation: 300 profiles (100GB) processed in 1hour

YAGOP– Alternative GOMOS O3 and Temp level-2

processor (internal research)– Validation against operational ESA

products RAIES Processor

– Assessment of the 2000 Hz individual echoes of Envisat/RA2

New Research applications MERIS/GOME inter-comparison

– Validation of Envisat/MERIS cloud fraction and water vapour measurements against ERS-2 GOME

Var2Grid– ESA/DMI cooperation (DK)– 3DVar data assimilation (T and specific

humid.) with CHAMP/GPS products (Background ECMWF ERA40)

SCIA-SODIUM– Internal prototype development– Extraction of Sodium profiles in

mesosphere from Sciamachy– Collaboration with ESO for the next

generation telescope ground site survey ALOS PRISM DEM Prototype Processor

– Internal Research (ESA Trainee)– Image co-registration of PRISM views and

height reconstruction by image matching between PRISM triplet images M

ou

nta

ins

aro

un

d S

hiz

uo

ka,

Jap

an

Earth Science G-POD

Call for Proposals

http://eopi.esa.int/G-POD

Promote the access and use of EO mission data available at ESA, offering on-line access to products with attached computing infrastructure and tools to assist the generation of “scientific added value products”

– new earth science algorithms – Build network of competences;

promote e-collaboration – transition from science

algorithms to new user products…

First call published beginning June 2006 on EOPI web

11 Proposals selected in first call, implementation ongoing

First very promising results expected mid 2007

Next call in few months

• Validation period: 11 day ensemble of GOMOS data (Sep., Oct., Dec. 2002)• total set of selected occ. events GOMOS: 1719 MIPAS: 4099 ECMWF: 1719

GOMOS to MIPAS temperature profile comparison (relative errors

shown in Fig.)

Time – Space coincidences criteria for analysis: 3h&300Km

YAGOP: Yet Another GOMOS Processor

GRID is very good for such analysis:

1 generic grid service

near future GRID plans

Operational Opportunities– Routine generation of “standard” high level

products– Extend infrastructure to other facilities and across

establishmentsConsider potential use of GRID

– In future EO missions preparation – GMES Service Elements– Contribute to GEOSS architecture– As a mean to get the science community more

involved in ESA programmesEarth Science presence in European e-

infrastructure initiatives – Strong participation to new EC FP7 initiatives such

as e-infrastructure, digital repositories, knowledge preservation, e-content... Knowledge

Infrastructures

GRID based RTD plans

Grid ES Community Applications

DL Applications …… as Knowledge InfrastructurePECS

GRID

ES Grid on Demandin operation @ ESA

CWE Applications

Thank you!

The ESA EO GRID TeamMore information at http://eogrid.esrin.esa.int