1 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp SIPAD New Generation: Dominique Heulet CNES 18, Avenue E.Belin 31401 Toulouse Cedex 9 France e-mail:[email protected].

1CEOS/WGISS20 – Kyiv – September 13, 2005 Paul

Kopp

SIPAD New Generation:

Dominique HeuletCNES

18, Avenue E.Belin31401 Toulouse Cedex 9

Francee-mail:[email protected]

A multi-field system for accessing scientific data and added-value services


Kopp

Introduction

SIPAD-NG– What it is:

» a complete operational system for preserving and accessing data,

» implements proven off-the-shelf products and a sophisticated data model,

» meets the requirements of a data center with a long-term mission,

» a centralized system with its own catalogue

– Why it has been developed:

» CNES wants to offer a complete generic software to be used as a basis for building data centers


Kopp

SIPAD-NG: follow-on of the SIPAD softwareThe SIPAD system

– Developed in the 1990s for the CDPP» CDPP: Space Plasma Physics Data Centre

» SIPAD enables users to retrieve data and metadata through a web server

– SIPAD software designed to be used by any scientific discipline:» Plasma Physics (CDPP & CASSINI projects),

» Oceanography (MERCATOR project),

» Atmospheric Chemistry & Astronomy

– Advantages of SIPAD:» SIPAD implements all the basic functions of a data access system

» Open-ended Data Model based on a Data Dictionary allowing the control of delivered metadata

– Disadvantages:» Limitations to implement new functions (e.g. interoperability),

» Web user interface not flexible enough (not easily customizable)


Kopp

The Data Model: graphs

For each project, definition of the Data Model

First goal of the Data Model: organize the catalogue in order to make data search & selection by navigation easy

– Definition of a data graph, a browse graph and a document graph

» Data graph: storage objects (archived data – e.g. a ‘tar’ file –)data objects (data files seen by end users)data sets (groups of data objects – e.g. instrument level 1

data –)collections (groups of data sets or collections)

» Browse graph: browse products, browse sets, browse collections

» Document graph: documents, document collections

– Definition of links between these graphs

Web server pages are automatically generated from the Data Model


Kopp

The Data Model: graphs

Project

Documentcollection

Datacollection

Browsecollection

Data setData

ObjectBrowse set

Browseproduct

Document DocumentData set

Data object Data objectBrowseproduct

Browseproduct

Datacollection

Data setBrowse

collectionBrowse set

Documentcollection

Storageobject

Storageobject

Storageobject


Kopp

The Data Model: attributes

Elements of the graphs can be described by attributes

Generic attributes» Size of a data object

» Number of data objects in a data set

Specific attributes according to scientific discipline» Plasma Physics: start time & stop time of an observation

» Earth Observation: geographical coordinates

Use of attributes– To present information associated to data

– To select data by criteria

» The web server provides selection pages for entering criteria values


Kopp

The Data Model: attributes

Project

Missiondocuments

Mission

DocumentData set

Data object

ExperimentExperimentdocuments

Data Object Id

Start time

Stop timeVersion

etc …

Title

Author

Name

Description

Measured parameters

etc …

Selectioncriteria

Dataset Id

Level

etc …

Browsecollection

Browseproduct

Browse set


Kopp

SIPAD & SIPAD-NG common functions

Data search & selection» Navigation in the Data Model,

» Search on criteria,

» Selection of product browse

User management» Access rights,

» Quotas

Order management» Data extraction from an archive,

» Media production,

» Network transfer,

» User workspace


Kopp

New functions implemented in SIPAD-NG

Integration of Added Value Services (AVSs)– SIPAD-NG allows to:

» associate AVSs to Data Model elements in order to propose them to users,

» call AVSs by giving them parameters,

» manage access rights and user quotas

– SIPAD-NG implements a specific interface for:

» Batch AVSs: process data which have already been ordered and deliver the result to the user (e.g. coordinate transformation),

» Interactive AVSs: process data available in the user workspace (e.g. graphic representation of data from several

data sets)

Interoperability– SIPAD-NG provides interfaces to send requests to or to receive requests

from another Data Centre

» Web Services & SOAP protocol


Kopp

Adaptability

Adaptation of the web server to the project needs– SIPAD-NG web server easily customizable to project needs

– SIPAD-NG web server can be replaced by a project specific web server

Implementation in Data Centres located inside CNES– SIPAD-NG compliant with CNES Computer Centre services:

» SUN SOLARIS operating system,

» interfaces with CNES archiving and media production facilities

– SIPAD-NG compliant with CNES security rules

Implementation in Data Centres located outside CNES– SIPAD-NG can run on a single machine or its components can be

distributed over several machines

– SIPAD-NG can run on both SOLARIS and LINUX operating systems


Kopp

Architectural design

SIPAD-NGkernel

Enum Type

Objet de stockage

idODtype(local,archive,url)chemin fichiernom fichier

Definition Attribut

idAttidEntitetype(INT,STRING,ENUM)nomstandard(OUI,NON)

1..n

0..1

1..n

0..1

Métadonnées

données

Entité Type

idEntite Typedescritiontype ( OBJDS,JDON,PBR,DOCU,COLLEC )

0..n

1

0..n

1

Attributs_Int

idAttributidAttidEntitevaleur

0..n

1

0..n

1

Jeux de données

idEntitedateDebdateFindroitaccesvolume

Object de données

idEntitedateDebdateFinvolumeradicalreference

1..n

1..n

1..n

1..n

Produit de browse

idEntitedateDebdateFindroitAccestypeBrowse

Collection

idEntitedroitAccesnomFrançaisnomAnglais

Attribut_String

idAttrributidAttidEntitevaleur

1..n

0..1

1..n

0..1Attribut_Enum

idAttributidAttidentitevaleur

1..n

0..1

1..n

0..1

SVA

idSVA

criteres

idCritere

11

11

table

Entite

idEntiteidAttnomdatecreation

0..n1

est de type

0..n1

1..n

1

1..n

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1..n

1

1..n

1

1..n

1

1..n

1

1

1

1

1

1

n

1

n

1

1

1

1

Document

idEntitedroit accesnomFrançaisnomAnglaisgluTagnbrePages

1

1

1

1

Catalogue

Basic services Administration& Supervision

IngestionUser

management

Catalogueconsultation

OrderManagement

User space

AdministrationWeb Server

Administrator

LocalData Centre

services

Clientapplications

On line archive Near line archive Media production

Web ServerAdded Value

ServiceRemote

Data Centre


Kopp

Client applications provided by SIPAD-NG

The SIPAD-NG software system is composed of:– a kernel:

» the six basic services,

» the Administration Web Server,

» the data base

– several client applications:

» complementary to the basic services (e.g. automatic ingestion),

» main client application: User Web Server

Characteristics of the SIPAD-NG User Web Server:» compliant with the six basic services interfaces,

» easily customizable,

» A project can replace it by its own web server and only reuse the communication layer with the basic services


Kopp

Time selection

Implementation examples

Internet machine

Geographic selection

Other SIPAD-NG services& Administration Web Server

STAF

SEM

Oracle serverCDPP catalogue

Solaris server

SIPAD-NGUser Web Server &

Catalogue Consultation service

CDPP at CNES Computer Centre

ICAREData Centre

File server

On-line data

SIPAD-NG basic servicesSIPAD-NG user web serverICARE catalogue

Linux server


Kopp


Kopp


Kopp

Technology

User Web Server&

Administration WebServer

Basic services

- JAVA- Apache Tomcat- Struts → MVC2

- Axis → Web Services

- JAVA- RMI → Inter-machines communication

- XML → Ingestion & Basic services interfaces

ORACLE 9i

‘Object-Relational Mapping’layer

‘Business’ layer

‘Application Logic’ layer

‘Client’ layer

‘Database’ layer


Kopp

Conclusion

Schedule» 2003: requirements, specification

» February 2004: beginning of development

» July 2005: first release

» December 2005: final release

SIPAD-NG contractor: CS SI

SIPAD-NG users– those already using SIPAD

» CDPP, MERCATOR, CASSINI

» Migration of the catalogues from SIPAD to SIPAD-NG

– Future potential users

» ICARE: Data Centre for Clouds & Aerosols

» SSALTO: Altimetry & Orbitography

1 CEOS/WGISS20 – Kyiv – September 13, 2005 Paul Kopp SIPAD New Generation: Dominique Heulet CNES 18, Avenue E.Belin 31401 Toulouse Cedex 9 France e-mail:[email protected].

Documents

scientific data

data dictionary

data model slide

sophisticated data model

document graph data

data search selection

building data centers

browse graph