Processing and dissemination of satellite remote sensing ...2007.foss4g.org/attachments/94/CID_FOSS4G2007_20070920.pdf · Processing and dissemination of satellite remote ... Community

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 1

Processing and dissemination of satellite remote sensing data in an heterogeneous environment.

Armin Burger, Paul [email protected], [email protected]

Community Image Data portal project

http://cid.jrc.it

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 2

Presentation structure

Project’s objectives

A few words about the Institution

Step 1: from a tape repository to a digital repository

Step 2: extending this digital repository by building services around it

Conclusion

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 3

European Commission (EC)

The Commission is independent of national governments.

Its job is to represent and uphold the interests of the European Union (EU) as a whole.

It drafts proposals for new European laws, which it presents to the European Parliament and the Council.

It is also the EU’s executive arm – in other words, it is responsible for implementing the decisions of Parliament and the Council.

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 4

Joint Research Centre (JRC)

The European Commission’s

Research-Based Policy Support Centre

The JRC provides customer-driven scientific and technical support to Community policy making.

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 5

Use of satellite imagery at the JRC

Twenty projects using Satellite Remote Sensing (SRS) data

Annual expenditure on SRS data > 7 M€

100 000’s of Low Resolution images

More than 12 000 High Resolution images

Over 500 000 km2 of VHR images

Overall, more than 55 TB of SRS data

Scattered storage and no central catalogue

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 6

First setup of an online archiveInitiated on the archive of one project end 2003.

MARS-PAC project is the largest civilpurchaser of satellite imagery in the EU.

Largest archive of HR and VHR data in the JRC

Goal: To be able to search for images from a web browser, to visualize them and get a linkto the files.

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 7

Characteristics of this first setupSRS data loading application

Works with raw HR and VHR dataExtracts metadata and stores them in a spatial DB (PostgreSQL/PostGIS)

SRS data pre-processingConversion of the data to TIFF format and georeferencing.Histogram stretching and conversion to ECW

SRS data disseminationSearch and browse imagery (ECW)Web mapping (Mapserver)Download of raw or georeferenced data

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 8

Outcome of this first setupSuccessful as initial objectives were reached

Several limitations were identified:disk space not easily extendablerestricted to raw HR and VHRno fine grained authentication possible (users and IPRs)no WMS or WCS servicesevere restrictions in usability by other projects at JRC

Management wish to build on it and provide a site-wide service

Expected cost savings in hardware and staff

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 9

Survey of potential usersInitiative welcomed!!!

Community Image Data portal project started!

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 10

CID portal project objectivesProvide a centralized storage facility with backup and a data upload application.

Provide a data acquisition service.

Provide a service with IT infrastructure in High Availability mode.

Build a central catalogue of all SRS data available at JRC with a CSW service and a web interface to the catalogue.

Provide secure access according to IPRs and user credentials viadirect file access at high speed (NFS v4, CIFS)OGC protocols (WMS, WCS)other (web mapping, KML...)

Provide some SRS data processing services:[semi-]automated orthorectification

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 11

CID portal – Staff and BudgetBudget to initiate the project:

Hardware: 200 K€Software development: 130 K€

Orthorectification softwareModules for authentication and catalogue interface

Annual resources:

IT consultancy and small developments: 50 K€1.5 engineers and 0.7 IT technician.

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 12

CID portal overview JRCData Owner

DataProvider

JRC staff

User

DATAstorage

METADATAdatabase

SRS dataprocessing

SRS dataacquisition

SRS dataloading

AUTH.service

NFS4CIFS

AUTH.service

CSWWMS WCS

KML

CID spec.service

CIDdata serving

serviceActor

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 13

CID services overview

AUTH: Authentication and Authorization service

SRS data acquisition: web application for SRS data ordering

SRS data loading: website, web service and desktop interface

SRS data processing: metadata extraction, registration in a central catalogue and orthorectification for HR and VHR data.

CID data serving: direct file access and OGC services

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 14

Open source software and CID

The preferred option at CID because:

Knowledge available within CID staff

Almost all functionalities required are available in FOSS

Some functionalities available only in FOSS!

Good support from the user community

No cost to acquire the piece of software

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 15

Base infrastructure

OS: Xen with Debian 4 (80%) and Windows 2003 server

Base services:Bind, NTP, MIT Kerberos, LVS

Monitoring and statistics:Zenoss, Awstats, JMeter

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 16

Authentication and Authorization Service

Extremely important service:Enforcement of Intellectual Property Rights restrictions Sensitivity of some datasets (e.g. crisis areas)

Technologies used:OpenLDAP with KerberosNFS v4 and CIFS with ACLsCentral Authentication Service (CAS)PostgreSQL with PostGISCustom authorization application interfacing with OGCservices (PHP/Drupal framework)

AUTH.service

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 17

SRS data acquisition service

Web based applicationinterfacing with 6 SRSdata providersfor 9 missions.

Technologies used:DotNetNuke framework

SRS dataloading

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 18

SRS data processing SRS dataprocessing

orthonode

1

orthonode

2

orthonode

x

JBoss cluster

raw image rectification process final ortho image

Digital Elevation Model

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 19

CID Data serving service

Web mapping [p.mapper, CAS]

Catalogue Web interface [PHP/Mapscript]

CSW [GeoNetwork]

WMS, WCS [PHP/Mapscript, CAS]

Direct file access [NFS, CIFS]

Download [PureFTPd]

Official CID website for referenceinformation [typo3]

CIDdata serving

service

Victoria, BC, Canada on 25 September 2007 – FOSS4G2007 Conference 20

In the coming months...By end 2007:

All backend and base services up and running in HAProcessing service placed in its production environmentCompletion of the web portal

Q1 / 2008:OGC services implementedSRS data loading service implementedSRS data acquisition service moved to the new environmentFirst import of data and intense testing

Q2 / 2008:Import of other datasetsPublic access to the CID portal (http://cid.jrc.it)