Research & Service Support Delft Software Days 2015, Deltares 28/10/2015 Processing services for Sentinel-1 data exploitation RSS Team member: J. Manuel Delgado
Research & Service Support
Delft Software Days 2015, Deltares
28/10/2015
Processing services for Sentinel-1 data
exploitation
RSS Team member: J. Manuel Delgado
Outline
1.Service description
2.Processing services
a. RSS processing on demand
b. RSS Cloud Toolbox
3.SNAP / Sentinels Toolbox
4.Examples of Sentinel-1A data processing
Service Objectives
• Support the Earth Observation community in exploiting EO data
• data provisioning
• data access
• data processing
• Empower effective and efficient research process
• Support exploitation environments
• enabling product generation
• specific derived information
instead of bringing “Big Data” to “users”
we bring “users” to “Big Data”
User community
RSS user community includes:
• Scientists (research centers, universities)
• Service developers (sme or industry developing technology projects)
• European or international institutions
who have started their work locally on their workstations with some data
samples and want to “scale up” to massive data processing or wide
service exposure.
Support in the Research Process
Support options
Provision of customized Sandboxes and Cloud Toolboxes
Customized Virtual Machines with pre-installed software (e.g. NEST, Sentinel
Toolbox, Matlab, IDL, other) to support user-specific processing requirements
can be provided on researcher request
Provision of EO data of interest
Data can be provided either in the sandbox/toolbox or on a dedicated Data
Farm for further processing. Reference datasets from ESA and Third Party
Earth Observation data on required areas of interest and time intervals can be
made available for download to the researcher's desktop
Processor Hosting / Processing
Algorithm integration and hosting service of selected processors on RSS
virtual infrastructure and support to processing campaigns on required areas
of interest and time intervals. Flexible processing resources (cloud) are used.
High Level Scientific and IT Competences
RSS team can support different stages of the algorithm/service development
with highly skilled competences
Research and Service Support
Sandboxes, Cloud toolboxes based on Customized Virtual Machines
with pre-installed software, mainly thought for post-processing and
analysis activities.
• Can be provided with the software required by the user
• Flexible amount of CPUs, RAM and dedicated storage, to be
tailored on the actual user needs and the type of machine
template required (virtual toolboxes)
Processing Resources
Grid and Cloud Processing On-Demand: g-pod
Generic GRID and Cloud-based processing on demand environment
with high-performance and sizeable computing resources.
Provides the necessary flexibility for building an application virtual
environment with quick accessibility to data, computing resources and
results.
RSS processing on demand
http://gpod.eo.esa.int
1. The Web Portal is the user access point where he/she can
a. Submit processing requests
b. Check the status of the task
(created/pending/active/completed/failed)
c. Perform some basic operations: abort/resubmit/copy the task
2. Catalogue Access Service (CAS) is the catalogue where the input files and
auxiliary data information (metadata and URL) are registered as well as the
path/address from which the data is stored
3. The Light Grid Engine (LGE) is the orchestrator of the GRID
a. It stores the applications (PI algorithms)
b. It receives processing requests from the G-POD web portal and
sends processing tasks to the computing clusters
4. Stores and datafarm are where the data belonging the different available
missions is physically stored
5. Grid and Cloud computing resources are where the processing are
performed
Architecture
RSS Flexible Resources
On-demand processing service:
Platform
Infrastructure
G-POD
ESRIN - 172 cores - 400 TB
UK-PAC - 96 cores - 300 TB
Flexible/ Unlimited Infrastructure - 10-200 cores - 1-10 TB
EO Scientists Principal Investigators
delivery Process
EO data
Volume accessed by PI projects in 2014: • Total Number Submitted Jobs 39,456 • Average Number of Products per Job: 48 • Average Product Size: 700 MB • Total Size Data Processed: 1326 TB
Catalogue
S1TBX service on G-POD
S1TBX service on G-POD
RSS CloudToolbox
RSS CloudToolbox Service Portal
http://eogrid.esrin.esa.int/cloudtoolbox
If approved the administrator creates the CloudToolbox
CloudToolbox request
start/stop button: the user manages the VM. Commands are launched through the APIs
User can require a machine update
CloudToolbox upgrade
1. Via SSH
a. Supporting the command lines operations
2. Via Desktop viewer (VNC viewer)
a. The user experience is not affected by the unpleasant
feeling of delay
3. Users are provided with detailed instructions on how to
connect
4. VMs are configured with an automatic shutdown after
30 minutes of inactivity.
How to connect
SNAP / Sentinels Toolbox
1. The Sentinel Toolboxes and full source code is distributed freely under the GNU
GPL license. The source code is available on Github.
2. The SENTINEL-1 Toolbox (S1TBX) consists of a collection of processing tools,
data product readers and writers and a display and analysis application to
support the large archive of data from:
a. ESA SAR missions including :
– SENTINEL-1, ERS-1 & 2 and ENVISAT, as well as
b. Third Party SAR data :
– ALOS PALSAR, TerraSAR-X, COSMO-SkyMed and RADARSAT-2.
c. The various processing tools could be run independently from the
command-line and also integrated within the graphical user interface.
d. The Toolbox includes tools for calibration, speckle filtering,
coregistration, orthorectification, mosaicking, data conversion,
polarimetry and interferometry.
3. The SENTINEL-1 Toolbox is being developed for ESA by Array in partnership
with DLR, Brockmann Consult and OceanDataLab.
Description
1. Shares the core architecture with the ESA BEAM and NEST toolbox;
2. Java implementation
3. Display and Analysis Tool (DAT): integrated graphical user-friendly
interface;
4. Graph Processing Framework (GPF): for creating user-defined processing
chains;
5. Graphical or command-line execution;
6. Tiled memory management for working with very large data products;
7. Data abstraction models to handle all SAR missions in a common way
8. Modular design for easy modifications and upgrades
9. Users are able to add their own modules via APIs
10.Multithreading and Multi-core processor support
Architecture Highlights
S1TBX Architecture: Data flow
Release of Sentinel Toolboxes
Sentinel Toolboxes
• Multi-mission Scientific Toolboxes
• Developed as open source software
• Common architecture (SNAP)
• Java/Python API and Java/Python plugins
• Portable engine to a Cloud infrastructure
Download https://sentinel.esa.int
Toolbox ServiceDeveloper
Toolbox Engine
Middleware
Processor
Plug-in A
Plug-in B
Plug-in C
Executable ZPlug-in C
Plug-in A
Plug-in B
Plug-in C
Plug-in C
Executable X
Executable Y
Executable Z
VisualisationUser Interface
AnalysisUser Interface
ProcessingUser Interface
Data I/OUser Interface
Toolbox Common Library and Extensibility Framework
Wrapper API(s)Stand-alone Tool
AdapterNative API
new processors (= new WPS capabilities)are deployed in the cloudwith the Toolbox Engine
as platform.
WebBrowser
MobileDevice
User’sToolbox
servicecall
Data Archives
cloudservice
interfaces
WPS
Toolbox “Server Configuration”
STEP
• EO science collaborative platform
• Technical forum and community animation
• Gathering user feedback and usage
• Communicating on results
• Cloud demonstrator (e.g. Land Training 2015)
Website http://step.esa.int
SentiNel Application Platform
Display and analysis tool: DAT
Graph Processing Tools
Readers
Utilities
SAR Tools
1. Absolute calibration (Sentinel-1A, Envisat ASAR, ERS 1&2,
ALOS, Radarsat-2, TerraSAR-X, Cosmo-skymed )
2. Coregistration of detected and complex products
3. Terrain Correction
4. Multilooking
5. Speckle filtering
6. Debursting of Sentinel IW SLC and ASAR WSS products
7. TOPSAR support
a. Split products in subswaths
b. Assembling consecutive products (same orbit and pass)
c. TOPSAR coregistration and interferogram formation
Example: Sentinel-1A data processing
Sentinel-1A data processing
1. Example 1:
a. TOPSAR Interferogram over The Netherlands
b. Georeferencing of Sentinel-1A derived products
2. Example 2:
a. Assembling of several Sentinel-1 slices (consecutive
products belonging to same orbit acquired during the same
pass)
b. Calibration
c. Thermal Noise Removal
d. Terrain correction
e. Subset
ESA UNCLASSIFIED – For Internal Use
Example 1
1. Using RSS processing on-demand:
a. Submission of a TOPS Interferogram
formation in G-POD using Sentinel-
1A IW SLC products over The
Netherlands
2. Using the RSS CloudToolbox:
a. Download of resulting interferogram
products and
– post-processing on RSS
CloudToolbox using the SNAP/
Sentinel-1 Toolbox.
– Visualization of results
TOPSAR Interferogram
TOPSAR Interferogram
1. Batch processing using Sentinel-1 Toolbox with xml graph (see figure).
TOPSAR Interferogram
S1TBX GUI on the RSS Cloud Toolbox
Georeferencing of Sentinel-1A derived products
Georeferencing of Sentinel-1A derived products
Sentinel-1A derived data visualization
Sentinel-1A derived data visualization
Example 2
1. By using the graph showed below, which has been created using the
Graph Builder module of SNAP/S1TBX, we obtain as output a subset
of two consecutive Sentinel products assembled, calibrated, thermal
noise free and terrain-corrected
Customized processing chain
Example 2
Example 2
Example 2
Example 2
Example 2
Example 2
RSS contacts
For general information:
RSS Website: rssportal.esa.int
Join&Share: wiki.services.eoportal.org
For feedback or enquiries:
RSS Team: [email protected]
G-POD Team: [email protected]