Page 1 Session 4: State of the art technology and standards, 12/1 0/2012 Web Processing Service Models for scientific exploitation of EO Data, Session 4: State of the art technology and standards Yves Coene, Spacebel s.a. Patrick Jacques, Spacebel s.a. Peter Baumann, Jacobs University Frascati - 12 October 2012
37
Embed
Page 1Session 4: State of the art technology and standards, 12/10/2012 Web Processing Service Models for scientific exploitation of EO Data, Session 4:
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1Session 4: State of the art technology and standards, 12/10/2012
Web Processing Service
Models for scientific exploitation of EO Data, Session 4: State of the art technology and standards
Page 2Session 4: State of the art technology and standards, 12/10/2012
Overview
• Introduction
• Web Processing Service
• Case Studies
– Decision Support (DREAM)
– Lunar Landing (LandSAfe)
– Processing On Demand (G-POD, SSEGrid, ngEO)
• Open-source implementations
• Issues and future work
• Standardisation work
– Ongoing
– Planned
• Conclusion
Page 3Session 4: State of the art technology and standards, 12/10/2012
Introduction
• HMA Standardisation
– Service Oriented Architectures
– For GSCDA and Science
• Service Taxonomy (HMA)
– Identity Management Services
– Discovery Services
– Invoke Services
• Feasibility Analysis
• Ordering
• Geospatial Processing: §3.4.7, §5.5.5
– Online Data Access Services
Page 4Session 4: State of the art technology and standards, 12/10/2012
Web Processing Service
• WPS is an OGC standard for:
– describing and executing a geoprocessing operation via a service,
– standardizing the requests and responses for a particular geospatial processing service.
• WPS is a generic model of a Web service supporting WSDL and REST, for which profiles need to be developed and standardised to support interoperability:
– Syntax and semantics of input parameters
– Input and output data format
Page 5Session 4: State of the art technology and standards, 12/10/2012
Web Processing Service
• Operations:
– GetCapabilities
• Discover which processes are offered by the service*
– DescribeProcess
• Discover input/output of a specific process
• Response can be used to build the user interface for capturing the input parameters**.
– Execute
• Execute a process
• Storing response or returning response document
(*) e.g. ProcessCoverages is a process that accepts WCPS request language as per OGC 09-045.
(**) similar to DescribeTasking in OGC 10-135 Earth Observation Satellite Tasking Extension of SPS.
Page 6Session 4: State of the art technology and standards, 12/10/2012
Web Coverage Processing Service
• The Web Coverage Processing Service (WCPS) specifies an expression language allowing for server-side filtering and processing of multi-dimensional raster coverages.
• It defines syntax and semantics of expressions for specifying extraction and analysis of coverages containing sensor, image, and statistics data.
• Not tied to a particular transmission protocol, the query paradigm can be embedded into any service framework, such as:
– OGC Web Coverage Service (WCS)
– OGC Web Processing Service (WPS).
Page 7Session 4: State of the art technology and standards, 12/10/2012
Web Coverage Processing Service
• Use Cases:
– On-demand retrieval
• “thresholded NDVI where threshold value is obtained from user input”
• Ad-hoc queries
– Processing
• “Fourier transform of scene X”
– Filtering
• “Scenes with less than x% cloud coverage”
– Summary data
– Sensor fusion and pattern mining
Slide 7
Page 8Session 4: State of the art technology and standards, 12/10/2012
WCPS By Example
• "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS"
for $c in ( M1, M2, M3 )return encode( abs( $c.red - $c.nir ), "hdf“ )
(hdfA,
hdfB,
hdfC)
8
Page 9Session 4: State of the art technology and standards, 12/10/2012
WCPS By Example
• "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS"
Page 10Session 4: State of the art technology and standards, 12/10/2012
WCPS By Example
• "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS"
– …but only those where nir exceeds 127 somewhere
– …inside region R
for $c in ( M1, M2, M3 ), $r in ( R )where some( $c.nir > 127 and $r )return encode( abs( $c.red - $c.nir ), "hdf“ )
(hdfA)
10
Page 11Session 4: State of the art technology and standards, 12/10/2012
Web Processing Service
• OGC 05-007r7, Web Processing Service, OGC Implementation standard.
• OGC 08-091r6, Corrigendum for ... Web Processing Service.
• OGC 08-068r2, Web Coverage Processing Service (WCPS) Language Interface Standard, version 1.0.0.
• OGC 12-029, Web Processing Service Best Practices, Discussion Paper, OGC Implementation standard.
• OGC 08-123: WPS-T Discussion Paper
• OGC 09-045, Web Processing Service – Application Profile for Coverage Processing. Version 0.2.0 (status: discussed in WPS.SWG).
*
(*) OGC Standards.
Page 12Session 4: State of the art technology and standards, 12/10/2012
Case Study 1: DREAM
• DREAM Project
– Dedicated to exploitation of EO data for Decision Support
– Based on use cases of European Institutions: EUSC and EMSA
– WPS used for Quality Assessment of ortho-images and Orthorectification of raw images:
• Horizontal Accuracy Maps
• Vertical Accuracy Maps
Slide courtesy DREAM Project (ASTRIUM-GEO)
Page 13Session 4: State of the art technology and standards, 12/10/2012
O1 – Online Data Access ClientO2 – Online Data Access Server
Q1 - Data Quality Proxy
Data Quality Service
Photogrammetry Library
Compute AOI WPS Processing
Reference3D Database
Reference Orthoimages
DEM
Compute RPCWPS Processing
OG
C WPS Service Interface
Web Application
Server
Compute Quality WPS Processing
OG
C WPS Client
OrthoRectifyWPS Process
AssessQualityWPS Process
Ortho-ImageProducer
OGC WPS Service Interface
Web JS Plug-in
ViewerWMS
….
File System
• Raw Acquisition• Ortho-image• Shift Shapefile• QualityMask Coverage
Q1 – Data Quality Client
WMS Server
WCS ServerIF-DREAM-
Q-DataQualityProxy
IF-DREAM-Q-AssessQuality,IF-DREAM-Q-OrthoRectify
IF-DREAM-Q-FileIO
IF-DREAM-O-RegisterData
[IF-HMA-Download]
IF-OGC-View
Slide courtesy DREAM Project (ASTRIUM-GEO)
Raw ImageHandler
Page 14Session 4: State of the art technology and standards, 12/10/2012
Case Study 1: DREAM
• DREAM, involving end users with massive data sets, allows to establish relevant demo scenarios. Sample WCPS use cases:
– „all coverages with timestamp younger than T within bbox B where cloud percentage is below P“
• User input: T, B, P
• Web client to allow input of T and P via sliders (or similar), B graphically
– „Bit mask of fires detected in coverage C“
– „band ratio between B1 and B2 for given coverage C“
• User input: B1, B2, C
– Scenario cases to be refined with users.
Page 15Session 4: State of the art technology and standards, 12/10/2012
Case Study 2: LandSAfe
• Landing Site risk Analysis software framework (ESTEC)
– ESA Lunar Lander Mission
– Vision-based Navigation technologies for safe precision landing.
– Hazard detection and avoidance.
Page 16Session 4: State of the art technology and standards, 12/10/2012
Case Study 2: LandSAfe
• Software modules for mission preparation stage and use during descent stage (live footage)
Page 17Session 4: State of the art technology and standards, 12/10/2012
Case Study 2: LandSAfe
• Processing requests sent over HTTP/SOAP to the WPS service which executes the request by running Python scripts from the LandSAfe Core.
pyWPS
See modules on previous
slide
See modules on previous
slide
Page 18Session 4: State of the art technology and standards, 12/10/2012
Case Study 3: G-POD
• ESA Grid - Processing On Demand
• WPS access point of G-POD installation is http(s)://<gpod_host>/wps/
• WPS process
– Refers to G-POD service
– Has identifier, title and abstract
– Output is RDF document referring to actual results
• G-POD service parameters are LiteralData (no ComplexData).
Page 19Session 4: State of the art technology and standards, 12/10/2012
Case Study 3: G-POD
• WPS as interface for ngEO Hosted ProcessingFacilities
WPS
e.g.G-POD
Page 20Session 4: State of the art technology and standards, 12/10/2012
Case Study 4: SSEGrid
• SSEGrid Project
– Introduce a Grid Processing on Demand model at VITO to be integrated with Vegetation processing chains
– Portal to integrate multi-Grid processing services to be used by the Principal Investigator (PI) for the systematic processing or re‑processing of EO data in a single or multi-mission environment
– On-demand Processing of EO data
• Large EO data size processing must be done where data is located
• Re-processing by PI processing chains must be deployable by PI
Page 22Session 4: State of the art technology and standards, 12/10/2012
Demonstration Scenario (FAPAR)
On VITO grid
For each product For each product
On ESRIN grid
For each product
ImportHDF_to_ENVI (5)
FAPAR (11)
Calculatedifference (21)
Projection (9)
Get list of SPOT-VGT products (3)
ImportMERIS_to_ENVI
(8)
Composite (10) Composite (10)
UI assign values to inputparameters (1)
Get list of MERISproducts (3)
MERIS FRarchive
(raw segm)
SPOT-VGTarchive
(P-products)
MODISarchive
(Level 1)
Get list of MODISproducts (3)
Projection (7)
ImportMODIS_to_ENVI
(6)Unzip script (4)
VIP MapProjection (15)
VIP Imageextractor (14)
SMAC (12)
NDVI (13)
SMAC (12)
NDVI (13)
(2)
(16)
(17)
(18)
(19)
(20)
(19)(19)
(20)
FAPAR (11) FAPAR (11)
(20)
Composite (10)
… (22)
Fraction of Absorbed Photosynthetically Active Radiation
for each = parallel
execution
data transfer
Page 23Session 4: State of the art technology and standards, 12/10/2012
SSEGrid WPS Hierarchy
PortalFederatingWorkflow
Engine
WPS-G
GridWorkflow
Engine
WPS-G
GridWorkflow
Engine
WPS-G
Grid
WPS-G
Grid
WPS-G
UserProcessing
Infrastructure(UPI)
FederatingProcessing
Infrastructure(FPI)
MissionProcessing
Infrastructure(MPI)
GridAccess
Infrastructure(GAI)
Grid
JavaSAGA
JavaSAGA
PI
One per mission Ground SegmentHierarchy of
WPS-G services
Page 24Session 4: State of the art technology and standards, 12/10/2012
SSEGrid WPS-G Interface
• Encapsulates all forms of processing– Grid Processes
– Workflows
WPS-G
Workflow Engine
Workflow
Workflow
DeploymentMonitoring
Execute
WPS-G
Grid
Met
a-sc
hedu
ler
DeploymentMonitoring
GRAM
MDS
RFT Scheduler
...
Nodes
job queue
Compute Elements
Execute
Page 25Session 4: State of the art technology and standards, 12/10/2012
Highlighting of thresholded data
• Date• Threshold• AOI• Target PNG (as URL)
Image thresholding
Display results
AOI & threshold selection
Image highlighting
• Highlight colour, fixed to (RGB=200,0,0)
• URL of thresholded PNG, from previous step
• Highlighted PNG, from previous step
Case Study 5: SSE
Page 26Session 4: State of the art technology and standards, 12/10/2012
Page 27Session 4: State of the art technology and standards, 12/10/2012
Open-source Implementations
• FP7 GENESIS Toolbox
– Evolution of SSE Toolbox
– Supports WPS 1.0.0 and corrigendum
– Multiple bindings + asynchr.
– http://code.google.com/p/toolboxenvironment/
• PyWPS
– Implementation in Python
• 52°North
– Supports WPS 1.0.0
– Supports WPS-T
– Extended with WPS-G On-demand Processing in SSEGrid
Page 28Session 4: State of the art technology and standards, 12/10/2012
Open-source Implementations
• SSEGrid contribution:
– Asynchronous Execute operation with callback (no polling)
Page 36Session 4: State of the art technology and standards, 12/10/2012
WPS with On-Demand Extension
• Feature/operations– Discovery of service metadata (WPS GetCapabilities operation)
– Discovery of process metadata (WPS DescribeProcess operation)
– Execution of processes (WPS Execute operation)
– Deployment of processes (WPS-T DeployProcess operation)
– Undeployment of processes (WPS-T UndeployProcess operation)
– Deployment of (auxiliary) data (WPS-G DeployData operation)
– Undeployment of data (WPS-G UndeployData operation)
– Discovery of data metadata (WPS-G DescribeData operation)
– Query of process status (WPS-G GetStatus operation)
– Cancellation of process (WPS-G Cancel operation)
– Query of process execution trace (WPS-G GetAudit operation)
– Process completion/failure notification (callback operation)
WPS-G
Workflow
1: Execute
2: Callback
Addressed by WPS 2.0
SWG
Page 37Session 4: State of the art technology and standards, 12/10/2012
Ongoing Standardisation Activities
• DREAM Project:
– “Identify the requirements and scenarios, extend the WCPS to cover the needed multidimensional queries in particular for what concerns space/time phenomena related to the joint exploitation of S-2, optical high resolution and other data including geo-statistical data as the one routinely provided by EUROSTAT and other international organisations. “
– WCPS work items:
• WCPS 1.0 based on WCS 1.1.2. Standardisation effort required to align with GMLCOV 1.0 and WCS 2.0 (Core)
• ATS to be updated
• ETS to be created
Page 39Session 4: State of the art technology and standards, 12/10/2012
Planned Standardisation Activities
• HMA-S Project:
– Revise WPS Specification taking into account:
• ESA project’s technical report “SSEGrid Processing On-Demand Extension (WPS-G)”
• ngEO needs in terms of process description and access to the processing outputs
• Relevant third party initiatives, about cloud computing standardisation, such as IEEE P2301.
• WPS 2.0 SWG ongoing work
• ESE Project:
– Create new WPS-G deployment profile for Map-Reduce processing environment on dynamic resources (cloud).
Page 40Session 4: State of the art technology and standards, 12/10/2012
Conclusion
• Two main « Processing » specifications: WPS and WCPS
• WPS:– Widely adopted in ESA projects
– Several open-source implementations available
– Many subjects to be resolved, improved or not yet a formal standard
• E.g. asynchronous, WPS-T, …
– Definition of OGC WPS 2.0 underway but slow process
• WCPS:– Less widely adopted
– To be used in DREAM and ESE projects
• Various ongoing or planned ESA projects contribute to standardisation effort of both specifications.
Page 41Session 4: State of the art technology and standards, 12/10/2012