Using python client libraries to access remote servers via Web Coverage Services

Using python client libraries to access remote servers

via Web Coverage Services

Dominic Lowe, BADCBen Domenico, Unidata/UCAR*

and

the GALEON Team

*Work partially supported by US NSF Atmospheric Science Division

Working Together on A Mosaic for Atmospheric Data

Ostia Antica circa 7 BCOstia Antica circa 7 BC

This presentation describes and draws on the This presentation describes and draws on the work* of many collaborating individuals and work* of many collaborating individuals and

institutionsinstitutions* Unidata’s contribution supported by the U.S. National Science * Unidata’s contribution supported by the U.S. National Science FoundationFoundation

Acronym Glossary

• GALEON (Geo-interface for Air, Land, Environment, Oceans NetCDF)

• FES (Fluid Earth Systems, mainly the data systems of the atmospheric and ocean sciences)

• http://www.unidata.ucar.edu/content/publications/acronyms/glossary.html

Outline

• GALEON Background

• OWSLib – OGC Web Services Library

• Simple Exercises for WCS Using OWSLib Clients

• Participating Institutions and Servers

• Future Directions

• References

Reconciling Disparate Data Models:Different Ways of Thinking about Data

• To the GIS (solid earth and societal impacts) community, the world is:– A collection of static featuresfeatures (e.g., roads, lakes, plots of land) with

geographic footprints on the Earth (surface).– The featuresfeatures are discrete objectsdiscrete objects with attributes which can be stored

and manipulated conveniently in a database.database.

• To the fluids (atmosphere and oceans) communities, the world is:– A set of parametersparameters (e.g., pressure, temperature, wind speed) which

vary as continuous functionscontinuous functions in 3-dimensional space and time.– The behavior of the parametersparameters in space and time is governed by a

set of equations.equations.– Data are simply discrete points in the mathematical function space.

Taking Advantage of Web Services for Data System Interoperability

GIS ClientGIS Client Applications Applications

FES ClientFES Client Applications Applications

OpenGIS Protocols:WMS, WFS, WCS, CSW

OGC or proprietary GIS

protocols

OGC or THREDDS, OPeNDAP, ADDE. FTP…

protocols

GIS ServerGIS ServerGIS ServersGIS Servers

Hydrologic, demographic, Hydrologic, demographic, infrastructure, societal infrastructure, societal

impacts, … datasetsimpacts, … datasets

THREDDS ServerTHREDDS Server

FES ServersFES Servers

Satellite, radar, Satellite, radar, forecast model output, … forecast model output, …

datasetsdatasets

Traditional GIS view

Attributes in DBMS

tables

Features as points,

lines, polygons

stefano

In the realm of geolocated datasets, data stored in Geographic Information Systems (GIS) are highly structured and most often stored in an underlying relational database. While this may be a gross simplification, GIS datasets typically consist of “features” on the surface of the Earth that can be represented by points, lines and polygons. An example is a county plat which can show natural features such as streams and rivers, infrastructure like roads and bridges and buildings, and plots of land such as towns, lots, and so forth. The attributes of these features lend themselves to storage in the tables of a relational database. There can be a table for the roads, another for the towns, yet another for the rivers, etc. Each specific feature is a record in a table which provides a very useful way of keeping track of the characteristics of each instance of each feature.Visualization is conceptualized in terms of a set of “layers.” In the physical world, transparent mylar sheets are often used to overlay various sets of features on a given base map. The same idea is used for manipulating the visualization of the classes of features in GIS visualization systems

Typical NetCDF Visualization

OGC GALEON• Phase 1:

– Many successful experiments with WCS 1.0– Gridded data

• Phase 2:– WCS 1.1 (and later) plus CS-W for catalogs– Non-gridded data collections via other Services

(WFS, SOS)– Dearth of client implementations slows progress

• OGC Network Rather than Interoperability Experiment

Apply GIS Tools ToAtmospheric Science Data

OWSLib – OGC Web Service utility library

• is a lightweight package for working with OGC map, feature, and coverage services using Python

• provides a common API for accessing service metadata and wrappers for GetCapabilities, GetMap, GetCoverage and GetFeature requests

• facilitates implementation of simple WCS clients as python scripts

OWSLib –Design Goals

• Instantiate an OGC web service proxy (ala xmlrpclib)

• Make a GetCapabilities request and marshal the response into a Python structure that models OWS Common metadata

• Proxy GetMap/GetFeature/GetCoverage/Get* requests to the service with Pythonic wrappers for those operations

• Note that OWS common is an initiative to separate out common elements of OGC web services -- likely important as GALEON attempts to serve non-gridded data

OWSLib History

• Originally developed by Sean Gillies as part of the Python Cartographic Library.

• Version 0.2 contained support for accessing WMS version 1.1.1 and WFS 1.0.0 servers.

• Version 0.3 developed by Sean Gillies & Dominic Lowe introduced support for WCS versions 1.0.0 and 1.1.0.

• Version 0.3 also introduced a harmonized OWS Common-like API to service metadata that is common across WCS, WMS and WFS. This API is a 'pythonic' interpretation of OWS-Common which provides easy access to the OWS-common metadata.

OWSLib Design Decisions

• Lightweight• Easy to use• Modelled on OWS-common but not verbatim;

“pythonic beats pedantic as long as significant meaning is not lost”

• Use python idioms: e.g. convenient dictionary like access to coverage metadata: wcs['temperature']

• Harmonize metadata between services where not harmonized (and where possible) in line with intentions of OWS common.

Rudimentary WCS Exercisesvia OWSLib Python Scripts

• Use getCapabilities for list of coverages

• Extract name of first coverage in list

• If multiple times are available, use the initial time

• Use getCoverage to request first coverage

• Use defaults wherever possible

• Focus on whether protocol works

• Worry about “correctness” of the returned coverages later

WCS Sites

• US Pacific Fisheries Environmental Lab• US Unidata THREDDS Data Server• US USGS Coast Environmental• US NCDC NOMADS• Netherlands KNMI Geoservices• Netherlands ARGOSS• UK BADC• US NSIDC• US George Mason U.• Italy U. of Florence CNR• US NNEW Weather.aero

General Results

• 3 servers returned a coverage on the first try• 2 required minor changes to the default

parameter list• The rest required interactions between Dominic,

Ben, and the WCS site administrator• 2 cases of missing required parameters• 2 cases uncovered bugs in python client library

which were subsequently fixed• All cases working in the end

Future Enhancements

• Include more servers in the exercise

• Augment exercise in the direction of a true test

• Retrieve representative coverages

• Retrieve representative times

• Use bounding box other than default

• Examine returned coverage

• Track evolution of WCS specification

More Inforamtion

• OWSLib wiki: http://trac.gispython.org/lab/wiki/OwsLib

• Subversion Repository: svn co http://svn.gispython.org/svn/gispy/OWSLib/trunk

• Python Cheeseshop: http://pypi.python.org/pypi/OWSLib/0.3

• GALEON Wiki page: https://sites.google.com/site/galeonteam/Home/GALEON%20WCS%20"Tests"

Using python client libraries to access remote servers via Web Coverage Services

Documents

data systems

gis datasets

natural features

classes of features

gis solid earth

earth surface

collection of static

various sets of features