Www.seegrid.csiro.au SEEGRID Roadshow Technical Workshop Stuart Girvan – Geoscience Australia Rob Atkinson – Social Change Online CSIRO - Andy Dent, Simon.

www.seegrid.csiro.au

SEEGRID Roadshow Technical Workshop

Stuart Girvan – Geoscience Australia

Rob Atkinson – Social Change Online

CSIRO - Andy Dent, Simon Cox, Robert Woodcock

SEEGrid Roadshow 2005

Disclaimer

Participants should only act on the information and advice in these presentations after independent consideration of their circumstances


Program

Demonstrator Architecture

OGC's WFS Specification

Introduction to Geoserver

GML Application Schemas

Implementing Geoserver - Database and Data Mapping Issues

Use Cases and Service Structure for Clients in an Open Ended Architecture

Further Technical Issues and Discussion


Demonstrator Project Participants


Contributors

Rob Atkinson, Peter Barrs – Social Change Online

Joan Esterle (UQ), Simon Cox, Rob Woodcock, Andy Dent – CSIRO EM

Greg Jenkins, Mark Jolly, Terry Hannant – PIRSA

Brendon Ward, Stephen Bandy – DOIR (GSWA)

Lesley Wyborn, Aaron Sedgmen, Tim Mackey – GA

Stuart Girvan – GA, pmd*CRC


Demonstrator Architecture


Did everyone see the demonstrator this morning?

http://cgsrv3.arrc.csiro.au/seegrid/savedapps/filter

http://www.ga.gov.au/wfs/reader/wfsGeochemReaderHome.jsp

Demonstrator

PIRSA DOIR GA

Web Map Composer

Web Feature Services (OGC)

GA

XML - GML\XMML

Web Map Composer

PIRSA DOIR

PIRSAWeb Feature Service (WFS)

Common Interface Binding – GML/XMML

GA Geochemistry

Feature Data Source

DOIRGeochemistry

FeatureData Source

DOIRWeb Feature Service (WFS)

GAWeb Feature Service (WFS)

Geoserver (Open Source)

PostGIS (Open Source)

OraclePostGIS (Open Source)

CLIENT APPLICATIONS

DATA ACCESS SERVICES

DATA SOURCES

WebMap Composer

GA Reports Application

PIRSA Geochemistry

FeatureData Source

Little or no change required here

Translation to standards here

Common Interface Binding – GML/XMML

WebMap Composer

GA Reports Application

PIRSAWFS

DOIRWFS

GAWFS

NTGSWFS

MRTWFS

NRMWFS

NSWDPIWFS

VICDPIWFS

??

?

CLIENTS

DATA SOURCES

DATA SERVICES


Services

Registry

ClientApplications

publish

bind

find

Data Access Services

Simple Web Services Architecture


ClientApplications

bind

• For the moment not using registries to find or publish services

• Using “Magic Knowledge”, ie applications will know the data services available

GA WFS

GSWA WFS

PIRSA WFS

Services

Simple Web Services Architecture


OGC's WFS Specification


Web Feature Service (WFS)

Specification https://portal.opengeospatial.org/files/?artifact_id=8339 (only 131 pages!)

Uses http

Accepts requests as either key-value pairs or XML

Responds in XML (GML)



Uses request/response not SOAP (although it can be wrapped in SOAP)

Three main calls:

GetCapabilities

DescribeFeatureType

GetFeature



WebFeatureService

Get Capabilities Request

Get Capabilities Response

Describe Feature Type RequestDescribe Feature Type Response

Get Feature RequestGet Feature Response

http protocol

XML/KVP

XML

XML/KVP

GMLSchema

XML/KVP

GML

Data Source

ConfigFiles

Application (web based or desktop)


Features

Features ≈ geographic features

Features are Geography Markup Language (GML) encoded

GML provides a standard for representing geometries (points, lines, polygons) and properties of geographic features


Features

Borehole collar location shape collar diameter length operator logs related observations …

Fault shape surface trace displacement age … Ore-body

commodity deposit type host formation shape resource estimate …

Observation location subject/specimen/station property/theme/measurand method operator date/time result (+ type/reference

system/scale/classification) …

Basin? formations shape – time

dependent resource estimate …


<abc:Bridge gml:id = “b1”> <abc:span>50</abc:span> <abc:heightAtCenterSpan>51</abc:heightAtCenterSpan> <abc:constructed>10-10-1996</abc:constructed> <gml:location> <gml:Point srsName = “#myRefSys”> <gml:coordinates>12435.2, 6756.34</gml:coordinates> </gml:Point> </gml:location></abc:Bridge>

Example: Instance of bridge feature encoded as GML

GML

attributes

Features


WFS Demo

GetCapabilities

DescribeFeatureType

GetFeature

http://www.ga.gov.au/ogcservices/wfs/GetCapabilities


Filter Encoding Specification

Defines how to write WFS queries when making a GetFeature call

http://portal.opengeospatial.org/files/?artifact_id=8340


Filter Encoding Example

<Filter>

<And>

<PropertyIsLessThan>

<PropertyName>DEPTH</PropertyName>

<Literal>30</Literal>

</PropertyIsLessThan>

<BBOX>

<PropertyName>Geometry</PropertyName>

<gml:Envelope srsName="http://www.opengis.net/gml/srs/epsg.xml#63266405">

<gml:lowerCorner>13.0983 31.5899</gml:lowerCorner>

<gml:upperCorner>35.5472 42.8143</gml:upperCorner>

</gml:Envelope>

</BBOX>

</And>

</Filter>


How would you use a WFS?

Using http calls from your programming language to access the WFS and native parsers

Using http calls and style sheets to transform the XML


Eg GA Report Pages

Java reads list of input URLs

Subs in search values to pre-prepared XML chunks

Calls URL of WFS(s) with XML parameters

Parses XML document returned by WFS

Data from XML document held in Java objects and sent to jsp pages for display


Introduction to Geoserver


Why Geoserver?

It’s the reference standard for implementation

Open Source – we could influence and directly upgrade if necessary

Open Source – no license fees/purchase cost


Other WFS Products

• Deegree (open source), Cadcorp SIS Map Server, CARIS Spatial Fusion Enterprise, RedSpider Web 3.0, Map Manager, GenaWare, SclFeatureServer, JCarnacGIS, GeoMedia WebMap, MapXtreme (MapInfo), MapServer(UMN) 4.2 + more

* Source: OGC Registered Products


Geoserver

Java based, servlets

Runs using Tomcat (or embedded Jetty)

Open source community size?

Reliant on and closely connected to Geotools

Lots of modules


Setting up Geoserver

Install into Tomcat using WAR file or use inbuilt Jetty servlet container

Works with PostGIS, Oracle, ARCSDE, Shapefiles, mysql as data sources

Config files catalog.xml, info.xml, schema.xml directory structure




Community/Application Schemas

Community schemas provide the common or shared model

The target community schema is essential


GML Community Schemas

GeologicalXMML - Exploration data (Example Geochemistry)

ADX – Assay data

GPML – plate tectonics

XMML/GeoTime – stratigraphy

CGI International Model

OtherLandGML

MarineXML



Design methodology can be learnt to develop community specific GML application schemas (Simon Cox, Andy Dent CSIRO)

Equally important is who will be the custodian and manager of a community schema

Also see GML 3.1 specification , Chapter 8 Rules for Application Schemes


Informational Requirements

Information models impact on real world performance

Ontology

Registries


Why XML?

Extensibility

Self describing

Ability to be (remotely) validated against schema

XML Schema provides “loose tolerances”

All software languages have tools to deal with XML


Implementing a WFS Using Geoserver –Database and Data Mapping Issues


Traditional GIS and Standard WFS

The WFS spec states the minimum requirements for a service interface

It says nothing about private schemas vs public (community) schemas

Hence most WFS software will only produce “flat” GML


Flat GML Example

Sample ID Coordinates Sample Type SiO2% TiO2% Al2O3% Fe2O3% MgO%

11111 127, -32 Whole Rock 50 1 14 14 3

<gml:featureMember> <GeochemSpecimen gmlid=”1111”>

<gml:Point srsName="http://www.opengis.net/gml/srs/epsg.xml#8307"> <gml:coordinates decimal="." cs="," ts="">118.78531,-21.122948</gml:coordinates> </gml:Point> <SampleType>whole-rock</ SampleType > <SiO2%>50</SiO2%> <TiO2%>1</TiO2%> <Al2O3%>14</Al2O3%> <Fe2O3%>14</Fe2O3%> <MgO%>3</MgO% >

</GeochemSpecimen></gml:featureMember>


Flat GML Example

<SiO2%>50</SiO2%>

<SiO2%ProcedureID>1</SiO2%ProcedureID>

<SiO2%Name> Proc1</SiO2%Name>

<SiO2%Description> Blah 1</SiO2%Description>

<SiO2%Method> X1</SiO2%Method>

<SiO2%DetectionLimit>0.01</SiO2%DetectionLimit>

<TiO2%>1</TiO2%>

< TiO2%ProcedureID>2</ TiO2%ProcedureID>

< TiO2%Name> Proc2</ TiO2%Name>

< TiO2%Description> Blah 2</ TiO2%Description>

< TiO2%Method> X2</ TiO2%Method>

< TiO2%DetectionLimit>0.01</ TiO2%DetectionLimit>

Nested XML Example

<GeochemMeasurement>

<resultMeasureExactoriginal UOM="wt%">50</resultMeasureExact>

<Analyte>SiO2</Analyte>

<procedureUsed>

<AssayProcedure gml:id="1">

<name>Proc1</name>

<description>Blah 1</description>

<Method>X1</Method>

<analyteDetails>

<AnalyteSensitivity>

<Analyte> SiO2</Analyte>

<lowerDetectionLimit uom="wt%">0.01</lowerDetectionLimit>

</AnalyteSensitivity>

</analyteDetails>

</AssayProcedure>

</procedureUsed>

</GeochemMeasurement>

<GeochemMeasurement>

<resultMeasureExactoriginal UOM="wt%">1</resultMeasureExact>

<Analyte>TiO2</Analyte>

<procedureUsed>

<AssayProcedure gml:id="2">

<name>Proc2</name>

<description>Blah 2</description>

<Method>X2</Method>

<analyteDetails>

<AnalyteSensitivity>

<Analyte> TiO2</Analyte>

<lowerDetectionLimit uom="wt%">0.01</lowerDetectionLimit>

</AnalyteSensitivity>

</analyteDetails>

</AssayProcedure>

</procedureUsed>

</GeochemMeasurement>


XML Structures

There are potentially many structures or information views that may be appropriate for describing data

These may be influenced by use cases and architectural considerations


Data source to community schemas

Community schemas provide the common or shared model

All data providers have their own local data model

All data providers must map data from local source (database) to community schema, irrespective of technology implementation


Requirement

We need to be able to map from private schemas to public in as flexible a manner as possible


WFS Spec Problem

WFS spec allows any query based on the feature description (community schema)

Comes from the point of view that you’d be able to use, for example, xpath against a ‘notional’ XML document

In reality almost all data sources will be databases or proprietary formats of some kind

In practise want to limit the potential queries that can be made against the data source


Query Profiles

Why you should only allow certain queries

efficiency

security

user friendliness

legal requirements

Eg phone books, amazon.com


Requirement

Limit the possible queries that can be made against the feature data source


Implementation

At beginning of project Geoserver (nor any other software we were aware of) could not perform mapping from private database to community schema

Geoserver could only serve “flat” GML

No restrictions on queries (follows the WFS spec)


Geoserver Enhancements

Initial stages of project included upgrading Geoserver to include plug in mapping to complex schemas and ability to configure pre-defined queries

Info.xml and Schema.xml files

Catch – only works from database source that you can run an SQL query against


info.xml


schema.xml


Mapping & Query Limitations

Limited amount of branching

Restricted in the positioning of branching

No data dependent tags

Only one predefined query per feature type


Coping with Limitations

The impacts of the limitations of Geoserver and WFS were accommodated by

agreeing to slightly modified target schemas

agreeing to commonly defined input parameters and patterns

a small number of technical hacks


Geoserver Development

The Geotools community is considering a comprehensive solution to the problems of complex objects.

See http://docs.codehaus.org/display/GEOTOOLS/Community+Schema+Support+and+Complex+Types

If you’re very keen get involved http://geoserver.sourceforge.net/html/index.php


Technical Lessons

https://www.seegrid.csiro.au/twiki/pub/Infosrvices/MCAProjectOutputs/TechnicalLearningfromtheMCA.pdf


Baseline Software Requirements*

Compliant with current WFS specification

Separation of back end system from common interface

Ability to restrict incoming queries

*For equivalent of current project


Baseline Software Requirements

Ability to access back end data source of choice (potentially through firewall)

Ability to produce GML documents with at least single branching complexity, element attributes, and nestable single elements


Software Wishlist

Ability to map to a GML document of any complexity and a GUI interface to configure the mapping

Ability to run various queries against a single defined feature type

Ability to map to an element based on a data source value


WFS Spec Changes

As a result of the project there is a formal change request in with OGC to allow for query profiles for a feature

This will also require a minor change in the Filter Encoding Specification


Moving forward

•The SEEGRID Roadshow will extend the test bed to the remaining state and territory surveys for Geochemistry

•Considering a 3 year ICIP (Industry Co-operative Innovation Program) grant which will take it from a test bed to a production system (network)

The Industry in ICIP is critical


Moving forward

•The establishment of a production level network of services for one data type will greatly smooth the way for other types of info, not just Geological

•Opportunities to cooperate with SEEGRID to progress technological and standards solutions

As part of ICIP proposal or independently

•SEEGRID is free, contributions to common goals come from sub sets of the community


Use Cases and Service Structure for Clients in an Open Ended Architecture

User wants to select sample data by some bounding box

Sample data must be queryable by bounding box

Scenario 1

User wants to see details of data selected in bounding box

Sample data must be queryable by bounding box or by id numbers of selected records

Scenario 2

User wants to select sample data by some bounding box and some threshold value

Sample data must be queryable by bounding box and threshold value property

Scenario 3


Client Requirements

Availability, SLA’s

Support

Response times

Data quality, metadata


Server Considerations

Not all use cases/query profiles can be known

But you can probably make a reasonable guess as to what may be required

Most importantly, like any project, need to involve clients (if possible)


Client Strategies for Using WFS

Querying everything vs multiple queries

Schema dependency

Caching vs going back to the web service


Further Technical Issues and Discussion


Technical Lessons

https://www.seegrid.csiro.au/twiki/pub/Infosrvices/

MCAProjectOutputs/TechnicalLearningfromtheMCA.pdf


WFS in the Notional Architecture

Notional Architecture is the end game perfect world

A brief note – what are registries about?

Imagine you’re an application!

Features Coverages ModelsReal-time

data


Ontology

Registry

ServiceMetadata

ClientApplications

PersistentBindings

FeatureType

Catalog

ModelManagement

Services

publish

bind

find

DatasetMetadata

ProcessingServices

Notional Architecture

Notional architecture

Dictionary ServiceMetadata

PersistentBindings

FeatureType

Catalog

DatasetMetadata

Yellow Pages + Semantics

MANAGED

Features Coverages ModelsReal-time

data


Registry

ClientApplications Model

ManagementServices

publish

bind

find

ProcessingServices

Managed Dataset MetadataSomeone has vouched that from this service you can

get a sparkplug that has a certain pedigree/qualityMetadata - it was made in Germany by Mercedes Benz

Managed Persistent BindingsSomeone has created instructions to use this service

in the following waysPersistent Binding (ie always true) - only ever put the sparkplug the right way up

For a Commodore you need sparkplug X

Managed Service MetadataSomeone has vouched that a spare parts dealer service

exists that supplies sparkplugs Service Metadata – Add in the yellow pages for the spare parts dealer

Including details of how to contact (use) that service

Managed Feature Type CatalogueFeature Type Catalogue – List of Feature Types eg Spark Plugs,

Petrol Caps, Oil FiltersFeature Type - The agreed set of attributes a sparkplug must have

Thread Diameter, Thread Reach, Gap etc

Www.seegrid.csiro.au SEEGRID Roadshow Technical Workshop Stuart Girvan – Geoscience Australia Rob Atkinson – Social Change Online CSIRO - Andy Dent, Simon.

Documents