GO-ESSP LLNL, June 2006 CSML – Stocktake and Forward Look Andrew Woolf Dominic CCLRC Rutherford Appleton Laboratory.

GO-ESSPLLNL, June 2006

CSML – Stocktake and Forward Look

Andrew Woolf ([email protected]), Dominic LoweCCLRC Rutherford Appleton Laboratory

with contributions from NDG Team (Bryan Lawrence, Roy Lowry, Kerstin Kleese van Dam, Ray Cramer, Siva

Kondapalli, Sue Latham, Kevin O’Neill, Ag Stephens)

mailto:[email protected]







Outline

• CSML: History• CSML: Present• CSML: Future


CSML: History• NERC DataGrid: the integration problem

– multiple organisations, formats, storage mechanisms (file, relational)

– only commonality is data semantics


CSML: History• The standards-based approach

– “Conceptual modelling is the process of creating an abstract description of some portion of the real world and/or a set of related concepts.” (ISO 19101)

NDG

Universe of discourse

...

startTime[1]bottomTime[1]salinity[*]temperature[*]depth[*]

CTDProfile

...

...

Feature types

ISO 19103

<?xml version="1.0" encoding="UTF-8"?> <schema targetNamespace="http://ndg.nerc.ac.uk/csml" xmlns="http://www.w3.org/2001/XMLSchema" xmlns:csml="http://ndg.nerc.ac.uk/csml" xmlns:om="http://www.opengis.net/om" xmlns:gml="http://www.opengis.net/gml" elementFormDefault="qualified" attributeFormDefault="unqualified" version="0.1"> <annotation> <documentation>CSML application schema</documentation> </annotation>  <import namespace="http://www.opengis.net/gml" schemaLocation="GML-3.1.0/base/gml.xsd"/> <import namespace="http://www.opengis.net/om" schemaLocation="phenomenon.xsd"/>    <complexType name="DatasetType"> <complexContent> <extension base="gml:AbstractGMLType"> <sequence> <element ref="csml:UnitDefinitions" minOccurs="0" maxOccurs="unbounded"/> <element ref="csml:ReferenceSystemDefinitions" minOccurs="0" maxOccurs="unbounded"/> <element ref="csml:PhenomenonDefinitions" minOccurs="0"/> <element ref="csml:_ArrayDescriptor" minOccurs="0" maxOccurs="unbounded"/> <element ref="gml:FeatureCollection" minOccurs="0" maxOccurs="unbounded"/> </sequence> </extension> </complexContent> </complexType> <element name="Dataset" type="csml:DatasetType"/>    <complexType name="ReferenceSystemDefinitionsType"> <complexContent> <extension base="gml:DictionaryType"/> </complexContent> </complexType> <element name="ReferenceSystemDefinitions" type="csml:ReferenceSystemDefinitionsType"/> <complexType name="ReferenceSystemDefinitionsPropertyType"> <sequence> <element ref="csml:ReferenceSystemDefinitions" minOccurs="0"/> </sequence> <attributeGroup ref="gml:AssociationAttributeGroup"/> </complexType>

Application schema

ISO 19109

FTC

ISO 19110 <gml:definitionMember> <om:Phenomenon gml:id="taxon"> <gml:description>The taxon name</gml:description> <gml:name codeSpace="http://www.vliz.be">taxon</gml:name> </om:Phenomenon> </gml:definitionMember> </NDGPhenomenonDefinitions>  <gml:FeatureCollection>  <gml:featureMember> <NDGPointFeature gml:id="ICES_100"> <NDGPointDomain> <domainReference> <NDGPosition srsName="urn:EPSG:geographicCRS:4979" axisLabels="Lat Long" uomLabels="degree degree"> <location>55.25 6.5</location> </NDGPosition> </domainReference> </NDGPointDomain> <gml:rangeSet> <gml:DataBlock> <gml:rangeParameters> <gml:CompositeValue> <gml:valueComponents> <gml:measure uom="#tn"/> <gml:measure uom="#amount"/> <gml:measure uom="#gsm"/> </gml:valueComponents> </gml:CompositeValue> </gml:rangeParameters> <gml:tupleList> 'ANTHOZOA',63.1,missing 'Scoloplos armiger',66.1,missing 'Spio filicornis',10,missing 'Spiophanes bombyx',60.3,missing 'Capitellidae',131.8,missing 'Pholoe',10,missing 'Owenia fusiformis',23.4,missing 'Hypereteone lactea',6.8,missing 'Anaitides groenlandica',13.2,missing 'Anaitides mucosa',6.8,missing

ISO 19118


CSML: History

• Is this standards stuff relevant?! One example:– EU INSPIRE Directive

• Establishes European Spatial Data Infrastructure – mandating interoperability of environmental data across EEA

• “[ISO/OGC] standards and specifications ... will be considered as reference material for the Consolidation Team and relevant Drafting Teams.” (INSPIRE Work Programme Preparatory Phase 2005 – 2006)

• Annex III Data Themes, incl.:– Atmospheric Conditions– Meteorological geographical features– Oceanographic geographical features– Sea regions


CSML: Historygrep ‘ISO|OGC|WFS|WCS|WMS|GML’ {AGU,AMS,GO-ESSP}/* | wc

0

1

2

3

4

5

6

7

1 (2003) 2 (2003) 3 (2004) 4 (2005) 5 (2006)

GO-ESSP

?0

1

2

3

4

5

1998 1999 2000 2001 2002 2003 2004 2005 2006

AMS (all)

0

5

10

15

20

25

1998 1999 2000 2001 2002 2003 2004 2005 2006

AGU (Spring and Fall)


CSML: History

British OceanographicData Centre

British AtmosphericData Centre

ISO standards

‘Governance Principle’

Climate Science Modelling Language

«FeatureType»AbstractGeometryFeature

«FeatureType»AbstractStaticCoverageFeature

«FeatureType»AbstractSeriesCoverageFeature

+domainSet : AbstractDomain+coverageFunction : MappingRule

«FeatureType»AbstractCoverageFeature

+domainSet+rangeSet+coverageFunction

«FeatureType»GML::AbstractDiscreteCoverage

+track : Trajectory

«FeatureType»TrajectoryFeature

+domainSet : ProfileDomain

«FeatureType»ProfileFeature

+domainSet : PointDomain

«FeatureType»PointFeature

«FeatureType»SectionFeature

+domainSet : GridDomain

«FeatureType»GridFeature

+domainSet : PointSeriesDomain

«FeatureType»PointSeriesFeature

+domainSet : ProfileSeriesDomain

«FeatureType»ProfileSeriesFeature

«FeatureType»SectionSeriesFeature

+domainSet : GridSeriesDomain

«FeatureType»GridSeriesFeature

{domainSet.domainReference = Position}

{domainSet.domainReference = Trajectory}

OGC Obs & Meas::Phenomenon

+parameter

{+,-}phenomenon{+,-}seriesFeature{+,-}gridI{+,-}gridJ{+,-}gridK

«Enumeration»IncrementOrder

+scanOrder[0..1] : Sequence<IncrementOrder>

«DataType»MappingRule

+id[0..1]+metaDataProperty[0..1]+description[0..1]+name[0..1]+boundedBy[0..1]

«DataType»GML::AbstractFeature

<?xml version="1.0" encoding="UTF-8"?><schema targetNamespace="http://ndg.nerc.ac.uk/csml" xmlns="http://www.w3.org/2001/XMLSchema" xmlns:csml="http://ndg.nerc.ac.uk/csml" xmlns:om="http://www.opengis.net/om" xmlns:gml="http://www.opengis.net/gml" elementFormDefault="qualified" attributeFormDefault="unqualified" version="0.1"> <annotation> <documentation>CSML application schema</documentation> </annotation>  <import namespace="http://www.opengis.net/gml" schemaLocation="GML-3.1.0/base/gml.xsd"/> <import namespace="http://www.opengis.net/om" schemaLocation="phenomenon.xsd"/>    <complexType name="DatasetType"> <complexContent> <extension base="gml:AbstractGMLType"> <sequence> <element ref="csml:UnitDefinitions" minOccurs="0" maxOccurs="unbounded"/> <element ref="csml:ReferenceSystemDefinitions" minOccurs="0" maxOccurs="unbounded"/> <element ref="csml:PhenomenonDefinitions" minOccurs="0"/> <element ref="csml:_ArrayDescriptor" minOccurs="0" maxOccurs="unbounded"/> <element ref="gml:FeatureCollection" minOccurs="0" maxOccurs="unbounded"/> </sequence> </extension> </complexContent> </complexType> <element name="Dataset" type="csml:DatasetType"/>    <complexType name="ReferenceSystemDefinitionsType"> <complexContent> <extension base="gml:DictionaryType"/> </complexContent> </complexType> <element name="ReferenceSystemDefinitions" type="csml:ReferenceSystemDefinitionsType"/> <complexType name="ReferenceSystemDefinitionsPropertyType"> <sequence> <element ref="csml:ReferenceSystemDefinitions" minOccurs="0"/> </sequence> <attributeGroup ref="gml:AssociationAttributeGroup"/> </complexType>

Conceptual model Schemas

Tooling


CSML: PresentConceptual model

numerical weather prediction model, ocean general circulation model

Timeseries of gridded parameter fields.GridSeriesFeature

vertical or scanning radar, shipborne ADCP, thermistor chain timeseries

Series of profile-type measurements.ProfileSeriesFeature

tidegauge, rainfall timeseriesSeries of single datum measurements.PointSeriesFeature

gridded analysis fieldSingle time-snapshot of a gridded field.GridFeature

wind sounding, XBT, CTD, radiosonde

Single ‘profile’ of some parameter along a directed line in space.ProfileFeature

raingauge measurementSingle point measurement.PointFeature

ship’s cruise track, aircraft’s flight pathDiscrete path in time and space of a platform or instrument.TrajectoryFeature

ExamplesDescriptionCSML feature type

‘Coverage’ feature typesPure geometry feature type

ProfileSeriesFeature

ProfileFeature

GridFeature

http://www.hpcx.ac.uk/images/HPCxPhoto.jpg


CSML: Present• GML Application schema

– all sections optional

CSML Dataset

csml:UnitDefinitions

csml:ReferenceSystemDefinitions

csml:PhenomenonDefinitions

Storage descriptors

gml:FeatureCollection

csml:NetCDFExtract, csml:AggregatedArray,csml:GRIBExtract, csml:NASAAmesExtract, etc

(cf. UDUNITS.xml)

GMLdictionaries

(cf. CFStandardNames.xml)


CSML: Present• Tooling

– Scanner

• uses ‘feature file map’ to characterise relationship between feature type and directory/files being scanned

• individual Python classes for GML classes, feature types, dataset, etc all have toXML() method for generating GML/CSML

• instances populated by Scanner

$ python csmlscan.py -d /badc/ecmwf-e40/data/gp/gp/ap-o era40gridseries.xml -c GridSeriesFeature -t 'time'

CDAT/cdunif+

csmlscanCSMLFile.xml


CSML: Present• Tooling

– Parser

• all GML/CSML Python classes have fromXML() method

• called recursively by Parser on ElementTree representation

tree = ElementTree(file=CSMLFile) dataset = Dataset()dataset.fromXML(tree.getroot())

...feature = dataset.featureCollection.members[0]variablePhenomenon = feature.parameterunits = feature.rangeSet.aggregatedArray.uom...

csml = dataset.toXML()

ElementTree+

parserCSMLFile.xmlDataset

NetCDFExtract

GridSeriesFeature

featureMember

AggregatedArray

rangeSet


CSML: Present

• Tooling– API

• experimental at present• defined against CSML data model• e.g.: Dataset.getFeatureList(),

GridSeriesFeature.getDomain(), GridSeriesFeature.subsetToGridSeries()

– Different feature types may support different operations (part of UML model!)

– ‘Subsetting’ regarded as ‘extracting a feature’


CSML: Present

• Higher-level services– BADC DataExtractor– OGC Web Coverage

Service• extension of GALEON

server

http://glue.badc.rl.ac.uk/cgi-bin/TPAC/WCS-CSML?SERVICE=WCS&VERSION=1.0.0&REQUEST=GetCoverage&COVERAGE=solar_3&CRS=EPSG:4326&BBOX=0,-90,356,90&TIME=2790-01-01/2790-05-01&FORMAT=CF-netCDF&RESX=2&RESY=2


CSML: PresentCSML (and friends): software stack


CSML PresentOther User Example: Norwegian Met Office


CSML: Future

• Immediate issues (v2)– Feature types:

• Swath• ProfileSeries{Radar, Section, ProfileSeries}?• Lose ‘composite domain pattern’?

– GML 3.2 ( ISO 19136)– remove ‘storage descriptors’ from core CSML

schema– deployment, testing


CSML: Future

• Longer-term issues– GML irregular grids – CSML extends gml:Grid

(evolving standards is slooow!)– ISO 19131 ‘Data Product Specifications’ (e.g.

CRS should be defined in DPS, not AS)– GML & file-based data (will be discussed next

week at OGC TC in Edinburgh)


CSML: FutureOther activities:

Universe of discourse

Feature types

ISO 19103

Application schema

ISO 19109

ISO 19110 <gml:definitionMember> <om:Phenomenon gml:id="taxon"> <gml:description>The taxon name</gml:description> <gml:name codeSpace="http://www.vliz.be">taxon</gml:name> </om:Phenomenon> </gml:definitionMember> </NDGPhenomenonDefinitions>  <gml:FeatureCollection>  <gml:featureMember> <NDGPointFeature gml:id="ICES_100"> <NDGPointDomain> <domainReference> <NDGPosition srsName="urn:EPSG:geographicCRS:4979" axisLabels="Lat Long" uomLabels="degree degree"> <location>55.25 6.5</location> </NDGPosition>'Anaitides mucosa',6.8,missing

ISO 19118

EUproject

‘MOTIIVE’

EUproject‘RISE’

GALEONMetOfficeoperationalmeteorology netCDF

datastandards

RISE and MOTIIVEare both pilots for INSPIRE


CSML: Future• The CSML ‘niche’

– set of base feature types for specialising or using as-is

cd CSML future

CF v ocab Obs & Meas

Climate CRS

CSML

+ GridSeriesFeature+ Profi leSeriesFeature

MyModel

+ Ensemble

+ EnsembleMember+ Mooring

«import»

«import» «import»

«import»

cd MyModel

«interface»CSML::ProfileSeriesFeature

+ domainSet: = ProfileSeriesDomain+ parameter: om:Phenomenon+ rangeSet:

Mooring

+ averaging: + blowover: + deploymentEnd: + deploymentStart:

«interface»CSML::GridSeriesFeature

+ domainSet: = GridSeriesDomain+ parameter: om:Phenomenon+ rangeSet:

Ensemble

+ ensembleParameters:

EnsembleMember

+ memberNumber: + paramValues:

*


CSML: Future

• CSML ‘out-of-the-box’– conceptual model– GML application schema– ‘geometric affordances’ (i.e. operations for

generic Grid, Profile, Point coverages + series)


Links

• CSML: http://ndg.nerc.ac.uk/csml• CSML validator: http://glue.badc.rl.ac.uk/cgi-bin/

csml/parseTest.py• MetOffice ‘Exeter Communiqué:

http://www.metoffice.gov.uk/informatics• NDG CSML subversion:

http://glue.badc.rl.ac.uk/viewsub/viewvc.cgi/ndg/TI02-CSML/

http://ndg.nerc.ac.uk/csml








http://glue.badc.rl.ac.uk/cgi-bin/csml/parseTest.py














http://www.metoffice.gov.uk/informatics

GO-ESSP LLNL, June 2006 CSML – Stocktake and Forward Look Andrew Woolf Dominic CCLRC Rutherford Appleton Laboratory.

Documents