BADC, BODC, CCLRC, PML and SOC Distributed Data, Distributed Governance, Distributed Vocabularies: The NERC DataGrid + + + + +[ ]= Bryan Lawrence (on behalf of a big team, and note also a substantial piece of work with specific authorship included herein)
54
Embed
Distributed Data, Distributed Governance, Distributed Vocabularies: The NERC DataGrid
Distributed Data, Distributed Governance, Distributed Vocabularies: The NERC DataGrid. Bryan Lawrence (on behalf of a big team, and note also a substantial piece of work with specific authorship included herein). +. +. ]=. +[. +. +. BADC, BODC, CCLRC, PML and SOC. Motivation Standards - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
units, etc.)– GML – canonical encoding [from ISO 19109 “Geographic information –
Rules for Application Schema”]
Semantic Workshop, Edinburgh, June 2006
Architecture: NDG Metadata Taxonomy
… not one schema, not one solution!
CSMLNCML+CF
MOLES THREDDS
DIF -> ISO19115
CLADDIER
Semantic Workshop, Edinburgh, June 2006
Architecture: Deployment Data
ProvidersNDG Core Services
Users
NDG GUI Interface(s)
Vocab Services
Semantic Workshop, Edinburgh, June 2006
Architecture: Deployment
NDG Core Services
Users
NDG GUI Interface(s)
Vocab Services
Semantic Workshop, Edinburgh, June 2006
Architecture: Deployment
Users
NDG GUI Interface(s)
Vocab Services
Semantic Workshop, Edinburgh, June 2006
Architecture: Deployment
UsersVocab Services
Current Status
Semantic Workshop, Edinburgh, June 2006
Discovery Service
http://ndg.nerc.ac.uk/discovery
NDG Products: Discovery Portal
NB: Web Service Interface (you can do the search from your own site and format and present the results there!
Semantic Workshop, Edinburgh, June 2006
Semantic Workshop, Edinburgh, June 2006
Semantic Workshop, Edinburgh, June 2006
Ugly as sin! A hint of things to come:
NDG Products: MOLES
Semantic Workshop, Edinburgh, June 2006
MOLES: implementation
Core linking concept is the deployment
Deployment
Activity
on behalf of an Activityof a Data Production Tool at an Observation Station
that produces a Data EntityData
ProductionTool
ObservationStation
Data Entity
Each of the main metadata objects has security data attached to it. This means that this can be applied to queries on the metadata
Links the metadata records into a structure that can be turned into a navigable structure
Semantic Workshop, Edinburgh, June 2006
Simulators as data production tools: NumSim
NDG Products: NumSim
Semantic Workshop, Edinburgh, June 2006
NumSim Example
NumSim Example
Semantic Workshop, Edinburgh, June 2006
Semantic Workshop, Edinburgh, June 2006
NDG Products: DataExtractor
Semantic Workshop, Edinburgh, June 2006
Semantic Workshop, Edinburgh, June 2006
Background activity being parallelised with GODIVA/CCLRC e-science collaboration (spectral -> gridpoint + CDMS + visualisation tools)
Download either plot or the data that went into the plot.
NDG Products: GEOSPLAT
Semantic Workshop, Edinburgh, June 2006
ERA40:•All driven from one CDML file, 9 TB online spherical harmonics, looking like 40 TB “virtual” gridded!
Semantic Workshop, Edinburgh, June 2006
NDG-A: Climate Science Modelling Language
• Aims:– provide semantic integration mechanism for NDG data– explore new standards-based interoperability framework– emphasise content, not container
• Design principles:– offload semantics onto parameter type (‘phenomenon’,
observable, measurand)• e.g. wind-profiler, balloon temperature sounding
– offload semantics onto CRS• e.g. scanning radar, sounding radar
– ‘sensible plotting’ as discriminant• ‘in-principle’ unsupervised portrayal
– explicitly aim for small number of weakly-typed features (in accordance with governance principle and NDG remit)
Semantic Workshop, Edinburgh, June 2006
Climate Science Modelling Language
• CSML feature types– defined on basis of geometric and topologic
structureCSML feature
type Description ExamplesTrajectoryFeature Discrete path in time and space of a platform or instrument. ship’s cruise track, aircraft’s flight pathPointFeature Single point measurement. raingauge measurementProfileFeature Single ‘profile’ of some parameter along a directed line in space. wind sounding, XBT, CTD, radiosondeGridFeature Single time-snapshot of a gridded field. gridded analysis fieldPointSeriesFeature Series of single datum measurements. tidegauge, rainfall timeseriesProfileSeriesFeature Series of profile-type measurements.
vertical or scanning radar, shipborne ADCP, thermistor chain timeseriesGridSeriesFeature Timeseries of gridded parameter fields.
numerical weather prediction model, ocean general circulation model
• Problem and Solution• NERC DataGrid Vocabulary Model• Vocabulary Technical Governance• Vocabulary Content Governance• Mappings and Thesaurus Server• Potential Role of Local Mappings
Semantic Workshop, Edinburgh, June 2006
The Problem
• NERC DataGrid cannot function operationally without metadata and data semantic interoperability
• This will never be achieved without:– Readily accessible standard terms whose
meaning is clearly understood– Readily accessible semantic maps both
within and between lists of standard terms– Semantic maps between local terms and
standard terms
Semantic Workshop, Edinburgh, June 2006
The Solution?
• Implementation of a Vocabulary Server• Building OWL ontologies mapping
between domain-relevant de-facto standard vocabularies
• Deploying the ontologies through a Web Service thesaurus server
• Making tools available for users to build and deploy local ontologies
Semantic Workshop, Edinburgh, June 2006
NDG Vocabulary Model
Entry (timestamped)
Key Term Definition
Entry (timestamped)
Key Term Definition
Entry (timestamped)
Key Term Definition
List (versioned, timestamped and labelled with key, name and definition)
List (versioned,timestamped and labelled
with key, name and definition)
Constraint (Aggregation of lists across which
entry keys are unique)
Semantic Workshop, Edinburgh, June 2006
NDG Vocabulary Model
• The vocabulary resource is built from Entries– The representation of a single object in the real world
comprising:• Key - A bit pattern that represents an entity. It must be unique,
permanent and free from semantics.• Term – Text used to label the entity to facilitate human
recognition.• Abbreviation – An shortened version of the term for use where
space is tight. Target size is 20-30 bytes.• Definition – Text that unambiguously specifies the entity.
– Entries are aggregated into Lists (entity class or subclass e.g. UK post towns)
– Lists are aggregated into Constraints (entity class e.g. post towns of the world)
Semantic Workshop, Edinburgh, June 2006
Vocabulary Technical Governance
• The story so far– Lists are available as flat ASCII files or XML
documents as URLs e.g.• http://www.cgd.ucar.edu/cms/eaton/cf-metadata/standard_name.xml• ftp://ftp.pol.ac.uk/pub/bodc/jgofs/datadict/new/parameter_group.csv• http://www.sea-search.net/cdi_documentation/cdi_sampling_codes.csv• http://gcmd.nasa.gov/Resources/valids//gcmd_parameters.html
– Some (BODC, SEA-SEARCH) include keys– Some (CF, BODC) include definitions– None are properly versioned
• Versioning should:– Provide a unique label for each instantiation of the list– Enable any previous instantiation of the list to be
recreated – Provide timestamp information for creation and
modification of every object in the vocabulary system• Delivery should
– Be from the master, not a copy– Be accessible to software agents to allow automated
synchronisation of local copies– Have a ‘hotline’ to content governance
Semantic Workshop, Edinburgh, June 2006
Vocabulary Technical Governance
• NERC DataGrid Vocabulary Server– Back End
• Fully automated record archive, timestamps and version numbering. Live April 2006.
• 47 (of 115) lists publicly accessible.– Front End
• Web Service API. Live June 2006.• XML list downloads from website (July 2006?).• Web-form tools (August 2006?).
Semantic Workshop, Edinburgh, June 2006
Vocabulary Content Governance
• Standard lists need to respond to ever expanding user requirements
• Change needs to be rapid or users lose interest
• Standard lists need to maintain information quality and internal consistency
• Content governance has to resolve these conflicting requirements
Semantic Workshop, Edinburgh, June 2006
Vocabulary Content Governance
• Content governance in oceanographic and atmospheric domains is based on:– Moderated e-mail discussion lists– ‘Benign Dictator’ and well-meaning volunteers
• Variable success depending on right people having ‘spare’ time at the right moments
• More formalism underpinned by more resources required
• But need to be careful about going too far or levels of service become unacceptable
Semantic Workshop, Edinburgh, June 2006
Mappings and Thesaurus Server
• There will never be a single list for a given topic
• Term mapping therefore an essential part of semantic interoperability
• Marine Metadata Interoperability (http://marinemetdata.org) have developed tooling and trialled mappings in the measurement phenomena arena
– Incorporate this map into the NDG Discovery Service to facilitate smart searching (e.g. ‘pigments’ finds dataset labelled ‘chlorophyll’) through MMI Web Service
– Integrate ontology maintenance into source list maintenance
Semantic Workshop, Edinburgh, June 2006
Role of Local Mappings
• There will always be local terms and understanding
• ‘Pigment data sets’ could mean:– Chlorophyll OR carotenoids OR
phaeopigments– Chlorophyll AND carotenoids AND
phaeopigments• Depends on point of view
Semantic Workshop, Edinburgh, June 2006
Role of Local Mappings
• Possible solution to this:– User builds an ontology reflecting local
perception of the mapping between local terms and standard terms
– Discovery or data integration tools use ontology as a ‘plug-in’ allowing user to operate with local terminology
• Tools (e.g. VINE) could be made available to facilitate this
Semantic Workshop, Edinburgh, June 2006
NDG Timeline
NDG2 runs until September 2007:• NDG-Alpha (June 2006)
– Not all components in place (particularly delivery broker)– Not many (maybe only DX) products will be deployable by non-NDG
participants(too much hard work installing things that haven’t been optimised for
installation)– Discovery portal will be (is now) usable, linking to NCAR data etc, but
isn’t very user friendly (options not obvious etc).• NDG-Beta (Feb 2007)
– Most components should work, but deployment of software may still be difficult by non-participants
• NDG-Prod (Jun 2007)– Should be deployable and far more user friendly (spending from Feb-
June working on deployment and friendliness, no new functionality)• Last few months working on sustainability etc