Common Data Index (CDI) Data Discovery & Access service

IMDIS conference23-25 September, 2013

Lucca, Italy

Common Data Index (CDI) Data Discovery & Access service

D.M.A. Schaap - SeaDataNet Technical Coordinator

Background

– Marine and oceanographic data are collected by hundreds of organisations from government, research and industry across Europe

– Many organisations manage their data themselves, while others make use of National Oceanographic Data Centres (NODCs) and other Marine Data centres.

– The CDI service has been initiated by the NODCs to provide a unified overview and access to all these distributed data resources

Common Data Index (CDI)

– CDI service development started early 2000 as part of EU FP5 framework programme in Sea-Search project (2002 – 2005)

– Further development in EU FP6 SeaDataNet (2006 – 2011) and EU FP7 SeaDataNet 2 project (2011 – 2015)

– Driven by the NODCs of 35 countries around European seas– Uptake by many other EU projects, such as FP6 - Up-Grade

Black Sea SCENE, FP6 - CASPINFO, FP7 - EuroFleets, FP7 – EuroFleets II, FP7 - JERICO, FP7 - Geo-Seas, FP7 – CitClops, FP7 – Micro B3

Common Data Index (CDI)

– It provides users a highly detailed insight and unified access to the large volumes of marine and oceanographic data sets managed by distributed data centres

– CDI describes an observation such as sample, core, timeseries, profile, trajectory, survey

– Metadata format is marine profile of the ISO 19115 geographical content standard, while XML coding follows the ISO 19139 Schema and it is fully INSPIRE compliant

Common Data Index (CDI)– Most info fields are marked up using the SeaDataNet Controlled

Vocabularies and other SeaDataNet European directories (EDMO – organisations; EDMERP – projects; CSR – cruises; EDMED – data set collections)

– Set of tools and web services to be used by each data centre • MIKADO: generator of XML descriptions of SeaDataNet

catalogues • NEMO: reformatting software to SeaDataNet formats • Download Manager: local software to control downloading • ODV: Ocean data view adapted to SeaDataNet needs • Web services for vocabularies and directories

– Online services for submission, validation, publication, retrieval, and visualisation

SeaDataNet NODCs and Marine data centres are leading partners• The SeaDataNet NODC’s and Marine data centres are mostly

divisions of major national marine research institutes and based in 35 countries, surrounding the European seas

• They are experienced in managing data, which includes quality control, documenting, long term stewardship and publishing, for their own institute and for many other institutes in their country

• NODC’s are members of IOC-IODE and ICES for sharing global practices

• They are encouraging and supporting other institutes in their countries to become part of the CDI infrastructure, through building of national infrastructures and through European projects

Pan-European directory services

CDI

CSR

EDMED

EDMERP

EDIOS

EDMO

Overview of organisations in Europe with their involvement in marine projects, data sets, cruises and monitoring

Pan-European directory services• EDMO : European Directory of Marine Organisations (>2900 entries)• EDMED : European Directory of Marine Environmental Datasets (>3900 entries from >700 data holding centres)• EDMERP : European Directory of Marine Environmental Research projects (>2800 entries)• CSR : Cruise Summary Reports (>44000 entries)• EDIOS : European Directory of Ocean Observing Systems (programmes and stations) (360 programmes; >16000 stations)

• Controlled Vocabularies for platforms, parameters, instruments, sea regions, data formats, units, datums, …….. (> 160.000 terms and international governance)

Portal with all services, standards and tools for users AND data providers

http://www.seadatanet.org

CDI service for discovery and unified access of data

[email protected] – www.seadatanet.org

IMDIS Conference, Lucca, Septembre 23-25, 2013

CDI database

User interfaces

DownloadManager

datadata

data

Shopping basket

Request Status Manager

DownloadManager

DownloadManager

Central UserRegister

Components of the CDI service

CDI service for discovery and unified access of data



• Tracking and tracing of all shopping requests• By users• By data providers• Analysis of transactions• Checking status of orders and downloading from data providers

Request Status Manager



Request Status Manager - reportingOverseeing all transactions and preparing reports

Monitoring of infrastructureMonitoring of the operational availability of the infrastructure for core services centrally based (website, catalogues querying) and for local services (downloading from connected data centres)



Monitoring of shopping process

Monitoring of the well functioning of the CDI data requesting and downloading process between central portal and connected data centres by ROBOT user, alert mailings and RSM logging of all steps

Interoperability• Support of OGC WMS – WFS, CSW and OpenSearch

• SeaDataNet CDIs can be retrieved from the IOC-IODE Ocean Data Portal (ODP)

• SeaDataNet CDIs can be retrieved from the GEOSS portal

• For refining and enriching the SeaDataNet standards• For making the SeaDataNet approach and standards fit for purpose of

many different user communities• For expanding the CDI infrastructure with more data centres and more

volumes of multi-disciplinary data • For developing added-value services on top of the CDI infrastructure• For developing interoperability to other infrastructures

Cooperation with other EU projects and EU initiatives

CDI service and EMODNet • CDI service engaged in EMODNet Chemistry, Bathymetry,

Physics, Geology and Biology • EMODNet stimulates wider adoption of standards, more

connected data centres and development of basin scale data products and services, fit for purpose of specific user communities such as MSFD implementation, industry users, ..

• EMODNet is management and policy driven which provides an excellent opportunity for CDI infrastructure to become a sustained operational infrastructure

> 1,56 million CDI entries from 34 countries and 103 data centres and >500 originators for physics, chemistry, geology, geophysics, bathymetry and biology; years 1800 – 2014; 85% unrestricted or under SeaDataNet licence

Already 103 data centres connected and more underway!

Name CountryPhysical Oceanography Chemistry Geology Bathymetry Biology Fisheries

Polytechnic University of Tirana - Institute of GeoSciences, Energy, Water and Environment Albania XFlanders Marine Institute Belgium X X X X

Management Unit of North Sea and Scheldt Estuary Mathematical Models, Belgian Marine Data Centre Belgium X X X X X

Bulgarian National Oceanographic Data Centre(BGODC), Institute of Oceanology Bulgaria X X X X XInstitute of Fishery Resources (IFR) Bulgaria X X X X

National Institute of Meteorology and Hydrology, Bulgarian Academy of Sciences Bulgaria X X X XUniversity of Mining and Geology - Saint Ivan Rilski Bulgaria XInstitute of Oceanography and Fisheries Croatia X X XCyprus Oceanography Center Cyprus X X X

Aarhus University, Department of Bioscience, Marine Ecology Roskilde Denmark X X X XGeological Survey of Denmark and Greenland (GEUS) Denmark XInternational Council for the Exploration of the Sea (ICES) Denmark X XGeological Survey of Estonia Estonia X X

Marine Systems Institute at Tallinn University of Technology Estonia X X XFinnish Meteorological Institute Finland X XSYKE Finland X

Already 103 data centres connected and more underway!Name Country

Physical Oceanography Chemistry Geology Bathymetry Biology Fisheries

BRGM / Office of Geological and Mining Resources France X XCentre de Données Géophysiques (CDG) / CNRS - Université de Strasbourg France XIFREMER / IDM/SISMER France X X X X XSHOM (SERVICE HYDROGRAPHIQUE ET OCEANOGRAPHIQUE DE LA MARINE) France X X XIv.Javakhishvili Tbilisi State University, Centre of Relations with UNESCO Oceanological Research Centre and GeoDNA (UNESCO) Georgia X X X XNational Environmental Agency of the Ministry of Environment Protection and Natural Resources Georgia XScientific - Research Firm GAMMA Georgia X X XFederal Institute for Geosciences and Natural Resources (BGR), Hannover. Germany X XGerman Oceanographic Datacentre (NODC) Germany X X X X XPANGAEA - Data Publisher for Earth & Environmental Science Germany X X X XHellenic Centre for Marine Research, Hellenic National Oceanographic Data Centre (HCMR/HNODC) Greece X X X XNational Center for Sustainable Development Greece XNational Observatory of Athens, Institute of Geodynamics Greece XMarine Research Institute (MRI) Iceland XCoastal and Marine Research Centre (UCC) Ireland XGeological Survey of Ireland Ireland X X X X XMarine Institute Ireland X



Israel Oceanographic and Limnological Research (IOLR) Israel X X X

CNR, Institute of Marine Science U.O.S. of Pozzuolo di Lerici (SP) Italy X X X XENEA Centro Ricerche Ambiente Marino - La Spezia Italy XInstitute of Marine Science (ISMAR) - Ancona Italy X X XInstitute of Marine Science (ISMAR) - Bologna Italy X X

ISAC - Institute of Atmospheric Sciences and Climate (Rome) Italy X X

ISPRA-Institute for Environmental Protection and Research Italy X X X

OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale), Infrastructures Division Italy X X

OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale), Division of Oceanography Italy X X X X X

Faculty of Geography and Earth Sciences, University of Latvia (LU) Latvia X X XLatvian Institute of Aquatic Ecology Latvia X XEnvironmental Protection Agency (EPA) Lithuania X

Institute of Geology and Geography of Nature Research Centre Lithuania X X

International Ocean Institute - Malta Operational Centre (University Of Malta) / Physical Oceanography Unit Malta X X XInstitute of Marine Biology (IMBK) Montenegro XNational Institute of Fisheries Research (INRH) Morocco X



IMARES, Wageningen UR, Institute for Marine Resources and Ecosystem Studies, IJmuiden Netherlands X X XNetherlands Institute for Ecology, Centre for Estuarine and Marine Ecology Netherlands X X X XNIOZ Royal Netherlands Institute for Sea Research Netherlands X X X XRijkswaterstaat Waterdienst Netherlands X X X XRoyal Netherlands Meteorological Institute Netherlands XRoyal Netherlands Navy Command, Hydrographic Office Netherlands X XTNO Geological Survey of the Netherlands Netherlands XGeological Survey of Norway (NGU) Norway XInstitute of Marine Research - Norwegian Marine Data Centre (NMD) Norway X X XUNEP/GRID-Arendal Norway X XInstitute of Meteorology and Water Management, Maritime Branch in Gdynia (IMWM MB) Poland X XInstitute of Oceanology, Polish Academy of Sciences (IO PAS) Poland XPolish Geological Institute - National Research Institute, Branch of Marine Geology (PGI BMG) Poland X XIHPT, Hydrographic Institute Portugal X X X X XNational Laboratory of Energy and Geology Portugal X X XPortuguese Institute of Ocean and Atmosphere Portugal X XNational Institute for Marine Research and Development Grigore Antipa Romania X X X X X XNational Institute of Marine Geology and Geoecology Romania XSC Marine Research SRL Romania X



All-Russia Research Institute of Hydrometeorological Information - World Data Centre (RIHMI-WDC) National Oceanographic Data Centre (NODC) Russian Federation X XP.P.Shirshov Institute of Oceanology, RAS Russian Federation X X

Russian State Hydrometeorological University, St-Petersburg Russian Federation X XSpace Research Institute of RAS Russian Federation XState Oceanographic Institute (SOI) Russian Federation X XNational Institute of Biology - NIBMarine Biology Station Slovenia X X X XBarcelona University(UB) Spain XHydrographic Institute of the Navy Spain X XIEO/Spanish Oceanographic Institute Spain X X X X X XIGME/Geological Survey of Spain. Marine Geology Service Spain XInstitute of Marine Sciences. Mediterranean Marine and Environmental Research Centre (CMIMA-ICM-CSIC). Department of Marine Geology Spain X X XMarine Technology Unit. Mediterranean Marine and Environmental Research Centre Spain X X XSwedish Meteorological and Hydrological Institute Sweden X X X X XInstitut National des Sciences et Technologies de la Mer – INSTM Tunisia X XAnkara University Turkey X X XDepartment of Navigation and Hydrography and Oceanography, Turkish Navy Turkey XDokuz Eylul University, Institute of Marine Science and Technology Turkey X X X

Institute of Marine Sciences, Middle East Technical University Turkey X X X XIstanbul University, Institute of Marine Science and Management Turkey X XKaradeniz Technical University, Faculty of Marine Sciences Turkey X X XPermanent Secretariat Commission on the Protection of the Black Sea Against Pollution Turkey XSinop University, Fisheries Faculty Turkey X X X X

Already 103 data centres connected and more underway!

Name CountryPhysical Oceanography Chemistry Geology Bathymetry Biology Fisheries

Danube Hydro-meteorological Observatory Ukraine X XInstitute of Biology of the Southern Seas, NAS of Ukraine Ukraine X X XInstitute of Geological Sciences - IGS Ukraine X

Marine branch of Ukrainian Hydrometeorological Institute Ukraine X XMarine Hydrophysical Institute Ukraine X X XOdessa National I.I.Mechnikov University Ukraine X X X X

Scientific Research Institute of Ecological Problems (USRIEP) Ukraine X X X

Southern Scientific Research Institute of Marine Fisheries and Oceanography Ukraine X XTaurida V.I. Vernadsky National University Ukraine X X XUkrainian scientific center of Ecology of Sea (UkrSCES) Ukraine X X X XBritish Geological Survey, Edinburgh United Kingdom X XBritish Oceanographic Data Centre United Kingdom X X X X XOceanWise Limited United Kingdom X X

Ingestion• EMODNet seeks ingestion to get on board data sets from

researchers and industry beyond existing data flows (crowd sourcing)

• Prevent to frustrate existing mechanisms and infrastructures • Make use of SeaDataNet core network with NODCs and

Marine data centres with presence in 35 countries riparian to European seas

• Set up a coordinated online submission system with national nodes operated by SeaDataNet centres

Ingestion• The SeaDataNet nodes will consider the incoming data and decide about:

• Contacting data providers about possibly becoming connected as a distributed data centre as part of CDI

• Routing the incoming data to the appropriate national data centre in the CDI network for intake, QA – QC, further documenting in dialogue with provider, and making arrangements for long term stewardship and wider availability through the CDI service

• QA – QC should not be underestimated and is required to make the new data fit for use and combining with other data!

• Budget needed for these uptake activities because data centres can not do this from their existing resources

DOI and Data Publishing• CDI is latest version of a data sets• DOI should be applied to give extra impulse to scientists to make their data

sets available• DOI at level of collections• NODCs follow IODE Cookbook on DOI• DOI and Data Publishing also subject of discussion at the ODIP Workshops

between Europe, USA and Australia to achieve a common approach

Common Data Index (CDI) Data Discovery & Access service

Documents