Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”

Eurostat

6. SDMX:A non-technical overview of the SDMX architecture and IT tools

1

Raynald PalmieriEurostatUnit B5: “Central data and metadata services”

SDMX Basics course, 27-29 October 2015

Eurostat

The SDMXComponents

2

IT Infrastructure for exchange and

sharing

Content Oriented

Guidelines

SDMX Information

Model

Eurostat

Who?

What?

When?Who?

How?

Describing the data exchange

DSDWhat's next?

Eurostat

EUROSTAT SDMX TOOLS FOR DESIGNERS

4

Eurostat

Eurostat SDMX tools for designers

5

Tools offered Use

Data Structure Wizard Used to create, edit and test SDMX objectsMaintenance of SDMX registry

SDMX Registry A central repository for storing and sharing SDMX objects

Eurostat

Eurostat SDMX tools for designers

6

Data Structure Wizard

SDMX Registry

DSD

Create/Edit

Store/EditWeb service

Work roles and processSDMX Registry

Create DSD

Publish DSD

Format / structure SDMX messages

Consult and get DSD

Send SDMX messages

COLLECTING ORGANISATION

Consult and get DSD

Interpret SDMX

messages

7

Data Structure Wizard

• Creation / edition of DSDs• Validation of DSD• Local repository

• Artefacts are stored in a local repository

• Remote repository• Can be used for retrieving existing artefacts

8

DSW: Management of SDMX artefacts

9

DSW: Artefact example: the codelists

10

Eurostat

SDMX Registry

11

• Metadata registry which implements the SDMX registry specifications

• Tasked with providing structure, organisation, maintenance and query interfaces for the SDMX components required to support the data sharing vision

• Provides a web-based user interface as well as web services for interacting with the SDMX structural metadata objects in use within the collecting organisation and with statistical partners.

Eurostat

SDMX Registry

12

• Only concerned with providing information needed to access the data and reference metadata sets.

• An application which wants a particular data or metadata set would then query the registry for the URL, and then go and retrieve the data or metadata set directly from the provider's web server.

• Example of existing SDMX registries: • Euro SDMX Registry (2.1)• SDMX Global Registry (2.1)

Eurostat

New Euro SDMX Registry (2.1)

• Eurostat's implementation of the SDMX Registry specifications as published by the SDMX initiative sdmx.org.

• Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts.

• Allows browsing, searching, editing and subscribing to artefacts.

• Advanced access control mechanism for distributed maintenance of artefacts controlling also their visibility.

Eurostat

New Euro SDMX Registry (2.1)

Home page

Most recent items

Access to the content of the

Registry by type

Access to the content of the Registry text

search

Eurostat

Registry Content

Number of objects of type selected

Filtering list

New Euro SDMX Registry (2.1)

SDMX Global Registry(SGR)

• Launched in 2011 by the SDMX Sponsoring institutions

• Originally to support the world-wide technical implementation of Balance of Payments and National Account implementations all international implementations

• To make metadata material, DSDs and related artefacts (concept schemes, MSDs, code lists, etc.), publicly and centrally available

SDMX Global Registry(SGR)

• Contains other SDMX DSD and artefacts that are used and maintained by national and international organisations.

• Designed in such a way that it can be developed into a global portal for SDMX structural data and metadata, providing not only information about how statistical information is structured, but also where the related data can be accessed.

Who should use the SDMX Global Registry (SGR)?

• Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard.

• Any end-user of statistical data can access the data structures posted on the SGR.

How to use the SDMX Global Registry (SGR)?

• All information published in the SGR is publicly available and freely viewable and downloadable.

• No need to log in to view or to download information - SDMX Global Registry Content.

• Users can view the full content of the Global Registry , see how the various objects are interlinked and use the SDMX Global Registry Technical viewer.

• Help is available in the Global Registry Wiki

Registry Content

Navigate the content of the

registry

Search for an item that contains the

text entered

Technical view of SDMX objects

Registry Content

Exploring by agency

Click on the agency to see the items maintained by it

Registry Content

Available SDMX objects

Click on a type

Registry Content

2. Select a version

1. Choose a DSD in the list

3. Click on View

Registry Content

Dimensions, measure and

attributes

Clicking on a concept on the left frame, displays

the corresponding list of codes (if exists)

Download the DSD (XML file)

Registry Content

Enter a text to be found in the

Registry content

Registry Content

Example of result for "NA-MAIN"

Two versions found

Registry Content

Click to access to the technical view

Technical view

Click on the icon to download a SDMX

object

Technical view

Select one or more SDMX objects

Click to download the xml file (unique file containing all

the selected objects)

Technical view

Access to the technical view of

the Registry content

Technical view

Select an agency

Select an object type

Select an object version

View of the content (here the DSD

elements)

Technical view

Access to the web service

Technical view

Select the SDMX version, the Object type and the

identifiers of the SDMX object (ID, agency, version)

Click to view the xml file in your browser

Select which referenced objects (e.g. Code lists) you

want to include

Same selection as above, the URL generated is the SDMX query to be sent to the web service to get the

xml file corresponding to the selected object

View the xml file in your browser (DSD NA_MAIN)

Eurostat

SDMX TOOLS FOR DATA PROVIDERS

35

Eurostat

SDMX Tools for Data Providers

36

Tools offered Use Action

Web Forms (EU) Excel-like templatesTransmission of low volumes of data

No costs for EU organisationsManual work for senders (type, copy/paste)

SDMX Converter(sender’s PC)

Converts data files between SDMX formats and other file formats

Installation on sender’s PCManual work for senders (convert)

SDMX Converter(batch mode)

Installation on server

SDMX Reference Infrastructure

A set of tools that allows to connect your IT systems to the SDMX world

Installation on serverMapping of database to DSDs

Own SDMX Implementation

Local development in organisations

Eurostat

Data provider view:One goal – different possibilities

WebformsEDAMIS

Single Entry Point

SDMX-ML file

Database export

SDMX Converter SDMX-ML file

Database SDMX Reference Infrastructure

SDMX-ML file

SDMX Web Service

Same DSDSame Format

Excel sheets

SDMX Converter SDMX-ML file

Eurostat

Data provider view:One goal – different IT Architectures

Webforms Web-based Push mode

Database export

SDMX ConverterLocal conversion

Push mode

Database SDMX Reference Infrastructure

Pull mode

Excel sheets

SDMX ConverterLocal conversion

Push mode

Eurostat

SDMX Converterhttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Converter

• File based conversion• Open source and platform independent (Java)• Different ways of using it:

• Graphical user interface• Batch file (server or client side)• Web Service interface• Reusing source code in your own Java application

• Formats: SDMX-ML 2.0 & 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates

Eurostat

Select the Input file Select the output file

Select the input and output formats

Select the DSD on the local driveIdentify a DSD to download from the SDMX Registry

Identify a dataflow linked to the DSD to download from the SDMX Registry Select / manage headers

for CSV input formats

Select mapping / transoding tables

CSV parameters

GESMES representation for GESMES output formats

Load / save the current settings

XML parameters for SDMX output formats

Eurostat

Exercice: conversions

41

• From CSV to SDMX Compact• From XLS to SDMX Compact• From SDMX Compact to CSV

Eurostat

SDMX Reference Infrastructure SDMX-RI

42

• Generalized service infrastructure that can be re-used partially or completely by any organisation interested in starting SDMX projects for data exchange.

• Set of pick-and-choose building blocks allowing a statistical office to expose data to the external world based on access rights

• Developed in both Java and .NET with well defined API

Eurostat

SDMX Reference Infrastructure SDMX-RI

43

• Provides data and structural metadata based on mappings to an organizations data warehouse

• Conform to SDMX Web Service guidelines

• Open Source package under the EUPL licence

• Support Census Hub and similar Eurostat projects

Eurostat

SDMX Query (XML File)

SDMX Reference InfrastructureCensus-HUB architecture – Eurostat to NSI

SDMX DataMessage (XML File)

Eurostat Census Hub National Statistics Institute

SDMX-RI

Web service

Eurostat

Data Provider Data Collector

Non-SDMXlocal database

SDMX data set

SDMX Registry

DSD

SDMX-RI

Mapping Assistant

Web Svc

Test Client

SDMX Reference Infrastructurehttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastructure_SDMX-RI

Web Client

Eurostat

Mapping Assistant

46

• Facilitates the mapping between the structural metadata provided by an SDMX-ML Data Structure Definition (DSD) and those that reside in a database of a dissemination environment.

• Maintains a Mapping Store for keeping the mappings between the SDMX and the local data storage scheme

• In the SDMX Reference Infrastructure, provides mapping information to the Data Retriever

DSD

EurostatEurostat

Local database

EurostatEurostat

MappingLocal variables to

DSD concepts

DSDLocal table

Mappingof composite

codings

Eurostat

Test Client

50

• Test mappings of data flows configured through the Mapping Assistant

• Test the NSI Web Service is correctly set up

• Export data sets to file in SDMX format

EurostatEurostat

Connection to the

database

Build SDMX query

Generate SQL and SDMX-ML file

Eurostat

Web Client

52

• Result of a SDMX query

EurostatEurostat

Display the SDMX dataset in an Excel-

like form

Eurostat

Support for data providers

• SDMX Reference Infrastructure Installed in ~32 countries for Census Hub Can be extended to other domains

• 1st contact point - Local Transmission Coordinators In National Statistical Institutes Experience with EDAMIS and SDMX

• 2nd contact point - Eurostat support teams

Eurostat

Tools Relationships – Example

55

DSW

SDMXConverter

SDMX Registry

SDMX-RI

Mapping Assistant

DSD

National DB

CSV File

SDMX-ML Dataset

Mapping

Produce

Produce use

Store

Produce

use

DSDDSD

Eurostat

SDMX Tools – Summary http://www.sdmxtools.org

• Eurostat tools Eurostat SDMX Info Space• SDMX Converter• SDMX Reference Infrastructure & Mapping Assistant• SDMX Registry & Data Structure Wizard

• Other tools available in the community• Fusion Weaver (validation, transformation, creation)• Fusion Transformer (command line conversion)• Fusion Registry• ECB Checker/Loader (conversion and validation)• FAO OpenSDMX project

Eurostat

Questions & Answers

• HELP – We do not have a database!

• No worries.• Excel templates are a possible use• Conversion is supported

through the SDMX converter.

• However: use the opportunity to start!

Eurostat

Questions & Answers

• I want to have a database. What should I do?

• Simple DB creation possible from Excel sheets• Sharing experience from other domains in your NSI

(e.g. Census hub)

Eurostat

Questions & Answers

• Which versions / formats should we use?

• SDMX-EDI (Gesmes/TS) supported, deprecated• SDMX-ML 2.0 (compact) most used• SDMX-ML 2.1 (structure specific)

Eurostat

Questions & Answers

• Where are the DSDs managed?

• DSDs will be managed in the SDMX registry

Eurostat

Questions & Answers

• Other questions?