1 Meeting on the Management of Statistical Information Systems (MSIS 2010) SDMX architecture for data sharing and interoperability Francesco Rizzo, ISTAT,

1

Meeting on the Management of Statistical Information Systems (MSIS 2010)

SDMX architecture for data sharing and interoperability

Francesco Rizzo, ISTAT, ItalyAdam Wronski, Eurostat

Daejeon, Republic of Korea, 26-29 April 2010

2

SDMX service infrastructure: rational and keywords

The idea is based on the analysis of the current level of

SDMX implementations within Member States and their

needs expressed during several TF, WG and other

meetings on the possibility to re-use software already

developed with the main aims of reducing costs and

increasing productivity

3

SDMX can help by ….

improving quality and efficiencies in the exchange and

dissemination of data and metadata;

reducing national reporting burden;

reducing costs through the re-use of the software;

facilitating and standardizing the use of new

technologies as XML and Web services.

4

Data Repository (Warehousing) Architecture

NSI

EurostatPull Requestor

eDAMIS

Data reception

SDMX Registry

Loadingpreparation

Verification /Conversion

o SDMX

Receiveddata in

SDMX-MLLoader

register

DataWarehouse

Database

query

Dissemination

XSL forSDMX-ML

PULL

PUSH

5

Data Hub Architecture

Dissemination

XSL forSDMX-ML

Data Hub

QuerySDMX

messages

cache

WebService

WebService

WebService

GUI

Data Providing Organizations Data collector Organization

6

Eurostat strategies to foster SDMX implementations within NSIs

A project aiming to design an SDMX service infrastructure

for NSIs and developing related building blocks;

Finance and support, through SDMX ESSnet , a group of Member States that have joined their resources in order to develop SDMX re-usable software.

Capacity building actions

7

SDMX NSI reference service infrastructure:main objectives

support Census Hub and other Eurostat project

facilitate SDMX implementation within NSIs with a particular attention to PC-AXIS community

stimulate a “SDMX community of developer”

8

SDMX NSI reference service infrastructure:deliverables

SDMX NSI reference service architecture document It represents the syntheses of several experiences worldwide and may be considered not as a strict specification but rather a guide or “best practice” document; It provides a description/specification of a generalized infrastructure that could be re-used partially or entirely by NSIs interested in SDMX projects;

A set of software building blocks;

Mapping Assistant tool;

Capacity building actions (Training and Technical Workshops).

9

SDMX NSI reference service infrastructure

Hub

PullRequestor

Data Collector SDMX NSI InfrastructureNSI

Disseminationenvironment

MappingStore

MappingAssistant

NSI Web Service

Web ServiceProvider

DataRetriever

SDMX DataGenerator

SDMXQuery Parser

DDB

Pull

DSDs

10

The mapping process

Data within Data Providers’ dissemination databases are described using “local” structural metadata (concepts, code lists, formats);

“local” structural metadata and SDMX structural metadata must be mapped:

concepts mapping

codes mapping

11

Concept and code mapping: an exampleLOCAL CONCEPTS

LOCAL CODES

SDMX CODES

SDMX CONCEPTS

Freq 12 M FREQ

IT REF_AREA

Type m W ADJUSTMENT

DomainSetCategory

eip63

PROD STS_INDICATOR

Ateco DADB

N100DAN100DB

STS_ACTIVITY

1 STS_INSTITUTION

Um pe 2000 STS_BASE_YEAR

PURE_NUMB UMIS

0 UMIS_MULT

Year, Month 2005, 3 2005-03 TIME_PERIOD

P1M TIME_FORMAT

Value OBS_VALUE

A OBS_STATUS

12

The Mapping Assistant tool: workflow

Map localDatabase schema

(Dataset)

Map localConceptsTo DSD

(Mapping Set)

Map localCodes

To DSD(Transcoding)

13

The Mapping Assistant tool: Dataset

14

The Mapping Assistant tool: Mapping Set

15

The Mapping Assistant tool: Transcoding

16

Software maintenance and governance

The software is offered as open source package under the EUPL licence

Up to now Eurostat is managing both the evolutive and adaptive maintenance.

How the versioning will be handled when NSIs will re-use and improve the software?

Is the model used by many “open source” community applicable to the “statistical community”?

know-how could come from NSIs participating in two ESSnet projects