Statistical data editing - UNECE work session – OSLO 24-26 September 2012 Proposal of a revised approach for data validation within the European Statistical.

Statistical data editing - UNECE work session – OSLO 24-26 September 2012

Proposal of a revised approach for data validation within the European Statistical System

Michel HENRARDEuropean Commission – EUROSTAT


The current situation

Validation process : 2 separated steps

– Member States

– Eurostat

Double work ? Zero Work ? Ping-Pong ?



Harmonisation

– Validation procedures• Harmonised?• Documented ?• Formally agreed? • Coordinated ?

– Validation rules• Common syntax ?• Formalised ?



Harmonisation of software solutions

– Standard building blocks within Eurostat• Partial degree on integration• Many ad-hoc solutions

– High number of supported solutions• Does not help streamlining of production processes• High cost of development and maintenance• Common validation solutions within the ESS ?



Compliance monitoring

– Assessment of data quality

• Subjective interpretation• Not structured


Yielding efficiency gains

Vision Infrastructure Project on Validation (end 2010)

– Validation solutions that can be used both by the Member States & Eurostat

– Validation as soon as possible

– Definition of the roles in the validation chain


Project proposals

Better & standard documentation of the validation process

– Should be part of the general documentation of the complete production system


Project proposals

Agreed & standard description of checking rules

– Common language (syntax)

– Guidelines for the selection of the validation rules


Project proposals

Availability of common software solutions

– Common standard IT tools and associated services

– Automation of procedures within the ESS


Project proposals

Clear & agreed distribution of responsibilities

– Validation where it can be done in the most efficient way


How ?

Process Description

– Systematic standard description of• The data validation• The validation rules using a common syntax


How ?

Validation review with Member States

– Gradual introduction of objective measurement of data quality

– Clear responsibility for each validation rule based on efficiency considerations (the sooner the better)

– Calendar of introduction of the new mechanism


How ?

Integration of validation in compliance assessment procedures

– Based on the agreed set of validation rules

– Reports on validation problems in the statistical domains


How ?

Improve the coherence of validation solutions within Eurostat

– Taxonomy of validation rules– Standard description of the validation rules (common

language/syntax) – Good practices– Current validation building blocks adapted to the users’

requirements


How ?

Provision of standard solutions to be shared in the ESS

– Deployment of a standardized validation language

– Adapting the IT tools to be shared in Eurostat & to be offered to Member States


What the VIP on Validation does provide ?

Templates & Guidelines

– Documentation of the validation process

– Error messages and reporting

– Selection criteria of validation rules ensuring minimum quality standard

– Attribution of responsibility of validation tasks


What the VIP on Validation does provide ?

Inventory of validation rules used in Eurostat

Typology of validation rules

Human understandable language for validation rules

User requirements & functional specifications to adapt the IT tools to the needs identified


Q&A

Thank you for your attention

Statistical data editing - UNECE work session – OSLO 24-26 September 2012 Proposal of a revised approach for data validation within the European Statistical.

Documents

ess data editing

validation end

validation processshould

validation problems

validation review

integration of validation

validation rulescommon

statistical domainsstatisti