1 DATA QUALITY The general method Data model Non-conform data Corrected data / improved IS Corrected programs Exceptions management measure correct prevent
Dec 24, 2015
1
DATA QUALITYThe general method
Data model
Non-conform data
Corrected data / improved IS
Corrected programs Exceptions management
measure
correct
prevent
2
MEASURE DATA QUALITY
DB
Dataacquisition
schema
?
??
?
Treatment
?
Extraction system
?
?
??
The data model is the central point for all actions
objectives questions what to measure
The data contained in databases are the result of a processing
Does the processes (collection, calculation, extraction) respect the structures, relations and data rules?
Data compliance with the data model
The data must allow the users to process tasks
Does the application meet the users requirements ?
Compliance of data model with users requirements
3
MEASURE DATA QUALITY
data
programs
data qualitymodel A
0-1
0-N
avenants-s inis tres
0-N
0-1
contrats -avenants
avenants
s inis tres
contrats
DB
data
programs
data qualitymodel A
0-1
0-N
avenants-s inis tres
0-N
0-1
contrats -avenants
avenants
s inis tres
contrats
DB
application B
information system quality
the organisation model(A+B+ functional links)
0-1
0-N
avenants-s inis tres
0-N
0-1
contrats-avenants
avenants
s inis tres
contrats
0-N
0-N
contrats-s inis tres
0-N
0-1
contrats-avenants
avenants sinis tres
contrats
organistion information system quality
application A
information system quality
real world
7
DATA QUALITYThe general method
Data model
Non-conform data
Corrected data / improved IS
Corrected programs Exceptions management
measure
correct
prevent
8
TO CORRECT
For the data
Concept inadequacy
Fields segmentation and normalization
Fields value cleaning orphan data detection
Occurrences deduplication
For the Information system
Data model and application improvements
10
TO PREVENT
The deployment of the data quality process must allow :
To clean up the bottom of the river punctually To dam up the arrival of new information flows of doubtful
quality
11
DATA QUALITYThe general method
Data model
Non-conform data
Corrected data / improved IS
Corrected programs Exceptions management
measure
correct
prevent
12
TO PREVENT
Objective :to (re)organize the data flows in order to guarantee a given quality level , so to minimize the corrective process.
Principle : data are products coming from a production line. For this reason, one should apply the quality control principles applied in the industry.
measure at different spots validation referenced with external world measures …
Involved the organization (management, administrative process) as well as technology
People and organisation resistance are important to consider
13
TO PREVENT
Technical issue Program correctionCorrection des programmes Data dictionary consolidation (complete méta-data) DB re-engineering
Organizational issue Identification of the processes and data flows Identification of the critical points and the
responsabilities Users training Organizational restructuring : flow
14
SYNTHESIS
The added value of the proposed approach
Dataprofiling
Reverse-engineering
Rulesdefinition
Datamerge
Programscorrection
Modelevolution
Datadictionary
Exceptionsmanagement
Conceptsprecision
The data quality steps according to Gartnerdata
profilingstandar-disation
deduplication
cleaning follow upenrichment
measure correct prevent
LogicalData
extraction
« Orphan » data
detection
15
Synthesis
What needs to be done
1
measure correct prevent
1 How to guarantee the conformity of the data after an IS merge ?
2 How to manage « old IS data » with respect with the new data management rules ?
3 How to manage the quality of the data flow entering or leaving the IS ?
4 How to manage the data rules with respect to the applications ?
Dataprofiling
Reverse-engineering
To specifyand complete
the rules
To manage the
data dictionary
To specifyand completethe concepts
To correct the data
2 Dataprofiling
Reverse-engineering
To specifyand complete
the rules
To specifyand completethe concepts
To correct the data
To manage the
data dictionary
3To manage
the exceptions
To correct the programs
Reverse-engineering
To specifyand complete
the rules
To specifyand completethe concepts
4 Reverse-engineering
To specify and complete
the rules
To specifyand completethe concepts
To manage the
exceptions
To correct the programs