Data collection and processing, data quality Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10 1 Heinrich Brüngger 10.

Data collection and Data collection and processing, data qualityprocessing, data quality

Management seminar on global assessment

Yalta, 23 - 25 September 2009Session 10

1

Heinrich Brüngger

10

Part I: Data collection and processing

Part II: Confidentiality (with emphasis on data collection and processing)

Part III: Data quality (quality management)

2

Nature of decisions on Nature of decisions on methods of data collection methods of data collection and processingand processingUnder professional independence (FP

2), therefore within the statistical system without outside interference once the decision about the “what” to measure in broad terms have been made

Decisions about what official statistics should measure may involve the political level, and have to involve discussions with a broad range of users

3

Professional Professional independenceindependence

The “how” part of statistical activities has to focus exclusively at obtaining the most reliable representation of the phenomenon of the real world with the given resources

Scientific methods, international standards, and empirically established good practices are the best guide for professional independence

The statistical law may require the adoption by government in the case of statistical surveys because of explicit legitimacy in the interaction with respondents and the burden caused to respondents, but governments should not interfere in methodological issues

4

Main data sources (FP 5)Main data sources (FP 5)Official statistics has to have the

right to use three types of sources:◦Statistical surveys: data collection

from individuals, households, corporate and unicorporated businesses, and public entities outside the own government structure, for the exclusive purpose of statistics (most of which official statistics)

◦Administrative records: secondary use of data collected primarily for administrative purposes about individuals or private businesses by other parts of the same government structure

5

Main data sources (ctd.)Main data sources (ctd.)

◦Secondary use of environment and territorial observation and monitoring data collected by specialised government agencies

For statistical surveys, the statistical law applies to all phases of the statistical activity; for the other two sources, primary data collection has to be based on other laws, and the statistical legislation becomes only applicable once data are handed over to a producer of official statistics

6

Statistical programmesStatistical programmesIf the law requires input-oriented

programmes, a clear distinction has to be made between statistical surveys, where the statistical law and the statistical programmes are the legal basis for data collection, and the second and third data source, where the primary data collection is for another purpose and has therefore to have a different legal basis

7

Minor data sources for Minor data sources for national official statisticsnational official statisticsOfficial statistics from a partner

country or from an international organisation (especially for international flows)

Registers in the hand of specialised private companies (as one of the sources used to keep statistical registers updated)

8

Statistical surveysStatistical surveys

Exhaustive vs. sample surveys (mixed forms possible)

Mail, face to face interview, telephone interviews, internet (mixed forms possible)

Compulsory participation of the respondent vs. right to decline participation (mixed forms possible)

Recall and follow-up policy for non-response, depending on type of survey

9

Response burdenResponse burdenResponse burden has to be

assessed in advance and taken into account:◦all new surveys have to be tested first

(pilots)◦all respondents have to be informed

about the purpose and legal basis of the survey, and especially about the confidentiality measures

◦forms and questions have to be understandable, non-intrusive, and make answers possible for him/her from memory or from existing material (businesses) 10

Response burden (ctd.)Response burden (ctd.)◦first reminders have to be

proportionate ◦response rates have to be closely

monitored

The NSI has also to act as custodian for the respondents; users will tend to minimise the issue of response burden

11

Surveys (ctd.)Surveys (ctd.)Information given by a respondent in

statistical surveys have no legal implications for him; they can (and should) therefore be checked and when necessary altered (edited) by the NSI

In many cases, the characteristics of interest to users cannot be taken over directly into questionnaires, but have to be compiled as derived characteristics using several questions

12

Non-response (valid for all Non-response (valid for all data sources)data sources)The NSI has to develop a policy for

treating general and item non-response, which has to be subsequently fine-tuned and documented for each survey

Imputations for missing (or unusable) information is legitimate, but a careful balance has to be found between “not enough” and “too much” (risk of introducing biases)

13

Administrative sourcesAdministrative sourcesThe statistical office must have the

legal and de facto right to receive, for its tasks, regularly and on an ad hoc basis, microdata sets from other ministries and public entities taken from their administrative sources

This does not mean that direct identifiers have to be included in all cases, but the possibility should not be entirely excluded

14

Administrative sources Administrative sources (ctd.)(ctd.)Statistical producers have the right to

alter (edit an impute) the administrative data received from other ministries to improve compatibility with statistical definitions and classifications

Data received in this way should never be given back to the data owner or transferred by the statistical producer to a third party for administrative purposes

15

Relationship between data Relationship between data sourcessourcesBefore considering introducing a new

survey, or expanding an existing one, for new information needs, other possible data sources (especially administrative) have to be examined for their potential

The possibility of combining questions about different areas in the same survey is limited: the survey system has to be structured carefully

16

Relationship between data Relationship between data sources (ctd.)sources (ctd.)Administrative sources are in constant

change, and some risk of disappearing completely

The final decision-making about administrative sources is outside the statistical system. Therefore, the NSI has the responsibility to constantly monitor (if possible anticipate) and react to planned changes that may affect the use of administrative sources for official statistics positively or negatively

17

Relationship between data Relationship between data sources (ctd.)sources (ctd.)

NSIs have to avoid considering data sources, and especially surveys, as stove-pipe type of parallel operations, but rather as a system of interrelated operations with n:m relationship between data sources and results

Very often, the best estimates are based on a judicious combination of sources, combining the individual strengths and reducing the individual weaknesses

Statistical registers of good quality are a cornerstone for a good survey system, and for matching different sources

18

Relationship between data Relationship between data sources (ctd.)sources (ctd.)When results on the same issue are

published from different sources separately (i.e. without prior integration), an accounting relationship between the results should be established, with the components quantified if possible

Otherwise, only the most complete aggregate should be released as results of official statistics in absolute terms

19

Confidentiality (FP 6)Confidentiality (FP 6)

Statistical confidentiality aims at protecting the privacy of individual units about which data are collected and processed. It has two components:◦Producers of official statistics do not disclose, either directly or indirectly, characteristics about protected units to any third party in such a way that any user might derive additional information about a protected unit 20

FP 6 (ctd.)FP 6 (ctd.)Producers of official statistics use data about

protected individual units only for statistical purposes (official statistics in the first place)

The statistical system, and in the first instance the NSI, have to make sure in an effective a credible way that the two elements of the confidentiality principle are strictly followed in all parts of the production and dissemination processes, and that confidential data are stored in a safe way

21

Confidentiality in data Confidentiality in data collectioncollection

◦Legal situation in countries may differ with respect to: Protected units: physical persons, private

households, private businesses (whether legal persons or unincorporated enterprises) should be fully protected; government units as institutions cannot invoke statistical confidentiality; public enterprises in a competitive market are a borderline case

All characteristics treated alike, or some considered to be free (e.g. for legal persons), or particularly sensitive (higher degree of protection, e.g. about health, crime, ethnicity)

22

Confidentiality in data Confidentiality in data collection (ctd.)collection (ctd.)

◦Data collection in statistical surveys: replies of respondents should go back as directly as possible to the statistical producer. No other government agency should be involved as intermediary (regional level!)

◦Data processing: except in the context of statistical registers, identifiers of units should be separated from the context variables by the statistical producer at an as early stage as possible, and either stored separately or destroyed

23

Confidentiality in data Confidentiality in data collection (ctd.)collection (ctd.)

◦Data security: final (edited) microdata sets have to be stored safely, and access limited to those within the NSO who use the data regularly. No direct access from outside the NSO should be possible. For files with identifiers (other than statistical registers), each access has to be documented.

◦Data security, update and access regulations for statistical registers have to be worked out carefully in a separate book of rules.

24

Quality management: Quality management: transparency as prerequisitetransparency as prerequisiteAll decision on methods

(collection/processing/dissemination) have to be documented and made accessible for the public (FP 3)

Quality management is addressed by several principles of the EU Code of Practice

Quality management refers to products, processes and staff

25

Quality management: Quality management: productsproductsProducts to be regularly assessed

with pre-defined quality criteria (and rejected if minimal quality level for fitness of use not met):◦Final micro-data sets from a data

collection instance◦Results of official statistics prior to

releaseProducts to be assessed occasionally

◦Questionnaires for surveys◦Dissemination products

26

Products to be releasedProducts to be releasedAn important element of

professional independence is the right of the NSI not to release official results if pre-defined minimal quality criteria are not met; this has to be a decision by the chief statistician

This does not preclude the use and release of such results as one input among others into a broader (e.g. geographical) aggregate

27

Quality management: Quality management: processesprocessesProcesses should be defined in

terms of who decides on what and when within the statistical system, and who has to be consulted and informed; they have to be documented for mainly internal use

Processes should be defined for each phase in a generic statistical business model (from design to archiving)

28

Processes (ctd.)Processes (ctd.)It is important to define the system of

processes and responsibilities in such a way that the tendency for creating “black boxes” by responsible organisational units is offset. Stove-pipes (same organisational unit responsible for all phases of the production process for a given activity such as survey) are problematic also from this point of view

29

Processes (ctd.)Processes (ctd.)The process definitions will be

different between on-going activities (with room for small improvements), and major redesigns/innovations, and between production activities ending in the release of results of official statistics and infrastructure activities such as dissemination or training

30

Cross-cutting evaluationsCross-cutting evaluationsAn overall evaluation of the portfolio

of activities and the extent to which the needs of users are met is necessary when the preparation for the next multi-annual programme starts

Extensive user consultations are necessary, and external expertise (such as in a GA) may help in identifying and prioritising gaps to be addressed in the next programme

31

Tendency in organisational Tendency in organisational structures of NSIstructures of NSIFirst-level division by broad subject

areas used to be predominantRecent shift to functional break-

down at first level (data collection; processing; dissemination) for various reasons, among others improved quality management

Senior management has to provide incentives for quality management leading to measurable improvements

32

Staff managementStaff managementQuality management has to become

an almost automatic part of the activity for each staff member

Regular training on management issues should include a substantial part on quality management

Regular assessment of the individual performance of each staff member can be very instrumental, if not implemented as a bureaucratic routine, but as open dialogue towards improvements

33

Thank you for your attention

34