MICS Data Processing Workshop Multiple Indicator Cluster Surveys Data Processing Workshop Data Archiving.

MICS Data Processing Workshop

Multiple Indicator Cluster SurveysData Processing Workshop

Data Archiving

Look familiar?

Today’s storage medium is no longer file cabinets

and storage rooms...

What is archiving and why do it?

Data Archiving—What is it?

• The long-term storage of data (raw data) and its affiliated documentation.

• Storage of metadata - the descriptive information about the survey and the data; essentially, “data about other data”.

The MICS Archiving Goal!• Accessible datasets that are free of errors—no

spurious or out-of-range values etc.• Datasets that are well-documented – all variables

and response codes labeled etc.• Information available on the survey methodology

that explains to external data users how the survey was designed and conducted.

• The final survey reports and materials used to disseminate results.

Group Work

Survey archiving brings huge benefits to both data users and data producers.

In 3 groups discuss the following:Group 1: What are the benefits to data users?Group 2: What are the benefits to data producers?Group 3: What should be included in a MICS survey

archive?

Archiving Benefits for Data Users• Facilitates the usage of data—without proper

documentation on the survey methodology and analysis, data cannot be fully understood which may lead to lack or inappropriate use of data;

• Proper documentation decreases the time a user needs to spend understanding the data and requesting information from the agency;

• Increases efficiency for users—less time can be spent producing pre-determined tables, more time can be devoted to higher-level analysis.

Archiving Benefits for Data Producers

• Increases visibility and credibility for the agency;• Allows for easier management of the data and survey

related materials;• Allows for easy and efficient sharing of information

with external audience; • Prevents loss of all survey related documents by

individuals and facilitates institutional memory.

MICS Documents to Archive• Survey plan and budget• Questionnaires• Pre-test report• Interviewer & field manuals• Sample selection, listing information

– Excel files, other supporting documents– Explanations of any event(s) that impacted the sample selection,

survey dates, etc• Steering/technical committee ToRs and minutes• Any other documentation that will help people understand what

happened in the design and implementation of the survey

MICS Programmes to Archive

Software Programmes & Files:• CSPro dictionaries• CSPro data entry & menu systems• CSPro secondary editing & menu systems• CSPro GPS Entry program• CSPro export & SPSS recode• SPSS tabulation programmes• Any country-specific programmes

MICS Datasets to Archive• SPSS Files:

‒ Household (HH)‒ Household listing (HL)‒ Insecticide-Treated Nets (TN, if included in survey)‒ Women (WM)‒ Birth History (BH if included in survey)‒ FGM (FG, if included in survey)‒ Maternal Mortality (MM, if included in survey)‒ Children age 0-4 years (CH)‒ Men (MN, if included in survey)

• CSPro Files:‒ Final (merged, not individual clusters) HH data file‒ GPS data (if included in survey)

MICS Reports and Dissemination Materials to Archive

• Key Findings Report• Final Report• Presentations• Factsheets• Press releases• Videos• And everything else!

When to Archive?

• Archiving is often the last step in data collection activities

• As a result, documentation is often of uneven or poor quality

Start archiving when you start your MICS survey

Archives for internal and external audiences

If some survey information is considered sensitive or for internal use only then at the end of the survey process it is advisable to create two separate versions of the MICS

archive.

Restricted MICS Archive(complete archive but

only accessible internally)

Public MICS Archive(with some documents

and information deleted)

What does a survey archive look like?

MICS Final Archive Examples…http://www.childinfo.org/mics3_surveys.html

How to Archive?

Preparing for archiving

• Save as much as possible electronically– Electronic documentation has a longer life and is more

readily usable– Paper documentation requires permanent storage areas– Paper documentation disappears!

• Save documents– in original format– in HTML– as PDF files

Preparing for archiving

• Save datasets– in original format (e.g., CSPro raw+final files)– in analysis file format (e.g., SPSS.sav file)

• Always include descriptions with data—provide labels within each file for:‒Variable labels (e.g., “Result of Interview”)

‒Value labels (e.g., 1=Complete, 2=Not at home)Don’t forget to document country-specific questions!

Creating the archive?

Recommend to use an archiving tool:

International Household Survey Network’s (IHSN) Microdata Management Toolkit

http://www.ihsn.org/toolkit

International Household Survey Network (IHSN)

• Established September 2004 as a recommendation of the Marrakech Action Plan for Statistics (MAPS)

• Users: International organizations actively sponsoring household surveys + local NSOs

• Governing committee members from DFID, ILO, Paris 21/OECD, UNICEF, UNSD, WHO, World Bank

• Distributes Microdata Management Toolkit and National Data Archive (NADA)

IHSN Toolkit 4 Main Components• Archive creation: Metadata Editor (also known as the Nesstar

Publisher)– Document your survey data, documentation, and metadata in accordance with

international metadata standards

• Dissemination: CD-ROM Builder– Facilitates the publication of survey data, documentation, and metadata onto CD-

ROMs and the Internet using HTML format

• Nesstar Explorer– Using their free program, access metadata and data, and export to common

statistical formats

• IHSN Report Center– Provides diagnostics on your metadata, dataset relations, external resources, etc.,

and creates a PDF of your study documentation (a summary report of all your documentation)

Step 1: The Metadata Editor • All data and documents are first archived in this

software. • Template-driven specialized editor • Imports and exports data to the most common

statistical formats• Integrated interface, multi-lingual support• Compiles metadata and data in single file

http://www.surveynetwork.org/home/software/ddi-metadata-editor

Installation

• Unzip IHSN_Toolkit_v1.1_full.zip into C:\MICS5\IHSN

• Double-click on the file “IHSN Toolkit Installer.exe”

• Select the first option: Metadata Editor

Getting Started

• Create a new study (ctrl+N)• Save your project (ctrl+S)—this gives your project a name

within the tree• Select the IHSN templates (MICS5 generic template) • Begin by customizing the fields under the:

– Document description– Study description

• Start saving key survey documents and files under the folders in:– External resources

Importing Datasets

• Once the datasets are final make sure all variables and values are properly coded and that the data has been anonymised in SPSS first.

• Delete the current datasets in (generic) archive and import the final datasets.

• Make sure that all variables are properly documented with manual/interviewer instructions and questions from questionnaire.– (This will be automatic for all standard MICS variables if using the

MICS template but will not be done for any non-MICS questions and modules)

Tips and Good Practices

• One person – ideally the survey manager – should be the main custodian of the survey archive and responsible for archiving all documents throughout the survey implementation.

• The most time intensive part of archiving is documenting all the variables once the final data has been imported. It is good practice to keep a complete record of all the changes from the standard questionnaires (changes to questions, interviewer instructions, new questions etc.) so it will be easy to go through the archive and make changes from the generic and add the information on variables that will not be there.

How to generating a final survey archive for internal and public

use?

Step 2: The CD-ROM Builder• Publishes survey metadata, data, and documents on a CD-

ROM or website• Stores archive in an HTML-based language• Users can customize the layout (branding) and content of the

CD (single or multi-surveys)• Integrates with IHSN’s

Metadata Editor

Generating the Final Archive(s)

• Only once all information is complete in the metadata editor should the CD-ROM builder be used. – Cannot make changes to the archive once generated by the CD ROM

Builder. If problems are found at this stage, need to go back to metadata editor and make corrections there.

• If different versions of a survey archive (public and restricted) will be generated, two “projects” need to be created in the metadata editor tree.

• To make a “public” archive delete all the documents that are for internal use only and give this archive a new name. Only need to do this at the end of the survey archiving process before using the CD ROM Builder.