Using Macro and Micro Data Sets in Cross National Analyses in Developing Countries

Using Macro and Micro Data Sets in Cross National

Analyses in Developing Countries

Workshop 1: Unit 2

Roy Carr-Hill

Universities of East London, London and York

Objectives: Participants able to:

– Understand the rationale for linking macro and micro data– Be aware of the typical factors that are likely to vary across

countries or regions – Be aware of the limitations on the macro data sets that are

currently available– Understand the differences between the International Micro

Data Sets that are publicly available in terms of purpose, technical data quality, user-friendliness

– [Describe some key quality control processes in collection and processing of survey data]

– Be able to critically assess the compatibility of data collected for similarly named variables at the micro, meso and macro levels]

Organisation of Presentation

I. Rationale for linking Macro and Micro Data

II. Availability of Data for Developing Countries

III. Limitations of Available Macro Data

IV. Limitations of Available Micro Data

V. Monitoring Poverty/ Socio-Economic Status through Household Surveys

VI. Reliability of Self-Reports of ill-Health biased

VII. [Assessing the Quality of Survey Data]

VIII. Examples of Pitfalls in Interpretation

I. RationaleTwo main reasons• Generalise ability of findings obtained from single –

usually individual level – sample• Improve understanding of association observed –

usually at country level.

Hence structure• Data Availability• Data Quality and Limitations• Measurement and Compatibility Issues

• Errors of interpretation

II. Availability of Data Sets

• Demographic Household Surveys sponsored by USAID (www.measuredhs.com)

• Living Standards Measurement Surveys - World Bank (http://www.worldbank.org/lsms/),

• Multiple Indicator Cluster Surveys - UNICEF (http://www.unicef.org/statistics/index_24302.html).

Activity A: Variation across countries

• In each group draw up a short list of the major factors that you think affect literacy/ smoking behavio0ur; and then discuss which of these are most likely to vary between countries or between regions

• Would it be sensible to carry out a combined cross country analysis? Are the answers different in the two cases?

III. Macro Data Sets: How reliable?; can they be compared?

Debates about Poverty• Ever more sophistication in concepts (absolute/relative)

and analysis of poverty (e.g poverty mapping;• Whilst recognised as a problem there is still very little

attention to quality of basic data

Millennium Development Goals – How will we know where we are in 2015?– How far are we away now?

III.1 Population Denominator Data in Developing Countries

• Only a few countries have functioning registration systems

• Current population estimates are based on Coale-Brass-Demeny population models

• As Chris Murray showed, in several countries, the estimates are based on parameters from neighbouring countries

NONE OF DATABASES COVER THIS

III.2 Quality of Data in Developing Countries

• International recognition that quality of statistics has deteriorated (e.g. Can’t Count Progress, ODI Review) BUT

• Majority of information systems are donor funded with minimal national involvement

• Routine administrative data systems are ‘thin’• Still little attention to assuring the quality of the

basic data

III.3 Deterioration of Statistical Systems

• Lack of attention since independence to ‘boring’ issue of infra-structure of statistical systems; but

• Current trend towards decentralisation usually means that district estimates are central to resource allocation

III.4. Donor Funded Systems

“The global statistical system is fragmented and characterised by poor inter-agency co-operation. Whilst more information is now available compared with previous years, this is usually through the medium of donor funded household surveys, which may by-pass domestic information systems and serve the needs of donors rather than developing countries themselves.”

Can’t Count Progress

III.5. Routine Administrative Data in Databanks

• Population Censuses – very high coverage but usually insufficient information

• Collection of Data on use of Education and Health services, rarely includes socio-demographic data

• Collection of data on receipt of income support/welfare also biased on both data collector and supplier sides

III.6. Quality of Basic Data

• Entrenched systems• Little or no inspection or quality assurance• Weak capacity – numbers and qualifications• No local use of data – so no incentive to verify

(Musgrove – data has to be used within 5km to ensure reliability)

III.7 Measurement of Income or Socio-Economic Status

• Lack of agreement over whether to use absolute or relative poverty

• Conventional levels like US$1 or US$2 a day per person are used with little evidence

• Lack of relation with other measures of well-being e.g. mortality, education

• Usually based on household expenditure surveys, i.e. what is consumed in the market omitting barter, black markets and exchange

III.8 Asset Indices

Difficulty of asking expenditure has led to development of ‘asset indices’ but:

• no information on quality and quantity of goods and services including the reliability of the asset

• distinguishing between household ownership, household based assets and individual access

• Routine Administrative Data• problems in generalising across different communities

IV. Monitoring Social Conditions: Household Surveys

MOST MACRO SOCIAL DEVELOPMENT DATA BASED ON HOUSEHOLD SURVEYS

IV.1 Three Difficulties

• Household surveys do not include the poorest of the poor (see next slide);

• Consumption expenditure is a poor substitute for measuring standard of living;

• The proxies used to measure poverty are almost impossible to compare over time because of changes in reach of formal economy) so that even within country trends are very difficult to assess

IV.2 Household Surveys: Omissions

• Those not in households because they are homeless• Those who are in institutions• Mobile, Nomadic or pastoralist populations• Many of those in fragile or disjointed or multiple

occupancy households.

IV.3 Limits of Self-Reporting• Focus on household rather than community – or intra-

household – poverty• Known associations between income and relative

reporting e.g. – reported illness rate higher in households with piped

water supply, with inside toilets with central heating, with TV;

– reported illness increases, reported deaths decrease with mother’s educational level

• Consumption and income poverty may not be the most salient (e.g. refugee camps, communication in Palestine)

IV.4 Possible Solutions?

• Extending the Sample - need to be confident about sampling frame

• Attributing - poverty mapping assumes initial relationship valid and relies on outside experts

• Modelling - comparison of trend estimates from different surveys, supplement with local evidence about excluded groups

Activity: Suitability of data from surveys

Issues to consider are:• Definitions and time periods (obviously)• Context and purpose of data collection especially in

micro study• Coverage of macro data compared to the sample you

are looking at• If your own study uses secondary data sources, you

should explore the extent to which the technical reports on those data sources cover these issues of coverage and excluded populations.

V. Different definitions of quality

• ‘Fitness for use’• ISO norm 8402: ‘Totality of features and characteristics

of a product or service that bear on its ability to satisfy stated or implied needs’

• Process (throughput) versus product (output) quality versus customer satisfaction (outcome?)

V.2 ESS Dimensions of Quality

• Relevance: are the data what user wanted• Accuracy: is the figure reliable• Punctuality and timeliness: on time according to pre-

determined schedule• Accessibility and Clarity: understandable?• Comparability: across countries• Coherence: with other data • (for details see HO 1A, 1B)

• DQAF: developed for National Accounts & Balance of Payments data (IMF webpage)

• Pre-requisites of Quality: e.g. Legal and institutional environment: supportive environment of statistics resources: adequate for the needs of the statistical programmes

• Components: Assurances of Integrity; Methodological Soundness; Accuracy and Reliability; Serviceability; Accessibility;

• Comparisons/ overlaps with ESS

• (See Annexes for comparison with ESS)

V.3 DQAF of IMF

V.4 Concerns about Data Quality

Official and Administrative Data• Focus on recording activity rather than processes or

outcomes• Motivation to collect reliable data• Hierarchy of data collectors• Pressures to manipulate data

V.5 Quality of Surveys

• Were standard contracts issued? (HO 4)• Do they have in-house quality control (HO 5)• Were power calculations of sample size carried out –

see HO 6 on Swaziland• Were questions appropriate to problem• Did they follow appropriate procedures (see HO 7)

V.6 Data Collection: Instrumentation

• Instrumentation• Has a check sum facility or similar been included?• Has instrument been piloted in a variety of contexts?• Piloting procedures of data collection• (not just for question content of surveys)• Team organisation • Timing and Cost• Acceptability to interviewees

V.7 Data Collection: Field Work

• Training• Assess commitment of interviewers to collecting reliable

data• During Fieldwork• Procedures for logging work, ensuring correct (unique)

identification of materials• Audit Trail

V.8 Processing and Analysis

Processing• Training of those entering data• Double Entry of Data (at least a sample)• Independence from Administrative Intervention

Analysis• Appropriateness of techniques used• Plausibility of results• Is there any confirmatory data (HO 8)

AND ALWAYS : Is there an Audit Trail

VI Examples of Pitfalls in Interpretation

• Comparison of educational performance of pupils in Anglophone and Francophone educational systems in Vanuatu

• Comparing inequalities in income within a country with inequalities of health within a country and GNP per capita

• Comparing Literacy achievement across regions of Uganda

• Bias in reporting health across 110 countries in DHS Surveys