Developing High Quality Baseline

7/30/2019 Developing High Quality Baseline

http://slidepdf.com/reader/full/developing-high-quality-baseline 1/36

Developing a High

Quality Baseline

Salimah Samji & Mona Sur World Bank, New Delhi

June 21, 2006



Overview

What is a baseline?

Why you should care

The phases of conducting a baseline

The errors to avoid in each phase

How to manage the common errors

Elements of a baseline survey (TOR)



What is a baseline?

Fixing the time at the base – a benchmarkfrom which you measure progress

Snapshot of indicators at a time

Instrument used to:

Test hypotheses of project (assess results)

Planning (refine targeting, indicators tomonitor)



Why you should care …

To identify whether there were any benefits for the investments made Were objectives met?

What factors explain the result?

How can the program be improved?

Compare alternative models to get the biggest

bang for your buck To inform next generation projects

Evidence-based policy making – demonstrationeffect for government



The phases of conducting a baseline

Design

Implementation – actual survey

Data entry and analysis Report writing



Phase 1: Design Phase



P1: Check-list

Clear objectives (what is the problem?)

Clear idea of how you will achieve the

objectives (causal chain or hypotheses)

Clear and measurable indicators

Clear (precise and unambiguous)

Relevant (to objectives)

Monitorable



Example of a causal chain

Impact

Outcomes

Outputs

Activities

Inputs Facilitators, Revolving fund (credit)

Forming, federating and organizing SHGs

Number of SHGs, decreased input prices

Increased use of credit for income generation, Loan

repayment rates

Higher income levels

Example: APDPIP



P1: Check-list (cont‘d)

Design survey instrument – keep it simple andrelated to the objectives and hypotheses youwant to test

Link surveys to GIS – use consistent units Select controls/counterfactuals to attribute

change (causality)

Timing of baseline

Before project (what if project never materializes)? 2 years into the project (intervention has begun)?

Other factors (seasonality)




Sampling strategy

Random allows inferences

about a population Random

Stratified random (include groups

which could be excluded)

Non-random – use a group

smaller than the population Introduces selection bias. Note:

every stratification introduces a

level of bias.

Population

Size

Sample

Size

10 10

50 44

100 80

500 217

1,000 278

3,000 341

50,000 381

1,00,000 385




Over sample for attrition

Design the database system for data entry

Translate the questionnaire Back translate to verify

Provide adequate training on

The objectives and importance of the study How the sections are linked and what the questions

mean

Attend the training if possible (stay involved)




Test the questionnaire

Ask yourself, can you answer these

questions? Are they relevant to the outcomes?

Will they be understood?

Field test: To test both how the surveyor

administers the instrument AND how therespondent understands the question.



P1: Why it is important to field test (i)

When asking a question about the level of awareness, the surveyor used a word that couldmean awareness or knowledge – therespondent understood it to mean education. :

The question was: ―Ram/ Gita knows about everything that happens(vikas) in the village. For instance, they know [the name of the

sarpanch, when and where the Gram Panchayat meets, nature and

type of development work in village, etc.]‖



P1: Why it is important to field test (ii)

CASE STUDY: Domestic Violence Study in India

―In studying domestic violence, a question in the survey instrument asked if

female respondents had ever beenbeaten

by their husbands in thecourse of their marriage. Only 22 per cent of the women respondedpositively to this question – a domestic violence rate much lower thanstudies in Britain and the US had shown. In probing the issue with in-depth interviews we discovered that the women had interpreted the word‗beating’ to mean extremely severe beating – when they had lostconsciousness or were bleeding profusely and needed to be taken to the

hospital. Hair pulling, ear twisting, etc, which were thought to be moreeveryday occurrences, did not qualify as beating. Reponses to a broader version of the abuse question, comparable to the questions asked in theUS and UK surveys, elicited a 70 per cent positive response.‖

Source: Vijayendra Rao (1998) – ―Wife-Abuse, Its Causes and Its Impact

on Intra-Household Resource Allocation in Rural Karnataka‖



Phase 2: Implementation



What would you do if you …

Were a city person who didn‘t speak the local

language very well

Had to travel to several villages and spendhours asking people questions that have no

relevance to you.

Were paid a small sum per questionnaire

Not monitored by supervisors

This is not your full time job



The answer is simple

Sit at home or in a bar and fillout the questionnaires!!



P2: Providing incentives and motivation

Sub-contracting surveyors from the statewho speak the language

Include women surveyors

Include a supervisor who conducts datascrutiny

If possible pay reasonable wages

Randomly verify questionnaires to reducethe likelihood of false responses (informthem beforehand - during the training)



Phase 3: Data Entry and Analysis



P3: Check-list

Data Entry

Make the data entry system as fool proof as

possible - has unique identifiers to link both

household, village and GIS data

Ensure database allows for merging of data

Do not change/erase data on questionnaires

Raw data should always be input as is, changes

can then be made in the database software

(programatically) with documentation




Often data entry is contracted out. Name variables corresponding to the question and section in the

questionnaire – include a dictionary

Code descriptive answers (to facilitate analysis)

All fields should be filled (NA or NR)

Units should be uniform by district

Totals calculated by formula not from summary column

Consistency checks – check for missing entries, wrong

entries, sample statistics, patterns (queries should beinbuilt)

Validity checks – similar questions in different places onthe questionnaire (RCH example)




Data analysis

Common mistakes in interpreting data

No analysis! No correlations, crosstabs, statistical significance levels or

regressions

Over generalizing the results

Mis-reporting statistics Using % when the numbers are small

Attributing causality when it is not demonstrated



Phase 4: Report Writing



P4: What the report should be …

Simple, Clear and Relevant

State limitations (attribution, causality)

Major findings should be upfront

Focus on quality rather than quantity Technical details in an appendix

Should always include the questionnaire in the appendix

ask for electronic copy of data Request copies of filled out surveys

Essential if you change consultants at midtermor want to conduct internal analysis to comparemodes of delivery (data lost example).




Phase 1: Design Clear objectives and hypotheses – know what you

want to test

Identify a person in your unit who will manage this

process Write a good TOR, remember the baseline determines

the quality of your panel

You can add questions as project evolves but cannotchange questionnaire – threat to internal validity

Identify consultants Procurement – focus on quality not the cheapest bid ―if you

throw peanuts you‘ll attract monkeys‖

Ideally you should have a black-list of organizations




Phase 2: Implementation

Organize an impact evaluation workshop if

necessary

Randomly verify questionnaires to reduce the

likelihood of false responses (no filling it in a

bar)

Pay reasonable wages to surveyors (if possible)

Show the client and firm that you care




Phase 3: Data entry and analysis

Double-data entry (2 separate organizations and

verify. Payment based on quality of data entry)

Select 15 questionnaires at random and check dataentry – person in your unit managing

Check data quality (consistency and validity checks)

Hold an IE workshop to build data analysis capacity (if

necessary)




Phase 4: Report writing

Agree on an outline beforehand

Dedicate a chapter on indicators you aretracking

Focus on quality not quantity

Think ―Big Picture‖



Elements of a Baseline Survey

Terms of References 1. Background: Project objectives and components

2. Survey design: Consult a sampling expert!!!

3. Survey instruments

4. Guidance on survey implementation

5. Data processing and analysis

6. Staffing

7. Duration and time schedule

8. Submission of reports and datasets

9. Support to the firm

10. Budget & Payment Schedule

11. Annexes: Draft questionnaires, Results Framework



Baseline Survey Design:

Typical Tasks for Consultants Recommend the methodology for sampling

Calculate the optimal sample size

Develop the sample frame and select thesample

The final sample and details of the statisticalmethodology used to select the sample need to

be cleared by the project Construct the sample weights and provide

documentation on the methodology used toconstruct the weights



Survey Instruments:

Questionnaires Design or refinement and adaptation of the data

collection instruments

Specify levels of data collection

Length of questionnaires Prepare all support documentation including

coding guides, interviewer and supervisor manuals and the data entry manual

Translation and back-translation Skip patterns, coding open ended questions



Guidance on Survey

Implementation Implementation plan

Selection and training of field workers: specifyminimum duration of training

Pilot testing should be explicitly specified in ToR

Responsibility for all field operations, includinglogistical arrangements for data collection and

obtaining household consent lies withConsultants.

Ask for field-work progress reports (bi-weekly/monthly)



Staffing

Sampling expert/statistician

Technical specialists as relevant

Economist Sociologist.

Core survey staff: the survey manager, the

field manager, the data manager Enumerators, supervisors and data entry

staff



Baseline Report & Data

Explicitly request final electronic

datasets—with complete documentation.

Agree on outline of baseline report up-front.



Managing a Baseline Survey

Consult the experts—survey specialist and

sampling specialist and develop the ToR in

consultation. Selection committee should include a survey

expert and social scientists in addition to

technical experts.

You can never over-supervise!!! Hire third-partysupervision consultant if needed.

Question the data and the findings.



Lets recap what you have learned

The devil lies in the detail

Be watchful

No pain, No gain

Developing High Quality Baseline

Documents