Beyond Experiments: General Equilibrium Simulation Methods for Impact Evaluation

Beyond Experiments: General Equilibrium Simulation Methods for

Impact Evaluation

Xinshen Diao International Food Policy Research Institute

J. Edward Taylor

University of California, Davis

Katmandu, Nepal, November 16-18, 2011

Outline

Why simulation approach? Why general equilibrium? What is a simulation approach? What is a local-economy general equilibrium

simulation model? Conclusions: simulation approach and its

implication to FtF

Why simulation approach? We need to look beyond experiments when…

Planning a large scale intervention (such as FtF) often requires an ex-ante assessment of its potential impact (no pilot rollout is possible)

Treatment and control groups are impracticable – Can’t randomize over large number of units – Investments (e.g., irrigation and rural road) to target certain

areas instead of individuals Economic impacts are indirect; higher-level effects

(e.g., poverty reduction and economic growth) We want to know “Why” & “if” there are impacts Multiple inputs and inter-related outcomes Impacts are heterogeneous, likely winners and losers

Why general equilibrium? - Externalities and linkages

What experimentalists call “externalities” or “control-group contamination” …GE modelers call “linkages”

Linkages transmit impacts from the treatment group to others in the same location

Can also create higher-level impacts outside the targeted locations

Transfer Policy

Transfer Policy

Treatment households adjust

External Linkages Transfer Policy

…affecting other households

External Linkages Transfer Policy

…which adjust

External Linkages Transfer Policy

…affecting still other households

What is a simulation approach? Thinking about a flight simulator

Flight simulator contains good model of mechanics and aerodynamics – If not, don’t fly with that pilot!

If we have a good model of how the local economy works, we can use it to – Simulate impacts of project, policy changes – Do an local-economy GE cost-benefit analysis – Estimate the distribution of impacts, winners and

losers, whom to compensate/provide adjustment assistance

– Experiment with project designs w/ specific goals Ex-post: We can use experimental results to

see whether the plane really flew

What is a local-economy simulation model? Recipe for simulation-based project evaluation

Understand the project or policy to be simulated – Elements of the project: E.g., cash transfer or input subsidy? Who’s the target

(i.e., treatment group)? Understand the actors and the economic system

– How is the treatment group connected with others in the zone of influence (ZOI) of the project?

– How do we model their behavior? – Sketch out a social accounting matrix (SAM) for each household group and/or

locality to reflect this Inventory existing data needs and availability to construct SAMs

– Baseline surveys fill data gaps (can modify pre-treatment surveys) Build the simulator: construct SAMs, use them to calibrate a general-

equilibrium (GE) model encompassing treatment and control groups Do simulations to evaluate high-level impacts of intervention Use the simulation results as inputs into CB or impact analysis, project

design Use experimental results for validation, recalibration of models

Examples of a local economy model: Malawi case

Challenges of this (like most) impact evaluation Three transfer mechanisms

– Input subsidy (IS) • Malawi Agricultural Inputs Subsidy Program (MAISP)

– Cash transfer (CT) • Malawi’s Social Cash Transfer Scheme, -SCTS

– Farm gate market price support (MPS) • Implemented historically

Can’t do an experiment for each of them

Challenges (continued)

Immediate indirect effects of transfers on the control group (linkages effect) – Experiments aren’t going to capture them

Heterogeneous treatment and control groups Sensitivity of outcomes to market structures

– E.g., will cash transfers create multiplier effects within households by loosening production constraints?

– Ex-post experimental evidence can help us parameterize this in the simulation model

Multiple goals of the analysis

To compare the effects of these three transfer mechanisms on incomes and welfare in rural areas – Including high-level effects, on non-beneficiary

households To assess differences in these effects across

household groups and market scenarios – The structure of the economy shapes outcomes

To understand why different transfer mechanisms produce different outcomes

Developing an economywide GE model in which…

A set of farm (and nonfarm) household models are defined

Each household model is representative of a group of households defined according to their eligibility for each transfer program

All these household models are embedded in an economywide GE model

Data Ideally, parameterize the model with data from a baseline (pre-

project) survey In this application, we had to rely on existing data…

– IHS2 (Second Integrated Household Survey) • 2004, immediately preceding the first round of the MAISP

– National agricultural production and consumption information available online from FAOSTAT

• 2003, the last completed cropping season before the IHS2 was conducted

Constructing a social accounting matrix (SAM) for each household group from the data

Nest the households within a “meta-SAM” for the ZOI (in this case, the entire rural economy)

Includes market accounts that link together the household groups

Simulations

Assumptions on market conditions matter 1. Perfect markets benchmark 2. With constrained input use 3. With unemployment 4. Combined 2 and 3

Under each type of market arrangements, simulating IS, MPS, CT separately at the given cost ($52 million)

Simulation results at the household level (perfect market benchmark) (1) (2) (3) (4) (5) (6)

Transfer Mechanism Ineligible, Non-farm households

Ineligible, Small farms

Ineligible, Large farms

Eligible for CT (ultra-poor labor-

constrained)

Eligible for MAISP (poor small-

holders)

Eligible for both CT &

MAISP Group's share of total households (%) 3 19 23 1 47 7

a) IS: Crop Inputs subsidies for eligible households

Group’s share of transfer (%) - - 93.0 7.0

Welfare (CV), % change 0.80 0.00 -0.30 0.01 5.47 4.50

Household-level efficiency - - 0.69 0.78

b) MPS: Market Price Support for Maize

Group’s share of transfer (%) 22.0 57.0 1.0 20.0 0.0

Welfare, % change -1.1 2.0 2.7 1.6 0.6 -1.9

Household-level efficiency 0.64 0.66 0.57 0.37 -

c) CT: Cash Transfer to eligible households

Group’s share of transfer (%) - 17.5 - 82.5

Welfare (CV), % change 0.0 0.0 0.0 50.8 0.0 69.7 Household-level efficiency - - - 1.00 - 1.00

Total production effects and efficiency measure under alternative market conditions

(a) (b) (c) (d)

Perfect markets benchmark

With constrained input use

With unemployment

With unemployment & constrained input

use Production effects (% change in total agricultural output)

Input Subsidy 4.0 2.3 13.4 5.0

MPS 1.0 -0.3 8.6 2.9

Cash transfer 0.0 0.8 0.0 2.0

Total transfer efficiency (welfare gain/transfer cost)

Input Subsidy 0.66 0.60 2.59 1.59

MPS 0.57 0.04 2.29 1.30

Cash transfer 1.00 1.17 1.00 1.47

Input subsidy becomes most efficient when households face unemployment and liquidity constraints

Which assumptions reflect reality?

Perfect markets benchmark seems to be overly optimistic

Effects of transfers depend on: – The elasticity of input supply – The responsiveness of wages to shifts in labor

demand – The extent to which there are cash constraints

on input demand All are likely to vary across project settings

Conclusions: Simulation approaches and FtF

Experiments have become the favored method of impact evaluation

Simulation methods will be increasingly important; and particularly important for FtF

Advantages of experiments

Verifiability – Create random treatment and control groups – Simply compare averages of outcomes of

interest to evaluate average effect of treatment on the treated

Disadvantages Experiments often are impracticable (cost, politics,

ethics) They almost never come out truly random (need for

econometrics) Control group contamination (due to GE linkages) Difficulty comparing impacts of several different

project designs Non-structural: Generally don’t tell us why

treatments have the impacts they do GE feedbacks change impacts once programs are

“ramped up”

Simulation approaches Designed to overcome these limitations of

experiments Ideal for

– Capturing higher-level impacts – Comparing alternative mechanism designs – Understanding the “Why?” – Evaluating differences in project impacts across

market environments Can be implemented before projects

The Simulation-experiment ideal Simulations: Use to evaluate likely impacts of

alternative project interventions ex-ante – Parameterize with data from baseline surveys

Carry out randomized experiment using most promising program designs

Use results of experiment ex-post to verify and (if needed) reparameterize simulation model

Use simulation model to provide a structural interpretation of experiment results (i.e., to answer the “Why?” question) – …and improve policy design

Thank You