Introduction to evaluation : The example of development aid.

Introduction to evaluation :The example of development aid

Plan

1.Definition : what is evaluation ?

2. Definition : main features of practical evaluation

3. Evaluation in practice : the scheme of a project evaluation

First discussion : constructivist vs positivist approach

Second discussion : Ambiguity of the impact criteria ?

Third discussion : micro vs macro level of evaluation

Next lesson on evaluation of aid effectiveness at the macro level

What is evaluation ?

• First of all it is an attempt to adress a need

The need is – production of legitimate and justified judgments on what have been done which serve as a basis for particular objective (decision making, accountability, learning)

Definitions

• A sytematic determination of merit, worth, and significance of something or someone using criteria against a set of standards (Wikipedia)

Definitions

• Evaluation is an applied inquiry process for collecting evidence that culminates in conclusions about the state of affairs, value, merit, worth, significance or quality of a porogram, product, person, policy, proposal or plan.

• Conclusions made in evaluation emcompass both an empirical aspects and a normative aspect (judgment about the value of something). It is the value feature that distinguishes evaluation from other type of inquiry, such as applied research, clinical epidemiology, investigate journalism, or public polling (Deborah M. Fournier)

Main features of practical evaluationsWhy evaluate ?

Two main objectives (plus transparency)

To prove. Accountability. To determine value of policies to inform important decisions (election, ressource allocation, changing, stopping or scaling up programs or policies)

To improve. Learning. To increase operationnal knowledge to inform current decisions (maintenance of policies, choice between operational options, knowledge management)

Two kinds of evaluation

To prove High level commission

Independant After completion

Summative

To improve Stakeholders consensus

Collaborative or participative

In itineri Formative

• When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative (Robert Stakes)

• In the real world, the two types of evaluation are not always so contrasted (when the restaurant manager tastes the soup it’s partly formative and partly summative)

Evaluation and applied research

Debate about evaluation and applied research, opposition between positivist and constructivist views on evaluation, cultural aspects

An attempt to characterize evaluation vs applied research by Rossi

• Quality. Pragmatic approach to quality• Initiative. External request• Field. Multiple areas of knowledge• Utilisation. Targeted towards decision maker• Ethics. Linked to the normative dimension

Evaluation in practice : an example of project evaluation

• Different kinds of evaluation in AFD

• Project evaluation, strategic evaluation, impact evaluation

• An exemple of a classical project eveluation : a rural road project in Sénégal

• A discussion of what could have been done if we had decided to undertake an impact evaluation

Needs Priorities

Project : objectives and design

Final effects income, local develop.productivity

Long term consequences of effects

Inputs financing, plan, training,

etc.

Process of implementation

Output : rural roads

Direct effects : traffic, accessibility, capacities

Spillover effects: effects on other zones or domains

PROJECT

CONTEXT

PROJECT LOGICAL CHAIN

PROJECT

CONTEXTNeeds Priorities

Project : objectives and design

Final effects income, local develop.productivity

Long term consequences of effects

Inputs financing, plan, training,

etc.

Process of implementation

Output : rural roads

Direct effects : traffic, accessibility, capacities

Spillover effects: effects on other zones or domains

EVALUATION TOOLS

Clarification of objectives

Results assesment and measurement

Analytical project narrative

Actors perceptions and strategies

Evaluation criterias• Relevance : The extent to which a development intervention (objectives and

design) is consistent with beneficiaries’ requirements, country needs, global priorities and partner’s and donors’ policies

• Efficiency : A measure of how economically resources/inputs (funds, expertise, time, etc) are converted into results

• Effectiveness : The extent to which the development ,intervention’s objectives were achieved or are expected to be achieved, taking into account their relative importance

• Impact : Positive and negative, primary and secondary long term effects produced by a development intervention, directly or infirectly, intended or unintended

• Sustainability : The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long term benefits. The resilience to risk of the net benefit flows over time.

(OECD, DAC)

Inputs financing, plan, training,

etc.

Output : rural roads

Direct effects : traffic, accessibility, capacities

Final effects : income, local develop.productivity

Process of implementation

Needs Priorities

Project : objectives and design

Spillover effects: effects on other zones or domains

Long term consequences of effects

PROJECTEFFICIENCY

SUSTAINABILITY

RELEVANCE

CONTEXT

IMPACT 1

EFFECTIVENESS

EVALUATION CRITERIAS

Inputs financing, plan, training,

etc.

Output : rural roads

Direct effects : traffic, accessibility, capacities

Final effects : income, local develop.productivity

Process of implementation

Needs Priorities

Project : objectives and design

Spillover effects: effects on other zones or domains

Long term consequences of effects

PROJECT

CONTEXT

IMPACT 2 additionality

IMPACT EVALUATION

First discussionConstructivist vs positivist approachs

Some questions

Does objective and neutral evaluation exist ?

Is measurement evaluation ?

Is measurement more objective than other kinds of evaluation ?

Positivist Constructivist

Reality is unique and tangible

Reality is constructed, subjective, multiple, relative

The knower and the known are independant of each other

Knower and the known are interactively linked

Inquiry is objective and value free

Inquiry is value bound

First discussionConstructivist vs positivist approachs

In evaluation positivist position is based on three claims :

1. Results count and could be objectively defined, all else is secondary

2. Objectivity stems from the scientific quality of measurement taken

3. the generalisation and discovery of social laws goes hand in hand with a rigourous treatment of the question of causality

All these claims are virougously contested by constructivits evaluators

Second discussion : complexity of the impact criteria

There are two commonly accepted definitions of what is the impact of a development intervention.

• The first is used by the Development Assistance Committee of the OECD (DAC) : “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”.

• The second is used more particularly by economists and can be summarised as: “All the effects that a development intervention has on beneficiaries that can strictly be attributed to this action”. (AFD evaluation site; http://evaluation.afd.fr/)

Practical difficulties

• Impact is dynamic

• Impact is multidimensional

• Impact is context dependant

Second discussion : complexity of the impact criteria

What kind of impacts should be evaluated ?

« Projects need to focus on creating and transmitting knowledge and capacity. The key role of development projects should be to support institutional and policy changes that improve public service delivery… In particular, the evaluation of projects from the donors' point of view should focus on whether they had a positive impact on the institutions and policies of the sector… We are more interested in the indirect than the direct effects of projects.» (Assesing Aid, World Bank, 1998)

That’s not a new question : Give a man a fish and he will eat for a day, teach a man to fish and he will eat for a lifetime (Confucius)

Aid is a special policy from the evaluation point of view : assessing impact of aid is not the same as assessing impact of development activities

Evaluating impact of aid effectiveness : historical perspective

Periods Main objectives

Main instrument

Aid effectiveness

Evaluation

1960-1980 Savings and investment, economic developement

Project, planification

Rate of investment, resource allocation

Ex ante analysis (IRR)

1980-1990 Reforms, liberalization

Structural adjustment, civil society support

Emergence of the issue : Good policies

Birth of evaluation function

1990-2000 Aid crisis : Diversification : poverty alleviation, environmental and social goals, institutional perspective

Program aid, budget aid

Burning question : importance of ownership, predominance of macro perpective

Cross countries analysis, program evaluation, rise of accountability issue

2000-2010 MDG Diversification (vertical funds, private funds, new donors, etc.)

Paris Declaration : process of aid delivery

Large evaluation, impact evaluation

Some global trends

• Diversification of actors : aid as a global hypercollective policy. Many actors at different levels without any institutional or economic regulation

• From countries to citizens perspective• Importance of the ex post level : what we get is not what we

expect• Importance of the issue of process (implementation,

transaction costs), difficulty to define results

• Going there and back on the level of evaluation (micro/macro) • Is there a progressive learning or a succession of paradigm

shifts ?

• These trends are probably relevant for most public policies

Third discussion : micro vs macro level evaluation

1990’s : The micro-macro paradox (Mosley) : most projects are deemed succesful after completion but global effects on macro variables are not detectable

« Projects financed by foreign aid are often highly visible and important successes – roads and highways, schools and health clinics, irrigation infrastructure, power plants. But success can be assessed at two levels-at the micro or project level, which typically shows high rates of success or at the macro level of economywide growth and poverty reduction, where, there is less visible success. » (Assessing Aid, World Bank, 1998)

• First explanation : fungibility : what you assess is not what you finance

• Second explanation : project evaluation doesn’t properly assess effects because of inappropriate methods

• Third explanation : sustainable effects are not measurable at the projet level (dominance of the context effect)

• Fourth explanation : There are large composition effects (aid dependance, lack of coherence, etc.)

2000’s : Even if you run « two millions regressions » you cannot elucidate the mystery of growth fron the experience of a hundred of countries (Duflo)

• The question of cross causalities is not tractable with econometric cross countries analysis

• A more fruitful way is to establish what works and what does not on the field an under which conditions (impact evaluation)

The current scene of aid and development evaluation

Very large evaluations, partly qualitative and process oriented, discussed in the development community and aimed at informing policy makers and public opinion

A strong emergence of impact evaluations at the micro level (project or segment of public policy), strongly quantitative, discussed in the scientific community

Classical project and program evaluations largely internal to donors institutions and their partners

Standing issues :

• Impact evaluation have a limited scope

• Communication between these different forms of evaluations

• Handling of the complexities of the impact measurement

Income

temps

ioWithout project

With project

Under-evaluationOver

evaluation

Impact ?

t1

i1

to

?

?