Monitoring and Evaluation System for CAADP Implementation_2010

Monitoring and Evaluation System for CAADP Implementation

CAADP Monitoring and Evaluation Framework Validation Workshop

Indaba Hotel, Johannesburg, South Africa

March 1-3, 2010

Babatunde OmilolaCoordinator, Regional Strategic Analysis and Knowledge Support System (ReSAKSS)

IFPRI

Outline• What is CAADP?

– Principles

– Processes and implementation

• Why a M&E System for CAADP?

• What is the role of ReSAKSS?

• Implementation of CAADP M&E Framework

– Key indicators

– Data collection strategy

– Collaborators

– Results and Outcomes

• Next Steps

What is CAADP?

• The Comprehensive Africa Agriculture Development Programme (CAADP) is the African Union (AU)/New Partnership for Africa’s Development (NEPAD) vision and strategy for the development of African agriculture.

• CAADP puts agriculture at top of priorities of African countries

• CAADP is African-owned and African-led

CAADP Principles

• Employ agriculture-led growth to achieve MDG1 of halving poverty and hunger by 2015

• Pursuit of 6% average annual sector growth at national level

• Allocation of 10% of national budgets to agriculture sector

• Exploitation of regional complementarities and cooperation to boost growth

• Policy efficiency, dialogue, review and accountability (evidence-based policymaking)

• Partnerships and alliances to include farmers, agribusiness, civil society

CAADP Process

• The principles are achieved through:– the strategic functions of CAADP, – the guidance and involvement of the Regional Economic Communities

(RECs), and– the national roundtable process

• These activities surround four key pillars, led by Africa-based technical institutions: – Pillar 1: extending the area under sustainable land management

(University of Zambia)– Pillar 2: Improving rural infrastructure and trade-related capacities for

market access (Conference of Ministers of Agriculture of West and Central Africa (CMA/AOC))

– Pillar 3: Increasing food supply and reducing hunger (University of KwaZulu-Natal)

– Pillar 4: Agricultural research, technology dissemination and adoption (Forum for Agricultural Research in Africa (FARA))

Why a M&E system for CAADP?

• To regularly monitor CAADP goals

– Are countries achieving the targeted growth rates? (6%)

– Are countries investing at the targeted level? (10%)

– Are these investments having their intended impacts on poverty and hunger?

• To inform the review processes established by CAADP-PP (mutual, peer and progress review)

• To further inform policy-making and dialogue

Idea behind M&E framework

Page 7

Greater/better distributedpoverty reduction & food and nutrition

security outcomes

Accelerated agricultural growth & Greater market access

More enabling policies &Greater/more efficient allocation of

agricultural investments

Global level

Commitments

Africa-wide level

Declarations

Decisions

Regional level

Early actions

8

7

6

1

2

3

5

P 4

P 3P 2

4

P 1

National level

Roundtable

Processes

Other factors

Other factors

Other factors

More effective

t

r

a

c

k

i

n

g

r

e

p

o

r

t

i

n

g

a

n

a

l

y

s

i

s

Page 8

Key questions for M&E• Delivering on commitments

– Have commitments and targets been met so far?

• Effectiveness of interventions (policies, investments, etc.)– How effective have different types of interventions been in any achievements

realized so far? What factors have shaped the achievements?

– What are the trade-offs and complementarities, if any, among different types of interventions?

• Consistency with initial targets– What are the projected impacts if interventions proceed as planned?

– Are the projected impacts compatible with the CAADP targets?

– If not, what adjustments are needed to get it on track?

• Exploring better interventions– Could greater or better distributed impacts be obtained by reconfiguring the

interventions?

– What are the different interventions that can lead to these outcomes?

Page 9

Main challenge and achievements to date

Main challenge– Need indicators that are standardized, consistent and measurable

across different countries and regions for cross-country comparisons and learning

• Main achievements– CAADP M&E working group established, beginning with a workshop

at the AU (Addis Ababa, Dec 3-4, 2007)

– Draft M&E framework developed and presented at CAADP PP meeting in Seychelles, March 2008 (www.resakss.org/publications/DiscussionP6.pdf)

– Framework based on CAADP principles and economic theory to address issue of attribution/causality (i.e. inputs outputs outcomes impacts)

– Developed data collection formats with which ReSAKSS nodes will work with network of partners to collect data on indicators

ReSAKSS Role in CAADP M&E• M&E of agricultural development initiatives in Africa is key

element of ReSAKSS strategic analysis

• CAADP M&E Technical Working Group met in 2008 and designed M&E Framework for CAADP– See ReSAKSS Working Paper No. 6

• ReSAKSS has now outlined a plan for implementing this framework

• As mandated by 5th CAADP-PP in Abuja, the CAADP M&E framework will be validated in collaboration with AU/NEPAD and other stakeholders in South Africa between March 1 and 3, 2010

Page 11

Types of indicators and rationale• Input indicators: what is the overall level of effort invested?

– CAADP processes, policies, institutions, investments, etc.

• Output indicators: what is the level of provision, coverage, and utilization of services?– Access to infrastructure and services, adoption of technologies, etc.

• Outcome indicators: what is the effect on outcomes that affect goals?– Yields, production, wages, prices, trade, etc.

• Impact indicators: what is the ultimate effect on goals?

– Growth, income, poverty, food security, hunger, etc.

• Conditioning indicators: how confident are we that any observed changes is due to the intervention?– Total budgetary resources, climate, natural disasters, wars, etc.

Page 12

Pillar 1Pillar 1

Land & water management

Pillar 4

Research &

Technology

Pillar 3

Food security

Pillar 2

Markets & trade

Programscaling up and

out, etc.

Key Indicators:

Credible and relevant evidence used in design of investment program

Inclusive participation of stakeholders in program design

Investment program aligned with CAADP principles and targets

Investment program technically reviewed

Mechanisms in place for implementation and M&E of the program

Input indicators:CAADP roundtable process

Program implementation

and M&E

Roundtable conference

held & compact signed

National compact

developed & discussed Stock taking

and gap analysis completed

Cabinet memo discussed &

approved

Steering & Technical

Committeesappointed

CAADP process

launched

National Focal Point appointed

Input indicators:enabling conditions (other processes, policies, institutions)

Key Indicators:

• Policies for private sector development (property rights, access to credit, contract enforcement, licensing, competition, …)

• Policies on equity(access of poor and vulnerable groups to resources, markets, food, and nutrition)

• Governance (political

stability, accountability, government effectiveness, regulatory quality, rule of law, control of corruption)

• Harmonized policies and strategies

• Commitments met

Page 13

Private Sector

Donors

Others

National Policy Processes/Events

CAADP, SWAP, MTEF, Exp Reviews, Donor

harmonization, Elections, Law, etc.

Gov’t

Regional Level Africa-wide level

Actors

RECs, RegOrgs., etc.

Actors

AU, Int’l Orgs., etc.

Global level

Actors

G-8, G-20, WTO, etc.

Process

Summit, Reviews,

etc.

Process

Assembly, Summit,

etc.

Process

Conventions, etc.

Input indicators:investments and disaggregation

AGRICULTURAL & RELATED

INVESTMENTS

Space

Province, district, rural/urban, Agro-

ecology

Agriculture

Research, extension, irrigation, input support,

markets, ...

Sub-sector

Crops, livestock, fishery, forestry

Economic

Salaries, capital, operations and maintenance, …

Commodity

Staples, traditional, high value, export, ...

Source

Government, Donors, Private

Sector

Other

Gender, socio-economic groups

CAADP Pillar

1, 2, 3, 4

Sector

Agriculture, roads, education, health,

water & sanitation, ...

Output indicators:coverage and utilization of services

Investment /Intervention

Provision /Coverage (e.g.)

Utilization (e.g.) Disaggregation

Research Number of technologies dev’d

Area under technology

Commodity, gender, space

Extension Extension-farmer ratio

Number of visits received per year

Gender, space

Irrigation Capacity of irrigation (irrigable area)

Area under irrigation Commodity, gender, space

Farm support Quantity of support Area under input Commodity, gender, space

Feeder roads Length or density of roads

Space

Market Distance to nearest market

Share of output sold Commodity, gender, space

Post harvest Capacity of storage Capacity utilized Commodity, gender, space

…

Outcome indicators:agricultural sector performance

AGRICULTURAL SECTOR

PERFORMANCE

Use of factors (land, labor, capital) and inputs by:

sub-sector, commodity, gender, socio-economic group, Space

Growth returns to different types of investments by:

sub-sector, commodity, space

Sub-sector growth and contribution to

AgGDP by:

Space

Commodity growth, contribution to

AgGDP by:

Space

Productivity of factors (land, labor, capital)

and inputs by:

sub-sector, commodity, gender, socio-economic

group, space

Sector growth and contribution to overall GDP by:

Space

Production, trade and prices by:

sub-sector, commodity, space

Impact indicators

INCOME, POVERTY,

FOOD AND NUTRITION SECURITY, HUNGER

Returns to different types of investments by:

gender, socio-economic group, space

Decomposition by:

sector (agriculture, services, industry); sub-sector (crops, livestock,

fishery, forestry);commodity (staples,

high value, export, etc.)

Distribution by:

gender, socio-economic group,

space

Returns to commodity growth by:

gender, socio-economic group, Space

Unit costs by:

gender, socio-economic group,

space

Returns to sub-sector growth by:

gender, socio-economic group, Space

Implementation of CAADP M&E Framework

• Implementation is a system consisting of:

– Linked country level and regional teams, working under clearly defined roles and using shared data standards and protocols

– Regular collection, documentation and processing of data at national and regional levels

– Timely publication of these indicators and related monitoring reports

Key indicators

Processes

• Formal communications, declarations

• Compact signings (country and regional)

• Review and dialogue

• Mutual accountability framework

Agricultural InvestmentsAgricultural Investments

• Levels and Shares

• Sub-sector

• Spatial

Agricultural Agricultural Growth

• Levels and growth rates

• Production and Productivity

Poverty Poverty Reduction

• Headcount index

• Absolute numbers

• MDG projections

Other factors: trade, hunger,

inputs, prices, etc.

Overview of methodology for M&E

Key question Tools Data

Delivering on commitments

• Trends• Simple correlations

• National surveys• Expert opinion surveys

Effectiveness of interventions

• Econometric methods• General equilibrium models

• National surveys• Targeted surveys• Expert opinion surveys

Consistency with initial targets

• Simulation models• Participatory approaches

• Assessment of effectiveness• Expert opinion surveys

Exploring better interventions

• Simulation models• Participatory approaches

• Assessment of effectiveness and consistency• Expert opinion surveys

Details in (www.resakss.org/publications/DiscussionP4&7.pdf)

Page 21

Methodology:delivering on commitments

• Trends, situation, and simple correlation analyses to monitor progress (i.e. no attribution to interventions)– CAADP Roundtable process

– Donor commitments

– 10% government agricultural expenditure

– 6% agricultural GDP growth

– Halving poverty and hunger, etc.

• Data– National household surveys, accounts, etc.

– International public datasets (AfDB, World Bank, FAO, etc.)

– Targeted surveys (CAADP national focal points)

– Targeted expert opinion surveys

– …

Page 22

Methodology:effectiveness of interventions (1)

• Main challenge is attribution

• Two complementary approaches– Before and after treatment

• Baselines (2003, compact signed, program implemented)

• Mid-term, end of project, long after project

– With and without treatment

• Treatment: direct beneficiaries; indirect beneficiaries through information/technology transfers, etc.

• Controls: can only be affected through general equilibrium effects, e.g. prices, wages, etc.

• For poverty (pov), impact of intervention (INV) measured by Average Treatment effect of the Treated (ATT):

ATTj = Ej [ povbefore, j – povafter, j | INVj =1] – Ei [povbefore, i – povafter, i | INVi =0]

Page 23

Methodology:effectiveness of interventions (2)

• Underlying relationships to be estimated– Intervention decision making and placement

– Household access to and utilization of services due to intervention

– Household production, marketing and consumption decisions

• Techniques– Econometric methods (double-difference, instrumental variables,

matching) to assess direct impacts (mid-term, end of project, long after project)

– General equilibrium modeling to assess economy-wide impacts

• Data– National surveys

– Targeted household, market and other surveys to fill gaps

– Expert opinion surveys

– Case studies (selected countries and/or programs)

Page 24

Methodology:consistency with initial targets

• Apply models developed for stocktaking and gap analysis

• What are the projected impacts if interventions proceed as planned?

– Use mid-term estimated ex-post impacts and parameters and growth patterns to project ex-ante impacts over period when target is expected to be achieved

Are the projected impacts compatible with the CAADP targets?– Compare above ex-ante impacts with initial targets

If not, what adjustments are needed to get it on track?– Use data and information from experts to identify plausible scenarios

– Simulate impacts under different scenarios to reach initial targets

Page 25

Methodology:exploring better interventions

• Could greater or better distributed impacts be obtained by reconfiguring the interventions?– Even if interventions are consistent with the initial targets, can

simulate impacts under different composition of investments to identify (in)efficiencies in implementation

– Use estimated ex-post impacts and parameters from cross-country reviews and information from experts to identify plausible scenarios

– Value of indicators associated with simulated impacts of the desirable scenarios can be used as guidelines to set new targets

What are the different interventions that can lead to these outcomes?– Composition of investments associated with desirable scenarios from

preceding analysis

Data collection strategy

• Short-term: – Secondary sources– These are already collected, updated and made

available on ReSAKSS website (www.resakss.org)

• Medium-term:– ReSAKSS surveys and collection from country partners

via regional nodes and established country SAKSS nodes• Already started in many countries and regions

• Long-term:– ReSAKSS data collection mechanism institutionalized

and implemented on regular basis (semi-annual, annual or bi-annual depending on indicator)

Page 27

Reporting and dissemination of M&E results

• ReSAKSS Annual Trends Report and briefs to be completed by end of September of each year―timely for CAADP PP meeting

• ReSAKSS website (www.resakss.org) to view and download trends, data, charts, supporting research publications, etc.

• Various other media and presentations to review M&E information and results of analyses:

– CAADP PP (Africa wide)

– CAADP advisory councils (REC level)

– ReSAKSS steering committee meetings (REC level)

– Other regional- and country-level policy dialogues

Tracking Progress of CAADP: ReSAKSS Websitewww.resakss.org

http://www.resakss.org/

Users can customizethe map and charts

based on the specific information they are looking for, whether

that be regional information or

country-specific information

Users can customizethe map and charts

based on the specific information they are looking for, whether

that be regional information or

country-specific information

The ReSAKSS website allows users to easily track progress

against the CAADP and MDG targets

while also accessing a wealth of knowledge and data on

agricultural development in Africa

The ReSAKSS website allows users to easily track progress

against the CAADP and MDG targets

while also accessing a wealth of knowledge and data on

agricultural development in Africa

Key collaborators• The main collaborator at the country level will be the

SAKSS nodes, which will liaise with the planning units of ministries, bureaus of statistics and private sector institutions and think tanks

• Data quality will be guaranteed by working with partners such as the AU, NEPAD, RECs, CAADP Pillar institutions, CAADP country focal points, external partners such as UN institutions, Universities, etc

• ReSAKSS will establish demonstrable agreements with RECs and national centers of expertise on shared platforms for benchmarking, collection…

Roles and Responsibilities in Implementation of CAADP M&E Framework

• Need for serious efforts by the RECs and their member countries to internalize the collection and provision of the data within their own M&E systems

• This should be tied with serious efforts to meet International standards in reporting

• Ultimately, the responsibility of collecting and analyzing the data and reporting the results must be with the country itself

• ReSAKSS and network of partners can facilitate this– Consulting with the RECS and CAADP country focal points

– Strengthening capacity of national statistical bureaus

– …

Results and Outputs of M&E System

• Data collected are openly available for review, analysis and download on ReSAKSS website

– 28 indicators already tracked and analyzed on website

Results and Outputs of M&E System

• Data are analyzed and published in key monitoring reports:

– Annual Trends report for each node

• Agriculture performance, investment and MDG targets

– Semi-annual report on CAADP process that will be made available at each CAADP PP

– Joint publications with key partners, such as AU/NEPAD

Where are we now?

• Almost 7 years after CAADP, has there been any progress toward the goals?

• The Process: – 13 countries and 1 region have signed CAADP compacts

• Agricultural Spending/Investment: – The number of countries spending at least 10% of budgets

on agriculture has increased since 2002

• Agricultural Growth:– At the continent level agricultural growth has increased

since 2002– The number of countries with annual agricultural growth

rates of 6% or more has increased since 2002

1. Government appoints Focal Point(s)

2. REC and Government launch

process

3. Country Steering and Technical

Committee

4. Cabinet Memo and Endorsement

5. Stocktaking, Growth, Invest.

Analysis

6. Drafting of Country CAADP Compact

7. Roundtable Signing of Compact

8. Elaboration of detailed investment

plans

9. Post compact review meeting and

validation of investment plans

10. Agreement on financing plan,

financing instruments, and annual review

mechanism

11. Operational design and other technical

studies and assessment for

program execution

12. Execution of new investment programs

13. First annual review meeting

14. Second annual review meeting

The National CAADP Roundtable Process & Country

Status

Libya, Eritrea

Zimbabwe, Egypt,

Mauritius, DRC

Comoros, Madagascar, Sudan

Cote d’Ivoire, Djibouti, Seychelles

Burkina Faso, Guinea-Bissau, Guinea, Kenya, Malawi,

Senegal, Uganda, Zambia

Benin, Burundi, Cape Verde, Gambia, Liberia, Mali, Niger,

Nigeria, Sierra Leone, Swaziland

Rwanda

Ethiopia, Ghana, Togo

Government spending on agriculture: Progress towards the Maputo Declaration target

• The African continent as a whole has not met the 10% target (current spending at 6-8 percent)

• But, this varies by country

0

5

10

15

20

25

Gu

ine

a B

issa

u*

**

Gab

on

**

*

DR

C**

Co

te d

'Ivo

ire

Mo

rocc

o*

*

Cen

tral

Afr

ican

…

Mau

riti

us*

*

Leso

tho

**

Rw

and

a

Egyp

t**

Cam

ero

on

**

Bo

tsw

ana

Bu

run

di*

**

Ken

ya*

***

Swaz

ilan

d**

Uga

nd

a***

*Su

dan

***

Nam

ibia

**

Tan

zan

ia**

Mau

rita

nia

***

Be

nin

***

*

Tun

isia

**

Nig

eria

Zim

bab

we*

*

Zam

bia

*

Togo

Mad

agas

car*

*

Gam

bia

***

Mo

zam

biq

ue*

*C

had

***

Gh

ana*

**

*

Mal

i

Mal

awi

Eth

iop

ia*

*

Sen

egal

Gu

inea

***

Nig

er*

Bu

rkin

a Fa

so*

%

Agricultural Expenditures as a share of total (%), 2007

CURRENT, 2007 (Unless otherwise noted)

*=2006; **=2005; ***=2004; ****=2008 estimates

Source: Omilola and Lambert, 2009.

Only 8 countries have met the 10%

target

Have countries increased their spending in response to the 2003 Maputo Declaration?

• At the continental level, agricultural spending nearly doubled between 2000 and 2005

• In 2003, only 3.2% of countries allocated 10% or more of their budgets to agriculture– This increased to 33.3%

in 2006 before slightly falling to 25% in 2007

• 9 countries increased their allocations from less than 5% spending to 5-10% spending

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

2002 2003 2004 2005 2006 2007

% o

f re

po

rtin

g co

un

trie

s

Level of agricultural spending as a share of total spending, 2002-2007

Less than 5% 5%-10% More than 10%


Agricultural expenditure as a share of agricultural GDP

• Measures government spending on agriculture relative to the size of that country's agriculture sector

• Under this measure, more countries fall into the category of low budget support to agriculture

0

20

40

60

80

%

Agricultural expenditures as a share of agricultural GDP, 2007

CURRENT, 2007 (Unless otherwise noted)

*=2006; **=2005; ***=2008 estimates

The range is considerable

(1 to 60%)

On aggregate , Africa spends between 5-7% of agricultural GDP on agriculture,

compared to 15% in Asia during its Green Revolution


Agricultural GDP growth

Page 39

-36

-30

-24

-18

-12

-6

0

6

12

18

24

Erit

rea

Gam

bia

, Th

e

Zim

bab

we

Sen

ega

l

Tun

isia

Mau

rita

nia

Mal

awi

Leso

tho

Cap

e V

erd

e

Gab

on

Mal

i

Ke

nya

Co

te d

'Ivo

ire

Eth

iop

ia

Zam

bia

Mad

agas

car

Alg

eri

a

CA

R

Ch

ad

Gu

ine

a-B

issa

u

Bo

tsw

ana

DR

C

Seyc

he

lles

Swaz

ilan

d

Nig

er

Bu

rkin

a Fa

so

Djib

ou

ti

Sud

an

Egyp

t

Cam

ero

on

Uga

nd

a

Gh

ana

Bu

run

di

Nig

eri

a

Co

mo

ros

Mau

riti

us

Mo

rocc

o

Tan

zan

ia

Gu

ine

a

Sou

th A

fric

a

Togo

Be

nin

Nam

ibia

Mo

zam

biq

ue

Sao

To

me

& …

An

gola

Rw

and

a

Equ

ato

rial

Gu

ine

a

2002: Only 9 countries achieved 6% or more annual growth

2008: At least 20 countries achieved 6% or more annual growth

6% CAADP target

6% CAADP target

Source: WDI

Burkina Faso CameroonCongo, D.R.

C. African Rep. Ethiopia Guinea Kenya Mali

Morocco Senegal

Swaziland Togo

TunisiaUganda

AngolaBotswana

EgyptGhana

MauritaniaNamibia

Countries on track towards halving poverty by 2015

Countries on track towards halving hunger by 2015Only 6 Countries on track

towards achieving both goals of MDG1

Algeria Malawi

Sao Tome and PrincipeTanzania

What about poverty and hunger?

Source: Omilola and Lambert, 2009

Next steps• Formal validation of M&E framework by all stakeholders under the

leadership of NEPAD and AU• Regional network already established

• Next step is to fully operationalize country-level surveys and data analysis in each region– Establish country SAKSS nodes

• Harmonize efforts with other development partners doing similar M&E work in Africa such as AGRA, World Bank, Pillar Institutions, Mutual Accountability Framework, etc

• Track the contributions of the CAADP process to the achievements of agricultural growth and poverty reduction

• Monitor how agricultural budgets are being spent

• Establish critical M&E information needed to enhance effective dialogue and policy processes at all levels

Thank you.