Sub-National Monitoring and Impact Evaluation Kai Kaiser PREM Public Sector & Governance DPWG 6 th Annual Meeting Brussels May 2011.

Sub-National Monitoring and Impact Evaluation

Kai KaiserPREM Public Sector & Governance

DPWG 6th Annual MeetingBrusselsMay 2011

The World Bank Page 2Public Sector & Governance

Motivation

• With Decentralization, Sub-National Governments Increasingly Responsible/Pivotal for Sectoral Outcomes

• Higher levels of government provide bulk of SN finance– Own source mobilization important for revenue

mobilization & accountability effects• Concerns about SN Outcomes

– Monitoring– Influencing


Decentralization – A Many Splendored Thing

• Devolution, Delegation, Deconcentration• Fiscal, Administrative, Political Decentralization• Different Motivations & Drivers

– Decentralization Theorem/Subsidiarity– Nation Building (Glue versus Solvent?)– Political Economy

• Design and implementation of system• Accountability for performance


Promoting Results– Diagnostic Frameworks & Instruments

• Recognizing Political Economy of Decentralization• Service Delivery Accountability• Results Chains

– SNGs & Public Expenditure Tracking– Tensions

• Country Donor Motivations & Drivers• Donor Donor Motivations & Drivers• Vertical Results “versus” Horizontal-Downward

Accountability Institution Building– Illustration: Intergovernmental Performance Grants– Selected Resources


A Results Framework

Decentralization Framework

Internal use of Performance Information

Sub-national Jurisdiction Sub-national Government

Inputs Process Outcomes Outputs

Central Government

Central Government Transfers

Sub-national Government Own Resources

Political Support

Sub-national Government Institutional Capacity

Resource Management Perspective

Ss. Delivery and Community Development Perspective

Measures of Effectiveness

Downward Accountability

Upward Accountability

Measures of Efficiency

Measures of Process

Performance Monitoring

System

E x t e r n a l

u s e

o f

P e r f o r m a n c e I n f o r m a t i o n

Citizens Voice & Democratic Participation

Unbundling Decentralization Dimensions

Policy Making Who sets the main policy guidelines for a service (e.g., free primary education as a national policy)?

Standards Setting Who sets the standards (e.g., national tests versus local building standards)?

Administration/contracting Who is the primary government authority that administers services on a day to day basis?

Financing Who provides the financing for services? Capital investments, wages, O&M?

Service deliveryWho actually produces/delivers the service (e.g., this can often

be private)?

Regulation Who regulates a service (e.g., professional accreditation in health)?

Monitoring & Evaluation Who monitors and evaluate a service (e.g., how can local citizens provide feedback)?


What Gets Measured Gets Done?

Central Government(Cross-Cutting/

Sectoral Agencies, Intermediate)

….. (Average)

SNGi

500++

Heterogeneity Ranking (e.g., systems, human, fiscal capacity, baseline performance )

SNGi#1

…..

Observed Performance

Indicators


Indicators


Indicators


Indicators


Indicators


Indonesia Decentralization Results?-1

00

01

00

200

300

ch

an

ge

in pa

ved

ro

ad

%

-100 0 100 200 300 400 500change in real Infrastructure expenditure %

Fitted values ch_paved

-5

00

50

100

150

ch

an

ge

in

acce

ss to cle

an

w

ater %

-100 0 100 200 300 400 500change in real Infrastructure expenditure %

Fitted values ch_clean_wtr


Bosnia-HerzegovinaMunicipal PFM and Service Delivery Benchmarking

0

5

10

15

20

25

30

BE

RK

OV

ICI

ST

OLA

C

MO

ST

AR

KA

LES

IJA

IST

OC

NI

ST

AR

I

BU

ZIM

VIS

EG

RA

D

ZV

OR

NIK

RIB

NIK

BR

EZ

A

JAB

LAN

ICA

SIR

OK

IB

RIJ

EG

LJU

BIN

JE

OD

ZA

K

DO

BO

J-JU

G

SA

MA

C

NO

VO

SA

RA

JEV

O

TU

ZLA

BA

NJA

LUK

A

ILID

ZA

02468101214161820

Service Outcome Score PFM Score (28+3 indicators)

Citizen Satisfaction Survey & PFM PEFA Type Benchmarking in 20 Pilot Municipalities (of 142) in 2007Source: From Stability to Performance: Local Governance and Service Delivery in Bosnia and Herzegovina (draft report)

Having the “ideal” counterfactual……

Y1 (observedl)

Y1

* (counterfactual)

Y0

t=0 t=1 time

Intervention

10

allows us to estimate the true impact

Y1

Impact = Y1- Y1*

Y1

*

Y0

t=0 t=1 time

11

Source: Skoufias


Strategies in Practice

• Pure Randomization Challenging– Political Economy of Decentralization

• Matching, Difference in Difference, Discontinuity• Layering Interventions• Natural Experiments• Commit of Baseline Platforms

– Across SNG Engagements– Across Time


• Can higher levels of government leverage specific transfers to sub-national governments to enhance local “performance” and “results”?

• What link of “results” chain to target?– PFM Process versus Outcome?– Compliance versus “Stretch” Reform

• Instruments & Interventions– Prevailing country flows & systems– WBG instruments/policies (e.g., new P4R OPs)

Application: Performance Grants


Selected References

Bird, Richard, (2010), Taxation and Decentralization, Economic Premise Note, No. 38 Eaton, Kent, Kaiser, Kai, and Smoke, Paul (2010), 'The Political-Economy of Decentralization Reforms: A Development Partner Perspective', Washington, DC, pp. 114 Dumas, Victor and Kai Kaiser, (2010), Sub-National Performance Monitoring Issues and Options for Higher Levels of Government, Washington, DC: Public Sector Group, June pp. 51 World Bank, (2004), World Development Report: Making Services Work for Poor People, Washington, DC, pp. 271 World Bank, (2009), Implementing Public Expenditure Tracking Surveys for Results: Lessons from a Decade of Global Experience, Washington, DC, pp. 16

Q & A

Sub-National Monitoring and Impact Evaluation Kai Kaiser PREM Public Sector & Governance DPWG 6 th Annual Meeting Brussels May 2011.

Documents

world bank page

performance slide

public sector group

local governance

performance grants slide

results framework slide

time slide

skoufias slide