Evaluation Fernando Castro

August, 2007Indonesia

Towards the Institutionalization of Evaluation in Planning and BudgetingManuel Fernando CastroEconoma Urbana, Ltda.

MotivationRelevant international experienceBasic M&E definitions Scope of proposed methodsInstitutionalizing Evaluation as a tool for planning and budgeting

-Selecting programs to be evaluated- Implementation and coordination of evaluations - Operational arrangements - Reporting and disseminating evaluation findings - Development of evaluation capacities

6.Organizing BAPPENAS Deputy of Performance Evaluation

Content

MotivationRelevant international experienceBasic M&E definitions Scope of proposed methodsInstitutionalizing Evaluation as a tool for planning and budgetingOrganizing BAPPENAS Deputy of Performance EvaluationContent

Motivation Estructural changes

The 1998 economic crisis took back poverty to its 1987 levels. In 2006:

- 16.7 percent (36,8 million people) in US$ 1.55 a-day poverty line. - 42 percent (92,8 million people) between US$ 1 and US$ 2-a-day. - Political democratization (direct elections, multi-party political system).

- Political reform (citizens and assemblies participation and control)

- Political, fiscal and administrative decentralization. Regions now:

- Manage 40% of total public expenditures - Execute more than 50% of public investment.

- Additional spending source of about US$ 15 billion from 2007 and beyond.

- Regaining of growth (prudent macroeconomic policies) - Public investment returned to its pre crisis-level of 6.5% of GDP - Reduction of fuel subsidies, (+ increasing revenues and declining debt service)

Motivation Main challenges

- RPJM targets reducing poverty from 18.2% in 2002 to 8.2 by 2009.

- Increasing secondary school enrollment from 79.5 percent to 98 - Diminishing maternal mortality rate from 307 per 100.000 live births to 226 - Increase and improve infrastructure

- Decentralization needs avoiding duplication of investment efforts. - Reducing corruption needs accountability, participation and public control - Consolidating structural changes will require governance (trust). - Quality of public spending and social services will have to be improved - Planning and Budgeting reforms seek integration and PBM

None of the later possible without proper and permanent M&E

MotivationRelevant international experienceBasic M&E definitions Scope of proposed methodsInstitutionalizing Evaluation as a tool for planning and budgetingOrganizing BAPPENAS Deputy of Performance EvaluationContent

Relevant international experience Main Drivers of M&E in Developing Countries:

- Frustration with the pace of economic development (Monterrey Conference, 2002) - Post-Washington consensus thinking on the content of reform - Re-valuation of role of institutions in development - Review of role and means of multilateral development organizations - Public sector reforms in OECD countries in last 20 years (benchmark)

Relevant international experience

Two Contrasting contexts in institutionalizing M&E

Relevant international experience

Institutionalization of M&E in Development Countries (LA)

MotivationRelevant international experienceBasic M&E definitions Scope of proposed methodsInstitutionalizing Evaluation as a tool for planning and budgetingOrganizing BAPPENAS Deputy of Performance EvaluationContent

Basic M&E definitions

Governments cycle: M&E feeds back the management model

Basic M&E definitions

M&E differences and complementarities

Basic M&E definitions

Variables and associated indicators for performance-based M&E

Rapid assessments (process or implementation evaluations): inform on whether the program is operating as it ought ( are we doing things right?). Analyzes activities, achievement of intermediate goals, administrative processes and use of resources. Impact evaluations: measure the achievement of intended (and un-intended) program or policy outcomes and impacts on beneficiaries and development goals. Seek to determine the links between inputs, activities, outputs, outcomes and impacts. Financial evaluations: analyses the way in which resources are programmed, disbursed and used over time as well as financial management capacities and tools of spending agencies. Basic M&E definitions

Three types of performance evaluations in planning and budgeting

Accountability: government agencies continuously producing and disclosing performance information for government officials, Congress, control agencies and citizenship. It diverts from the so called normative accountability which focuses on legal and procedural compliance and wrongdoing.

Institutional incentives: recognition of public agencies results and best managerial practices.

Budgetary incentives: establishing links between performance planning and budget allocations (budget ceilings, additions, reductions or re-allocations). Basic M&E definitions

Other key concepts in performance-based M&E

MotivationRelevant international experienceBasic M&E definitions Scope of proposed methodsInstitutionalizing Evaluation as a tool for planning and budgetingOrganizing BAPPENAS Deputy of Performance EvaluationContent

Scope of proposed methods- Based on international experience, context and needs of Indonesia, two methods proposed: rapid assessments and impact evaluations. - Highly complementary methods that can allow Indonesia at once to : - Fulfill law mandates to evaluate all existing programs - Strengthening linkages between work plans, budgets and performance. - Count with opportune, economic and reliable evaluation method to feed the budget needs - Respond to demands of in depth evaluations for planning and policy formulation - Enhance the production of performance information - Promote effective accountability Estimated Costs - Rapid assessments - Impact Evaluations Between US$ 15.000 US$ 25.000 (without field work)Up to 1% of program value

Scope of proposed methods

Rapid Assessments - Asses the quality of the programs in seven areas :1. Design2. Operation3. Organizational Structure4. Financial Management5. Performance Information 6. Efficiency 7. Effectiveness - Provide results in a short period of time (3 months) for influencing:

- Programs design adjustments- Implementation improvements- Resource allocation decisions

Standardized and low cost method that generates uniform and comparable information for different programs

Scope of proposed methods

Rapid Assessments: contentsProgram EvaluationOrganization and gathering of Information Analysis of Information and evaluative judgementPlan of adjustments proposalSemi-AnalyticMethodologyProgram DescriptionAnalytic MethodologyAnalytic Methodology

Scope of proposed methods

Rapid Assessments: the methodEvaluation of the ProgramDescription of the program based on internal InformationIs Internal information enough to describe the program?Construction of lacking informacin using other sources (secundary, direct gathering)Complement Description of the ProgramYesNo

Scope of proposed methods

Rapid Assessments: one methodCoherence between institutional mechanisms and objectivesBudget, sources and allocation mechanismDesigns external coherence (i.e. need for, aligned objectives, duplication?Design and operational internal coherence (P.model, conditions?) Quality of organizational structure and financial managementEfficiencyEffectivenessQuality of performance information and M&E tools 42 focused evaluation questionsAdditional

Scope of proposed methods

Impact Evaluations- Broader scope and deepness. - Sophisticated tool of analysis which provides the stronger evidence for decision making- Demand considerable efforts, large amounts of good quality information and financial resources- Require time for validating results (some time years)Not all of the programs require this type of in-depth analysis

None standardized method, depends on the program, available information and resources. Evaluators should propose method according to technical criteria

Scope of proposed methods

Impact Evaluations: the method

Scope of proposed methods

Impact Evaluations: the methodExperimental - Random allocation of a program to individuals - It creates controlled variation that can be used to obtain credible results - Single differences (t-c)

2. Cuasiexperimental - No ramdomization posible, then constructed control: - Comparing treated to untreated taking into account pre-program differences by having a pre-program measure - Difference in difference (o double difference) - Instrumental variables: Propensity Score Matching estimating beneficiaries probability of being in the program

Scope of proposed methods

Impact Evaluations: the methodDesign- Selecting object of analysis (treatment and control)- Formulation of questions to be answered- Defining the sampleImplementation- Identifying tools for information gathering- Data collectionData Analysis

- Quantitave- QualitativeImplementing recomendations

MotivationRelevant international experienceBasic M&E definitions Scope of proposed methodsInstitutionalizing Evaluation as a tool for planning and budgetingOrganizing BAPPENAS Deputy of Performance EvaluationContent

Institutionalizing Evaluation as a tool for planning and budgeting

- Selecting programs to be evaluated- Implementation and coordination of evaluations - Operational arrangements - Reporting and disseminating evaluation findings - Development of evaluation capacitiesContent

Institutionalizing Evaluation as a tool for planning and budgeting

Selecting programs to be evaluated: main criteria

Rapid Assessments Impact Evaluations1. No impact evaluation taking place, neither requirements of it based on programs characterisitics.

2. Although impact evaluation required, it cannot be done due to technical, political or financial problems.

3. Recurrent investment in the related sector, allocation of resources in three or more previous annual budgets and likely to continue.

4. Recent implementation but likely to present performance problems.

5. Subject to possible budget modifications (increases or cut backs) or design adjustment in response to central or sector authorities decisions. 1. Ammount of budgeted resources involved,

2. Programs size (coverage) and characteristics of beneficiary groups

3. Priority for government as defined in Mdium Term Development Plan (RPJM) and/or Annual Work Plans (RKP).

4. Strategic priority for sector policy as defined in Ministry/Agency Mdium Term Development Plan (Renstra-KL) and/or Annual Work Plans (Renja-KL)

5. Innovation features of programs. Intervention model challenging previous or current approaches or never applied.

6. Potential for replicating or escaling up in the same or other sectors based on evaluation lessons.

Institutionalizing Evaluation as a tool for planning and budgeting

Selecting programs to be evaluated - Main Impact evaluations according to development priorities:

1st Phase. Social programs to meet Millennium and RPJM poverty reduction goals. These include primarily:

- Nutrition- Maternal health- Transition to junior education - Water and sanitation- Descentralization - Regional investments DAU and DAK funds

2nd Phase. Infraestructure programs Due to their importance for promoting a competitive economy and the ammount of resources involved. They might include:- Housing- Telecommunication (connectivity)- Transport (roads)

Institutionalizing Evaluation as a tool for planning and budgeting

Selecting programs to be evaluated (examples)

Rapid Assessments Impact Evaluations1.

Education1. 9 years of basic education programHealthIndividual health effortPublic health effortService of health for mother and childSocial ProtectionRepairment of nutrient societyFood program*Housing Development and community programsLag areas development programBussines Micro-scale programDestitute community programSocial prosperity programInfraestructureRoads work programIrrigation chain pprogram

Institutionalizing Evaluation as a tool for planning and budgeting

- Selecting programs to be evaluated- Implementation and coordination of evaluations - Operational arrangements - Reporting and disseminating evaluation findings - Development of evaluation capacitiesContent

Institutionalizing Evaluation as a tool for planning and budgeting

Implementation and coordination of evaluations Due to time and capacities required, as well as to guarantee independence and credibility, evaluations must preferably be done by external organizations or experts (universities, reasearchers, etc.)

However, if time and capacities exist, they may also be performed by technical officials from the Nacional Planning Agency -BAPPENAS.

The selection of evaluators should be competitive and transparent, taking into account experience, technical capacities, and costs.

- In the case of impact evaluations, additionally, methodological proposals, data collection and analysis strategy must be considered

Institutionalizing Evaluation as a tool for planning and budgeting

Implementation and coordination of evaluations

- The Deputy of Performance Evaluation at BAPPENAS, will be responsible of coordinating and supporting evaluation processes. It includes:

- Defining the scope (TOR) - Supporting contracting and negotiation processes - Carry out qualification of proposals - Technical supervision - Ensuring that findings are translated into policy decisions

The Deputy of Performance Evaluation at BAPPENAS should ensure that responsible agencies participate in all stages.

Institutionalizing Evaluation as a tool for planning and budgeting

Implementation and coordination of evaluations - BAPPENAS as head of NPS will conforme a Performance Evaluation Commette to:

- Discusse and approve a pluri-annual evaluation agenda which might be presented to cabinet appovation, if considered. - BAPPENAS (DPE) will prepare a draft of the agenda for discussion at the Committe - The committe should be integrated by: MoF (DG Budget), line ministries and agencies as invited, BAPPENAS (Deputy of Development Funding). The Technical Secretariat will be chaired by BAPPENAS (Deputy of Performance Evaluation). - Evaluation findings and recomendations should be presented to the Committe which will establish curses of action and might propose them to upper levels of governemnt through BAPPENAS (Minister of Planning)

- The DPE will estimate the costs and identify possible sources. It will present a proposal for approval to the Commette, ensuring that agencies contribute

Institutionalizing Evaluation as a tool for planning and budgeting

- Selecting programs to be evaluated- Implementation and coordination of evaluations - Operational arrangements - Reporting and disseminating evaluation findings - Development of evaluation capacitiesContent

Institutionalizing Evaluation as a tool for planning and budgeting

Operational arrangements: roles and responsibilities It is proposed that, for institutionalizing evaluation in Indonesia, institutional arrangements should involve eight (8) key levels:

The Cabinet, The Performance Evaluation Commette, The National Planning Agency, (Deputy of Performance Evaluation and Deputy of Development Funding), The Ministry of Finace (DG Budget), The Line Ministries, the Agencies and Public Enterprises,The Parliament, control agencies and citizenry. The Nacional Bureau of Statistics BPS, and The Performance Evaluation Fund (financial tool).

Institutionalizing Evaluation as a tool for planning and budgeting

Operational arrangements: levels involved

Institutionalizing Evaluation as a tool for planning and budgeting

- Selecting programs to be evaluated- Implementation and coordination of evaluations - Operational arrangements - Reporting and disseminating evaluation findings - Development of evaluation capacitiesContent

Institutionalizing Evaluation as a tool for planning and budgeting

Reporting and disseminating evaluation findings - Initial purpose: to improve quality of existing reports by introducing performance-based orientation based on evaluation findings. Including:- Presidents Annual Report to Parliament- Quarterly and annual government plans reports and RAPBN documents.- Annual Budget Law (motivation)

Tailor evaluation reports to the needs and capacities of different publics. TOR should demand evaluators executive summaries and BAPPENAS (DPE) should traslate them to basic and friendly language

DPE must produce web reports on selected programs and disseminate them to a list of contacts strategically defined (media, universities, research organizations, key stakehoders and all goverment agencies at technical and directive levels).

Institutionalizing Evaluation as a tool for planning and budgeting

Reporting and disseminating evaluation findings Performance Evaluation Committee should present an annual report of carried out evaluations. It will brief main findings and actions taken.

- Elements of such informe should be included by MoF (DG Budget) as annex to the budget proposal sent to Congress

- Consolidated report should be published and web disclosured and sent to control agencies and, specially, to the media.

- Reports could also be distributed, as considered useful, at plans public consultations (Musrenbang)

- Seminars and discussions with specialized audiences are desired on impact evaluations, depending on their findings and implications.

- Aliances with civil society might create active demand for such reports.

Institutionalizing Evaluation as a tool for planning and budgeting

- Selecting programs to be evaluated- Implementation and coordination of evaluations - Operational arrangements - Reporting and disseminating evaluation findings - Development of evaluation capacitiesContent

Institutionalizing Evaluation as a tool for planning and budgeting

Development of evaluation capacities BAPPENAS (DPE),should prepare a DEC plan and present it to the Committee for discussion and approval. It might include:

Official definition of concepts, methods and tools to be used

Disemination means to all central government agencies (formal and informal)

Training agenda in the use of such methods and concepts to key sectors initially and then wide spread. It may require external support to implement (universities, experts, etc).

Funding required

Actions to creating capacity for responding on demand, and as methodologies are up-dated

Institutionalizing Evaluation as a tool for planning and budgeting

Development of evaluation capacities Competence mechanisms and procurement rules are a key aspect for the generation of capacities.

These should award expertice and know-how stimulating international/national partnerships for conducting evaluations.

The following criteria might be observed:

- Promoting competitive supply of Ntnal evaluators to establish a diversified expertice. - Establishing a national evaluators registry based on technical criteria - All national evaluators must be registered in order to be contracted (M. Requirements) - Evaluators should carry out specific actions for transferring capacities. - All information should be transfered to government agencies. - Information should be in all cases property of the government and PUBLIC.

MotivationRelevant international experienceBasic M&E definitions Scope of proposed methodsInstitutionalizing Evaluation as a tool for planning and budgetingOrganizing BAPPENAS Deputy of Performance EvaluationContent

Organizing BAPPENAS Deputy of Performance EvaluationObjectives

As technical head for results-based management and M&E activities BAPPENAS through this unit should contribute to: i) improving the effectiveness and impact of plans, programs and state agencies;

ii) increasing the efficiency and transparency of resource allocation; and

iii) promoting effective accountability and public control on governments performance.

Organizing BAPPENAS Deputy of Performance EvaluationMain functions

-Formulating policies, standards, methods and procedures for M&E and PBM.

-Support institutionalization of M&E in P. Sector

-M&E the performance of governments long (RPJP), medium (RPJM) and annual (RKP) term development plans, and the Presidents agenda

-Coordinating technical definition of goals and indicators for monitoring (RKK-KL) documents

-Proposing and coordinating implementation of a pluri-annual agenda of performance evaluations

-Guarantying that evaluation influences planning, budgeting and implementation -Preparing and disseminating periodic Ev. Reports -Enhancing governments performance information

-Promoting development of M&E capacities

-Providing technical support to provinces and district levels in M&E and PBM as required.

-Providing information for defining budget ceilings to integrate budgeting and planning

-Stimulating accountability through enhancing demand and supply of performance information

Organizing BAPPENAS Deputy of Performance EvaluationBasic structure

***********************************************