The political side of social program evaluation Ministry of Social Development MEXICO

The political side of social program evaluation

Ministry of Social Development MEXICO

Gonzalo Hernndez Licona

ObjectiveAnalyse the institutional challenges facing Mexico, specifically SEDESOL, in constructing a Monitoring and Evaluation (M&E) system.

How can we institutionalize an M&E system?The political constraints and challenges

How can we go beyond the impact evaluation of Oportunidades?: More programs evaluated in a regular basisConstructing a Results-based management systemInsert M&E within the Social Policy process

OutlineAnalytical framework

What type of M&E system were aiming for?

The need of creating and designing institutions in order to build an M&E system

What have we done and whats next?

Conclusions

Social PolicyIdentification of social problems and objectives

Analysis

Program designProgram operation and resourcesBudget

Monitoring and Evaluation

Evaluation: why and howHelps re-designing and improving programs

Supports efficient use of public funds

Adds objective and technical elements to the social policy debate

Promotes Transparency and social accountability

Emphasis on resultsWe need to measure indicators but consider qualitative evaluations(Long run) impact evaluation and (frequent) monitoring of every day operationExternal (and good) evaluatorsObjective and useful evaluations: search for programs stakeholders cooperation and participation

Decision taking Firm Social Program ProcessI. Identifying benefits

II. Measuring Impact

III. Information

IV. Who should evaluate?

V. Monitoring

VI. Who demands evaluations?

VII. What do we do with the results? Well-beingpoverty health infrastructure income nutrition perceptioneducation social cap. satisfaction I-C. IRRWhat would have happened without.. .? Counter-factual In order to know the whole processIt doesnt work usually looking for results ProfitsOwners; share-holdersEfficient use of informationResults: what for? Not clear Public resources In general preciseThe firm pays for it Lots of information Who pays for it? The firm itselfExternal auditors The programExternal evaluators

I. Identifying benefitsPrecise Rules of OperationBetter rules now Still problems due to the fear for auditors

We still have social programs that have only political objectives: Opciones productivas, PET, Acuerdos para el Campo: Vivienda rural, adultos mayores del campo

II. Measuring Impact: How?Guidelines for the annual Evaluation of Programs issued in 2002 by the Ministry of Finance and the Ministry of Audit (Contralora)The guideline indicates the need to measure impact in every program every year Very ambitious... but helpful in the short run to accelerate the creation of a culture of evaluation Pressure from donors ProgresaCIMOProbecat..

II. Measuring Impact: Promoting long-run impact evaluations

External support: technical and financial (WB, IDB, Conacyt, international academics) At least 9 impact evaluations in SEDESOL Progresa-Oportunidades 1997-2004Liconsa fortified milkMicrosimulation: Oportunidades, Liconsa, DiconsaFood programMicro-regions StrategyHabitatHousing program Tu CasaCoahuila States Piso FirmeJvenes con Oportunidades

II. Measuring Impact: The politics of the evaluation designProgresa: Centralized program with relatively little participation from beneficiaries or local authorities: it favoured randomization.Decentralized programs sometimes should seek for other methodologiesAn experimental design requires political supportWe tried to include in the SDL the possibility to have randomization, when feasible.

III. InformationFrom the institutional point of view it is not clear who should pay for the information The program?Sedesol?Hacienda?Donors?

Sometimes programs hide information from evaluators Next step: La Contralora

IV. Who should evaluate?Presupuesto de Egresos de la FederacinExternal Evaluators National evaluatorsInternational evaluators not allowed sometimes Creation in 2002 of the Under Secretariat of Planning, Prospective and Evaluation Social Development Law External evaluators The SDL explicitly bans consulting firms from evaluating social programs

IV. Who should evaluate?: The Human capital of External evaluatorsNot enough (good) evaluators Impact evaluationMonitoringA good evaluator should be able to evaluate a program with the existing ingredients: experiments are rare.Impact vs Monitoring Seminars (impact evaluation, monitoring, qualitative evaluation, power calculations, etc.) Were promoting partnerships between national and international evaluators

IV. Who should evaluate?: The bidding processThe bidding process favours the cheapest proposal Public universities dont have to go through the bidding process MonopoliesExcess transparency Changing the external institution every yearAnnual contractsIncentives to present good results in order to evaluate again in the future

V. MonitoringThere is no obligation to do this Indicators demanded by Hacienda, Funcin Pblica, Presidencia, Congress, with little management purpose: Highly inefficient

International support: WB, IDBCreate a true Results-based management system for every program New Direccin General de Evaluacin y Monitoreo de Programas Sociales, in order to promote internally the construction of a monitoring system

VI. Who demands evaluations?1. International donors IDB support: the need for evaluationReceptive authorities: Levy, Gmez de Len Internal battle (tcnicos vs rudos)

2.The opposition in Congress Since 2000, Congress demands annual external evaluations for every public program. Presupuesto de Egresos de la Federacin (PEF)In 1999 the PRI was not a majority in CongressThe opposition feared the use of social programs for the 2000 electionThere were good and reasonable academics in Government

VI. Who demands evaluations?The Social Development Law (2003-2004), institutionalizes the evaluation process National Council for the Evaluation of Social PolicyEvaluation of programs not only in SedesolPoverty measurementWere including in the Social Development Law (SDL) the obligation to evaluate every new federal program Law of Transparency and Public Access to Information Democracy

VII. What do we do with the results?

In the past almost nothing The results were useful to Progresa in order to survive, but it was not useful for the everyday management

A small institutional change: Our internal indicator % of programs evaluated every year% of external recommendations attended by programs The weaknesses and recommendations (summary) are officially sent to every program managerThe programs have to give an official answer on what actions will they implement The office of internal affairs (contralora interna), demands to see proofs of the actions taken every year

VII. What do we do with the results?

Evaluations are taken more into consideration

Better reports by evaluators

Programs make changes

Evaluations became this year a tool for the budget process within Sedesol We still need to link more closely (and formally, evaluations with the budget process)

VIII. The politics of the evaluation process: very importantIf we aim for an objective but yet useful M&E, we need to take into account stakeholdersWe need the participation of stakeholders involved with evaluationsThe DGEM published internal rules for the evaluation process: In the process operators should participate in TORs, analysis and reviewing sessions of developed workContinuous dialogue with external agentsDGEM runs the party

Decision taking Marks Social Program ProcessI. Identifying benefits

II. Measuring Impact

III. Information

IV. Who should evaluate?

V. Monitoring

VI. Who demands evaluations?

VII. What do we do with the results? Well-beingpoverty health infrastructure income nutrition perceptioneducation social cap. satisfactionWhat would have happened without.. .? Counter-factualIt doesnt work usually looking for resultsResults: what for? Not clear Public resources Lots of information Who pays for it? The programExternal evaluators????

ConclusionsThe Evaluation must be part of the Social Policy processBuilding a Monitoring and Evaluation System is a political task, that requires technical elementsIt is important to institutionalize the process and take into consideration the programs stakeholders for the evaluation processIn Mexico, it is crucial to build-up on the evaluators, policy-makers and congress officials technical abilities

Too many programs, not many evaluationsBetween 1990 and 2002 Mexico spent almost 550 billion dollars in social policyThere were very few evaluations: Probecat, Liconsa, Diconsa, ProgresaThe market itself doesnt solve the need for evaluation in social programs