3 5 7 28 29 76 90 From the Director’s iPad Rakesh Nangia Defining Capacity and Evaluation Capacity Development Towards a Shared Framework For National Evaluation Capacity Development Perspectives from Around the Globe Africa • Côte d’Ivoire • Ethiopia • South Africa • Uganda • CLEAR Initiative for Francophone Africa • African Development Bank Latin America Food for Thought • How to Speed up Global Development’s Learning Cycle • Developing Evaluation Capacity, but Which Capacity? • Checklist for Developing a National Evaluation System In This Issue: September 2013 eVALUatiOn Matters “It’s a bit like the Titanic: Do you want to course correct and miss the iceberg, or do you want to set up a comprehensive review of why it hit the iceberg and what lessons you could learn?” —Trevor Davies in How to Speed Up Global Development’s Learning Cycle A Quarterly Knowledge Publication of the Operations Evaluation Department of the African Development Bank Group
100
Embed
Evaluation Matters - How can we Strengthen National ......• Strengthening National Evaluation Systems Samuel Kouakou, Côte d’Ivoire • The importance of Strengthening National
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
3
5
7
28
29
76
90
From the Director’s iPadRakesh Nangia
Defining Capacity and Evaluation Capacity Development
Towards a Shared Framework For National Evaluation Capacity Development
Perspectives from Around the Globe
Africa •Côted’Ivoire •Ethiopia •SouthAfrica •Uganda •CLEARInitiativeforFrancophoneAfrica •AfricanDevelopmentBank
Latin America
Food for Thought • HowtoSpeedupGlobalDevelopment’s
How can we Strengthen National Evaluation Systems?
—Perspectives from around the globe
“It’s a bit like the Titanic: Do you want to course correct and miss the iceberg, or do you want to set up a comprehensive review of why it hit the iceberg and what lessons you could learn?”
—Trevor Davies in How to Speed Up Global Development’s Learning Cycle
A Quarterly Knowledge Publication of the Operations Evaluation Department of the African Development Bank Group
A QUARTERLY KNOWLEDGE PUBLICATION OF THE AFRICAN DEVELOPMENT BANKSeptember 2013 . Volume 2 . Number 3
3 From the Director’s iPad Rakesh Nangia
5 Defining Capacity and Evaluation Capacity Development
7 Towards a Shared Framework For National Evaluation Capacity Development
Marco Segone, Caroline Heider, Riitta Oksanen, Soma de Silva, and Belen Sanz
26 Framework for Understanding NECD Definitions and Terminology
28 Perspectives from Around the Globe
29 Africa• StrengtheningNationalEvaluationSystems
Samuel Kouakou, Côte d’Ivoire• TheimportanceofStrengtheningNationalEvaluationSystems
F. Kelemework Fasika, Ethiopia• DevelopingaNationalEvaluationSysteminSouthAfrica,
Ian Goldman, South Africa• PatternsandInfluencesintheSupplyandDemandofEvaluation
andMonitoringinUganda’sPublicSectoroverthePastTwoDecades David Rider Smith, Uganda• StrengtheningNationalEvaluationSystems
El Hadji Gueye, Senegal• TowardsanEvaluationCultureintheAfDB:Practicesand
Fiscal pressures, greatly magnifiedbytheglobalfinancialcrisis,brought not only pressure on officialdevelopmentassistancesupportbutalsoincreaseddemandsfromexternalstake-holders.Thefocusonresultsisunprec-edented,andgoeshand-in-handwiththedemandforuseable,results-basedmonitoringandevaluation(M&E)systemstoprovidetangibleevidenceofoutcomes,aswellassupportforevidence-baseddecisions.
Ideally, we should begin with an analysis of the existing M&E system, including identifying the gaps. This should, of course, be done with counterparts in govern-ments, which one would hope includes some champions.
Rakesh Nangia is the Director of the Operations Evaluation Department of the African Development Bank. Prior to joining the AfDB, he spent 25 years at the World Bank, where he held several positions includ-ing Director of Strategy and Operations for the Human Development Network and Acting Vice-President for the World Bank Institute.
He attended the Indian Institute of Technol-ogy in Delhi and Harvard University and holds degrees in business administration and engineering.
Defining Capacity and Evaluation Capacity
Development
5A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
the united nations Evaluation Groupdefinescapacityasthe
7A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Marco Segone, UNICEF Evaluation Office, Co-chair EvalPartners, Co-chair UNEG Task Force on National Evaluation Capacity Development
Caroline Heider, Director General and Senior Vice-President, Independent Evaluation Group, World Bank
Riitta Oksanen, Ministry for Foreign Affairs, Finland; Chair OECD/DAC Task Force on Evaluation Capacity Development
Soma de Silva, President, IOCE, and Co-chair EvalPartners
Belen Sanz, Chair, UNEG, and Head, Evaluation Office, UN Women
This paper was published by EvalPartners in the book Evaluation and Civil Society.
Reprinted here by permission
IntroductionNational ownership and leadership as overarching factors for ensuring relevant development outcomesTheParisDeclaration onAid Effectivenessendorsedin2005,andthe2008Accrafollow-upmeeting,statethatnationalownershipandleadershipareoverarchingfactorsforensuringgooddevelopmentoutcomes.Theimplicationfortheevaluationfunctionisfundamental.
National evaluation capacities as part of good governanceAscountriestakegreaterownershipof,andleadershipin,theirdevelopmentprocesses,theyhavealsoincreasinglydevelopedtheirsystemsto lead,manage and account for resourcesinvestedintheseprocessesandresultspro-ducedwiththem.Results-basedbudgetingandmanagementhavebeenpartoftheagendatostrengthengovernanceandgohandinhandwithcapacitiesformonitoringandevaluation.Evaluation capacities empower stakehold-ers—fromnationalgovernmentstoCSOs—toquestion,understandandtakechargeofnecessarychangestodevelopmentprocessesasandwhenevidenceindicatespolicies,pro-gramsandprojectscanbemoresuccessful,
In a number of coun-tries, public policy evaluation functions are in place. Examples are those set up through the implementation of Poverty Reduction Strategy Papers, which link monitoring and evaluation systems to poverty reduction.
9A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
The growing role of civil society organizations in national evaluation capacity developmentAlongthelinesoftherecentBusanDeclaration,CSOscanandshouldplayacentralroleinadvo-catingfortransparencyintheallocationandexpenditureofpublicbudgets;accountabilityfortheimplementationofpublicpolicies;strength-eningthedemandanduseofevaluationtoinformevidence-basedpolicy-making;and,strengthen-ingcapacitiesofqualifiedevaluatorstoproducecredibleandusefulevaluationsbasedonnationalandinternationalevaluationstandards.
While national evaluation capacity is oftenunderstoodasgovernmentcapacity,itshouldbeacountry-basedcapacity,includingVoluntaryOrganizations for Professional Evaluation2 (VOPEs),universities,thinktanksandNon-governmentalOrganizations(NGOs).CSOsandparliamentariansneedevaluationstosupporttheirunderstandingofissuesandparticipationindecisionmaking.Thiscanstrengthenthequal-ityofdemocracywherebyinformedcitizensareabletoinfluencedecision-making.
In this context, 25 organizations launchedEvalPartners4,aninternationalcollaborativeinitiativetocontributetotheenhancementofthecapacitiesofCSOs—notablyVOPEs—toinfluencepolicy-makers,publicopinionandotherkeystakeholderssothatpublicpoliciesareevidence-based,equitableandeffective.ThemainobjectiveistoenhancethecapacitiesofCSOs/VOPEstoengageinastrategicandmean-ingfulmannerinnationalevaluationprocesses,tobeabletoinfluencecountry-ledevaluationsystems.
Towards a shared framework for national evaluation capacity developmentIn addition to governments, VOPEs andNGOsatcountrylevel,amultitudeofstake-holders is currently engaged in supportingNationalEvaluationCapacityDevelopment:
2 Sinceanumberofdifferentnamesareusedtodescribethesegroups, the term Voluntary Organizations for ProfessionalEvaluation(VOPEs)hasbeenintroduced.VOPEsincludeformallyconstitutedassociationsorsocieties,aswellasinformalnetworksandcommunitiesofpractice.Theirmembershipsareopennotonlytothosewhoconductevaluationsbutalsotothosewhocommissionandutilizeevaluationsandthoseengagedinbuildingtheevaluationfield.
theUnitedNationsEvaluationGroup(UNEG)anditsmembers,theEvaluationCooperationGroup(ECG)oftheInternationalFinancialInstitutions(IFIs),the Development AssistanceCommittee (DAC) of theOrganisation forEconomic Cooperation and Development
(OECD),theregionalCLEARcenters,amongothers.However, in certain cases differentactorshavenotcoordinatedtheirsupportandinitiatives,makingitdifficulttocreateposi-tivesynergiesandefficiency.Inextremecases,duplicationsandparallelprocessesaresup-portedbydifferentagencies.Therefore,there
Defining Capacity and Evaluation Capacity DevelopmentThe United Nations Development Group(UNDG)definescapacityastheabilityofindi-viduals,institutionsandsocietiestoperformfunctions,solveproblems,andsetandachieveobjectivesinasustainablemanner,andcapacitydevelopmentastheprocessthroughwhichtheabilitiestodosoareobtained,strengthened,adaptedandmaintainedovertime.
ForOECD,evaluation capacities aretheabil-ityofpeopleandorganisationstodefineandachieve their evaluation objectives (OECD,
Evaluation Capacity Development (ECD)isunderstoodas theprocessofunleashing,strengthening and maintaining evaluationcapacities.Capacitydevelopmentisalong-term,endogenouschangeprocessthattakesplaceinthecontextofongoingpartneranddonoreffortstostrengthenrelatedsystemsofmanagement,governance, accountability and learning, toimprovedevelopmenteffectiveness.Thebestcapacitydevelopmentapproachesareflexible,adaptedandsustainable.
The United Nations Development Group (UNDG) defines capacity as the ability of individuals, institutions and societies to perform functions, solve problems, and set and achieve objectives in a sustainable manner, and capacity development as the process through which the abilities to do so are obtained, strengthened, adapted and maintained over time…For OECD, evaluation capacities are the ability of people and organizations to define and achieve their evaluation objectives.
11A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
A Systems Approach to National Evaluation Capacity DevelopmentNational Evaluation Capacity Development(NECD) is a complex field inwhich severalstakeholdershavedifferentrolestoplaybasedontheirrespectivevalueadded.Thiscomplex-ityencouragestheuseofasystemsapproachtoNECD.Thismeansthatitisnecessarynotonlytolookatactorsatdifferentlevelsandacross
Individual and institutional evaluation capacities enabled by a supportive environmentInthepast,evaluationcapacitydevelopmentfocusedonstrengtheningthecapacitiesofindi-viduals’knowledgeandskills.However,itisbynowclearthatcapacitydevelopmentshouldbebasedonasystemicapproachthattakesintoaccountthreemajorlevels(individual,institu-tional,andexternalenablingenvironment),and
Figure 1: A systemic and integrated approach to national evaluation capacities development
Individual Level
EnablingEnvironment
InstitutionalLevel
Equity-focused and Gender-responsive
Dem
and S
up
ply
– Tailored to the specific context of each country– Country leadership and ownership
12 eVALUatiOn Matters
Enabling Environment: the enabling environment provides a context that fosters (or hinders) the performance and results of individuals and organizations
Institutional Level: the institutional framework in which individuals work needs to provide a system and structures to perform and attain results individually and collectively, as an organization
Theenabling environment for evaluation isdeter-minedbyacultureoflearningandaccountabil-ity,meaningthedegreetowhichinformationissoughtaboutpastperformance,theextenttowhichthereisadrivetocontinuouslyimprove,andtoberesponsibleoraccountableforactionstaken,resourcesspent,andresultsachieved.Suchcultureisembeddedintacitnormsofbehavior,theunderstandingofwhatcanandshould—orshouldnot—bedone,inmanycasesbehaviorsbeingrole-modeledbyleaders.Throughasetofvaluesandattitudessupportingevaluative(critical)thinkingwithinanorganization,indi-vidualsaremoreself-directedlearnersanduseinformationtoact;totakehigherrisksbut,alsotodevelopagreatersenseofpersonalaccount-abilityandresponsibility;and,toconsult,coach,andsupporteachothermore.Inthiscontext,organizationswithacultureofevaluationareabletodevelopinnovativeideasandstrategies;changemorequicklyaccordingtovariationsintheexternalenvironment;and,increaseeffi-ciencyandeffectivenessbysystematicallyusing
An enabling environment is also supportedorcreatedthroughgovernancestructuresthatdemandindependentevaluation,beitthroughparliamentsorgoverningbodies,andisfurtherenhancedthroughVOPEsthatsetstandardsandstrivetowardsgreaterprofessionalisminevalu-ation.Therefore,VOPEsshouldbesupportedtoenablethemtofosterindigenousdemandandsupplyofevaluation,includingbysettingnationalevaluationstandardsandnorms.TherearealsoexamplesofgovernmentssolicitingtheadviceandinvolvementofVOPEsinnotonlytheformulationofevaluationpoliciesandsystems,butalsointheimplementationofevaluationsconsistentwiththosepolicies(PresidencyofSouthAfrica,2011).
The institutional framework for evaluationensuresthatasystemexiststoimplementandsafeguardtheindependence,credibilityandutil-ityofevaluationwithinanorganization.Suchaninstitutionalframeworkhasthefollowingcharacteristics:
• Includes a system of peer review orassur-ancethattheevaluationfunctionissetuptosafeguardandimplementtheprinciplesofindependence,credibilityandutility.
• Establishes safeguards to protect individual evaluators—evaluators,evaluationmanagers,andheadsofevaluationfunctions—whenexercisingtheir independence, includingtransparentandcredibleprocessesfortheselection,appointment,renewal(ifapplica-ble)andterminationofcontractoftheheadofevaluationandassurancesthatevaluationstaffdonotsufferinpromotionexercises.
• Ensures a multidisciplinary team existsintheevaluationfunction,oronanevaluationteam,thatensurescredibilityofevaluationbyunderstandingmultipledimensionsofevalu-ationsubjectsandcombiningthenecessarytechnicalcompetence.
• Secures the independence of funding of evalu-ations,atanadequatelevel,toensurethatnecessaryevaluationsarecarriedoutandthatbudgetholdersdonotexerciseinap-propriate influenceor controloverwhatisevaluatedandhow.Therefore,fundingshouldbeunderthedirectcontroloftheheadoftheevaluationfunction,andshouldbeadequateforareasonableworkprogram.Theadequacyoffundingforevaluationcanbedeterminedthrougharatioofevalua-tionresources(humanandfinancial)overthetotalprogramsize(someorganizationsaimtoallocate1to3percentoftheirtotalbudgetforevaluation)orthecoveragerateofevaluations(forinstance,evaluating25percentoftheoverallprogram).
• Combines measures for impartial or purpo-sive selection ofevaluationsubjectstoensureimpartiality,ontheonehand,andincreasedutility,ontheother,bymakingdeliberatechoiceslinkedtodecision-makingprocesses.
15A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
• Sets out a system to plan, undertake and report evaluation findings in an independ-ent, credible and useful way.To increaseobjectivityintheplanningandconductofevaluation,systemsareneededthatincreasetherigor,transparencyandpredictabilityofevaluationprocessesandproducts.Suchsystemscanincludemoreorlessdetailedprocessdescriptionsorguidelinesforthedesignofevaluations,forpreparatoryworkthatneedstobeconductedandreportedon,andforreportingfindings.Theprocessesshouldhavebuilt-instepsforcommunica-tion,consultationandqualityassurance,andbecommunicatedtostakeholderstoenhancetransparencyandtosecuretheirwillingnesstoshareinformation.
• Institutes measures that increase the useful-ness of evaluations, includingthesharingoffindingsandlessonsthatcanbeappliedto other subjects. Evaluations should beundertakenwiththeintention—ofstake-holdersandevaluators—tousetheirresults,andmanagement’sresponsesdevelopedandimplemented.Thetimelinessofplanningandconductingevaluationsandpresent-ingtheirfindingsisequallyimportantto
Evenwithstructuresandsystemsinplace,theindependenceandimpartialityofevaluationdependsontheintegrityandprofessionalismof individualsasevaluators,evaluationmanag-ers,andtheheadofevaluation.Theprofession,anditsethics,requireslimitingpersonalbiasestotheextentpossible.Insomecircumstances,external evaluators are believed to exercisegreaterindependencethanthosewhoworkinanorganization,becausetheyarelessexposedtoinstitutionalorpeerpressuresorhavenotabsorbedthecorporatecultureinanunques-tioningway.However,individualorintellectualindependenceisdependentontheindividual,whodemonstratesinhis/herbehaviortheadher-enceto,andpracticeof,theevaluationprinciples:avoidingconflictofinterest;actingwithintegrityandwithanindependenceofmind;engaginginevaluationsforwhichtheyarecompetent;actingimpartially;and,undertakinganevalu-ationwithaclearunderstandingoftheclientsordecisionmakingprocessandhowtheseneedtobeinformed(seeAEA,2003).ManyVOPEsandevaluationfunctionsofnationalandinter-nationalorganizationshaveadoptedcodesofconductforevaluators.Inadditiontothese,theUNEvaluationGroupdevelopedethicalguide-lines(UNEG,2005).Debatesaboutprofessionalstandardsandtheaccreditationofevaluatorsandevaluationmanagershavebeenongoingformanyyearswithinprofessionalforums.
16 eVALUatiOn Matters
Development of national evaluation capacityAninstitutionalframeworktoinstitutionalizetheevaluationfunctionshouldbedevelopedwithintheorganization.Thisshouldincludeassistancetodevelopanevaluationpolicy,whichforeseesthesetting-uporstrengtheningofacredibleevaluationdepartment,includingendorsementofevaluationstandardsandqualityassurancesystems(includingpeerreview).Adiagnosisoftheexistingevaluationfunctionand/orsystem(or,ifnotexisting,ofthepreconditionstodevelopit),whichwouldincludefunctionalclarity,effec-tivehumanandfinancialresourcesmanage-mentandarobustcoordinationmechanism,shouldbecarriedouttoenablethecraftingofacontext-specificevaluationpolicyandsystem.
Attheindividuallevel,acapacitydevelopmentstrategyshouldstrengthensenior management capacity to strategically plan evaluations(andtoidentifythekeyevaluationquestions);tomanageevaluationforindependenceandcredibility;and,to use evaluation.
MacKay(2007)underlinestheimportanceofidentifying and supporting leaders or natural champions whohavetheabilitytoinfluence,inspireandmotivateotherstodesignandimple-menteffectiveevaluationsystems.Leadershipisnotnecessarilysynonymouswithapositionofauthority;itcanalsobeinformalandbeexer-cisedatmanylevels.Therefore,theevaluationcapacitydevelopmentstrategyshould,especially
inthe initialstages, identifyandsupportasappropriate,nationalandlocalleadersinthepublicadministration,ininter-governmentalmonitoring, and in evaluation groups andnationalVOPEs.
19A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Fostering demand for and supply of evaluationAdistinctionshouldbemadebetweenthecapacityofpolicy-makers/policyadvisorstouseevidenceandthecapacityofevaluationprofessionalstopro-videsoundevidence.Whileitmaybeunrealisticforpolicy-makers/policyadvisorstobecompetentexpertsinevaluation,itisbothreasonableandnecessaryforsuchpeopletobeabletounderstandanduseevidenceproducedbyevaluationsystemsintheirpolicyandpractice.Integratingevidenceintopracticeisacentralfeatureofpolicy-makingprocesses.Anincreasinglynecessaryskillforpro-fessionalpolicy-makers/policyadvisorsistoknowaboutthedifferentkindsofevidenceavailable;howtogainaccesstoit;and,howtocriticallyappraiseit.Withoutsuchknowledgeandunderstandingitisdifficulttoseehowastrongdemandforevidencecanbeestablishedand,hence,howtoenhanceitspracticalapplication.
• Vicious circle countries.Evidenceistech-nicallyweakandpolicymakershavelittlecapacitytomakeuseofit.Policydecisionsaremainlytakenonpoliticaljudgmentandpersonalexperience,whichmayresultinpoorpolicydesignand,consequently,poorresults.Inthiscase,itisnecessarytoadoptmeasureswhichwillsimultaneouslyincreaseboththedemandandsupplyofevidence,aswellasimprovethedialoguebetweenpro-ducersandusersofevidence.
• Evidence demand-constrained countries.Thequantityandqualityofevidenceisimprov-ing,butitisnotdemandedfordecisionmak-ingbecausepolicy-makerslacktheincentivesand/orcapacitytoutilizeit.Policy-makersarelikelytobeattheveryleastwaryof(ormayevenactivelydislike)havingmoreandbetterevidencepushedatthemwhenitmaynotsupportdecisionstheyhavealreadytakenorwishtotake.Inthiscase,priorityshouldbe given to the adoptionofmeasures to
The quantity and quality of evidence is improving, but it is not demanded for decision-making because policy-makers lack the incen-tives and/or capacity to utilize it.
21A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
• Virtuous circle countries.Evidenceistechni-callyrobustandisbeingusedincreasinglyfordecision-making.Theproductionofgood(oratleastimproved)evidenceismatchedbyitswidespread(oratleastincreased)useindecision-making.Thesetwoprocessesaremutuallyreinforcing,resultinginbet-terpolicydesignandimplementation,andultimately,betterdevelopmentresults.
Key Principles of National Evaluation Capacity DevelopmentThenumberofstakeholdersandtherangeofcapacitiesandcapabilitiesdiscussedaboveillus-tratethecomplexitiesofcapacitydevelopmentandflagtheneedforplanningandimplemen-tationtoolsthatareabletoaddresscomplexity,somethingthatblueprintstendnottobegoodat.Instead,spaceneedstobecreatedforanalyzingcapacitygapsinaparticipatoryway,agreeingoncommongoalsandtranslatingthemintoajointstrategy.Atthesametimetheflexibilityandcapabilityshouldbemaintainedtotakeupopportunitiesastheyarise,learnfromexperi-enceandchangetactics,andworkonvariouslevelsofcapacityatthesametime.
Theprinciplesofnational ownership and leader-ship havetodowiththerealizationthatcapacitydevelopmentneedstocomefromwithinand/orhavestronginternalchampions,whetheritiswhendevelopinganationalsystemorafunctioninanorganization.Ifanideaisimposedfromtheoutside,chancesarethatcapacitydevelopment
a) Ownership,whichmanifestsitselfinhav-inglocalorinternalchampions,resourceallocations,andengagement.Itis,however,difficulttomeasure,nothomogenousacrossinternalstakeholders,andnotconstantovertime;
b) Leadership,whichisimportant,especiallyforsettingthecultureoflearningandaccount-ability,butalsomustfitwiththecontextanditsculture;and
c) Collectiveaction,motivationandcommit-ment,meaningthatcapacitydevelopmentwillnotoccur if it isonepersonwritingdocuments,butrequiresengagingvariousstakeholderswhosecapacitiesandcapabili-tieswillbedeveloped.
22 eVALUatiOn Matters
Capacity development should therefore beunderpinnedbythefundamentalcharacter-isticofnationalownership.Takingacapacitydevelopmentstrategytoscalerequireslinkingittonationalandlocalplans,processes,budgetsandsystems.Tobesustained,acomprehensivecapacitydevelopmentresponsemustlinkto,anddrawfrom,relevantnationalreforms.
The process of developing and agreeing onclearexpectationsin“country-specific”capac-itydevelopmentisinitselfpartofthecapacitydevelopmentprocess.Developingacommonunderstandingofcapacityweaknessesthroughaparticipatory,structureddiagnostic—usingtoolsliketheconceptualframeworkinFigure1(p.11)—canserveasananalyticalframeworktomoveattentionfromresourcegapstorecognizebroaderissues,andasharedvisionoftheneededcapacitiesareimportantstepsinchannellingresourcestowardsclearobjectives.Veryoften
23A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Aligning with national evaluation capacity development strategies and systemsStrongnationalownershipandleadershiparethefoundationforaligningexternalsupportthatleadstosustainableresults.Nationalevalu-ationcapacitydevelopmentstrategiesprovidethestartingpoint.Whenthenationalstrategiesareofhighqualityintermsofhavingclearpri-orities,beingresults-orientedandoperational,theyareanaturalbasisforcooperationandalignment.Anotherimportantindicationofnationalcommitmentistheresourceallocationsintheinstitutions’budgets,andstaffingfortheimplementationofthecapacitydevelopmentstrategies.
Use of external expert inputs (technical assistance) for demand- driven evaluation capacity developmentCapacitydevelopmentimpliesintentionalitytostrengthencapacities.Forexample,technicalassistancemaystrengthencountrycapacitiesonlyifitisclearlyorientedtowardsthisaim,byengagingcountrystakeholdersintheevaluation
Initiatives and institu-tions for national evaluation capacity development have rapidly increased both in the public sector and in civil society.
25A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Working jointly for evaluation capacity development led by national institutionsInitiativesandinstitutionsfornationalevaluationcapacitydevelopmenthaverapidlyincreasedbothinthepublicsectorandincivilsociety.Therearealsosignsofgrowinginteresttosupporttheseefforts.Theexpandingnumberofpartnershipsforevaluationcapacitydevelopmentbringswithitthetraditionalchallengesofcoordinationandpromo-tionofjointwork.Itisimportantthatnationalinstitutionsexerciseleadershipincoordinatingexternalsupport.Thisimpliesanactiverolefor
Managing for resultsCapacitydevelopmentmonitoringandreportingoftensufferifthefocusissimplyonwhathasbeendoneandimmediatelyachieved.Reportsontrainingsessionsorganizedandnumbersofpeopletrainedarefamiliartoall.Ashifttoemphasizingmediumandlongertermresultsisnecessary.Theimportantthingistoreportonwhatactuallychangesinindividuals’knowl-edge,skillsandwill,ininstitutionsandintheoperatingenvironmentasaresultofcapacity
ConclusionsIn the past, several stakeholders worked tostrengthenNationalEvaluationCapacitiesfocusingontrainingindividuals,ratherthanstrengtheninginstitutionalcapacitiesandanenablingenviron-ment.Inaddition,thankstothechangingexternalenvironmentandtherecognitionandinvolve-mentofnewstakeholders(notablyCSOs,including
Evaluation function of Africa, JSukai Prom-Jackson,
Joint Inspection Unit of the United Nations System
LATIN AMErICA • SpeakingTruthtoPower:TheRoleofNationalEvaluation
Systems in Democratic Governance, Claudia Maldonado
• JointInspectionUnitoftheUnitedNationsSystem
Perspectives From
29A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Samuel Kouakou, Deputy Director of Evaluation and Projects Sustainability, Directorate of Evaluation and Projects, Ministry of Agriculture, Côte d’Ivoire.
Why is it Important to Strengthen National Evaluation Systems?The Paris High-Level Forum on AidEffectiveness, hosted by the Frenchgovernment and organized by theOrganisationforEconomicCo-operationandDevelopment(OECD)inFebruary2005,andtheThirdHigh-LevelForumonAidEffectiveness(HLF-3),heldinAccra,Ghana,September2–4,2008,havegivenarenewedimpetustomonitoringandevaluation(M&E).
Why Should We Invest Time and Effort in National Evaluation Capacity Development?Itisimportanttoinvesttimeandeffortinnationalevaluationcapacitydevelopmentbecausecapacitydevelopmentisastrategicinvestmentinthefutureofthecivilservice.Thiscanenhancethewell-beingofcitizensandtheprosperityofthecountry.Itcanalsoimprovegovernmentperformance.1
2 N.Holvoet,S.Dewachter,M.Gildemyn.Strengthening National Evaluation Societies: An Exploratory Survey,2011(Manchester,U.K.:InstituteforDevelopmentPolicyandManagement,2012).3 J.RughandM.Segone,eds.Voluntary Organizations for Professional Evaluation (VOPEs): Learning from Africa, Americas, Asia, Australia, Europe, and Middle East (EvalPartners, UNICEF, IOCE, 2013), pp. 14–15.
IttakestimetocreateorstrengthennationalevaluationcapacitiesandtobuildsystemsforsharingM&E information among relevantministries.
Good practices, tipsExtensiveexperiencewithcapacitydevelopmentinitiativeshashighlightedthreemajorlessonslearned:
• Capacity development should be under-pinnedbythefundamentalcharacteristicofnationalownership.Takingacapacitydevelopmentstrategytoscalerequireslink-ingthestrategytonationalandlocalplans,reform,processes,budgets,andsystems.
• Capacitydevelopment is about transfor-mations and must address how best tomanagechangewithintheexistingpolicyenvironment.
Examples of models of national evaluation systems that are performing wella) South Africa caseInabout2000,M&EinSouthAfricatookonincreasedprominence.6TherehadbeenM&Eactivitysince1995,butitwasnotcoordinated.In2005,theSouthAfricancabinetapprovedrecommendationsfromthePresident’sOfficeon“animplementationplantodevelopamonitor-ingandevaluation(M&E)system.”Thesystemwastoincludefunctionssuchasmonitoring;evaluation;earlywarning;dataverification;anddatacollection,analysis,andreporting.
5 “DesignofMonitoring&EvaluationSystemforStrengtheningthe Sri Lankan Vocational Training System by GTZ SVTPProject—Colombo.”6 IndranNaidoo.Unit:“TheMonitoringandEvaluationinSouthAfrica.ManyPurposes,MultipleSystems.”On-linee-learningprogramoncountry-ledM&Esystems.
b) Australian caseTheelectionofareformistlaborgovernmentin1983providedanenvironmentfavorabletoevaluation.Aseriesofpublic-sectormanagementreformswereimplementedinthefirstseveralyearsofthenewgovernment.Oneaspectofthesereformswasthedesiretolet the managers managebyadevolutionofpowersandresponsi-bilitiesthatreflectedthephilosophythatpublicsectormanagerswouldbestronglyencouragedtoimprovetheirperformanceiftheywerepro-videdgreaterautonomyandthepotentialtomanagetheirdepartmentswithfewercentralagencycontrolsandlessinterference.Thecabi-net’sagreementtotheevaluationstrategy8wasexpressedinaformalcabinetdecision.
7 RonettaEngelaandTaniaAjam.”ImplementingaGovernment-wide Monitoring and Evaluation System in South Africa”(Washington,DC:IEGECDWorkingPaperNo.21,July2010).8 KeithMackay.“TheDevelopmentofAustralia’sEvaluationSystem”(Washington,DC:IEGECDWorkingPaperNo.4,May1998).
c) Canada caseEvaluationinpublicsectormanagementextendsbackto1969inCanada,9withtheinitiationofformalizedandcentralizedevaluationpractices.Theintroductionofthepracticeofevaluationwasinspiredbythenotionoflettingthemanag-ersmanage;thatis,allowingdeputyministersoffederalgovernmentdepartmentstoassumegreaterresponsibilityfortheirdepartmentsandprograms.Buttheywerealsotobeheldaccount-ablefortheperformanceofthoseprogramsandtheprudentuseofpublicfunds.Themodelisbasedonastrongcentralmanagementboardthatoverseesandholdsdeputiesaccountable,andoneofthemechanismsintendedtodothisistheperformanceevaluation.
The2009EvaluationPolicyhasputrenewedemphasis on evaluating all direct programexpendituresofgovernmentand,intheprocess,focusingontheissuesofprogramrelevanceandperformance—thatis,effectiveness,efficiency,andeconomy.Theintentisthatevaluationwillbe
Over the past two decades, the World Bank and other inter-national financial institu-tions, and more recently some prominent UN agencies, have devoted resources to building evaluation capacity in selected countries.
33A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
d) Columbia caseColombia’snationalM&Esystem(SINERGIA)isoneofLatinAmerica’smorevisibleeffortstoinstitutionalizeM&Eatthegovernmentlevel.
Establishedin1994,followingthemandatesoftheNationalConstitution,SINERGIA,10thenational system for evaluation of management and results,hasendurednumerousperiodsofprogress,stagnation,andsetbacksbecauseoftheinstitutional,political,andfiscalchallengesfacedbythecountryovertime.Nonetheless,SINERGIAhasattainedasignificantlevelofconsolidationandcustomizationandisoftenreferredtoasagoodpracticesystembymulti-lateralorganizations,donoragencies,andaca-demicanalysts.
Finally,itappearsthatmanyfactors,suchasthestrengthofleadershipforthereform,theresourcesprovided to support it, the estab-lishmentofroutinerulesandprocedures,and
The new global realityOverthepasttwodecades,theWorldBankandotherinternationalfinancialinstitutions,andmorerecentlysomeprominentUNagencies,havedevotedresourcestobuildingevaluationcapacityinselectedcountries.However,inaworldwhereresourcesaretighterinallglobalsituations,thecontextfornationalevaluationcapacitydevelop-menthaschangedforUNagenciesandpartnercountriesalike.ThenewglobalrealityisthatUNagencies,bothontheirownandunderthecoor-dinatingbodyoftheUnitedNationsEvaluationGroup,willneedtobequiteselectiveinsearchingforthemostcost-effectiveapproachtosupport-ingnationalevaluationcapacitydevelopment.
AlongthelinesoftherecentBusanPartnership,11 civilsocietyorganizations(CSOs)canandshouldplayacentralroleinadvocatingfortransparencyintheallocationandexpenditureofthepublicbudget,accountabilityfortheimplementationofpublicpolicies,andthestrengtheningofthedemandfor theuseofevaluation to informevidence-basedpolicymaking,andthecapaci-tiesofqualifiedevaluatorstoproducecredibleandusefulevaluationsbasedonnationalandinternationalevaluationstandards.
a) The Ivorian M&E Network (RISE)RISEhasgainedofficialgovernmentrecogni-tionandispresentatallthemajormeetingsorganizedbytheAfricanEvaluationAssociation(AfrEA).
c) The International levelAsamemberoftheEvalPartnersTaskForceontheenablingenvironment,Iamdeeplyinvolvedintheimplementationofactivitiestoadvocateandpromoteevaluationandevidence-basedpolicymakingattheinternational,regional,national,andlocallevels,andindesignating2015astheInternationalYearofEvaluation(EvalYear).
Manyothercapacity-buildingactivitieswillbe conducted by the European EvaluationSociety(EES)WorkingGrouponSustainableDevelopment, as well the InternationalDevelopmentEvaluationAssociation(IDEAS)
35A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Urgent issues in national evaluation capacity developmenta) The issues of certification of evaluatorsTheprofessionoftheevaluatorisyoung.Thereisnoprofessionalorderofevaluators.Therearenationalevaluationassociationsinseveralcoun-tries,generallyindustrializedcountries.
ThenewlyaccreditedCLEARcenters in theLatinAmerica,Africa,andAsiaregionsthatareplanningtoofferuniversitycertificationsinthefuturewillassuredlyhelpaddresstheseissues.
b) The issues of enabling environmentWemustensurethat:
• Public administration is committed totransparencyandmanagingforresultsandaccountabilitythroughresults-basedpublicbudgetingandevidence-basedpolicymaking.
Key elements for successful national evaluation capacity developmentInthepast,evaluationcapacitydevelopmentmainlyfocusedonstrengtheningindividualcapacities.However, it isclear thatcapacitydevelopmentshouldbebasedona systemicapproachthattakesintoaccountthreemajorcomponents:theindividual,institutional,and,externalenablingenvironments.12
12 UNICEF. “Evidence for Children: Developing NationalCapacitiesforCountry-LedEvaluationSystems,AConceptualFramework.”(NewYork:UNICEF,July2010).13 J.RughandM.Segone,eds.Voluntary Organizations for Professional Evaluation (VOPEs): Learning from Africa, Americas, Asia, Australia, Europe, and Middle East (EvalPartners, UNICEF, IOCE, 2013), pp. 14–15.
Actions that would strengthen evaluation capacity developmentAcountrystrategyfornationalevaluationcapac-itydevelopmentshouldstrengthentheenablingenvironment bysupporting:
Agoodinitiativethatwillstrengthenevalu-ationcapacitydevelopment is theorganiza-tionParliamentariansForumonDevelopmentEvaluation in SouthAsia.14 The forumwasinitiatedbyasmallgroupofparliamentariansaimingtoestablishnationalevaluationpoliciesintheSouthAsianAssociationforRegionalCooperationregiontoensureaideffectiveness,achievementofresults,andsustainabilityofdevelopment.
Mr. Samuel Kouakou is Deputy Director of Evaluation and Projects Sustainability in the Directorate of Evaluation and Projects of the Ministry of Agriculture, Côte d’Ivoire.
He has extensive experience in evalua-tion in public and private service at the national and international levels. He is an active member of the African Evaluation Association, International Development Evaluation Association, European Evaluation Society, and the Ivorian M&E Network (RISE).He holds a diploma in international environmental law from the United Nations Institute for Training and Research (UNITAR), a master’s degree in rural engineering, as well as several post-graduate certificates in the areas of integrated water resources management, gender mainstreaming, and results-based M&E.
Mr. Kouakou is currently e-learning visit-ing lecturer at the International Institute for Water and Environmental Engineering . He earned his e-learning tutor certifica-tion from the Association of Universities of the Francophonie.
37A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
F. Kelemework Fasika, European External Action Service-Addis Ababa
since the initiation ofPovertyReductionStrategyPapers(PRSPs),povertyreductionhasbecomethemainfocusofthenationalpoliciesandstrategiesofmostdevelopingcountries.Moreover,thisfocushasbeensharedbyregionalandglobaldevelopmentplayersforquitesometime.Althoughmuchhaschangedovertime,povertyandinequalityremainchallenges,andtheneedforstrongandsustainablegrowthisstillhighonthedevelopmentagenda.
SincetheadoptionofthePoverty-ReductionStrategyPapers,resultsframeworksandM&Esystemshavebecomeroutineannexes.Itistruethat almost all the organizationsprovidingservicesinEthiopiahavesomekindofbuilt-inevaluationaspartoftheirprocessofplan-ningand/oroperations.Butthisimpliestwoimportantthings.Ontheonehand,itindicatesthatevaluationiscommonlyused.Ontheotherhand,itleadstoageneralunderstandingthatevaluationiscommon,anditssignificancehasknowinglyorunknowinglybeeneroded,andinsomeorganizationsithasbeennothingmorethanacliché.Suchpracticesriskovershadowingtherealbenefitsofevaluation.
role of Professional Associations in Evaluation Capacity Development
TheEthiopianEvaluationAssociation(EEvA)hasbeenestablishedandlegallyregisteredinEthiopiasince2008.Itsvisionistobecomeacenterofexcellenceincompetencyandstand-ardsofM&EinEthiopiaandinAfrica.Itsmis-sionistoengageandenableM&Eprofessionalsinthecountrytocontributetoevidence-baseddecisionmakingandpositivelyinfluencethesocioeconomic development of the countrythroughknowledgegeneration,awarenesscrea-tion,capacitybuilding,advocacy,anddemon-stratingmodelserviceinevaluation.MembersoftheassociationareM&Eprofessionalsworkinginthegovernment,nongovernmentalorganiza-tions,academia,andtheprivatesectorinthecountry.
Dr. Ian Goldman, Head, Evaluation and Research in the Department of Performance M&E, South African Presidency. [email protected]
Backgroundduring the 2000s therewasagrow-inginterestinmonitoringandevaluation(M&E)inSouthAfricaandtheM&Eroleinthepresidencybegantostrengthen.In2005thecabinetapprovedaplanforthedevelopmentofagovernment-wideM&Esystem.Itwasenvisagedasasystem of systems,inwhicheachdepartmentwouldhaveafunctionalmonitoringsystem,andthenecessaryinformationcouldbeextracted.
43A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
DPME started by building on the UnitedKingdommodel establishedbyTonyBlair’sadministration,focusingonalimitednumberofpriorityoutcomes,developingministerialtargets for these,plans todeliver them,andthenworkingwith relevantdepartments tounblockdeliveryoftheseplans.Thenextareaofwork—developedbasedontheexperienceofCanada—wastheassessmentofmanagementperformanceofdepartments,theManagement
Performance Assessment Tool (MPAT). In2011 the focus shifted to evaluation. VisitsweremadetoMexico,Colombia,theUnitedStates,andAustralia.TheexperiencesofMexicoandColombiawereusedtodevelopaNationalEvaluationPolicyFramework(NEPF),whichwasapprovedbycabinetinNovember2011,andtheexperienceofAustraliapre-1996wasusedtodevelopthemodelforaNationalEvaluationPlan.
BuildingEvaluationCapacityAsquotedearlier,only17percentofdepartmentsareundertakingorplanningtoundertakeevalu-ations,andthislowparticipationratemakesexperiencearoundevaluationquite limited.Thereisalsolimitedtrainingaboutevaluation(Podems,Jacob,andGoldman2013).HencethedevelopmentoftheNationalEvaluationStrategy(NES)iseffectivelystartingwithacleanslate,althoughithasbeenabletousetheexperienceof a fewdepartments thathavebeencarry-ingoutevaluation,notablythePublicServiceCommissionandtheDepartmentsofSocialDevelopmentandBasicEducation.
In57percentofcases,M&Einformationhadlimitedornoinfluenceondecisionmaking.Nearlyhalf of the respondents (46percent)regardintegrationwithpolicydevelopmentas
With these implications inmind,DPMEhasdecidedtoconducttrainingforseniormanagers.Acase-basedthree-daycourseisbeingdevelopedfortopmanagers(directorsgeneral/permanentsec-retariesanddeputydirectorsgeneral/deputysecre-taries),whichisbeingpilotedinNovember2013.Tomakeitmoreinterestingtotheparticipants,thefocusisonevidence-basedpolicymakingandimplementation,andinternationalexperienceisbeingbroughtinwiththesupportfromtheCenterforLearningonEvaluationandResults(CLEAR).
Inaddition,intensiveworkisbeingundertakenwithparliament.DPMEactually reports totheStandingCommitteeonAppropriations.Workshopshavebeenrunforthecommittee,and study toursundertaken—to theUnitedStates andCanada in 2012, andKenya andUgandain2013—tobuildtheunderstandingofthecommitteearoundM&E.Thishasbeenverysuccessful,andDPMEhasnowdonepresenta-tionstothechairsofallportfoliocommittees,toparliamentaryresearchers,andtospecificportfoliocommittees,tostimulatetheiraware-nessaroundevaluationsandhowtheycanusetheresultstofacilitatetheiroversightrole.InJune,thefirstcaseswerepresentedtoportfoliocommittees,andthisledtorequeststodepart-mentstosubmitspecificevaluations.DPMEisnowplanninganextensiveprogramofcapacitydevelopmentforparliamenttoassistcommitteesinusingDPME’sM&Einformation.
Capacity in a number of departments willincrease,becausetheyareinvolvedinseveralshortcoursesandwillcompleteanumberofevaluations(forexample,ruraldevelopment,tradeandindustry,andhumansettlements).Thisexperiencewillcreateacadreofexperiencedevaluationmanagers.
———. 2012. “Evaluation CompetencyFramework for Government.” Pretoria,DepartmentofPerformanceM&E,avail-ableatwww.thePresidency-dpme.gov.za.
Goldman, I., S. Ntakumba, and C. Jacob.2013.“Reflectionson theSouthAfricanExperiencewithEvaluationandtheUseofEvaluativeEvidence toOrientPublic
Policy Formulation.” Paper for theUNDPEvaluationCapacityDevelopmentConference,SaoPaolo,September2013.
Goldman,I.,R.Engela,I.Akhalwaya,N.Gasa,B.Leon,H.Mohamed,andS.Phillips.2012.“EstablishingaNationalM&ESysteminSouthAfrica.”PREMnotesNo.21,SpecialEdition on Nuts and Bolts of M&E.Washington,DC,WorldBank.
Podems,D.,C.Jacob,andI.Goldman.2013.“Evaluator Competencies: The SouthAfricanExperience.”PapersubmittedtoCanadian Journal of Evaluation.
Dr. Ian Goldman started his career as an agronomist in Mexico in 1978-79, and has since worked in 18 countries in Africa, Europe, and Latin America, and with national, provincial, and local government, as well as in the NGO and small and medium-size enterprise sectors.
In October 2009, Dr. Ian Goldman became Team Leader of the Monitoring and Learning Facility of the Programme to Support Pro-Poor Policy Development (PSPPD) and in January 2011 he was appointed a deputy director general in the Department of Performance M&E (DPME) in the presidency, where he is responsible for evaluation and research. In this capacity he is leading the establishment of the National Evaluation System. Many elements of the system have been estab-lished, including a National Evaluation Policy Framework, guidelines and templates, evaluation standards, evaluation competencies, and 5 courses; 20 evaluations are under way.
50 eVALUatiOn Matters
David Rider Smith, UK Department for International Development’s Evaluation Adviser in Uganda and Acting Head of the Capacity and Quality Group in its Evaluation Department 1
1 DavidRiderSmithisanEvaluationAdviserwiththeUKDepartmentforInternationalDevelopment.Allviewshereinaresolelythoseoftheauthoranddonotrepresenttheviewsofhisemployer.2 Inthispaper,evaluativeresearchconductedinthepublicsector,i.e.ledbypublicorquasi-publicsectorinstitutions,referstoanalysisnotonlyoftrends,butalsoofcausesandpotentialpolicyresponses.TheseincludethePovertyStatusReports(PSRs)andrelatedanalyticalproducts.3 Thispaperbuildsononepresentedatthe2012AmericanEvaluationAssociationConferenceentitled‘politicaleconomyofevaluationinUganda’spublicsector’andthearticlebyRider Smith, Nuwamanya and Nabbumba Nayenga, 2010,Policies,InstitutionsandPersonalities:LessonsforUganda’sexperienceinMonitoringandEvaluationinFrom Policies to Results: Developing capacities for country monitoring and evaluation systems,UNICEF.
Patterns and Influences in the Supply and Demand
of Evaluation and Monitoring in Uganda’a Public Sector over
the Past Two Decades
51A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Poverty analysis and its impact on public policyTheGovernmentofUgandastartedtoproducepovertymonitoringdatain1992,throughtheUganda National Household Survey(UNHS)reports,andhassinceupdatedthisinformationeverytwotothreeyears. Thisdata,however,didnotplayavitalrolein assessingpublicpolicyuntilthelaunchofthePoverty Eradication Action Plan(PEAP)in1997.Heavyinvestmentsinedu-cationandhealthservicedeliverythroughthe
4 Weiss,1999,“Theinterfacebetweenevaluationandpublicpol-icy,”Evaluation5:468;Bamburger,2009,Institutionalizing Impact Evaluation within the Framework of the Monitoring and Evaluation System,WorldBank;GaarderandBriceno,2010,InstitutionalizationofGovernmentEvaluation:BalancingTrade-Offs,Working Paper 8, International Initiative for Impact Evaluation;WeyrauchandLangou,2011,“Soundexpectations:fromimpactevaluationstopolicychange”,Working Paper 12, International Initiative for Impact Evaluation.5 Thefullnormativeframeworkcanbefoundatthepaperpre-sentedatthe2012AmericanEvaluationAssociationofthesamename.
In1999,theGovernmentofUgandadesignedapovertymonitoringstrategythatguidedtheproductionofbiannualpovertystatusreports,andassociatedmonitoringreportsandpublica-tions.ThePoverty Status Reports(PSRs)werehighqualityanalyticalpieceswhichdrewuponquantitativeandqualitativehouseholdandfacili-tiessurveydatatodeterminethepatternsandchangesinruralandurbanpoverty.
WithintheMinistryofFinance,PlanningandEconomicDevelopment(MFPED),aPovertyMonitoringandAnalysisUnit(PMAU)wasestablished, tomonitor, analyse and reporton data generated on poverty and relatedissues,includingthepreparationofPSRs(withfinancingfromUK).Thisworkwassupportedby the Government-sponsored Economic
Eachoftheseestablishments:UBOS,PMAU,UPPAPandEPRChadhighlyqualified,com-mittedandmotivatedindividualsinkeyposi-tions.Theability toproducehighqualitymonitoringreports ina timelymanner forpolitical andadministrative consumptionref lectednotonlytheir individualabilities,but their willingness towork together todeliverdemanddrivenmonitoringreports.WithinMFPED,astrongworkingrelation-shipbetweenPMAUandthetopmanagementof theMinistrymeant that findings fromtheseproductsmadetheirwayintopolicyandallocativedecisions,and in turn,attractedincreasedofficialdevelopmentassistance.ManyoftheissuesraisedinthePSRsinf lu-enceddecision-makingatbothattheCabinetand theParliamentary level,andhelped infocusingexpendituresinareasthatweremostmeaningfulforpovertyreduction.
The window of opportunity and practice in the produc-tion of PSRs in Uganda reflected the priority and progress made in poverty reduction from the President down, and the relation-ships and abilities of the institutions and individuals involved.
53A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Shift to budget and performance monitoring for accountabilityWhilst theevidencefromhouseholdsurveysandPSRsbegantorevealthattheoverallpov-ertyheadcountwasreducinginthecountry,itwasalsobeingnotedthatgrowthanddevelop-mentwasbecomingincreasinglyimbalanced(MFPED,2005;UBOS,20067).Asthepublicpurseexpanded,basedbothonastrongandstableeconomicgrowthrate,relativelylowinfla-tion,andaconsiderableriseinofficialdevelop-mentassistance,sotoodidconcernsontheappli-cationandaccountabilityforpublicspending.Effortstostrengthenpublicfinancialmanage-mentincludedtherecognizedneedtounder-standhowpublicresourceswerebeingappliedcentrally,andlocallyunderthedecentralizedsystemofgovernmentandpublicprogrammeimplementation,introducedin1997.
ofGovernmentareunderway.APresidentialdirectivetoinitiatepublicforatoholdlocalGovernment to account (so called public‘barazas’)hasbeeninitiatedbytheOfficeofthePrimeMinister,andtheUBOSisseekingtoexpanditsmandatetoimprovethequalityofadministrativestatistics.
Simultaneously,theOfficeofthePrimeMinister(OPM)hasreinvigorateditsownconstitutionalroleoncoordinatingtheimplementationofpub-licpoliciesandprogrammesthroughestablishingarobustmonitoringcoordinationandoversightfunction.Buildingonanearlyattemptatpro-ducinganoutcome-basedreviewofthePEAPin2007,bi-annualGovernmentPerformanceReportswere initiated in theFinancialYear2008/09andtwo-dayretreatsofallMinisters,PermanentSecretariesestablishedunderthePresidenttodiscusstheperformancereport,holderportfolioMinisterstoaccount,andpro-posecorrectivemeasures.Since2011,thesebi-annualretreatshavebeenexpandedtoincludeallLocalGovernmentCouncilChairpersonsandChiefAdministrativeOfficers.Thishasexpandedthebasisofdebatearoundpublicservicedelivery.
Efforts to strengthen the analytical and the evaluativeTheconcertedeffortstostrengthenmonitoringhavecomeatacost.Thepracticeofpublicsec-torevaluationhasneverbeeninstitutionalizedinthecountry,butwasreasonablywellservicedinthelate1990sandearly2000sthroughthePSRsandotheranalyticaltoolsandproducts.Thishasledtoadeficitintheanalysisofresultsandconstraints,andinthe identificationofpolicylessonsandchoices.Monitoringitselfdoesnotanswerthesequestionsoraddresstheseissues.
Between2005and2008,areviewofevaluationinUgandafoundeighty-fiveevaluationscom-missioned,ofwhichtenwerecommissionedorco-managedbytheGovernment(OPM,20098).Whenreviewingthese tenreports indetail,severalwerefoundnottomeetbasicqualitystandardsforevaluation,andweresubsequentlyreclassified as performance assessments orreviews.Intermsofcoverageofpublicinvest-ments,Governmentestimatedin2009thatlessthan10percentofprojectsovertheperiod2005-089werebeingsubjectedtoevaluation.Inasam-pleofMinistries,DepartmentsandAgencies,thesamereviewfoundlittleexplicitdemand
Oftheevaluationsthatwereconductedduringthatperiod,thereislittleevidenceoftheirimpactduetoalackofappraisalbyGovernmentortheinternationalpartnersondisseminationoruse.Anexceptionweretheevaluationscoveringtheagriculturesector,startingwithonecoveringthePlanfortheModernizationofAgriculture(PMA)in2005,asecondlookingattheperfor-manceoftheNationalAgriculturalAdvisoryandDevelopmentService(NAADS)in2008,andathirdbeinganimpactevaluationalsoonNAADSin2009.Eachoftheseindependentevaluationsgaveabroadlypositiveassessmentofprogress,withtheimpactevaluationon2009showingpositiveresultsonadoptionofimprovedtechnologies, productivity and per capitaincomes.Thestudy(Benin,200910)alsofoundthatbetween2004-08,NAADSwasassociated
10 BeninS.(2009).Impacts of and Returns to Public Investment in Agricultural Extension: the Case of the NAADS Programme in Uganda.IFPRIResearchReport.Washington:InternationalFoodPolicyResearchInstitute.
56 eVALUatiOn Matters
withanaverageof24-53percentincreaseinpercapitaagriculturalincomeoftheprogramme’sdirect participants compared to their non-participantcounterparts.However,asnotedbyothercommentators,thatdespitereportedsuccessesofNAADS,overallindicatorsforagri-culturalgrowthwerenotimproving(KjaerandJoughin,201111).ThishaspresentedaproblemforGovernment,andresultedinchangingstrate-giesonagriculture,andNAADS,includingtherenationalizationofextensionworkers,despitethefindingfromthe2008evaluationthat ‘areturntousingpublicsectorextensionwork-ersforthemajorityofserviceswasnotaviableoption’(Ibid,2011).Thisraisestwoissues,onerelatestotheunitofanalysisoftheevaluation,whereNAADSasaninitiativemaybeseentoberelativelysuccessful,butdoesnottakeintothebroadercontext,whichmaybelesspositive,andhencepointsatmorefundamentalstructuralchallenges.Thesecond,reflectstheuseofevalua-tiveevidenceincaseswhetherthemajorityofthepopulation,includingpolicymakers,havedirectpersonalknowledgeaslandownersandfarmers,andwherethemajorityoftheelectorateliveof
11 Kjaer,A.M.&Joughin,J.,2012,“TheReversalofAgriculturalReforminUganda:OwnershipandValues”,Policy and Society,Vol31,Issue4,November2012:pp.319-330.
The paucity of rigorous public sector evaluations has been identified as a constraint to improving the culture of debating empirical evidence in public policy.
57A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
The withdrawal of donor financing and the increasing role of politics in civil administrationThesereformstookabackwardstepinlate2012whenacaseofgrandcorruptionwasidentifiedintheOfficeofthePrimeMinisterandotherpartsofGovernment,resultinginalarge-scalewithdrawalofdonorfinancialaidandbudgetsupport.Thishadtwoimmediateimpactsontheevaluationagenda.First,themajorityshareof therecurrentanddevelopmentbudgetofthemonitoringandevaluationactivitieswasfinancedbydonors,andthustheaidfreezehadanimmediateandbigimpactonstaffingwithnumbersreducingbyoverhalfinthedepart-ment12andthemajorityofactivitiesstopped,includingseveralevaluations.Second,thecred-ibilityoftheOfficecameintoquestionwithinvestigationsintoseniorandmid-levelstaff.Duringthisperiodofinvestigation,courtcasesandinternalreorganization,theinstabilityand
A feature of Uganda’s public sector governance arrange-ments has historically been the power of the Presidency and the relative weakness of the Cabinet system.
59A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Asabreakpoint,thecorruptioncaseandthefreezingofaidenabledobserverstoseetheextenttowhichtheGovernmentiscommittedtothemonitoringandevaluationagenda,havingtofinancealloperationsfromitsowncorebudget.Government monitoring, the performancereportandCabinetretreatshavecontinued,albeitwithsomechallengestoqualityposedbythesubstantialreductionintrainedstaff.Theevaluationagendahassuffered,withdwindlingnumbersattendingthecross-Governmentevalu-ationcommitteeanddespiteahealthypipelineofevaluations,slowprogressbeingregistered.
Inthe lastthreeyears,notwithstandingthecontinuedfocusonmonitoringforaccount-ability, theOPMhas led the drive towardsreintroducingmore analytical work in thepolicydebatethroughtheestablishmentoftheGovernmentEvaluationFacility.ShiftsinthepoliticaleconomyoftheCabinet,withanewandpowerfulPrimeMinisterandareinvigorated
David Rider Smith is the UK Department for International Development’s Evaluation Adviser in Uganda and Acting Head of the Capacity and Quality Group in its Evaluation Department. Prior to joining DFID in 2012, he worked for five years in the Uganda Bureau of Statistics and the Office of the Prime Minister in Uganda as an advisor to the Government, leading on the consolidation of performance monitoring, Cabinet reporting, strengthening district statistics and setting up the Government Evaluation Facility. He has previous worked as an evaluation specialist in the Evaluation Office of UNDP, as a researcher at the Natural Resources Institute, and as a survey manager for USAID-supported programmes in Mozambique. His background is in geography, sociology and development economics.
The opportunities for evalu-ation to influence decisions may now lie primarily outside the public sector and with the public themselves, supporting citizens to demand better services and rights.
61A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Dr. El Hadji Gueye, BA, MA, PhD, Coordinator of CLEAR Initiative for Francophone Africa
strengthening national evaluation systemsisimportantforgoodgovernance,improvementoftheeffectivenessofpublicpolicy,andbet-termanagementofdevelopmentwork.Transparencyintheallocationandmanagementofstatefundsandofficialdevelopmentassistance,theobligationtobeaccountable,andtheemergenceofacultureofresultsaresomeofthemanyreasonsthatjustifytheimportanceofcreatingandencouraginganevaluativecultureinthecountriesofFrancophoneAfrica.
Lessons learned with respect to NECDSuccesses, failuresUnfortunately,littlehasbeenimplementedsuc-cessfully.Thisisoftenduetoalackofpoliticalwill.Toachieveownershipofevaluationasarealtoolthathelpsindecisionmaking,suchbehav-iorshouldbeencouragedandsupportedbythecentralgovernment,butalsobytheparliament,whichhasafundamentalroletoplay.
Unfortunately, little has been implemented successfully. This is often due to a lack of political will. To achieve ownership of evaluation as a real tool that helps in decision making, such behavior should be encouraged and supported by the central government, but also by the parliament, which has a fundamental role to play.
63A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Examples of models of national evaluation systems that are performing well Ghana’sexamplehasalottoteachus,asdoestheSouthAfricanmodel.Benin’smodelcanalsobeusedasanexample,althoughmuchremainstobedonethere.
• Senegal: The General Delegation forStateReform(DREAT), theDepartmentofNationalPlanning in theMinistryofEconomyandFinance.
The road ahead Urgent issues in national evaluation capac-ity development include the following:Theindependenceofnationalevaluatorsandthepositioningofthegovernmentstructuresinchargeofevaluation.
ofpolicies,projects,andprogramsandequipthemwith the approaches andmethodsneeded to conduct successful evaluationmissions.
Key elements for successful national evalu-ation capacity developmentFirst, commitment by the state to efficientnationalM&Eandeffectiveinvolvementofthestate,alongsidedonors.Thisshouldbefollowedbythedefinitionofalegalframeworkforthesystematicevaluationofpublicpolicies,projects,andprograms.
65A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Dr. El Hadji Gueye’s expertise includes methods of impact evaluation for programs and projects, M&E for health expenditures, management based results, and economic evaluation for programs. He has provided consulting services for the World Bank, UNICEF, UNAIDS, UNDP and technical assistance on data collection and analysis for evaluation and management based on the results with Marginal Budgeting for Bottlenecks tool in Senegal, Burkina Faso, and Gabon.
He has coordinated and facilitated workshops on National Health Account, M&E for health programs, and strategic planning for HIV programs. He is the head of the Economics Department and the coordinator of CLEAR Initiative for Francophone Africa at the Centre Africa in d’Etudes Superieures en Gestion (CESAG) in Dakar, Senegal.
He has a PhD in econometrics and statistics, a master sin econometrics, and a bachelors in quan-titative methods at the faculty of Economics and Social Sciences, University of Geneva, and a bachelor in Mathematics from UCAD in Senegal.
iv) Createadegreecourseinevaluationandprovidefundingforappliedresearchinthefieldofevaluationinordertochangepractices; develop new tools that aremoreadaptedtotheparticularcontextofFrancophoneAfrica.
67A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
developing an evaluation culturewithintheBankandinregionalmembercountriesisoneofthecorner-stones of the independent evaluationpolicyrecentlysubmittedforapprovalto theCommitteeonOperationsandDevelopmentEffectiveness(CODE).Thisobjectiveisbasedonthenotionthatpro-motinganevaluationculturewithinaninstitutionsuchastheAfricanDevelopmentBankplacesanemphasisonevaluationoutcomesandonthelessonslearnedfromsuccessesaswellasfailures.Aninstitutionwithanevaluationcultureassumesthatitsmembers:
Promoting an evaluation cultureTheBankanditsstakeholdersseektopromoteacultureofevaluatingoutcomes,whichcanonlyoccurifevaluationisatthecenterofthiseffort.DespitetheincreasingengagementinfavorofoutcomesalreadyrecordedwithintheBank,promotinganevaluationculturepresupposesalargeprogram.TheOperationsEvaluation(OPEV)Departmentwillhavetocollaboratewithotherpartnersinthiseffortandensurethatitsworkcompletesotherinitiatives,suchasthequalityassuranceworkdonebytheBank’sQualityAssurance&ResultsDepartment(ORQR),anaturalstrategicpartnerinachievingthisobjective.
Toward an Evaluation Culture in the AfDB: Practices and
Challenges
Mohamed Manai, Manager, OPEV, AfDB
68 eVALUatiOn Matters
Reinforcing stakeholder engagementIt is important tomention that engagementoccursinthreemainareas:
• Within the framework of individual evalu-ations.Engagingmanagementandrelevantpersonnelatcriticalmomentsoftheevalua-tioncycle(evaluationframeworkdocument,identificationofstakeholderandreferencegroups,disseminationofknowledgefromfindingsinthefieldandfrominteractionswithbeneficiaries) isabestpractice that
Increasing the impact of evaluations in regional member countries Thereisincreasingdesireforevaluationknowl-edgeinAfDBregionalmembercountries.MakingsurethattheknowledgegeneratedbyOPEVis
• Regional member country participation in the evaluation process.Participatingincountrystrategyevaluationsisencouragedbutnoteasilyachievableduetocountries’lowcapacities.Nationalstakeholderinvolvementthroughouttheevaluationprocessmustbeaccompaniedbyawareness-raisingsessionsonthemeritsofnationalevaluationsystems,withinaframeworkofimplementingaman-agementapproachthatisbasedondevelop-mentresultsandtherationaluseofpublicresources.
• Raising awareness, by targeting regional member countries in communication efforts.Knowledge-sharingsessions,con-ductedinregionalresourcecenters,facilitateparticipationbyregionalmembercountries.TherecentworkshoporganizedattheAfDBSouthernAfricaResourceCenter(SARC)atteststothehighlevelofinterestofregionalcountryrepresentatives,who,inthecaseoftheSARC,madesubstantivecontributionsonsubjectssuchasprivatesectordevelop-ment,Bankassistancetoregionalintegra-tion,economicandsector-basedreviews,and
• Supporting regional member country efforts to develop national evaluation capacities.Regionalmembercountriesarealsolookingtogenerateevaluativeknowl-edgefromtheirownprogramsanddevel-opmentprojects,whethertheyarefinancedexternally or not.They are increasinglyinterestedinreinforcingnationalevalua-tioncapacities.OPEVhasalreadybeguntoundertakereinforcementactivitiesincoun-trieswhereatangibleneedhasbeendem-onstrated(suchasEthiopiaandTanzania),focusingitseffortsonministriesinchargeofplanningbyensuringthepracticalandhands-ontrainingofpersonnelinchargeofpost-evaluationmonitoringfornationaldevelopment programs. In otherwords,OPEVwillsupportthedevelopmentofanevaluationcultureinregionalmembercoun-triesbysupportingtheirevaluationsystems.Suchsystemshelpensurethatevaluationsmakeadifferenceinpoliciesandinpractice.
69A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
toparticipateinmajorinternationalevaluationcommunitiessuchastheevaluationnetworkof theDevelopmentAidCommittee of theOrganisationforEconomicCo-operationandDevelopment,andtheEvaluationCooperationGroup.OPEVcontinuestodevelopstrongandmutuallybeneficialrelationswithbilateralaid
entitiesandengageswithorganizationsfocusedonevaluation,suchastheAfricanEvaluationAssociation,variousregionalcentersforlearningfromevaluationsandoutcomes,EvalPartners,the International Initiative for EvaluationImpact,andphilanthropicorganizationsinter-estedinsupportingevaluationefforts.
Managing and disseminating knowledgeOPEVusesvariousmediumsandtechnologiestofosterknowledgesharing:
• The OPEV website,thedepartment’sprimaryinterfacewiththeworldhasbeenredesignedtobemoreuser-friendlyandtoallowuserstoquicklylocatetheknowledgeproductstheyneed.
• A database of lessons learned, whichwillallowusers to search for informationbythemeorbycountry.OPEVwillcollectles-sonsfromotherorganizationstomaximizetheusefulnessofthelessonsdatabase.ThissystemwillincludeindependentevaluationsaswellasCompletionReportReviews,andit
• Regional member country awareness willbeincreasedthroughregionalworkshopsandvideoconferences,whichallowfordissemina-tionanddiscussionofknowledgegeneratedbyevaluationwork.Tofurthertheseefforts,OPEVwillworkinclosecollaborationwithcountryteamsandwiththeBank’sregionalresourcecentersandoffices.
Mohamed Manai is Division Manager in the Operations Evaluation Department of the AfDB. Before joining the Bank, Mohamed worked as a Management Consultant for PriceWaterhouse in Kuwait, Tunisia and Morocco. He is trained in finance and holds an MBA degree.
72 eVALUatiOn Matters
Dr. Sukai Prom-Jackson, Inspector, Joint Inspection Unit, United Nations
ConceptualFramework for Evaluation and National Evaluation Capacity Development1
Why Africa Has Not Developed to Its Full PotentialMoussa Seye (African savant) on challenges to African growth and development (August 7, 2013) (translated from Wolof)
—If someone lends you his eyes, you will only see what he wants you to see. Theeyesarethewindowstothemind.Theywilldirectwhatyouthink.
—If someone lends you his language, you will use the models he has developed to express reality based on his own ecological and historical background. Thereisnosovereignthathasdevelopedonthelanguageofanotherculture.
—If someone lends you his mind based on how he defines value and truth, you will work for it and continue to fix it to fit.Youwillneverknowandyouwillneverbecomeafithumanbeing.
EvaluationandCapacitiesforCriticalThoughtEvaluationisjudgmentofvalueorworthandprovidesinformationtosupportdecisionmak-ing. Indevelopment evaluation, it supportsaccountabilityfortheeffectiveuseofresources,lessonsforimprovement,knowledgesharing,andthedistillationofthisknowledgeforusewithinorganizationsandinthewiderglobalcommunity.
The main factor that makes evaluationstandout fromotheroversight functions in
organizations is thecritical mode of think-ing it requires. It is a rigorous process ofinquiry.Itexaminesactionsandresultsandasks thequestions:Arewedoing the rightthings?Arewedoing thingsright?Arewegettingresults thatmakeadifference?Arethesetherightresults,andwhatistheimpactandvalue?Whatexplainsthedifferencemadeandwhatenhancesimpactandsustainability? In providing answers to these questions,evaluationusesthesystematicmethodsand
Toenhancecredibilityandutilityinsupport-ingdecision-makingprocessesassociatedwithaccountability, improvement, learning, andknowledgemanagement, governments andorganizations have put in place evaluationfunctionswithdefinedstructures,institutionalframeworks, and systems andmechanisms.Figure1providesanillustrationoftheevalua-tionfunctionusingthekeyelementsofdemand, supply, and resultsandtheirrespectivecapabili-tiesasaframework.2
Evaluation Capacity Development and role of Evaluation for the Development of Africa National evaluation capacity development isanendogenousnationalprocess todevelopthecapacity for“evaluative thinking.”Thisencompasses theusecritical thinkinganddivergentandconvergentreasoningtopro-ducecredible(impartial,valid,andreliable)andtrustworthyevidencetosupport(i)deci-sionmakingand(ii)actionthatwouldbenefitnationalgrowthanddevelopment,includingthedevelopmentof social and intellectualcapital.Thisnationalprocess iscarriedoutatalllevelsofsocietyandisinstitutionalized,inparticularaspartofthebusinessmodelforeducationandschooling(formaland infor-mal),governmentandnationalgovernance,universities,thinktanksandresearchcenters,theacademiccommunity,andtheportionsof
Approachestoaddressingtheseframingcondi-tionsmayvary.Itissubmittedthatforanysug-gestedframework—includingthoseforevalua-tion—toserveasacriticaltoolortooutlinethecapacitiesneededtoguidenationaldevelopment,itmusthavethefollowingfourelementsforahome-grownframeworkofaction.3Thesearereferredtointherestofthepaperasthe four Rs(4Rs)
Thefirstpriorityofthisapproachmustberebirth. Themeaningofrebirthmayvary,butwhatitentailsisabreakfromthepastanditsdisempow-eringfeatures.Thisisnecessarytoallowspaceforideas,practices,androutinesthatarenotladenwithnegativelegacies.However, for there to be a meaningful break, rebirth must be total, purposeful, and home-grown. It is given expression in the French phrase: enracinement puis overture.
Restoration:Africa’sencountershaveexposedandsubjectedittoagonies,violation,anddep-ersonalization.There is an urgent need forrestoration:restorationofpride;restorationofheritageindevelopmentnarratives;restorationofambition;andrestorationofgenuineleader-ship.Butitisimportanttonotethatrestoration isnotaboutnorms-localizationorintroducingforeignmodels.Rather, it is about putting back the missing links that were compromised by the dominance of Eurocentrism in Africa.
Reclamation:Thefaultlinesincurrentdevel-opmentmodelsandevaluationmethodsmeanthatAfricamustnowturntoandexplorehome-grownknowledgesystems.ThisisnecessaryforthereclamationofAfricanspacesandownershipindevelopmentnarratives.ApotentialbenefitisthepossibilityofseeingAfricathroughatotallynewlensthataddspride,confidence,andrel-evance tohome-grownknowledge systems.Achieving this requires conceiving methods and models that are particular to, and reflective of, African sensibilities.
The first priority of this approach must be rebirth. The meaning of rebirth may vary, but what it entails is a break from the past and its disempowering features. This is necessary to allow space for ideas, practices, and routines that are not laden with negative legacies.
77A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Responsibility: In rebirth, restoration, andreclamation,thereispotentialintheformula-tionofanAfrica-centered,Africa-specific,andAfrica-drivenevaluationfunctionandmod-elsforevaluation.ThesethreeRsalsoprovidetheframeworkfornationalevaluationcapacitydevelopment.Howtomakethishappenisacriticalconsideration.ItrequiresleadershipandactionfromAfricans.Whatisdifferentthistimeinnationalownershipisthenatureofthevisionthatwoulddrivetheprocess—onethatisfocusedontheinterestsofAfrica.Thisvisiondrawson
ConceptualFrameworkfortheEvaluationFunctionandCapacities for Evaluation in AfricaDemand, supply, results for accountability, improvement, and knowledge management within a well-defined, integrated framework that is “Africa-centered”Definition:Evaluationisa judgmentof thevalueorworthofanentity.Theroleofevalu-ationfordevelopmentofAfricamustincludedefiningwhatisvaluedformovingAfricafor-ward.Evaluationisasystematicprocessbasedonscientificandmanagementprinciples.Theprocessofgeneratingevidenceandknowl-edgemustrespectwaysofknowinginAfrica,includingtheimportanceofactiveinvolvementintheconstruction,management,anduseofknowledge.
Systems model—the national evaluation sys-tem:Giventheroleofevaluationforaccount-ability,improvement,learning,andknowledgemanagementintheinterestofthenation,thenational evaluation function and capacitiesrequiredmustnecessarilyusean integratedandholisticframeworkthatappliesasystemsmodel.Thisincludes:
Figure 2: A systems perspective for national evaluation capacity development
*Supportingevaluationcapacitydevelopment
Guiding principles for the evaluation func-tion: Rebirth, Restoration, Reclamation, and Responsibility: Thefollowingsetofguidingprinciplesfromthe4Rsprovidetheframeworkthatmustbepartoftheconstituentelementsindefiningthekeyelementsofthefunctionofnationalsystemsandthatshouldguidecapacitydevelopmentandsupport.The4Rsareinter-related.However,thewaytheyfitwiththekeyelementsoftheevaluationfunctionfordemand,supply,andresultsisillustratedinFigure1.
Rebirth: Defining demand, purpose, and added value of evaluation.ThevisionforevaluationinAfricamustpromoteevaluationthatinformsdevelopmentthatisAfrica-centeredandrootedinAfricanrealitiesandtheissuesthatarecriticalformovingAfricaintothefuture.
various forms of knowing recognized inAfricansocietiesaswellasinvariouspartsoftheworld.Itsbusinessmodelmustbeinclu-siveinsettinganagendaofcriticalissuesimportantforthefutureofAfrica,address-ingkeychallengesandallaspectsofsocietyandallgroups,beyondthosepertinenttodevelopmentpartners.
79A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
• Aparamountgoalofevaluationistobuilddynamic learning groups and learningnationsthatareresilientandadaptabletofast-movingchangeandfocusedontheuseofknowledgeforinnovationandchange.
Reclamation and Responsibility for Results: EvaluationmustassumeleadershipforAfricaandtakeitsplaceindefininghigh-impactevalu-ationfunctionsthatrespondtovaryingdemandsinabalancedway.Itmustreclaimitspositionintheconstructionofknowledgeandinservingasalegitimateagentofchange.Itshouldseektodevelopavisionforevaluation,withkeyelementsthatincludebalancingthefourfoldpurposeofevaluationforaccountabilitytonationsandtodevelopmentpartners,improvementofpractice,buildinglearningnationsandenhancinginno-vationsinsociety,andcontributingtoglobalknowledgeandtothefieldofevaluation.
81A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
• Itshouldtakealeadroleindirectingsub-stantialfocusonformativeevaluationtoenhance understanding of what works,why,andhow.Thisis importantfortest-ingdevelopmenthypothesesandfortimelyengineeringandpreventionofthenotionof
Sukai Prom-Jackson is an Inspector with the Joint Inspection Unit of the UN System. She has over 20 years of professional experience and leadership in the management and conduct of research and evaluation, in policy formulation and strategic planning and program development, and in the facilitation of learning.
She has worked for the UNDP, World Bank, U.S. government, universities, and research centers in the United States. She has a PhD in educational psychology (1982) with focus on policy research, measurement, and evaluation from Howard University (U.S.).
82 eVALUatiOn Matters
Dr. Claudia Maldonado, General Coordinator, CLEAR Center for Spanish-Speaking Latin America1
Monitoring and evaluation (M&E)havelongbeenpromotedbythedevelopmentcommunityasnecessaryfortheenhancementofresults-orientedpublicmanagement,tofosteraccountability,andtoenableevidence-basedpolicymak-ingtopromotedevelopmenteffectiveness.Inthedevelopedworld,thistrendresultedintheadoptionofnewpublicmanagementconceptsandprinciplesintheprocessofadministrativereformandmodernization.Inthedevelopingworld,however,thisagendahasbeenmoreforcefullypromotedbythedemandforaccountabilityofdonorsandexternalinstitutions.
What s Democracy Got to Do With it?Thedevelopmentofnationalevaluationsystemsandtheadoptionoftechnicaltoolstoassessprogramandgovernmentperformancecanbemistakenlyassociatedwithatechnocraticpushforplacingpoliticaldecisionsinhandsofanar-rowgroupofexperts,orthebackdoorforovertpolicyinfluencebyexternalactors.Inmyview,nationalevaluationsystemsrepresentpreciselytheopposite:theyplacebettertoolsandknowl-edgeinthehandsofdemocraticdecisionmak-ing,providinganinformationalpublicgoodforaccountabilityandcitizenempowermentand,ultimately,securingthelegitimacyofdemocraticregimes.
Inademocraticsystem,citizensdelegatesov-ereignpowerstoelectedofficials,whoreceiveamandatetopromotesubstantiveoutcomesforthecommunityandaseriesofinstruments—publicfundsandpowers,authority,regulation,administrative structures, and so forth—toadvancethem.Originally,thepreservationofthismandatecouldtheoreticallybesecuredbytheelectoralincentive(Downs1957)andtheruleoflawoftheWeberianbureaucracy(O´Donnell
Inaddition,onthesupplyside,decisionmak-ersfacemultipleincentivesandrestrictionsinadoptingthelogicofresults-orientedmanage-ment.Theycanbebudget-maximizersthatcom-peteforbudgetswiththelogicofbureaucraticcontrolandpower(Niskanen1971);theycanoptforshort-termsolutionsthatprovideimmediatepoliticalbenefitsinsteadofsecuringlong-terminvestmentsthatmaximizepublicvalue;theycaninvestpublicresourcesintheconstructionof clientelistic andcorporatistnetworks for
In other words, we need to track whether the theories of change implicit in any intervention and the conse-quent mobilization of public funds to tackle development challenges are progress-ing toward producing the expected result.
85A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Furthermore, what we should expect anddemandascitizensarepolicyoutcomes,notsimplypublicprograms.Decisionmakersandgovernmentagenciesproducepolicyoutputs,notnecessarilylinkedtoexpectedpolicyoutcomes.Forthisreason,thelinkbetweengovernmentactionandthetransformationofthecorevari-ablesaroundapublicproblemcannotbeheldasaxiomatic.Aslawyerswouldsay,theburdenofproofisapublicresponsibilityofgovernments,becausecitizenscannotsystematicallygivethebenefitofdoubttopoliticians,withoutstringsattached,andbecausetheopportunitycostsofill-conceivedandpoorlyimplementedpublicinterventionsthatutilizescarcepublicresourcesaretoohigh.
policytoachievingthesocialobjectivesthatgave rise to it.Anddecisionmakers shouldnotbeentitledtomakethesechoicesbymereintuitionorgoodintentions,inanevidence-freezone.Theactualexistenceandverificationoftheinstrumental-causallinkbetweenpublicchoicesandpublicoutcomesisaninescapablepublicresponsibilityfordemocraticgovernance.Butthislink,ananchorofresults-orientedmanage-ment,needstobeconstructed.Itrequiresentireadministrativeandinformationmanagementprocessesthatspeakthelanguageofresultsandfeedintobureaucraticroutinesandpoliticallanguage.
The link between institu-tional frameworks (national evaluation systems) and actual capacity for evalua-tion has been at the heart of institution-building theory and debates.
87A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
M&E systems are information systems andprocessesthatareanchoredinresultsandevi-dence-basedinputs.Theyallowbothcitizensandgovernmentstoknowthetruthabouttheeffectivenessandefficiencyofpolicies,tofindoutwhetherpublicactions,instruments,andstrategiesarecorrectorwhetherbasicpremisesandassumptionsshouldberevised.ThatistheprimaryroleofM&Esystems.
National Evaluation Systems and the Quest for Evaluation CapacitiesThe link between institutional frameworks(nationalevaluationsystems)andactualcapacityforevaluationhasbeenattheheartofinstitu-tion-buildingtheoryanddebates.Shouldexist-ingbaselinecapacitiesprecedetheenactmentofformalsystemsforevaluation?Aretheyadirectresultofinstitutionalframeworks,ordotheyalwaysinteractinfarfromlinearrelationships?
However,thisimportantstepwillnotbesuf-ficienttosecurestrongandreliableevaluationcapacitiesacrosspolicysectorsandbureaucraticlevels, and extendingbeyondgovernmentalactorsandsmallgroupsof stakeholders.Asmentionedabove,M&Esystemsareafirststepfortheproductionforpublicinformationthatisreliable,relevant,andresults-oriented.Buteffectivecapacitiesforevaluationrequiresystem-levelchangesandpositivefeedbackandlearn-ingcycles,andtheydependonfactorsthatareexternaltotheevaluationsystems.
Fortheseconditionstodevelop,theremustbeaclearexternaldemandforresults-orientedinformationthatischanneledintothepoliticalandpolicyprocesswithsomedegreeofsaliency.Ifbothcommissionersandevaluatorsexpecttheproductoftheirevaluationeffortstobeforgotten,neglected,orignored,theywillhavenoreasontoinvestinthequalityandcredibil-ityofevaluation.Furthermore,ifthedecisionmakersresponsibleforkeyperformanceandprocessindicatorsoftheprogramstobeevalu-ateddonotexpectevaluationresultstoresonatewithinthebureaucraticand/orpoliticalarenas,thereisnoreasonforthemtargettheireffortsatinducingbetteroutcomesthatcanbeidentifiedandrecognizedthroughM&E.Inturn,citizensandattentivepublicswhoarenotinformedandawareofthepotentialroleofevaluationresultsandmonitoringdatawillnotunderstandtheirlinktosubstantivedevelopmentpromotion.Withoutsuchunderstanding,theywillfailtoutilizetheseresultsintheirpushforaccount-ability andbetter government responses topublicproblems.
AttheCLEARCenterforLatinAmerica,wehaveamandatetopromotecapacitybuildingforevaluation,monitoring,andperformance-basedmanagementtopromotedevelopmentattheregionallevel,aswellastopromoteglobalcross-dissemination of good practices andlearningspacesrelatedtoourcorefunctions(technicalassistance,knowledgegeneration,andtraining).Inordertofulfillourmission,wehaveadoptedasystemicviewoftheroleofcapacity-buildingliketheoneproposedabove:weseektopromotetheunderstanding,adop-tion,andmobilizationofM&Einformationasanewlanguageof thedemocraticpoliti-calinteractionthatisalreadytakingplaceinpluralistsystems,yetdoesnotusuallyoccurinthelanguageofresultsandevidence-basedpolicymaking.Tolowerbarriersofentrytothis language,wepromotetraining,aware-ness,anddevelopmentoftechnicalexpertiseamongwideraudiencesandthediversestake-holderswhoalreadyoccupycenterstage inpolicydebates—civilsociety,activists,watch-dogorganizations,andgovernmentagenciesatthenationalandsubnationallevels—andseektofosterleadershipamongkeystakeholdersinthepromotionofthisagenda.
Dr. Claudia Maldonado is currently Professor-Researcher at the Public Administration Division of the Centro de Investigación y Docencia Económicas (Center for Research and Teaching in Economics) a Mexican center for research and higher education specialized in the social sciences. Her research focuses on the political economy of conditional cash transfers in Latin America and comparative development (Mexico and Brazil), social policy and program evaluation.
Prior to her current position, Dr. Maldonado coordinated M&E capacity-building efforts for Mexico’s Consejo Nacional de Evaluación de la Política de Desarrollo Social (National Council for the Evaluation of Social Development Policy, CONEVAL) and worked in the General Direction for Evaluation at Oportunidades, the largest conditional cash transfer program in Mexico
Dr. Maldonado earned her Ph.D in Political Science from the University of Notre Dame, and holds a Master s Degree in Public Affairs from Princeton University.
How to Speed Up Global Development’s Learning Cycle
Trevor Davies
… we do not learn enough, or learn quickly enough, from the work that’s already happening—or happened—on the ground… this knowledge management gap means we are constantly missing opportunities to improve the quality of our work and its impact on the world
Ofcourseeveryonewill say theyaredoingthis type ofmonitoring, but often it’s justanexerciseintickingabox.It’snotasexcit-ingorsexyasdoingthenewquasi-scientificthings likerandomcontrol trials,and long-termevaluation—bothofwhichhavearoletoplay—butitdoesgiveyoutheopportunitytocoursecorrect.
There’s always a tendency to try andpolishthings,togetmethodologiesto100percent.Infact,whatisimportantistolearnfromwhatwe’realreadydoingsowecandoitbetter.
KPMG is walking that walk of knowledgemanagementthroughournewInternational
DevelopmentCenterofExcellence.BecauseKPMGisorganizedaroundclients,wehaveteamswithinKPMGworkingwithkeystake-holders in the development communityaroundtheworldincludingAusAID,theAsianDevelopmentBank,DfID,SIDA,theUnitedNations,theWorldBank,andmanyphilan-thropicorganizations.
This column was originally published by Devex Impact. Published here with the author’s permission.
Trevor Davies leads KPMG’s International Development Assistance Services Global Center of Excellence and works across the firm’s global network of member firms. He has more than 23 years of experience working with governments and develop-ment agencies.
My First encounter withthecon-ceptofcapacitydevelopmentdatesafewyearsback.Comingfromtheworldofmanagementconsulting,Iusedtoreen-gineer processes, diagnose organiza-tions,andmanagechange.Movingintotheworldofdevelopment,Iwastoldthatourjobwastobuildordevelopcapacity.Butwhatisthiscapacitywearesupposedtohelpdevelop?Whatkindofassistancecanmakeaninstitutionrunitselfsustainably?AndhowamI,aguywhohasbeendoingmonitoringandevaluation,supposedtohelp?
Developing Evaluation Capacity, but Which Capacity?
Samer Hachem
How harmoniously does the evaluation function operate with the other entities? How clear are the processes and structures surrounding evaluation? How well is knowledge packaged and delivered for decision making? How far does leadership accept this knowledge as a guide for action?
95A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Morgan’sbookisbasedonthepremisethatalmostallourthinkingaboutorganizationsisbasedononeoracombinationoftheeightmetaphors.Aswelookattheinstitutionsandorganizationsaroundusintheworldofdevelop-ment—developmentagencies,ministries,civilsocietyorganizations,and the like—wecanindeedseethesemetaphorsatplay.
Samer Hachem is a manager in the Operations Evaluation Department of the African Development Bank. Prior to joining OPEV, he worked with the AfDB’s Quality Assurance and Results Department, where he focused on monitoring the Bank’s performance on aid effectiveness, and helped introduce the Development Effectiveness Review and a new results management system.
Samer started his career in management consulting, focusing on the financial services sector, and leading monitoring and evaluation in large-scale change management programs. He then joined UNDP HQ in NY, contributing to the roll out of new results-focused management policies and systems. He also worked as a consultant supporting West African countries’ efforts to evaluate their capacities in managing for development results and developing improvement plans.
considerable progress has beenmadeinM&Eoverthelast20years.Duringthisperiod,theUnitedNationsDevelopmentProgramme(UNDP)hasplayedakeyroleinstrengtheningM&Ecapacityandbolsteringtheroleofotherfunders, includingthatoftheAfricanDevelopmentBank.
National Evaluation Capacity Development: Past and Future
Directions
Mohamed Manai, Manager, AfDB
Any national strategy for evaluation capacity building must emanate from the country itself through consultation with all stakeholders regarding national development plans or strategies for poverty alleviation and growth. Donors must coordinate their actions based on the resulting guidelines and national choices and work together with governments.
97A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group
Anynational strategy for evaluationcapac-itybuildingmustemanatefromthecountryitselfthroughconsultationwithallstakehold-ersregardingnationaldevelopmentplansorstrategiesforpovertyalleviationandgrowth.Donorsmustcoordinatetheiractionsbasedontheresultingguidelinesandnationalchoicesandworktogetherwithgovernments.
A good example of this is the case of theDemocratic Republic of of Congo (DRC),wherethegovernmentsoughtsupportfromtheAfricanDevelopmentBank(AfDB),theWorldBank, and theUNDP tomodernizepublicadministration.Theseinstitutionswereengagedtoacttogetherinacoordinatedframe-work.TheUNDPcompleted thediagnosticstudyofnationalcapacity,whiletheAfDBandtheWorldBanksetupaframeworkforjointimplementationofresults-basedmanagementsystemassistance.Suchorganizationoftasksavoidsduplicationandoverlapofefforts.TheCLEAR(theRegionalCentersforLearningonEvaluationandResults) initiative,promotedbytheWorldBankandsupportedbytheAfDBandotherdonors,willbuildonthediagnosisofnationalcapacityassessmentscarriedoutbytheUNDP.Itwillnotreinventthewheel,but,instead,buildonexistingachievements.
Source:Lahey(2006).National Evaluation Capacity Development: Practical tips on how to strengthen National Evaluation Systems,pps.14-15.
FONDS AFRICAIN DE DÉVELO
PPEMENT
AFRI
CAN D
EVELOPMENT FUND
BANQUE
AFRICAINE
DE DÉVELOPPEMENT
About the AfDB: The overarching objective of the African Development Bank Group is to foster sustainable economic development and social progress in its regional member countries (RMCs), thus contributing to poverty reduction. The Bank Group achieves this objective by mobilizing and allocating resources for investment in RMCs and providing policy advice and technical assistance to support development efforts.The mission of the Operations Evaluation Department is to enhance the development effectiveness of the AfDB in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge
Operations Evaluation Department, African Development BankWebsite: http://operationsevaluation.afdb.org/Write to us: [email protected]
From Experience to Knowledge … From Knowledge to Action … From Action to Impact