A Guide to Monitoring and Evaluation of Capacity-Building ...

A Guide to Monitoring and Evaluation ofCapacity-Building Interventions in the Health

Sector in Developing Countries

March 2003

MEASURE Evaluation Manual Series, No. 7

MEASURE Evaluation Project

Anne LaFond, MS, JSI Research and Training

Lisanne Brown, PhD, Tulane University

The manual series is made possible by support from USAID under the terms of CooperativeAgreement HRN-A-00-97-00018-00. The opinions expressed are those of the authors, and donot necessarily reflect the views of USAID.

March 2003 Printed on recycled paper

Other Titles in the Manual Series

NO. 1 Evaluando Proyectos de Prevención de VIH/SIDA: Un Manual con Enfoqueen las Organizaciones No Gubernamentales. July 2000.

NO. 2 Quick Investigation of Quality (QIQ): A User's Guide for Monitoring Qual-ity of Care. February 2001.

NO. 3 Sampling Manual for Facility Surveys for Population, Maternal Health,Child Health and STD Programs in Developing Countries. July 2001.

NO. 4 Measuring Maternal Mortality from a Census: Guidelines for Potential Us-ers, July 2001.

NO. 5 A Trainer's Guide to the Fundamentals of Monitoring and Evaluation forPopulation, Health, and Nutrition Programs, 2002.

NO. 6 Compendium of Indicators for Evaluating Reproductive Health Programs,August 2002.

Recommended CitationLaFond, Anne and Brown, Lisanne. A Guide to Monitoring and Evaluation of Capacity-Building Inter-ventions in the Health Sector in Developing Countries. MEASURE Evaluation Manual Series, No. 7.Carolina Population Center, University of North Carolina at Chapel Hill. 2003.

Acknowledgements i

Acknowledgements

We wish to acknowledge the contributions and support of a number of individuals and institu-tions that enabled the successful completion of this document. Ray Kirkland and Krista Stewartof USAID were instrumental in the conception of the Guide. Sara Pacque-Margolis of USAIDprovided the support to see it through to completion. Our sincere gratitude also goes to severaltechnical reviewers for their constructive and instructive comments on earlier versions of theGuide. They are: Alfredo Fort (PRIME II), Diane Catotti (IPAS), Alison Ellis (MSH), Leo Ryan(CSTS/ORC Macro), Eric Sarriot (CSTS/ORC Macro), Fred Carden (IDRC), and Doug Horton(ISNAR). Kate Macintyre contributed her ideas and encouragement, as well as provided theSAIDIA case material. Catherine Elkins and Kate Macintyre contributed to the MEASUREworking paper on measuring capacity in the health sector, which provided a basis for this guide.Thom Eisele and Cira Endley reviewed and analyzed capacity-measurement tools and practices.Case examples of capacity measurement were developed with the cooperation of PRIME /INTRAH; SAIDIA; NGO Networks for Health; and PATH (in a workshop setting). Finally, weare grateful to the many adventurous organizations and individuals working to build capacity inthe health sector in developing countries. Their experimentation in capacity-building monitoringand evaluation is commendable and deserves further study. This guide would not have been pos-sible without the support of the Offices of Health and Population at the United States Agency forInternational Development (Contract Number: HRN-A-00-97-00018- 00).

Prologue iii

Prologue

Capacity development1 has moved to center stage of the agendas of development organizations.Substantial sums are being invested in capacity-building programs. Yet, their design and man-agement leave much to be desired. Marred by untested, unrealistic assumptions, the results ofmany programs fall short of their goals and expectations.

“Evaluations are needed to test the theories and assumptions on which capacity developmentprograms are based, to document their results, and to draw lessons for improving future pro-grams. However, few capacity development programs have been systematically and thoroughlyevaluated” (Horton et al., 2000).

1 Capacity building and capacity development are used interchangeably throughout this document.

Acronyms and Abbreviations v

List of Acronyms and Abbreviations

API AIDS Program Effort IndexBASICS Basic Support for Institutionalizing Child SurvivalCHW Community Health WorkerDHS Demographic and Health SurveyDOSA Discussion-Oriented Organization Self-AssessmentFHI Family Health InternationalFP Family PlanningFPE Family Planning EffortFPEI Family Planning Effort IndexFPMD Family Planning Management Development ProjectFPPE Family Planning Program EffortHR Human ResourcesIAI Institutional Assessment InstrumentIDA Institutional Development AssessmentIDRC International Development Research CentreIEC information, education, and communicationIHFA Integrated Health Facility AssessmentIISD International Institute for Sustainable DevelopmentISNAR International Service for National Agricultural ResearchM&E Monitoring and EvaluationMDA Management Development AssessmentMEASURE Monitoring and Evaluation to ASsess and Use REsultsMES Materials, Equipment, and SuppliesMFSS Management/Financial Sustainability ScaleMIS Management Information SystemMOH Ministry of HealthMOST Management and Organizational Sustainability ToolMSH Management Sciences for HealthNGO Nongovernmental OrganizationOCAT Organizational Capacity Assessment ToolOSI Outcome Sustainability IndexPASCA Program for NGOs that provide HIV/AIDS services in Central AmericaPHR Partnership for Health ReformPI performance improvementPROSE Participatory, Results-Oriented Self-EvaluationPSI Program Sustainability IndexRH Reproductive HealthSAIDIA Local Kenyan NGOSFPS Santé Familiale et Prévention du SIDASTD Sexually Transmitted DiseaseTOT Training of TrainersWHO World Health Organization

Table of Contents vii

Table of Contents

Acknowledgements.......................................................................................................................... i

Prologue ......................................................................................................................................... iii

List of Acronyms and Abbreviations.............................................................................................. v

About This Guide............................................................................................................................ 1Structure of the Guide................................................................................................................. 2

Introduction..................................................................................................................................... 3Defining Capacity-Building Monitoring and Evaluation ........................................................... 4Capacity-Building M&E Has Many Roles ................................................................................. 5

Part 1. Concepts, Definitions, and Attributes of Capacity and Capacity Building ........................ 7Why Build Capacity?.................................................................................................................. 7What is Capacity Building? ........................................................................................................ 7Useful Definitions ...................................................................................................................... 7Attributes of Capacity and Capacity Building............................................................................ 7Capacity Building Is Behavior Change ...................................................................................... 9Why Monitor and Evaluate Capacity Building?....................................................................... 11What Is Different about M&E of Capacity Building?.............................................................. 11Implications for Capacity-Building M&E ................................................................................ 12Summary for Managers and Evaluators.................................................................................... 12

Part 2. Understanding the Role of Capacity in the Health Sector: Introducing a ConceptualFramework ........................................................................................................................ 15

Overview Framework: The Role of Capacity in the Health Sector .......................................... 15Capacity at a Single Level ........................................................................................................ 17Defining Variables Related to Capacity and Performance ....................................................... 18Using These Conceptual Frameworks ...................................................................................... 25Summary for Managers and Evaluators.................................................................................... 26

Part 3. Monitoring and Evaluating Capacity-Building Interventions........................................... 27STEP 1 Define the Purpose of the Evaluation......................................................................... 28STEP 2 Define Performance Objectives.................................................................................. 30

Defining Performance........................................................................................................... 30STEP 3 Mapping Capacity: Build a Conceptual Framework for a Specific Capacity-Building

Intervention ................................................................................................................ 32When to Map Capacity ......................................................................................................... 32How to Map Capacity ........................................................................................................... 33Single-Level Capacity Mapping ........................................................................................... 34Multi-Level Capacity Mapping............................................................................................. 36Dealing with Context ............................................................................................................ 36Interpreting and Using Capacity Maps ................................................................................. 41

viii

STEP 4 Identify Capacity Indicators ....................................................................................... 44What Are Capacity Indicators?............................................................................................. 44Working with Capacity Indicators ........................................................................................ 45Lessons for Indicator Development ...................................................................................... 45

STEP 5 Identify Appropriate Methodological Approach and Sources of Data....................... 55Methodological Approaches and Challenges........................................................................ 55Tackling Methodological Challenges ................................................................................... 56Sources of Data ..................................................................................................................... 57Tools for Measuring Capacity at Different Levels ............................................................... 58

STEP 6 Develop an Implementation and Dissemination Plan ................................................ 65

Part 4. Summary Checklist: Steps for Designing a Capacity-Building M&E Plan ..................... 67Checklist: Steps in Designing a Capacity-Building M&E Plan ............................................... 67

Annex A. Example of Scoring Used for Measuring Capacity Building in Training, PRIME I . 71

Annex B. Example of Results of PRIME Training Capacity Index .......................................... 75

Annex C. Key Internet Resources for Monitoring and Evaluating Capacity-BuildingInterventions ............................................................................................................... 77

Annex D. Capacity Mapping and Performance Improvement Compared .................................. 91

Glossary ........................................................................................................................................ 93

Bibliography ................................................................................................................................. 95

Tables, Figures and Boxes ix

Tables

Table 1 The Use of Assessment vs. M&E in Capacity-Building Intervention ........................ 4

Table 2 Capacity and Performance Variables Defined .......................................................... 20

Table 3 Questions Posed by Different Types of Capacity-Building M&E............................ 28

Table 4 Examples of Capacity Indicators in Current Use in Health Programs ...................... 52

Table 5 Examples of Performance Indicators in Current Use in Health Programs................ 53

Table 6 Example of a Table of Data Sources for an Organizational Assessment .................. 60

Table 7 Capacity Measurement Tools .................................................................................... 61

Figures

Figure 1 Overview of Capacity in the Health Sector............................................................... 16

Figure 2 Health System Capacity ............................................................................................ 19

Figure 3 Health Service and Civil Society Organization Capacity ......................................... 21

Figure 4 Health Program Personnel Capacity.......................................................................... 23

Figure 5 Individual/Community Capacity .............................................................................. 24

Boxes

Box 1 Capacity Measurement Case Examples ...................................................................... 1

Box 2 Measuring the Effectiveness of Capacity Building in Training: PRIME I ................. 8

Box 3 Examples of Organizational Capacities .................................................................... 10

Box 4 Six Steps for Developing a Capacity-Building M&E Plan ....................................... 27

Box 5 DO'S AND DON’TS of Developing an M&E Plan for a Capacity-BuildingIntervention ............................................................................................................... 29

Box 6 Characteristics of a Good Performance Objective .................................................... 31

Box 7 The Process of Capacity Mapping............................................................................. 32

x

Box 8 Questions to Guide Discussion for Capacity Mapping ............................................. 34

Box 9 Guidance on Capacity Mapping ................................................................................ 35

Box 10 Questions to Guide Discussion on the External Environment and Its Influence onOrganizational Capacity ............................................................................................ 42

Box 11 Examples of Capacity Indicators from Non-health Sector Capacity-BuildingInterventions.............................................................................................................. 46

Box 12 PASCA: From Self-Assessment to External Assessment ......................................... 56

Box 13 Advantages and Disadvantages of Self-Assessment and External AssessmentTechniques................................................................................................................. 59

Maps

Map 1 Organizational Capacity Map - Single Level ............................................................ 37



Map 4 Community Capacity Map on Multiple Levels ......................................................... 40

Map 5a Mapping Capacity First Iteration .............................................................................. 48

Map 5b Mapping Capacity Second Iteration .......................................................................... 49

Map 6 Community Capacity Map on Multiple Levels with Indicators................................ 50

About This Guide 1

About This Guide

This guide has grown out of the collectiveexperience of health and development organi-zations working to build health sector capac-ity in developing countries. The focus of theGuide is the measurement of capacity for thepurpose of monitoring and evaluating capac-ity-building interventions. It responds to ademand among public health planners, evalu-ators, and practitioners for advice on assess-ing the many aspects of health programmingthat fall under the rubric of capacity building.

The purpose of this guide is to assist healthplanners and evaluators to

� gain a clear understanding of the conceptsof capacity and capacity building

� critically evaluate the strengths and limi-tations of current approaches to capacitymeasurement

� design a capacity-building M&E plan thatoutlines a systematic approach to meas-uring capacity and assessing the results ofcapacity-building interventions in thehealth sector

The Guide was developed based on a requestfrom the United States Agency for Interna-tional Development under the MEASUREEvaluation Project.

Many readers of this guide may not be awarethat there is a lack of widespread experiencein the field of capacity-building M&E in thehealth sector. Capacity-building programsproliferate. Yet, methods for testing andtracking their results are rare. We have there-fore based the advice in this guide mainly onlessons learned from current practices in ca-pacity assessment (see Table 1 for discussionof the differences between assessment andM&E). Sources include: a review of the stateof the art of capacity measurement (Brown,

LaFond, and Macintyre, 2001); a review ofcapacity-building measurement tools and in-dicators; formal and informal consultationswith practitioners; and an in-depth explorationof four different capacity measurement expe-riences (Box 1). The Guide also draws onlessons learned about capacity-buildingmonitoring and evaluation in other sectors,such as agriculture and housing, and on newevaluation approaches designed to supportlearning in development programming (Hor-ton et al., 2000; Morgan, 1997; Earl, Carden,and Smutylo, 2001).

From the discussion that follows on the con-cept of capacity building and capacity meas-urement techniques readers will come to un-derstand why this guide is neither prescriptivenor exhaustive. Standardized approaches tomonitoring and evaluating capacity-buildinginterventions are not found because of thewide variety of circumstances in which ca-pacity building takes place. Capacity buildinghas been applied to actions as distinct as pol-icy formulation, supplying basic health com-modities, and identifying danger signs of

Box 1: Capacity Measurement CaseExamples

� SAIDIA, a health and community devel-opment nongovernmental organization(NGO) in Kenya;

� the PRIME I and PRIME II index ofcapacity of training institutions;

� A Workshop on Sustainability and Ca-pacity Building hosted by PLAN Inter-national in May 2001, in Dakar, Senegal;and

� MEASURE Program Technical Assis-tance to NGO Networks for Health.

2

malnutrition. In short, capacity building de-mands adaptation to its context and capacity-building evaluation techniques must reflectthis potential variation. The Guide acknowl-edges this and other challenges by providing alink between the theoretical and practical as-pects of capacity measurement in the healthsector and offering an approach to monitoringand evaluation that is relevant in a variety ofsettings.

It is also important to keep in mind that themonitoring and evaluation of capacity build-ing, while singled out for discussion in thisdocument, is normally part of an overall planor system for monitoring and evaluating ahealth program or health sector intervention.This guide should therefore be used as a toolfor orienting planners to capacity measure-ment in the context of developing a project-level or overall program-level performance-monitoring plan (particularly programs wheresustainability and scaling-up are a centralconcern). As such, it will aid the process ofthinking through the role capacity and capac-ity measurement play in improving perform-ance.

Structure of the GuidePart 1 of the Guide briefly discusses attrib-utes of capacity and capacity building, andhow these attributes influence M&E ap-proaches.

Part 2 introduces a series of conceptualframeworks for understanding the role of ca-pacity in the health sector and illustrates pos-sible capacity variables (�) at each level ofthe health system.

The heart of the Guide is found in Part 3,which suggests a 6-step approach to devel-oping an M&E Plan for Capacity-buildingthat centers on the process of capacity map-ping (�). Mapping involves the constructionof a visual framework that helps the evaluatorunderstand relationships (or assumed relation-ships) among the many factors that contributeto or detract from capacity and, ultimately,performance. Mapping can be used to identifyuntapped, constrained, or missing elements ofcapacity. It also can be used to guide inter-vention choices and to build a monitoring andevaluation framework. Part 3 also commentson indicator selection for M&E and practicallessons from field experience, as well asmethods and data sources, and disseminationof results. The indicators and tools referencedin this section are provided as examples tostimulate thinking and discussion about ca-pacity-building and M&E strategies ratherthan as prescribed approaches.

Part 4 concludes the Guide with a summaryand checklist for developing a capacity-building M&E plan. Annexes contain detailsof M&E approaches and a summary of Web-based resources on capacity-building M&E.The Glossary at the end of the Guide explainsmany of the technical words and jargon usedin the field of monitoring and evaluation. Inthe text they are marked with the followingsymbol: �

Introduction 3

Introduction

Over the last decade, capacity building hasbecome as central to the business of develop-ing health systems in lesser-developed coun-tries as providing financial resources and ap-plying the latest science. Capacity is believedto contribute directly to improving perform-ance in the health sector, and is thought toplay an important role in sustaining adequateperformance over time. Despite increasedattention to capacity, experience in gaugingthe effectiveness of capacity-building inter-ventions in the health sector is still limited.Unlike other aspects of health-related moni-toring and evaluation (M&E), capacity meas-urement is not supported by a comprehensivehistory of theory and practice. While methodsfor monitoring and evaluating health servicecoverage, access, and quality are well ad-vanced, there are few tried and true ap-proaches for capturing “the interim state orprocess that reflects the ability to achieve andsustain coverage, access, and quality overtime” (Brown, LaFond, and Macintyre, 2001).Thus, capacity measurement in the healthsector is both new and experimental.

There are intrinsic challenges to measuringcapacity that are reflected in the concept androle of capacity itself. For example, capacityderives its relevance from the contribution itmakes to performance. There are endless ar-eas where performance is required in thehealth sector, and an equally wide range ofpossible capacity variables that influence per-formance. In addition, contextual factors (orfactors outside the control of most healthsector actors �) can have a strong influenceon capacity or the desired outcome of capac-ity-building intervention. These and othercharacteristics of capacity and capacitybuilding explain why there are no gold stan-dards for capacity-building M&E. There is noshort list of valid indicators of capacity in thehealth sector, nor are there standardized

measurement tools applicable to every capac-ity-building experience.

Many of these challenges have also discour-aged widespread testing of methods of capac-ity-building monitoring and evaluation. Theextent of experience is so limited that, at thisstage, capacity measurement is considered tobe an art rather than a science. Evaluatorsmust therefore approach M&E of capacity-building interventions with a willingness totest strategies and share what they havelearned in order to build a body of theory andpractice.

Despite the conceptual and practical chal-lenges of tackling capacity measurement,there are a number of reasons to put energyand time into developing a sound approach tomonitoring and evaluation of capacity-building interventions. The most significantreason is that measurement is an importantpart of achieving capacity-building and per-formance goals. Monitoring and evaluationcan help health program professionals under-stand the relationship between capacity-building interventions, capacity and perform-ance, and to focus strategies used for im-proving performance. Specifically, monitor-ing and evaluation can help answer a range ofquestions about

� the process of capacity change (how ca-pacity building takes place),

� capacity as an intermediate step towardperformance (what elements of capacityare needed to ensure adequate perform-ance), and

� capacity as an outcome (whether capacitybuilding has improved capacity)

4

In this guide, when we talk about monitoringand evaluation of capacity building orcapacity development, we are mainly inter-ested in the last question, that is, measuringchanges in capacity and linking them (directlyor indirectly) to capacity-building interven-tions.

Defining Capacity-BuildingMonitoring and EvaluationMost capacity measurement experience todate has emphasized capacity assessmentrather than M&E (Brown, LaFond, and Mac-intyre, 2001). Assessment normally takesplace at the beginning of an intervention aspart of an organizational diagnosis or forma-tive design process. Evaluators can learn agreat deal from capacity assessment tools (aswe have in developing this guide). However,it is worth noting that while capacity assess-ment is an important first step in planning acapacity-building intervention, capacity-building M&E differs from assessment byvirtue of its explicit focus on measuringchange. Capacity-building monitoring andevaluation tracks or identifies changes in ca-pacity that take place in the course of a ca-pacity-building intervention. It uses statedobjectives for capacity building and perform-

ance improvement as a reference for gaugingprogress. As such, it guides program man-agement as well as informs funding agenciesabout the results of capacity-building invest-ments. A final aspect of M&E (as opposed todiagnosis or assessment) is the use of con-ceptual frameworks that make assumptionsabout the relationship between different vari-ables that influence capacity and perform-ance. Table 1 describes many of the differ-ences between capacity assessment and M&E.

Capacity monitoring normally would be usedto understand the effectiveness and efficiencyof a capacity-building intervention duringimplementation (i.e., is capacity improvingand at what cost?), to contribute to strategic oroperational decisions related to capacitybuilding, or to enable a periodic look at a pro-gram or system. Capacity evaluation is nor-mally more complex than monitoring, and is

Table 1: The Use of Assessment vs. M&E in Capacity-Building Intervention

“In the evaluation of capacity devel-opment, the impact metaphor shouldbe avoided. The militaristic impact

metaphor fails to capture the essen-tial features of capacity development,

which is a process of change andgrowth.” (Horton, 2002).

Capacity Assessment Capacity Monitoring and Evaluation� Purpose: diagnostic or descriptive; defines

constraints� Purpose: predictive; for accountability or

comparisons; gauges results� Measures gap between actual and desired

performance� Measures results or progress toward de-

sired results� Findings are used for internal purposes

(design and planning)� Findings are used for internal and external

purposes (management; accountability)� One time measurement � Often uses repeat measurement� Action oriented � Action, analysis and accountability ori-

ented� Looks broadly at existing situation � Uses conceptual frameworks to discern

relationships between variables

Introduction 5

conducted to gain understanding of the rela-tionship between capacity-building interven-tions and capacity outcomes, or the links be-tween capacity and performance variables.The term “impact evaluation” � is not ap-propriate or useful in the context of capacity-building M&E because of the difficulty ofquantifying many elements of capacity andattributing capacity change to any single in-tervention or even a range of interventions.

Capacity-Building M&E Has ManyRolesA final introductory observation relates to therole that measurement plays in a capacity-building intervention. Many experienced ca-pacity-building practitioners feel that capacitymeasurement cannot be separated from the

process of building capacity itself. Programmanagers often use capacity assessment toolsto raise awareness about capacity problems,stimulate commitment to improving capacityamong stakeholders, and for setting self-determined benchmarks. The focus is internal.In practice, capacity-building M&E is oftenencouraged (or required) by externalstakeholders to be used mainly for account-ability. Defining the purpose of M&E istherefore not always easy for managers andevaluators. The discussion that follows con-siders the pros and cons of these various ap-proaches and informs critical measurementchoices. It begins with a discussion of therationale for capacity-building M&E and ex-plores the concept of capacity and its role inimproving performance.

Concepts, Definitions, and Attributes of Capacity and Capacity Building 7

Part 1 Concepts, Definitions, and Attributes of Capacity andCapacity Building

Why Build Capacity?In the context of results-based programming,resources are invested in different aspects ofthe health sector with the ultimate aim of en-hancing health system performance and im-proving the health of populations. Translatingthese resources into sustained performanceoften requires new or improved capabilities inindividuals and organizations (includingcommunities) operating in the health sector.Capacity represents the potential for usingresources effectively and maintaining gains inperformance with gradually reduced levels ofexternal support.

What is Capacity Building?Used alone, the term capacity building is in-tangible and vague. What constitutes capacitybuilding in practice can vary enormously, andthe concept continues to develop as field ex-perience grows. In the early days of capacity-building intervention, many practitionersequated capacity building with training. Ifthere was a gap in performance, the solutionwas often to hold a workshop to “retrain” or“refocus” the individuals whose performancewas faltering. Organizational developmentexperts and field-level capacity-building ef-forts inform us today that individual skills areonly part of the complex mixture of elementsthat constitute capacity to perform a certainfunction or groups of functions effectivelyand consistently over time. Individual healthworkers, no matter how skilled, are unlikelyto deliver essential health and family planningservices effectively without adequate suppliesand equipment, proper motivation and sup-port, a good relationship with the communityserved by the health center, and so on. Ca-pacity building may be required in all of these

and other areas to ensure performance goalsare met.

Useful Definitions �It is useful to start with definitions of capac-ity, capacity building and performance toguide measurement efforts and M&E plan-ning. This guide returns frequently to suchissues because meaningful capacity measure-ment depends on clear understanding of ca-pacity and its role in the health sector.

Capacity is “the ability to carry out statedobjectives” (Goodman et al, 1998). It has alsobeen described as the “stock of resources”available to an organization or system as wellas the actions that transform those resourcesinto performance (Moore, Brown, and Honan,2001).

Capacity building (or capacity development)is a process that improves the ability of aperson, group, organization, or system tomeet objectives or to perform better.2

Performance is a result or set of results thatrepresent productivity and competence re-lated to an established objective, goal orstandard.

Attributes of Capacity and CapacityBuildingThe definitions of capacity and capacitybuilding above reflect certain attributes ofeach concept that inform this guide’s ap-proach to monitoring and evaluation. Theseattributes are as follows: 2 For other definitions of capacity building, see Brown,LaFond and Macintyre, 2001.

8

� Capacity building can be defined onlyin terms of a specific objective or goal.

In the health sector, capacity does not existfor its own sake. Health planners and manag-ers are concerned with capacity because itenables performance. For example, a healthfacility that experiences regular stock-outs ofpharmaceuticals might require additional ca-pacity in financial planning or supplies man-agement (i.e., interventions that are specific tothe particular performance goal of commoditysupply). It follows that a capacity develop-ment strategy for improving pharmaceuticalsupply would call for a different approachthan one aimed at strengthening communityinvolvement in health. The link between ca-pacity and performance, therefore, serves asthe guide for both programming and evalua-tion of capacity-building interventions. Im-proved performance, in turn, is a good indi-cator of success in capacity development.

� Capacity (and capacity building) aredynamic and volatile.

Capacity can be perceived as a moving target.At any given time, capacity can improve ordecline. It often develops in stages that indi-cate improved readiness to influence perform-ance (Goodman et al., 1998). Capacity build-ing, therefore, is an ongoing process (the de-velopment of abilities), whose stages can bemeasured as “development outcomes” �

through monitoring and evaluation. The dy-namic nature of capacity is often a reflectionof the many different forces that influence itsdevelopment or decline.

� Capacity building is multidimensional.Capacity building can be described in termsof levels. In the health sector, capacity is re-quired at four different levels: health system,3organization, health personnel, and commu-nity. Yet, to date, most capacity-building ex-perience and measurement have focused onorganizational and health personnel capacity.In practice, capacity at one level is often in-fluenced by actions at other levels. A singlemissing aspect of capacity rarely explainsperformance failures. The PRIME project(Box 2), for example, constructed an index ofthe capacity of training institutions that in-cluded 13 critical elements, ranging from po-litical support for training in reproductivehealth to community involvement in training(Fort, 1999).

Analysis of capacity levels through measure-ment encourages evaluators to think in termsof complex, multifaceted systems. Connec-tions and forces within a system are critical to 3 Some have labeled this level institutional develop-ment (Kotellos, 1998; INTRAC, 1998), while othersuse the terms organization and institution inter-changeably. To avoid confusion, we have adopted theterm system.

Box 2: Measuring the Effectiveness of Capacity Building in Training:PRIME I

The PRIME I project provided technical assistance to strengthen the capacity oflocal institutions in developing countries to train health personnel for reproductivehealth (RH) service delivery. A key M&E strategy for this project was developmentof a capacity index specific to the features of RH training institutions. The indexwas tested in 14 countries and later revised and applied to monitor the results ofPRIME’s capacity building in training activities. Detailed reports of these evaluationsin El Salvador, Dominican Republic, Ghana, and other countries are available from thePRIME II project (Catotti, 1999; Ampomah, 2000; Luoma, 2000; www.prime2.org).


understanding constraints to capacity and howto overcome them. Paying too much attentionto one part of the organization or system maylimit results at the overall organization orsystem level and fail to improve overall per-formance (Morgan, 1997). Take the exampleof delivering immunization services at theorganization level. The effectiveness of thisservice depends on elements that go beyondthe capacity of the facility alone. The ColdChain � must function from the central levelto the facility to ensure vaccine viability.Civil service norms, regulations, and salarylevels can influence health worker motivationand acceptance of the value of immunizationamong caregivers and encouragement fromcommunity leaders can affect service utiliza-tion. If performance falters (i.e., coveragedeclines), it may be the result of limited ca-pacity at the facility or other levels. Anevaluation framework should consider allthese variables, although it may focus meas-urement efforts on a smaller number of them.

� Capacity depends on the context.Contextual factors or elements of the externalenvironment influence capacity directly andindirectly. Contextual influences include cul-tural, social, economic, political, legal, andenvironmental variables. The influence ofthese factors may be crucial to the success ofcapacity building, yet they are often difficultto control or measure. For example, SierraLeone’s Ministry of Health (MOH) may havethe capacity to deliver childhood immuniza-tion services. However, frequent political in-stability in the country can challenge that ca-pacity and reduce performance (e.g., immuni-zation coverage) dramatically. Taking a moregeneral example, the stagnation and decline ofeconomic growth that occurred in Africa inthe 1980s severely undermined public sectorcapacity to meet recurrent costs for salariesand supply of basic health commodities. Evenwell-established health systems, such asGhana’s, were unable to withstand the decline

in health sector financing, and capacitygradually eroded to a very low level (LaFond,1995).

Capacity Building Is BehaviorChangeIn addition to these attributes, current thinkingabout capacity building reflects two ways ofcapturing the changes expected as a result ofintervention. Traditional approaches to ca-pacity building concentrate on the internalfunctioning of organizations and systems(structures, strategies, staff, and skills).Morgan (1997), however, notes the necessityof considering the “macro” aspect of capacitybuilding that relates to the behavior and op-erations of groups of organizations or indi-viduals and their role in wider systems (suchas the role of public sector health systems,ministries of health, or district-level healthunits in rural health improvement). In general,there is more experience working on andmeasuring capacity at the micro level than atthe macro level.

Taking both a micro and macro look at ca-pacity building suggests that capacity devel-opment goes beyond a simple technical inter-vention. It is to a great extent focused on in-ducing behavior change, a process that in-volves learning, moderating attitudes, andpossibly adopting new values at individual,organization, and system levels. Therefore,the focus of capacity-building interventionsand M&E must capture related conditions andconcepts such as motivation, culture, andcommitment, as well as changes in resourceavailability, skill levels, and managementstructure (Morgan, 1997; James, 2001). Ex-amples of different types of organizationalcapacities are found in Box 3.


Why Monitor and Evaluate CapacityBuilding?Given the nature of capacity development—the volatility of capacity, its many levels, andlinks to performance—some authors describecapacity building as a high-risk investment(UNICEF, 1999). Yet, most developmentorganizations agree that facilitating growth incapacity among local partners’ systems, or-ganizations, and communities is key to thesuccess of social development overall. Assuch, all stakeholders need dependable meth-ods for answering such questions as� What capacity exists now, and how does it

affect performance?� What improvements in capacity or new

kinds of capacity are required?� Is capacity being built? Is the capacity-

building intervention focused on the rightelements?

� What has been learned about capacity-building strategies?

� How does capacity contribute tosustainability?

In addition, there is value in not restrictingmonitoring and evaluation of health and de-velopment interventions to a few importantoutcomes or results (i.e., quality, coverage,and health status). Organizations and systemsproduce many different and critical effects.For strategic purposes, and to manage changein programs, organizations and systems ef-fectively, regular information on a number ofoperational indicators is required (Moore,Brown and Honan, 2001). A well-definedmonitoring and evaluation strategy will helpmake sense of these many facets of capacityand performance. Monitoring and evaluationshould help local practitioners and their ex-ternal partners to think strategically aboutcapacity development and to learn, throughpractice, what works under different circum-stances. At the same time, systematic meas-urement of capacity contributes to results-

based management of programs where capac-ity building is part of the overall strategy forimproving performance.

What Is Different about M&E ofCapacity Building?Traditionally, monitoring and evaluation fo-cuses more on measuring performance andless on the way performance is achieved orsustained. In contrast, capacity-building M&Efocuses fundamentally on processes (e.g.,building alliances, mobilizing communities,decentralized planning, learning) and otherqualitative aspects of individual or organiza-tional change (e.g., motivation to perform)that contribute to better performance. Conse-quently, M&E of capacity building oftenseeks to capture actions or results that oftenare not easily measured.

That said, results of capacity building are asimportant as processes. In capacity-buildingintervention, the process and result of capac-ity building becomes the “intermediate out-come” that is expected to lead eventually toimproved and sustained performance. Ex-ploring the links between changes in capacityand changes in performance is therefore key.However, it often involves considerablespeculation about the capacity needed toachieve those goals. One of the main gaps inthe knowledge base that informs capacitymeasurement is the lack of common under-standing of the relationship between capacityand performance. Little is known about whatelements or combinations of elements of ca-pacity are critical to performance. Moreover,there is considerable variation in what con-stitutes “adequate” performance.

12

Implications for Capacity-BuildingM&EClearly, the attributes of capacity and capacitybuilding noted above have implications formonitoring and evaluation. Broadening theconcept of capacity building beyond technicalskills and resources and thinking about ca-pacity building in terms of multiple levels andinfluences helps planners and evaluators tohypothesize about what aspects of capacityare critical to performance and to define entrypoints for targeting capacity-building inter-ventions. A measurement approach shouldalso reflect a clear understanding of the inter-action among different aspects of capacityand how they work (or fail to) work together,particularly with respect to individual andorganizational behavior. These types of vari-ables may be represented by indicators in anevaluation plan, but may require additionalinterpretation to ensure a complete grasp ofcapacity and its role in improving perform-ance.

As noted in the Introduction, it is also impor-tant to keep in mind the conventional wisdomabout how to monitor and evaluate capacity.Conventional wisdom notes that it is not pro-ductive to separate measurement practicesfrom capacity building itself (Morgan, 1997;Horton, 2001; Earl et al, 2001). Because ca-pacity-building M&E focuses on behaviorchange, the success of capacity developmentis often directly related to the extent of own-ership and commitment to the process on thepart of the participants. This commitmentincludes, in some cases, ownership of thedesign, procedures, and reporting of moni-toring and evaluation activities. Applied inthis way, monitoring and evaluation of capac-ity can become a key strategy for improvingperformance. However, many of the M&Emethods that promote ownership (i.e., involveself-evaluation and relying on respondents’perceptions) may also affect the validity offindings. Specifically, they may compromise

the use of capacity-building M&E for ac-countability, predicting performance, ormaking comparisons between different inter-ventions or sites (common reasons for con-ducting evaluation). This theme surfaces oftenin the discussion of capacity-building M&E,and will be addressed in Part 3 of this guide.

Summary for Managers andEvaluators� Capacity is a pre-condition for perform-

ance. Capacity building is used to improveperformance in a variety of ways andsituations.

� Capacity-building M&E is normally partof an overall plan or system for monitor-ing and evaluating a health program orhealth sector intervention.

� There are no standardized approaches forcapacity-building M&E because of thewide variety of circumstances in whichcapacity building takes place. There is noshort list of valid indicators of capacity inthe health sector, nor are there standard-ized measurement tools applicable toevery capacity-building experience.

� Monitoring and evaluation should helplocal practitioners and their external part-ners to think strategically about capacitydevelopment and to learn, through prac-tice, what works under different circum-stances. At the same time, systematicmeasurement of capacity contributes toresults-based management of programswhere capacity building is part of theoverall strategy for improving perform-ance.


� Capacity building in the health sector canbe described and measured in terms offour levels: health system, organization,health personnel, and community. Capac-ity at one level can be influenced by ac-tions at other levels.

� Contextual factors or elements of the ex-ternal environment influence capacity di-rectly and indirectly.

� Capacity development goes beyond asimple technical intervention, focusing onbehavior change in individuals and or-

ganizations. Thus, capacity-building M&Emust capture conditions and concepts suchas motivation, culture, and commitment,as well as changes in resource availability,skill levels, and management structure.

� Any strategy monitoring capacity shouldreflect a clear understanding of the inter-action among different aspects of capacityand how they work (or fail to work) to-gether.

Understanding the Role of Capacity in the Health Sector: Introducing a Conceptual Framework 15

Part 2 Understanding the Role of Capacity in the Health Sector:Introducing a Conceptual Framework

The first step in developing a vision of capac-ity development, and a plan to measure it, isto understand the role capacity plays in thehealth sector in developing countries. Whatare the expectations and assumptions sur-rounding capacity and its relationship to per-formance and health outcomes? Clear think-ing about these variables helps planners de-fine realistic objectives for capacity-buildinginterventions and express desired capacityoutcomes explicitly and precisely. Evaluatorsmust rely on these parameters of capacitybuilding in order to develop a capacity-building M&E plan.

The following series of conceptual frame-works are provided as a reference to helpplanners and evaluators develop their ownvision of the role capacity (and capacitybuilding) plays in the health sector. We havefound that directed discussion using thesetypes of frameworks prior to M&E planningcan stimulate strategic thinking within projector work teams, clarify individual and collec-tive expectations and thereby improve capac-ity-building M&E. Figure 1 – The Overview– illustrates the critical role capacity plays ininfluencing and sustaining performance in thehealth sector. It takes a system-wide view ofcapacity, including all possible levels wherecapacity building might take place. The fourother frameworks (Figures 2-5) take capacityat each level and break it down into definedcomponents: inputs, processes, outputs, andoutcomes (See Table 2). In breaking downcapacity at each level, the frameworks pro-vide a starting point for identifying the keyvariables that influence capacity and perform-ance at that level.

Overview Framework: The Role ofCapacity in the Health SectorHealth system performance depends on ca-pacity. Figure 1 provides an overview of thatrelationship and specifies four levels wherecapacity is needed to ensure performance:system, organization, health personnel, andindividual/community. The diagram suggeststhat capacity contributes to performance at alllevels, and capacity at each level collectivelyenables overall health system performance.

Figure 1 also implies that capacity plays arole in sustaining health system performance.If health system performance remains ade-quate over time (supported by consistent ca-pacity), performance is said to be sustained.Although few health systems in developingcountries can boast this accomplishment, theunderlying aim of capacity developmentshould be a sustained change in resources orbehavior that leads to improved and sustainedperformance. The goal is not short-term gainbut a lasting or robust change in ways of do-ing business that becomes imbedded in thesystem or organization itself.

“Understanding capacity andperformance of individuals andorganizations demands carefulconsideration of their role inlarger systems, and their rela-

tionships within those systems”(Morgan, 1997).

16

Figure 1. Overview of Capacity in the Health Sector

a

Sustainable Health System

Performance

ImprovedHealthStatus

HealthProgramPersonnel

External Environment

Health SystemPerformance

T

I

M

E

Individual/CommunityBehavior Change

Health System

OrganizationOrganizationalPerformance

PersonnelPerformance

Capacity Levels Performance

Individual/Community Capacity

SustainedIndividual/Community

Behavior Change

Sustainability



At the center of the framework is the ultimategoal of capacity building in the health sector:improved health status. Capacity does notdirectly influence health status but contributesto it through its link to performance at system,organization and health personnel levels. Inthis illustration, the health system interactswith individuals or groups of individuals (e.g.,the community) to influence health status.Individuals and communities contribute tohealth system capacity by interacting withproviders and organizations (receiving care,determining priorities, or providing resources)and to health system performance by usinghealth services. In addition, individuals andcommunities can improve their health statusindependent of the health system by promot-ing and adopting preventive measures, such asregular hand washing, not smoking, or eatingwell. Improvements in individual and com-munity capacity should result in sustainedbehavior change over time, representing thislevel’s contribution to sustained health systemperformance and improved health status.

At the perimeter of Figure 1 we mark the in-fluence of environmental or contextual fac-tors, including cultural, social, economic,political, legal, and environmental variablesthat influence capacity and performance at allfour levels (Africa Bureau, 1999; Horton,2001; James 2001). The obvious importanceof these factors for improving and sustainingboth capacity and performance suggests thatspecial efforts are needed for tracking theirstatus overtime. In this guide, we focusmainly on variables that donors, governments,private agencies, and individuals can influ-ence through health sector interventions.However, we also encourage evaluators toidentify and monitor key contextual variablesand examine their relationship to programoutcomes.

Capacity at a Single LevelThe four levels of capacity are detailed furtherin the following related frameworks (Figures2-5).

These conceptual frameworks take a broadlook at capacity at one level to illustrate manyof the potential factors that might come to-gether to influence capacity and performance.The purpose of these frameworks is to showhow capacity can be broken down at eachlevel into inputs, processes, outputs, and out-comes in order to

� identify the different factors that contrib-ute to capacity, and performance

� hypothesize about the potential relation-ships among these factors within a singlelevel

Conceptual frameworks like these differ fromlogical or strategic frameworks in that they donot reflect the linear logic of a particular ca-pacity-building intervention, and its presumedeffect on capacity outcomes. Rather, theyshow the range of all possible variables thatmight influence capacity and performance. Inthis way they help planners at the early de-sign stages to determine the scope and focusof a capacity-building intervention, andevaluators to design valid measures for de-termining the success of those interventions.Conceptual frameworks can become gradu-ally more specific as decisions are made aboutcapacity-building interventions and the ca-pacity and performance changes expectedfrom them.

18

Defining Variables Related toCapacity and PerformanceCapacity inputs represent the resources (hu-man, financial, material, etc.) that contributeto capacity and performance. Processes repre-sent the activities or behaviors at each capac-ity level that transform resources (inputs) intocapacity outputs and outcomes. Capacity out-puts and outcomes are the results of inputsand processes, and indicate products (outputs)and “an ability to carry out stated objectives”(outcomes). In many cases, capacity outcomesare expressed as knowledge, skills and be-havior. Performance is the expected result ofcapacity (a “stock of resources”) and the envi-ronment, the final link in the hypothesizedchain of causality. Performance is defined asresults that represent productivity or compe-tence related to an established objective, goal,or standard.

System LevelFigure 2 refers to the health system. It in-cludes the resources, actors, and institutionsrelated to the financing, regulation, and provi-sion of health actions (Murray and Frenk,1999; WHO 2000).4 The system is seen as acollection of institutions or organizations,plus the personnel in those organizationsworking together to deliver health care and/orpromote better health. The health system per-forms certain functions independent of thoseperformed by the organizations, and person-nel within it, and therefore possesses its owncapacity that can be assessed over time andtargeted for intervention.

Performance at the health system level is of-ten defined in terms of access to services,quality of care, equity, and efficiency, al-though there are many other possible indica-

4 A health action is defined as “any set of activitieswhose primary intent is to improve or maintain health”(Murray and Frenk, 1999).

tors of performance at this level.5 The frame-work includes a range of possible capacityinputs, processes, outputs and outcomes thatcontribute to performance at this level.

The system level is a complex area in whichto define or address capacity development orto assess changes in capacity resulting fromexternal or internal intervention. Despite theuse of an inputs-process-outputs-outcomesframework, in practice, relationships amongelements of capacity are not perfectly linear.Change (or the lack of it) in capacity resultsfrom multiple influences, some of which canbe unexpected (Sarriot, 2002a). Contextualfactors such as political and economic stabil-ity can also play a dominant yet poorly under-stood role in ensuring system capacity. Goodexamples come from health sector reformactivities that seek to improve national healthsector performance by changing sector priori-ties, laws, organizational structures, and fi-nancing arrangements. For instance, the actualresults of legal reform in Zambia wereachieved but not well communicated to healthworkers, which led to internal resistance to“delinking” or separating health workers fromthe civil service (Lake et al., 2000). Despiteaddressing key constraints such as laws orregulations, capacity to manage human re-sources more effectively did not emerge asplanned.

5 The World Health Organization proposed new indi-cators for monitoring health system performance in theWorld Health Report 2000, including measures ofstewardship, financing, resource generation, and serv-ice provision.


Figure 2: Health System Capacity

Infrastructure

Public/private compositionof services

Organizational structure(public sector)

Existing health-relatedlaws, regulations, andpolicies

Information/communicationsystems

Human resources

Leadership

Financial resources (public/private, internal/external)

History and culture of thesystem

Published health policiesand regulations

Formal and informalcoalitions

Sector-wide strategy

Increased local financingof recurrent costs

Improved human resourceavailability in rural areas

Coordinated donorinterventions

Timely analysis anddissemination of nationalhealth information

Health policy making

Enforcement of healthrelated laws andregulations

Health sector strategicplanning

Resource allocation

Resource generation

Financial management

Human resourcedevelopment andmanagement

Donor coordination

Multi-sectoralcollaboration

Information coordination& dissemination

Inputs Process Outputs CapacityOutcomes

Effective health policies

Accountability(financial and programtransparency)

Capacity to assess andcope with internal andexternal change

Financial self-reliance

Effective monitoring ofquality of care

Responsiveness to clientneeds and demands

Efficient/appropriateresource allocation

Use of information forstrategy and learning

Performance



20

Organization LevelFigure 3 depicts a similar categorization ofcapacity variables at the organization levelthat contribute to organizational performance.Performance at the organization level mightbe described in terms of the ability of the or-ganization to produce goods and services toan acceptable standard (e.g., the quality ofcare; coverage of the catchment population).This framework relates to organizationswhose main function might be health servicedelivery (in the public or private sector) andthose considered to be civil society organiza-tions (nongovernmental or nonhealth serviceagencies). Civil society organizations gener-ally are not involved in the direct delivery ofhealth services, but they do influence healthservice delivery, policies, and behaviors inmany societies throughout the world. Civilsociety organizations of particular importance

could be cooperatives, community develop-ment organizations, advocacy groups, infor-mal pressure groups, and others. The MOH isa unique organization for conceptualizingcapacity building since it can be a significantactor at both the system and organization lev-els. The contextual factors influencing or-ganizational capacity are represented at theperimeter of the diagram and include systemlevel factors as well as typical political, eco-nomic, cultural, and other variables.

Table 2: Capacity and Performance Variables Defined

Input

Set of resources, including health personnel, financial resources, space, policyorientation, and program service recipients, that are the raw materials thatcontribute to capacity at each level (system, organization, health personnel,and individual/community).

Process Set of activities, practices, or functions by which the resources are used inpursuit of the expected results.

Output Set of products anticipated through the execution of practices, activities, orfunctions.

Outcome Set of results that represent capacity (an ability to carry out stated objectives),often expected to change as a direct result of capacity-building intervention.

PerformanceSet of results that represent productivity and competence related to an estab-lished objective, goal or standard. The four capacity levels together contributeto overall system-level performance.

Impact Long-term results achieved through improved performance of the health sys-tem: sustainable health system and improved health status. Impact measuresare not addressed in capacity-building M&E.


Figure 3: Health Service and Civil Society Organization Capacity

Infrastructure

Organizationalstructure

Mission

Leadership

Financialresources

Equipment andSupplies

Human resources(technical &managerial)

History andculture oforganization

Strategic and operationalplanning

Human resource managementand development


Logistics/suppliesmanagement

Research and evaluation

Coordination with other units

Resource mobilization

IEC

Advocacy

Community relations andmobilization

Strategic and operationalplans

Staff trained and supported

Functional managementsystems (i.e., suppliesavailable, supervision done)

Functional financialmanagement system (i.e.,resources available, costscontained)

Functional healthinformation andcommunication system(information collected,analyzed and used)

Functional service deliverysystems (i.e., servicesavailable)

Regular IEC and communitymobilization activities


Able to assess and cope withinternal and externalchange

Responsiveness to clientneeds and demands

Financial self-reliance

Stakeholder involvement

Regular supply of essentialcommodities/No stock outs

Acting and learning withinformation

Ability to monitor servicequality and correct gaps asneeded

Able to develop andmaintain workingrelationships with otherorganizations and groups

Performance



22

Health Program Personnel LevelFigure 4 presents the health program person-nel level. The term health personnel refers toall those who perform clinical, managerial,advocacy or other work within the healthsystem. In contrast to the system and organi-zation levels, comprehensive interventions tobuild and maintain capacity are more com-mon at the health personnel level. Ideally, ineach health system there is a plan for pro-ducing and maintaining a cadre of qualifiedpersonnel (personnel with capacity) and pro-viding them with an adequately supportiveenvironment in which to perform effectively.It is less common to find comprehensive or-ganization- and system-level capacity-building plans, although one could argue theyare equally important.

The vast majority of capacity-building inter-ventions in the health sector focus on chang-ing the skills and behavior of health personnelbecause managers and providers play a criti-cal role in ensuring organization and systemlevel capacity and performance. This frame-work attempts to tease out some of the keyvariables at this level that relate directly toindividual health personnel capacity, but wemust acknowledge that organizational contextis equally important. Organizations and sys-tems are often responsible for the inputs andprocesses that enable health personnel to per-form effectively. Thus, there is a significantoverlap between the inputs and processes thatcontribute to capacity at the organization andthe health personnel levels. Many of the vari-ables listed in system and organization levelframeworks also contribute to health person-nel capacity.

Inputs such as sufficient funds, space andmaterials for professional development aretransformed into capacity outcomes throughprocesses such as educational and trainingevents or other opportunities for improving ormaintaining health personnel capacity. Ca-pacity outcomes relate to the knowledge,skills, experience, and motivation resultingfrom inputs and processes. Performance atthis level includes the application of knowl-edge and skills in management, health serv-ices delivery, training, and other related ac-tivities.

Individual/Community LevelThe final figure, Figure 5, represents the “de-mand side” of the equation for capacitybuilding as well as the role individuals andcommunities play in shaping health systemsand improving health status. In addition tosystem, organization, and health personnellevels, capacity is required within individualclients and communities to ensure demand forappropriate services to promote their role incontributing to or influencing service deliv-ery, and to encourage the practice of certainbehaviors conducive to good health. For ex-ample, clients’ capacity to demand improvedor new services or to engage with health carepersonnel and organizations is vital to healthsystem performance and achieving adequatehealth status of the population.


Figure 4: Health Program Personnel Capacity

Financial resources(i.e., salaries, benefits,incentives)

Physical resources• venues• materials• supplies• equipment

National/organizationaltraining policies, plans,and guidelines

Up-to-date informationon appropriate clinicaland managerialpractices

Curricula

Human resources

Pre-service and in-servicetraining events (training oftrainers and trainees)

Training events formanagers (includingsupervisors)

Staff performanceevaluations

Experiential learningopportunities

Professional networking

Staff trained/retrainedas required

Trainerstrained/retrained asrequired

Managerstrained/retrained asrequired

Supervision received

Professional or peersupport networks

Access to information


Knowledge and skillsof trainees

Trainers and traineescontinue to gainexperience

Motivated healthpersonnel

Performance



24

Figure 5: Individual/Community Capacity

Exposure to programs/services

Past experiences with healthservices and preventionpractices

Utilization-enhancing activities(e.g., IEC, accessible services)

Recognition ofsymptoms anddanger signs andactions needed

Ability to articulateneeds and demands

Knowledge ofprevention behavior

Community supportfor preventionbehaviors

Community supportfor community-based health care

Community-basedmobilization andempowerment forinteracting withhealth system

Needs identification andproblem solving

Collaboration

Achieving consensus

Critical reflection

Securing resources

Negotiation

Communication

Inputs OutputsProcess CapacityOutcomes

Recognition ofneed for services

Intention to useservices

Participation incommunity healthcommittees

Community plans

Individual/familyEducationIncomeFamily historySexPerceptions of need/riskWillingness to seek careAbility to pay

Community dimensions•Community history•Citizen participation•Cohesiveness•Leadership•Material and financial resources(internal and external to community)•Social and interorganizationalnetworks•Communication channels•Values•Skills

Performance




Here the individual/community level repre-sents all those who could benefit from andparticipate in the health care system; thus itincludes all current and potential clients of theservices offered and the communities inwhich they live. The inclusion of individualand community capacity in this frameworkrepresents a departure from conventionalthinking on capacity in the health sector. Ref-erences to community capacity are foundmostly in literature on communityempowerment and strategies for improvingcommunity mobilization and participation(Goodman et al. 1998; Israel et al, 1994; Is-rael et al. 1998; and Eng and Parker, 1994).The inputs in this framework represent theresources available to individuals and com-munities. They include individual/familyfactors, community factors, and factors out-side the immediate influence of the commu-nity, such as exposure to health and educationprograms. Processes explain how individualsand communities use their resources to act insupport of their own capacity development.Capacity outcomes relate to knowledge, mo-tivation, skills and behavior that support indi-vidual and the community’s health and well-being. Performance is the actual behavior onthe part of individuals or communities thatmight include interaction with the health sys-tem (participation or advocacy), as well asbehavior that directly influences health out-comes: utilization of health services, selftreatment, compliance, prevention behavior.

Using These ConceptualFrameworksWhile it is useful to separate levels of capac-ity for facilitating M&E planning, these levelsare clearly interdependent, as shown in thenesting of health personnel and organizationlevels in the system level, and the arrowsconnecting individuals/communities to thehealth system and its parts. A health system ismade up of organizations and health person-

nel, and organizations cannot function with-out health personnel. Without individual usersof health services, the other levels cannot be-gin to perform effectively. Going beyond one-dimensional diagrams to understand the dy-namics of capacity building at each level andbetween levels will guide the development ofM&E strategies and techniques.

For example, the processes listed at the sys-tem level in practice are often activities car-ried out by the MOH with support from do-nors and in collaboration with other actors inthe health sector (e.g., NGOs, private compa-nies). There is a clear overlap between systemand organizational capacity since the capacityof the system to carry out certain functionsmay depend directly on the capacity of theMOH to play its organizational role effec-tively. An M&E plan should attempt tomonitor changes at both levels to explain ca-pacity development (or lack of it) well.

The overview diagram that describes the rela-tionship between capacity, performance andsustainability also suggests a logical progres-sion from capacity to performance to sus-tained performance, when in fact both capac-ity and performance can improve or decline inuncoordinated or illogical ways. Because ca-pacity is a fluid notion that responds to manyinfluences, linear frameworks, often used inresearch and evaluation, are sometimes con-sidered too mechanical for monitoring andevaluating capacity. Cause and effect chainsrelated to capacity are seldom linear, sug-gesting the need to break out of a rigid, in-flexible way of thinking.

Figures 2 – 5 suggest one way to look beyondthe linear representation of capacity variablesby depicting the process of capacity develop-ment as a cycle. Once one stage of capacitydevelopment is achieved, capacity outcomesbecome the new inputs and processes for thenext stage of improvement. Indicators in thissense become relative, in that an indicator of

26

capacity expressed as an outcome might bedescribed as another type of variable as ca-pacity improves or declines.

This guide recommends the development ofconceptual frameworks as a useful process forthinking through a capacity-building inter-vention strategy, clarifying expectations ofstakeholders and in hypothesizing the vari-ables that are considered important to pro-gram results in a specific context. However,these tools should be used along with strate-gies such as creative thinking, revisiting as-sumptions, and reflecting on results withstakeholders when conducting capacity-building M&E. Part Three of the Guide willelaborate on the use of frameworks or maps inM&E and discuss these and other strategiesfor understanding changes in capacity andtheir relationship to performance.

Summary for Managers andEvaluators� The first step in developing a vision of

capacity development, and a plan tomeasure it, is to understand the role ca-pacity plays in the health sector in devel-oping countries.

� We have found that directed discussionusing conceptual frameworks or mapsprior to M&E planning can stimulatestrategic thinking within project or workteams and clarify individual and collectiveexpectations, and thereby improve capac-ity-building M&E.

� The conceptual frameworks (Figures 1 –5) illustrate the critical role capacity playsin influencing and sustaining performancein the health sector, including the fourlevels where capacity is needed in thehealth sector: system, organization, healthpersonnel and individual/community.

� Figures 2 – 5 depict capacity at each level.The purpose of these frameworks is toshow how capacity can be broken downinto inputs, processes, outputs, and out-comes in order to identify the differentfactors that contribute to capacity and per-formance, and hypothesize about the po-tential relationships among these factorswithin a single level.

� The frameworks provide a starting pointfor identifying the key variables that in-fluence capacity and performance at thatlevel, and will help evaluators define ca-pacity variables to track in the M&E plan.

Monitoring and Evaluating Capacity-Building Interventions 27

Part 3 Monitoring and Evaluating Capacity-Building Interventions

Part 2 described a generic conceptual frame-work for understanding the role of capacity inthe health sector and suggested possible ca-pacity variables for each level. This part pres-ents the six steps for developing a monitoringand evaluation plan for a specific capacity-building intervention. At the heart of this pro-cess is the development of a “capacity map”or conceptual framework that applies to theparticular capacity-building intervention un-der study. The six steps are listed in Box 4.

Ideally an M&E plan should be formulatedduring the design and planning of a capacity-building or performance improvement inter-vention. Evaluators and program plannersshould work together with key stakeholders toconduct a needs assessment, define the inter-vention strategy, and construct an M&E plan.Since capacity building is often one strategyin a broader approach to improving perform-ance, capacity-building M&E should fit intothe overall performance-monitoring plan.

An M&E plan for capacity building stateswhat is to be evaluated, what evidence isneeded to answer key evaluation questions,how the data will be used, who will use thedata, and for what purpose. The intended re-sult of the planning steps is a clearly definedguideline for data collection, analysis, and usefor assessing the effectiveness of a capacity-building intervention. In general, capacity-building M&E plans contain the following:

� a conceptual framework� a definition of essential variables of ca-

pacity and performance� hypotheses on important links between

these capacity and performance variables� identification of the stages of capacity� indicators, and methods� a timeframe, and� a dissemination strategy

Box 4: Six Steps for Developing a Capacity-Building M&E Plan

1. Define the purpose of the evaluation

2. Define performance objectives

3. Map capacity: Build a conceptual framework for the specific capacity-buildingintervention

4. Identify capacity indicators

5. Identify appropriate methodological approach and sources of data

6. Develop an implementation and dissemination plan

28

STEP 1 Define the Purpose ofthe Evaluation

There are different types of evaluation, eachwith a different purpose. In designing anevaluation strategy, the evaluator first needsto identify the key question(s) that he/shewishes to answer and thus the type ofmonitoring or evaluation to conduct. Table3 illustrates some of the research questionsaddressed by different types of capacity-building M&E.

A second question to address at the outset ofplanning is: who are the intended users ofevaluation results? M&E of capacity-buildinginterventions can be used for different pur-

poses and to meet the needs of many differentstakeholders. It is advisable to specify theprimary and secondary users at the outset ofplanning to avoid confusion and aggravation.In the NGO Networks for Health Project, theproject partners and the donor expected to usecapacity-building monitoring data in differentways. The NGOs sought information tomonitor the results of detailed internal organ-izational capacity-building plans. The fundingagency desired information on more generalcapacity changes related to the quantity andfocus of programming in order to demonstratethe overall results of the project. Until themain purpose of collecting data was specified,it was impossible to define the methods orindicators in the M&E plan.

Table 3: Questions Posed by Different Types of Capacity-Building M&E

Type of Evaluation Key Questions AnsweredNeeds assessment What is the current level of capacity?

Where are the gaps in performance and capacity?What capacity is needed?How can the intervention best address the gaps in capacity and per-formance?

Monitoring Inputs: Are inputs available to the program in appropriate quantitiesand at appropriate frequency? Did the type or quantity of inputschange?

Processes: Are key processes carried out to an acceptable standard orat an acceptable frequency? Did the processes change?

Outputs: Are products related to capacity available? Did the productsexpected emerge or change?

Outcomes: Is capacity appropriate and adequate? Did capacity im-prove?

Performance: Is performance appropriate and adequate? Did per-formance improve?

Evaluation Did the capacity-building intervention lead to changes in capacityand/or performance?


In practice, one finds an inherent tension indefining the purpose of capacity-buildingM&E. Managers generally use capacity-building M&E results for two main reasons.The first is primarily an internal function, thatis, improving capacity and capacity-buildingstrategies. The second is primarily an externalfunction, that is, reporting on the progress of acapacity-building intervention to various fun-ders and other external stakeholders. Whilethe two purposes are not mutually exclusive,managers must guide the M&E process care-

fully to ensure the best possible outcome. Toomuch attention to serving external (often do-nor) needs has been found to dilute the use ofM&E for improving capacity-building strate-gies and organizational learning (Horton,2001; Morgan, 1997). Lack of attention tovalid measures of change (or relying toomuch on self-reported perceptions of capac-ity) can undermine the credibility of evalua-tion results. Box 5 summarizes key advice onconstructing a capacity-building M&E plan.

Box 5:DO'S AND DON’TS of Developing an M&E Plan for a Capacity-BuildingIntervention

DO� Develop capacity-building M&E plan during the intervention design phase

� Develop capacity-building M&E plan with respect to broader performance objectives

� Involve all stakeholders, both internal and external, in developing the M&E plan, par-ticularly the purpose of the evaluation

� Be prepared to negotiate with stakeholders on the purpose of the evaluation and makeall expectations transparent

DON’T� Base M&E plans only on the needs of external stakeholders (mostly donors) at the ex-

pense of meeting internal information needs

� Miss opportunities to reflect and learn about capacity development through M&E

30

STEP 2 Define PerformanceObjectives

Before launching into monitoring and evalua-tion of any capacity-building program or in-tervention it is critical to step back and fullyunderstand its focus and strategy. It is par-ticularly crucial to understand how the statedcapacity-building strategy is expected to im-prove performance and what signs of im-proved effectiveness are expected from ca-pacity building. Although it is not possible toprove causality, it is important to clearly de-fine the expected pathways between capacitybuilding and performance.

To begin, evaluators should address the fol-lowing questions:

� What is the purpose of the capacity-building intervention?

� What type of performance is expectedin a given period and at what level:health system, organization, healthpersonnel, or community?

� What processes or activities are beingused to build capacity?

� What external influences should betaken into consideration?

� Who has a stake in capacity buildingand capacity measurement?

Defining PerformancePerformance objectives should relate to themandate or specific purpose of a system, or-ganization, or community, or to health per-sonnel functions. The more specific one canbe about performance expectations, the easierit will be to construct a capacity map. If theM&E plan is being developed after a capac-ity-building intervention has been designed,then articulating the performance focus andexpectations should not be difficult (assumingthe design document is sufficiently explicitabout performance objectives). Moreover,some organizations already may adhere to a

set of performance indicators for internalmonitoring or reporting to externalstakeholders. Thus, there may already beclearly stated performance standards. If, how-ever, M&E planning takes place as part of thedesign process (starting with needs assess-ment and intervention design) then focuseddiscussion among program planners, manag-ers, and evaluators about what would consti-tute adequate performance in this context willbe needed.

In practice, perceptions of performance canvary widely among stakeholders. For exam-ple, a manager of a clinic may define per-formance in terms of benefits to the clients;whereas the clinic’s financial managers mightdefine performance as the acquisition of newclients (and a correlating increase in income).There is a growing body of literature aboutPerformance Improvement in the health sec-tor, particularly organizational performance6

that can be useful for defining performanceexpectations and identifying gaps in perform-ance and possible reasons for those gaps. Per-formance objectives should be expressed asvariables or indicators that can be measuredagainst international or national standards, orlocally determined expectations. Normally,the definition of performance objectives re-flects both external and internal criteria. SeeBox 6 for characteristics of a good perform-ance objective and two examples of perform-ance objectives that will be used to illustratecapacity mapping in Step 3.

6 See Lusthaus, C., M. Adrien, G. Anderson, and F.Carden. 1999. Enhancing Organizational Perform-ance: A Toolbox for Self-Assessment, Ottawa: IDRC;http://www.pihealthcare.org; McCaffrey, J., M. Luoma,C. Newman et al. 2000. Performance Improvement:Stages, Steps and Tools, Chapel Hill, NC: INTRAH.


Box 6: Characteristics of a Good Performance Objective

� Measurable� Reflects a needed change� Relates to a clear product or action� Relates to a defined target population� Performed by specific delivery agent (e.g., organization, community group, etc.)� Relevant to a particular context/situation

Examples

� Consistent delivery of a package of family planning services by X organization to a definedpopulation (defined in terms of coverage, quality, and consistency)

� Improved demand for immunization services in communities served by community healthworkers (CHW) (defined in terms of utilization and coverage)

32

STEP 3 Mapping Capacity:Build a ConceptualFramework for aSpecific Capacity-Building Intervention

Once performance objectives and expecta-tions are defined, planners and evaluatorsmust make assumptions about the capacityrequired to meet these objectives. Capacitymapping is a structured process of thinkingthrough the role capacity plays in ensuringperformance by developing a conceptualframework that is specific to a particular ca-pacity-building intervention. During capacitymapping, all the possible factors of capacitythat influence performance and the relation-ships between them must be identified. Oncethe factors are all laid out, the program staffor evaluator can focus on those that are mostessential for the evaluation.

Mapping capacity can be a critical step indeveloping an M&E plan. The map is a toolthat guides the design of the plan, from selec-tion of indicators and methods to presentationof evaluation results. As stated by Morgan(1997), evaluation designers and their pro-gram partners need “a sense of what capaci-ties they need to develop and for what reason.Most groups and organizations can articulatesuch a vision of the future given sufficienttime and productive discussion.” Mappingcapacity makes plain to all stakeholders as-sumptions about key variables that affect thedesired outcome of a capacity-building inter-vention. A mapping exercise is an excellentway to bring all stakeholders to a commonunderstanding of the scope and focus of acapacity-building intervention, the perform-ance outcomes expected from capacity devel-opment, and the role of M&E in tracking andinfluencing change.

For the evaluator, the objective of this stageof M&E planning is to create a conceptual

framework that links capacity-related inputs,processes, outputs, and outcomes to perform-ance of a system, organization, health per-sonnel, or community. The advantage to theevaluator of developing a capacity map istwofold. First, through mapping, the evaluatorgains a better understanding of how key deci-sion-makers and stakeholders believe thesystem, organization, health personnel, orcommunity should be working. Second, map-ping enables evaluators to define exactlywhich capacity variables are to be evaluatedover time.

When to Map CapacityAs noted above, an M&E plan should be for-mulated during the design and planning of anintervention. If program planning and M&Edesign are conducted simultaneously, capacitymapping can contribute to the choice of inter-vention strategies and to the M&E strategy.However, sometimes circumstances do notpermit this ideal type of coordination onprogram and M&E design. Frequently,evaluation designers are brought in well afterprogram planners have defined the interven-tion strategy and specific activities. In thiscase, evaluators must still work with programplanners to understand the intervention strat-egy and the role of evaluating it. Capacitymaps should reflect and/or inform this overallstrategy. If a conceptual framework already

Box 7:The Process of CapacityMapping

1. Identify primary level of capacitybuilding

2. Define outcomes for that level3. Develop a one-dimensional level

capacity map4. Develop a multi-dimensional level

capacity map


exists for the intervention, designers shouldreview the assumptions and relationshipamong variables depicted in this diagram tounderstand the expected role of capacitybuilding. If an overall conceptual frameworkfor the intervention does not already exist, itis essential to construct one to support capac-ity mapping.

How to Map CapacityThe process of developing a capacity map isoutlined in Box 7. During this process, plan-ners, evaluators, and key stakeholders mightlike to use the series of questions in Box 8 toguide discussion. At a minimum, they shouldconsider the following two questions:

1. At which level is capacity required to en-sure the stated performance objectives?In other words, what level is likely to be themain focus of capacity-building efforts? Thegeneric capacity map (Part 2, Figure 1) de-fines four different levels where capacity isneeded in the health sector: system, organiza-tion, health personnel, and individ-ual/community level. Careful definition of theperformance objectives in Step 2, and a clearunderstanding of the capacity-building strat-egy should help evaluators answer thesequestions. For example, if performance gapsare found in a specific health facility, then itis likely that capacity-building interventionswill seek to improve capacity outcomes at theorganization or individual level. The first mapwould focus on one of those levels.

2. What capacity outcomes are expected atthat level to improve performance?Once the level has been specified, designersshould identify aspects of capacity that mightinfluence the specific performance objectiveat that level and express them as capacity out-comes. Morgan (1997) defines capacity out-comes as the “product of new learning andabilities that eventually become part of theorganization or system, and support new lev-

els of performance.” Designers can refer toguides on organizational capacity develop-ment, for example, to help guide the choice ofcapacity outcomes. However, capacity out-comes should always be tailored to perform-ance objectives or standards of the particularintervention or organization under study.

At the intervention design phase, it is worthcasting a wide net to consider all possibleaspects of capacity that might relate to desiredperformance. Brainstorming on capacity canthen lead stakeholders or participants in thismapping process to begin to prioritize areasfor capacity-building intervention. Whereparameters of an intervention are already setor where a structure for brainstorming isneeded, designers might choose two or threedifferent areas of capacity development, ex-press them as capacity outcomes, and thenmap them. Although capacity building oftentries to address multiple capacity gaps simul-taneously, for measurement purposes, it isadvisable to choose a limited number of keycapacity outcomes for capacity mapping.

For example, in Maps 1-3 below, the per-formance objective for the (fictitious) FamilyHealth Organization is defined as “consistentdelivery of a package of essential, good-quality family planning services to a definedpopulation.” Performance variables mightinclude coverage, quality, and consistency,which would be expressed as indicators. Thethree key capacity outcomes for this specificperformance objective are defined as financialself-reliance, quality assurance practices in-stitutionalized, and health services able torespond to client needs. Although many otheraspects of capacity might influence coverage,quality and consistency in the delivery offamily planning services, this organizationhas chosen to concentrate on these three ar-eas.

34

Single-Level Capacity MappingOnce the two questions about levels and out-comes have been answered, it is necessary todraw up a table or matrix that maps each ca-pacity outcome at a single level. The processinvolves identifying the variables that influ-ence the specific capacity outcome at thatlevel. Capacity variables include inputs, suchas physical and human capital (defined byMorgan, (1997) as “knowledge, infrastructureand skills”) and processes representingchanges in human behavior (such as growthof new skills, attitudes, values, and relation-ships) that are reflected in the functions per-formed by individuals or groups. These inputsand processes come together to produce im-proved capacity outputs and outcomes. It isoften expected, in the course of capacity de-velopment, that individuals or groups add toor build on their existing assets to make posi-tive changes with respect to managing thoseassets. A capacity map tries to capture thesecritical assets and behaviors and link them to

capacity outcomes and new levels of per-formance.

Once completed, the map illustrates concep-tually the pathway to achieving desired per-formance results. It includes specific variablesthat may be targeted for intervention and thenmonitored over the course of the interventionto understand changes in inputs and processesand any resulting improvements in capacityoutcomes. Evaluators are reminded that thevariables depicted in the capacity map arethose that relate to the inherent or desiredcapacity of the system, organization, healthpersonnel, or individual/community targetedfor intervention. They do not represent ele-ments of the capacity-building interventionitself.

Box 8: Questions to Guide Discussion for Capacity Mapping

Describing the link between capacity and performance� What elements of capacity are needed to ensure performance?� Where are the capacity gaps?� What might be the cause of poor capacity?� What are two or three key aspects of capacity required for performance?� At what level is capacity required?

Identifying capacity variables� What essential inputs and processes contribute to capacity at that level?

Describing the process of capacity development� Could capacity develop in stages?� How would one define possible stages of capacity?� What benchmarks might be used to mark these stages?� How would stages of capacity development manifest themselves in terms of improved

performance?


To build such a capacity map, planners andevaluators can use a facilitated discussionamong stakeholders as well as tap existingdata from needs assessments, capacity diag-noses and prior monitoring. Evaluators mightalso draw on the experience of system andorganizational theory, theories of adult learn-ing, and community development to hypothe-size the most likely causes of poor perform-ance. Box 9 provides some general guidancefor capacity mapping.

The following three diagrams (Maps 1, 2, and3) provide examples of capacity maps thatdefine in a very general sense some possibleinputs, processes, and outputs related to thethree particular organizational capacity out-comes for the hypothetical Family HealthOrganization: financial self-reliance, qualityassurance practices institutionalized, andhealth services able to respond to client needsand demands.

Box 9: Guidance on Capacity Mapping

� Capacity mapping should refer to the logic of the overall program, project or intervention. Hor-ton et al. describe this approach as “referring to a theory of action” � that binds interested par-ties into a single vision (Horton, 2001). Whether mapping capacity during intervention design or inthe context of an already defined intervention strategy, it is advisable to refer to existing data onthe intervention area, including needs assessment, capacity assessments, etc.

� When mapping capacity it may be helpful to refer to the conceptual framework in Part 2 for ageneral review of the role capacity plays in improving performance in the health sector and exam-ples of capacity variables.

� Be realistic about your expectations of the role of capacity. There is a tendency to considerevery aspect of resources and behavior in an individual, organization, or system as a capacity vari-able, and to risk measuring too much.

� Look beyond individual capacity and training solutions to identify capacity variables. For exam-ple, during discussions on the capacity framework with SAIDIA, a Kenyan NGO (nongovernmentalorganization) that provides health services and community development opportunities, staff at firstclaimed that training health workers and community members was their only work in capacity build-ing. Yet, with further discussion, participants illustrated a wide range of capacity-building activitiesat all levels, including their work in coordination and collaboration with the public sector, and court-ing relations with donors that fund the NGO.

� Map capacity with a wide range of stakeholders to inspire a sense of ownership of capacitybuilding and appreciation of the use of evaluation in programming. Since capacity-building M&Edelves into many internal characteristics and processes found within systems, organizations, andcommunities, it requires considerable investment on the part of the members of these groups toachieve success. The quality of information obtained from evaluation, therefore, is directly af-fected by the extent to which participants develop a feeling of ownership of the M&E activity andvalue the data being collected.

36

Multi-Level Capacity MappingThe three single-dimension capacity mapsprovide a list of possible variables that influ-ence capacity outcomes at one level. How-ever, it is equally important to consider theconnections among levels where capacitybuilding might take place and their role withrespect to realizing capacity outcomes andperformance objectives. Although perform-ance may be faltering at the facility, the strat-egy used to improve performance may requireadditional capacity improvements at both thehealth personnel and system levels. In thiscase, designers may choose to construct acapacity map that includes several levels andthat will provide even greater detail on possi-ble variables that contribute to capacity out-comes. Thus, once the single-level map iscompleted a second map is developed thatincludes more than one dimension to illustratethe interdependence among different levels ofcapacity and determine which factors at otherlevels might influence capacity outcomes atthe focus level. The two types of maps (sin-gle-level and multiple-level) will be used toidentify the variables to be assessed as part ofthe M&E plan.

In Map 4, we have taken the same basic ma-trix but added a second axis to account for thefour possible levels of capacity. This examplefocuses on the community level but the mapdepicts variables at the four different levelsthat might influence the specific community-level outcome.7 As noted in Map 4, the over-all performance goal is to “improve demandfor immunization services at the communitylevel,” expressed as immunization serviceutilization and coverage. The capacity ofCommunity Health Workers (CHW) to de-liver IEC services was chosen as the capacityoutcome for mapping. In this case, the de- 7 This matrix is adapted from an exercise completed byparticipants at a Workshop on Sustainability and Ca-pacity Building hosted by PLAN International in May2001 in Dakar, Senegal.

signers began by listing a large number ofpossible capacity variables and then narrowedthem down to the key variables to be moni-tored over the course of the intervention.Shaded areas represent an explicit decisionnot to monitor an indicator in that category.

Dealing with ContextWhen assessing the effectiveness of capacity-building interventions it is also critical to un-derstand the environmental or contextualfactors that influence capacity and perform-ance. Horton and colleagues (2000) describecontext as “formal and informal rules of thegame and how they are used.” As noted inPart 2, context can relate to the administra-tive, legal, political, socio-cultural, economic,and technical forces that shape capacity andperformance. Clearly, many of these forcesare well beyond the reach of a typical capac-ity-building intervention. Nevertheless, it isadvisable for program managers to track envi-ronmental changes periodically. Organiza-tional theory describes a successful (and sus-tainable) organization as one that understandsits environment and is able to adapt to envi-ronmental changes to ensure its survival.Thus, tracking changes in the operationalcontext informs strategy for capacity devel-opment, even if planners or managers feelthere is little they can do to change it. Thepublication, Enhancing Organizational Per-formance, published by the International De-velopment Research Centre (IDRC) providesa useful list of questions related to environ-mental influences on organizational capacity.These questions are reproduced below in Box10. In each map found in this guide there isan additional box at the bottom where keyenvironmental variables are recorded.


Map 1: Organizational Capacity Map - Single Level Capacity outcome: Financial self-sufficiency

InterventionPerformance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency).Capacity-building objective: Improve financial self-reliance of health facilities in District One.Strategies and activities: Improve leadership and financial planning skills of district managers; introduce new procedures for strategic planning; develop linksbetween health facilities and communities leading to joint planning and management; develop skills in grant application writing and reporting to funders.

Inputs Processes Outputs Capacity Outcome

Performance Objective

Leadership

Finances

Infrastructure

Human resources

Finance policy

Organizational culture

Strategic & operationalplanning


Resource mobilization

Human resource management& development

Research, monitoring &evaluation

Coordination with otherinternal units

Creation & maintenance oflinkages with external groups

(specifically, funders)

Advocacy

Managing quality of care

Community mobilization

Strategic & operational plansdeveloped and implemented

Staff trained

Functioning financialmanagement system

External linkages established(to donors, partners,

individuals, community)

Financial self-reliance (ability togenerate resources

& maintain ahealthy funding

base)

Consistent delivery ofessential package of good-

quality family planningservices to a defined

population (coverage,quality, and consistency)

Context or operational environmentNational policy on fee-for-serviceNational financial management procedures

38

Map 2: Organizational Capacity Map - Single Level Capacity outcome: Quality assurance practices institutionalized

InterventionPerformance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency).Capacity-building objective: Improve quality assurance practices in health facilities in District One.Strategies and activities: Improve leadership of facility managers and supervisors; introduce norms and procedures, clarify job descriptions and expectations;improve links to supplies and logistics unit.

Capacity Inputs

Capacity Processes

Capacity Outputs

Capacity Outcome


Leadership

Financial resources

Infrastructure

Human resources

Technology


Operational planning

Human resourcemanagement &development

Incentive practices

Training and supervision

Research, monitoring &evaluation

Logistics/suppliesmanagement

Creation & maintenanceof linkages with other

organizations(specifically, managers

and suppliers)

Operational plansdeveloped andimplemented

Staff, managers &supervisors trained

Quality assurancestandards clearly stated &

reference materialavailable

Staff expectations clear tothem

Monitoring reports onquality, utilization, &

client satisfaction

Functional relationshipsbetween facilities and

suppliers

Quality assurancepractices

institutionalized

Consistent delivery of essentialpackage of good-quality familyplanning services to a definedpopulation (coverage, quality,

and consistency)

Context or operational environmentPublished norms and standards for careNational health information system use of data to assess qualityCentral stores policies and procedures


Map 3: Organizational Capacity Map - Single Level Capacity outcome: Health services able to respond to client needs and demands

InterventionPerformance Objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality and consistency).Capacity-building objective: Improve the ability of the health services to respond to client needs in District One.Strategies and activities: Introduce incentives for quality of care practices; improve client provider communication skills; research and design optimal mecha-nisms for communication and interaction between communities and health facilities.

Capacity Inputs

Capacity Process

Capacity Outputs

Capacity Outcome


Leadership

Finances

Infrastructure

Human resources

History of health serviceorganization


Human resourcemanagement &

orientation

Organizational incentivepractices

M&E, research

Coordination andcommunication with

referral units

Creation & maintenanceof linkages with

community groups

IEC

Community mobilization

Staff trained in technical& communication skills

Functional communityoutreach &

communicationmechanisms

Feedback from routineclient satisfaction &

community monitoring

Quality of referral servicemonitored

Health services ableto respond to clientneeds and demands

Consistent delivery of essentialpackage of good-quality familyplanning services to a defined

population (coverage, quality, andconsistency)

Context or operational environmentNational policy on consumer roles and rightsPublished norms and standards of care

40

Map 4: Community Capacity Map on Multiple LevelsCapacity outcome: Effective delivery of IEC services

InterventionPerformance objective: Increase demand for childhood immunization in Sierra Leone.Capacity-building objective: Improve capacity of CHWs working with local NGO to provide IEC on childhood immunization.Strategies and activities: Develop curricula for training of trainers and training of CHWs; conduct training of trainers and supervision; health personnel supportCHWs from health centers; NGO supervises and supports health center personnel working in service delivery.

Level CapacityInput

CapacityProcesses

CapacityOutputs

CapacityOutcomes

Performance

SystemNational policy onimmunization and

community-based workers

Organizational(Local NGO)

Health center personnel(quantity/basic training)

Community health worker(quantity)

Designing & planninga training program

Supervision andmentoring of CHWs

Training plan developed

Training materials developed

Successful organization &execution of training of trainers

Ability to recognize trainingneeds and meeting them

Personnel Curricula for:Training of Trainers & for

Community HealthWorkers

Participation inTraining of Trainers

Participation in CHWtraining on IEC

Trainers meet standardsfollowing course

CHWs meet standardsfollowing course

IEC session provided

Capacity of CHWs to deliver IECon immunization:

- CHWs skilled & motivated toprovide services

Effective delivery ofIEC services (Quality

of IEC sessions)

Community Exposure to immunizationprogram

Community meetingswith CHWs

Level of participation inhealth care learning activities

Recognition of need forimmunization

Community knowledge ofimmunization benefits and side

effects

Caregivers value immunization

Improved demand forimmunization in

communities servedby CHWs(coverage)

Context or operational environmentNational economic growthNational health expenditures on immunizationDonor support for immunization


Interpreting and Using Capacity MapsThe examples of capacity maps above illus-trate how the different factors of capacitywork together to drive or influence perform-ance. They enable designers to view theseelements in a more systematic way that pro-motes common understanding and evaluation.When capacity mapping is conducted after anintervention has been planned, it can be usedto help evaluators understand the intentions ofmanagers in terms of their strategy for capac-ity development. During mapping, managersare encouraged to pinpoint and define clearlythe areas of potential change that will serve asindicators of progress in capacity develop-ment. Used after the design phase, the map-ping exercise can reinforce existing capacity-development strategies, thereby increasingtheir specificity. Sometimes mapping can alsoprompt planners to reexamine strategicchoices and change their tactics. Indeed, thisuse of capacity mapping for strategic plan-ning, and the linking of M&E with programstrategy should be encouraged throughout thecourse of the capacity develop-ment/performance improvement intervention.

Each type of mapping (single-level or multi-ple-level) can be done in two or three itera-tions. The first iteration of a map should at-tempt to provide a full list of capacity vari-ables that may influence capacity outcomes

and performance. It should present capacityvariables in a general way. Planners andevaluators then can discuss these variablesand narrow them down to priority areas ofintervention or measurement, and describethem more specifically. The second or thirditeration of a map should be more precise indepicting the variables to be monitored overthe course of the intervention. Map 5a pro-vides an example of the first iteration of mul-tiple-level capacity mapping. It contains awide range of general categories. Map 5billustrates the second iteration in which vari-ables are specified in greater detail.

Through mapping, evaluators can identify andorganize the key questions to be addressedregarding expected changes in the quantity,quality, cost, and other key aspects of capac-ity which require monitoring over time. Asplanners and evaluators interpret the map,they will narrow down the focus of monitor-ing and evaluation activities. In Step 4, be-low, evaluators define indicators that measurethese variables and build them into a moni-toring and evaluation plan.

42

Box 10: Questions to Guide Discussion on the External Environment and ItsInfluence on Organizational Capacity

Administrative� Is your organization influenced by the rule of other organizations, institutions, and groups to

which it is related or might be expected to be related?� Is your organization influenced by expectations of consumers, policymakers, suppliers, com-

petitors, and other organizations in its external environment?� Are your organization’s objectives and activities influenced by governments, donors, and

other organizations?� Is your organization influenced by important sector rules and regulations?� Do administrative norms/values in your country support or hinder the work your organization

intends to carry out?Legal� Do the laws of the country support the role played by your organization?� Does the legal framework support the organization’s autonomy?� Is the legal framework clear?� Is the legal framework consistent with current practice?� Is the legal regulatory context conducive to your organization’s work?� Does your organization monitor changes in the legal context that could affect the position of

the organization?Political environment issues� Do the political and ideological trends of the government support the kind of work the or-

ganization does?� Does the government system facilitate collaborative arrangements?� Does the organization play a role in national or sector development?� Does the organization have access to government funding?� Does the organization have access to international funding?� Does the organization have access to the government’s knowledge and publications?� Do government policies and programs support the organization?Sociocultural environment� Is equity in the workplace a social value?� Does the organization account for the effect of culture on program complexity?� Do values found in the sociocultural environment support the work of the organization?� Does the organization have access to a pool of capable human resources to recruit staff?� Does the organization analyze and link demographic trends to its work?Economic environment� Does the government’s economic policy support the organization’s ability to acquire technolo-

gies and financial resources?� Is money available to do the organization’s work?� Do donors support the organization?Technological environment� Is adequate physical infrastructure (telecommunication, transport) in place to support the

organization’s work?� Is the technology needed for your work supported by the overall level of national technology

development?


� Does the government system facilitate the organization’s process for acquiring needed tech-nology?

� Is the level of human resource development in your organization adequate to support newtechnology?

Stakeholder environment� Is the community involved in the organization?� Are partners involved in the organization?� Do governments value the organization’s products and services?� Do governments request or use the organization’s products and services?� Do similar organizations compete or cooperate with your organization?� Do donors influence the organization?� Do funders support the organization?

The questions above are adapted from Enhancing Organizational Performance (Lusthaus et al.,1999). While they are focused on the organization level, many of them can be adapted for anylevel of the health system.

44

STEP 4 Identify CapacityIndicators

The next step in developing an M&E plan forcapacity building is to define indicators forthe elements of capacity identified duringcapacity mapping. � Indicators are specificvariables that describe a given situation andcan be used to measure inputs, processes,outputs, and outcomes at any level (system,organization, health personnel, or individ-ual/community). They can be constructedfrom qualitative or quantitative data accordingto the type of variable one is interested intracking. For example, the indicator “numberof personnel per health facility trained incontrol of sexually transmitted infections(STI)” tracks the inputs that influence capac-ity of a public health system. Alternatively,measures of provider knowledge of appropri-ate treatment for different sexually transmit-ted infections and the availability of key STIpharmaceuticals at each facility are outcomeindicators signaling capacity in service deliv-ery. All three of these indicators could betracked to determine whether capacity existsto meet system-level performance objectives,such as “quality of STI care.”

What Are Capacity Indicators?Capacity indicators generally project an aspi-ration or a sought-after state or ability. Theycapture the current “stock of resources avail-able” for various uses or an individual or or-ganizational behavior that puts those re-sources into action (Moore et al., 2001). De-fining or choosing indicators for M&E en-courages planners and evaluators to be preciseabout the inputs and processes that influencecapacity and performance and what types ofchanges might result from capacity-buildinginterventions. Well-defined indicators providea reference framework for guiding allstakeholders toward the same goals. Indica-tors also allow for standardized measurementof change during implementation, which en-

ables evaluators to understand the process ofcapacity development over time and its rela-tionship to capacity-building intervention.

There is no agreed upon menu of “standard”indicators of capacity development. AsMorgan (1997) states, “It is difficult to finduseful examples of indicators that have beenused effectively to measure or assess capacitybuilding.” Examples of common health sec-tor-related indicators are found in theMEASURE Evaluation Compendium of Indi-cators for Evaluating Reproductive HealthPrograms (Bertrand and Escudero, 2002) andother indicator handbooks. However, no sin-gle indicator manual focuses exclusively oncapacity building or differentiates betweencapacity and performance measures. The ob-vious consequence is the need to work care-fully and systematically during M&E plan-ning to develop indicators that accurately re-flect capacity development in each particularcontext. Some capacity indicators can bedrawn from experience in human performanceimprovement, organizational assessment andtheory, and other disciplines. Others will re-quire testing through practice. When thePRIME project developed an index of capac-ity in training organizations, it built on yearsof experience working in this area and thecollective understanding of what it takes toprovide good-quality training on a sustainablebasis (Pyle and LaFond, 2001).

Even with the benefit of a generic indicatorreference material, most indicators used incapacity-building M&E require some moldingor adaptation to a particular situation. Forexample, if evaluators would like to study theprogressive stages of capacity development ina specific organization, they might chooseindicators based on defined scales of organ-izational development, as in the Managementand Organizational Sustainability Tool(MOST) developed by Management Sciencesfor Health (MSH, 1996). However, they


should also adapt these indicators to a par-ticular organization’s baseline assessment ofcapacity and its particular product or service.Expectations for improved performance andthe timeframe of a specific capacity-buildingintervention also matter. An organization pur-suing capacity improvement in reproductivehealth service delivery would choose differentmeasures of change from one seeking capac-ity improvement in networking and partner-ing. Thus, at the outset of M&E planning, oneshould begin defining indicators based on thecapacity variables identified in mappingrather than selecting indicators from a genericlist. Map 6 illustrates how indicators can beadded for each capacity variable, using theformat from Map 3. The discussion on indi-cators below begins with general guidance onindicator design, provides examples of capac-ity indicators, and concludes with lessonslearned from a variety of capacity develop-ment experiences (in health and other sec-tors).

Working with Capacity IndicatorsBy now most program managers and evaluat-ors at least have heard about what makes agood indicator. In general, all indicatorsshould share the following traits:� Validity: Validity refers to whether theindicator is measuring what it is supposed tomeasure. Indicators should have a close con-nection with the intervention.� Reliability: Reliability refers to the de-gree of random measurement error in an indi-cator. Error may result from sampling or non-sampling; whether the response is inherentlyobjective or subjective.� Well-defined: Indicator definitionsshould use clear and precise terms so every-one involved can understand what is beingmeasured.� Sensitivity: A sound indicator is sensitiveto the changes in program elements beingassessed.

Evaluators also need to take into account theavailability of data for “operationalizing”indicators and the potential costs of gatheringdata, in terms of financial resources and time.

Table 4 provides examples of health-sectorcapacity indicators by level (system, organi-zation, health personnel, and individ-ual/community) and measurement variable(input, process, output, and outcome) takenfrom various sources (Morgan, 1997; Hortonet al, 2000; Bertrand and Escudero, 2002;Brown, LaFond, and Macintyre, 2001). Itsuggests wide variation in the indicators cur-rently used to measure capacity and the needfor both quantitative and qualitative datasources. The table is not intended to representrelationships among these specific indicators.Box 11 provides examples of capacity indi-cators used in non-health sector programs.Table 5 gives examples of performance indi-cators at each level for reference.

Lessons for Indicator DevelopmentThe following lessons on indicator develop-ment are drawn from field experience in ca-pacity measurement in health and other sec-tors (Morgan, 1997; Horton et al. 2000; Fort,1999; Luoma, 2000; Ampomah, 2000; Ca-totti, 1999; Pyle and LaFond, 2001).

Lesson 1: Indicators should reflect an un-derstanding of the change strategy for ca-pacity development.

The process of choosing capacity indicatorsshould feed into the overall change strategydesigned for building capacity and improvingperformance. Indicators should be developedalongside capacity mapping while designing acapacity-building intervention. Evaluatorsalso might seek to understand how informa-tion is currently used in the organization orsystem to ensure that indicators become in-centives for change and not barriers.

46

Box 11: Examples of Capacity Indicators from Non-health Sector Capacity-Building Interventions

Example 11. Capacity indicator related to decentralized payment functions administered by local officials,district assembly members, and financial and political employees:Ability of the system to transfer funds between authority levels (for example, within 45 daysof the end of the quarter) and/or produce audited statements within six months of the end ofthe fiscal year.

2. Capacity indicator related to community water management committee’s role in water pumpmaintenance:A functioning Pump Management Committee that meets at least once a month and keeps thepump functioning 90 percent of the time in normal circumstances.

3. Capacity indicator related to coordination of information among six ministries working on soilerosion:Twenty-five percent increase in the number of projects that require contributions from two ormore departments.

4. Capacity indicator related to government department to carry out joint surveys of clientfarmers in delta area of cotton region:Acceptance of survey methods as an effective tool by senior research officers and their incor-poration into the work program of the agencies.

Source: Morgan, 1997

Example 2Indicators related to motivationMotivation to implement the strategic approachMotivation to undertake strategic planningInterest in improving the management information systemInterest in designing and managing competitive projectsIndicators related to capacityKnowledge of the strategic approachSkills to undertake strategic planningKnowledge about designing and managing competitive projectsKnowledge about the foundations of an information management systemIndicators related to context or environmentDegree to which tasks demand conceptual and methodological creativity and innovationPositive appreciation of performance in institutional evaluationsDegree of autonomy to undertake workContribution to improvement of the management information system

Source: Horton et al, 2000


Lesson 2: Capacity indicators should cap-ture organizational and behavioral changeas well as material and technical change.

The most challenging demand of capacitymeasurement is constructing meaningfulmeasures of human and organizational be-havior change. There is a tendency, particu-larly in the health sector, to advance technicalexplanations for what are just as likely to beorganizational or human behavioral problems.For instance, it is often presumed that traininghealth providers alone will address perform-ance gaps in service delivery when the rootcauses of poor performance can range fromunreliable sources of supplies to low healthworker motivation. Capacity developers andevaluators need to have a sense of how peopleand organizations change, what brings aboutlasting change, and why change in certainvalues and practices makes a difference. Ca-pacity indicators should capture the essenceof these changes in human and organizationalbehavior.

Lesson 3: In planning capacity-buildingM&E, it is important to monitor not onlycapacity but also key aspects of perform-ance and the environment.

Improved performance serves as the mainreference for mapping capacity and is the goalof capacity building. Evaluators should re-view changes in performance alongside ca-pacity to examine the relationships amongdifferent capacity and performance variables.In addition, evaluators should track environ-mental changes. Environmental factors typi-cally help to explain changes (or lack ofchange) in capacity and performance. Indi-cators that monitor external conditions serveas a warning to organizations that capacityand performance may be in jeopardy.

Lesson 4: Indicators should encourageownership and appreciation of the capac-ity-building and M&E process.

Indicators should be designed to promoteownership of the capacity-building process.Evaluators should work with capacity-building stakeholders to define indicators thatreflect locally determined and accepted no-tions of change. Keeping indicator definitionssimple and relevant to local needs will en-courage widespread use of M&E for capacitydevelopment. Designing indicators to serveexternal (often donor) needs rather than localdecision making can adversely influenceownership of capacity development (Morgan1997). This type of approach can “diminishthe contribution that capacity indicators canmake to project effectiveness.” Evaluators areadvised to balance the desire for more infor-mation for accountability purposes with thevalue of using information to motivate posi-tive behavior changes in individuals and or-ganizations.

Evaluators should also keep in mind thatmeasuring capacity can also be a sensitiveissue. Organizations, and people, do not relishhaving their “weaknesses” documented. Theyfeel even less enthusiastic about having theirweaknesses broadcast to their superiors, part-ners, and funders. The quality of data gath-ered for constructing capacity indicators couldbe distorted and/or obstructed unless the pur-pose of monitoring and evaluation is clear toall stakeholders, including the usefulness ofcertain indicators. Indicators should be asnon-threatening as possible.

48

Map 5a: Mapping Capacity First IterationInterventionPerformance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency).Capacity-building objective: Improve ability of health services to respond to client needs and demands in health facilities in District One.Strategies and activities: Introduce incentives for quality of care practices; improve client provider communication skills; research and design optimal mecha-nisms for communication and interaction between communities and health facilities.

Inputs Processes Outputs Outcomes PerformanceSystem Civil service administration

practices

Supplies & delivery of essentialgoods

Organization Leadership

Human resource

Supplies

Supervisors

Incentives

Referral

Quality of referralsystem

Feedback

Supplies management

Health services ableto respond to clientneeds and demands

Consistent delivery ofessential package ofgood quality familyplanning services to adefined population (cov-erage, quality, and con-sistency)

Personnel Number of staff Outreach

Learning

Provider-client interactionCommunity Experience with family planning

Local health organizations

Leadership

Links to community Number of contacts Outcome of contacts



Map 5b: Mapping Capacity Second Iteration Capacity outcome: Health services able to respond to client needs and demands

InterventionPerformance objective: Consistent delivery of a package of family planning services to a defined population (coverage, quality, and consistency). Capacity-building objective: Improve ability of health services to respond to client needs and demands in health facilities in District One.Strategies and activities: Introduce incentives for quality of care practices; improve client provider communication skills; research and design optimal mecha-nisms for communication and interaction between communities and health facilities.

Inputs Processes Outputs Outcomes Performance

SystemCivil service administration prac-tices that support counseling andprovision of family planning

Supplies & delivery of essentialgoods family planning supplies

OrganizationLeadership within managementteams with knowledge and trainingin family planning

Human resource (quantity & qualityof existing training/skills)

Supplies of family planning and IECmaterials (quantity & reliability)

Behavior of supervisors (content,communication & modeling of desiredbehavior among health workers)

Incentives for supervisors & providers toperform adequately

Referral system (designating, enabling &following up referrals)

Number of commodity reports

Worker feedback on supervi-sion

Client feedback on services

Supplies management check-list used

Frequency of needed referral

Health services able torespond to client needsand demand:(Expressed as: Utiliza-tion; Client satisfaction;and Supplies availability/stockouts)

Consistent delivery ofessential package ofgood-quality familyplanning services to adefined population (cov-erage, quality, and con-sistency)

PersonnelNumber of staff in each professionalcategory related to family planning

Community outreach activity(frequency and quality)

Availability & use of learning opportuni-ties for improving communication onfamily planning

Provider-client interaction index (quality)

Number of outreach visits Health workers moti-vated to address clientneeds

Health workers’ abilityto conduct client inter-view

CommunityExperience with family planning

Local organizations/unit focused onhealth

Leadership

Mechanisms for linking health services &community groups (frequency & quality)

Number of contacts withhealth facilities

Outcome of contacts interms of client satisfac-tion


50

Map 6: Community Capacity Map on Multiple Levels with Indicators (in Italics)InterventionPerformance objective: To increase demand for childhood immunization in Sierra Leone.Capacity-building objective: Work with a local NGO to improve Community Health Workers (CHW) capacity to provide Information, Education, &Communication (IEC) on childhood immunization.Strategies and activities: Develop curricula for training of trainers and training of CHWs; conduct training of trainers and supervision; health person-nel support CHWs from health centers; NGO supervises and supports health center personnel working in service delivery.

Level Capacity Inputs Capacity Processes Capacity Outputs Capacity Outcomes Performance

SystemNational policy onimmunization & CHWs(Policy exists & is favorable)

Organization(Local NGO)

Health center personnel(Quantity/ basic training)

Community health workers(Quantity/ basic training)

Designing & planning atraining program(Planning mechanismsexist & planning skillsdemonstrated)

Training plan developed(Plan exists)Training materials developed(Quantity/quality of materials)

Successful organization &execution of Training ofTrainers (TOT completed;trainees’ knowledge improves;trainees satisfied)

Ability to recognize trainingneeds and meet them(assessment process leads totraining)

Personnel Curricula for:- Training of Trainers

and- Community Health

Workers(curriculum exists)

Participation in Trainingof Trainers

Participation in CHWtraining on IEC(% of personnel orCHWs completingtraining)

Trainers meet standards following course(Post-test scores)

CHWs meet standards following course(Post-test scores)

IEC sessions provided(Number/frequency of IEC sessions)

Capacity of CHWs to deliveryIEC on immunization

- CHWs motivated to provideservices (attitudes of CHWs toIEC)

Effective delivery of IECservices (Quality of IECsessions)

Community Exposure to immunizationprogram (Past experiencewith childhoodimmunization)

Perceptions of CHWs(Community relationship with CHWs andacceptability of their role)

Community knowledge ofimmunization benefits and sideeffects (Index of immunizationprogram message recall)

Improved demand forimmunization services incommunities serviced byCHWs (Immunizationservice utilization &coverage)

Context or operational environmentNational economic growth (GDP)National health expenditures on immunization (% of health budget spent on immunization; total expenditure on immunization)Donor support for immunization (% of immunization expenditure from external sources)


Lesson 5: The results of indicator-basedcapacity-building M&E should be inter-preted wisely.

There are documented challenges to usingindicators to monitor and evaluate capacitybuilding. Evaluators can manage each chal-lenge with careful planning of M&E. Some ofthese challenges are detailed below.

Capacity development is context specific. Itreflects qualitative (as well as quantitative)changes in resource availability and behavior.Given the wide range of possible scenariosand capacity/performance objectives, it isoften not possible to establish objective stan-dards that would allow local or regional com-parisons in capacity across similar types ofentities. Internal benchmarks can be set, butthey may not be valid for other entities orcontexts. It follows that aggregation of indi-cators on a district, regional, or national scaleis not likely to result in useful information forM&E.

Selection of capacity indicators is oftenhighly subjective. To encourage ownershipand relevance, evaluators often rely on per-ceptions of capacity and capacity changeamong participants in the capacity develop-ment process as the basis for measuring prog-ress. Thus, there is a need to balance thesesubjective measures with a range of objectiveindicators and data-gathering strategies.

Capacity is influenced by many differentvariables. Hence, there is a tendency to try tomonitor a number of indicators at the sametime. We encourage the use of multiple indi-cators for each level within the capacity mapbecause they provide greater insights into thestate of capacity and can serve to validatefindings. Use of multiple indicators is oftenrecommended to explain what can be an im-precise situation or occurrence. At the sametime, however, evaluators should prioritize

indicators based on program objectives anddevelop a manageable set to monitor overtime.

Evaluators are experimenting with indices orcomplex indicators that combine a short listof essential indicators (sometimes weightedby strength of influence) into a single measureof capacity. Of the few examples in the healthsector, the PRIME project used a single indexto assess capacity dimensions of organizationsthat conduct training in reproductive health(Fort, 1999; Ampomah, 2000; Catotti, 1999).This index also takes into account differentpossible stages of capacity by using a scalefrom 0 to 4 to assess progress of an organiza-tion for each indicator under study. An exam-ple of the indicators and scales used in theTraining Organizations Index and a presenta-tion of the results of a capacity assessment inEl Salvador are found in Annexes A and B.The PRIME Project did not use this index toconduct routine monitoring and evaluation oftraining organizations; however, it hasadapted many of these indicators and thescaling approach for use in its performancemonitoring plan (PRIME II, 2001). Otherexamples that use scales or scoring as part ofa capacity index can be found in the Man-agement and Organizational SustainabilityTool (MOST) (MSH, 1996), and tools devel-oped to evaluate the capacity of agriculturalresearch organizations (Horton et al., 2000).8

8 It is important to note that indices can be difficult tointerpret if they are presented out of context or to anaudience that does not understand how the index isconstructed.

52

Table 4: Examples of Capacity Indicators in Current Use in Health ProgramsLevel Inputs Process Outputs OutcomesHealth system � Doctors per population

� Ratio of health care spendingon primary health care vs.tertiary care

� Percent of health budget fundedby external sources

� Percent of national budgetallocated to health

� Donor coordination committee meetsevery 6 months

� Collaborative “arrangements” existbetween social sectors – e.g.,meetings between health &agriculture or health & education

� Percent of districts with decentralizedbudgeting

� Number of multisectoral meetingsheld

� Number of collaborative projectsinitiated in sectors outside health

� Existence of national standards forprofessional qualifications

� Existence of sector-wide strategy

� Widely distributed sector-widestrategy

� Regular auditing of system-wideaccounts by independent company

� Percent of recurrent costs coveredthrough local resource generation

Organization � Existence of clear missionstatement

� Number of trained managersper unit

� Percent of district medicalofficers with public healthdegree/training

� Clearly defined organizationalstructure

� Organizational culture thatvalues and rewardsperformance

� Coordination with other organizationsevident through internal reportingmechanisms

� Number & quality of jointlyadministered activities with partnerorganizations

� Job descriptions updated regularly toreflect real work requirements &responsibilities

� Team planning (frequency andquality)

� Supervisors playing mentoring role

� Presence of financial managementsystem that regularly providesincome/revenue data & cash flowanalysis

� Number of commodity trackingreports

� Individual work plans are preparedfor all staff

� Sufficient number of sitesfunctioning as clinical training sitesto meet clinic practice needs

� Percent of MIS reports complete andon time

� Supervisors able to guide on-sitelearning

� Ability to adjust services inresponse to evaluation results oremergencies

� Cost-sharing revenue as aproportion of the annual MOHnon-wage recurrent budget

� Percent of facilities with stock-outof essential commodities in the last6 months

� Regular review of MIS data forroutine planning

HealthPersonnel

� Adequacy of trainingmaterials/supplies has beenassessed in one or moreinstitutions

� Adequate training suppliesavailable in sufficientquantities to support ongoingRH/FP training in one or moreinstitutions

� Up-to-date curricula� Percent of training budget from

external assistance

� Number of training sessions toimprove human resource managementaddressing needs expressed byproviders

� Managers trained in and usingperformance evaluation

� Percent of courses where trainingmethodology is appropriate fortransfer of skills/knowledge

� Professional networking (frequencyand quality)

� Number of providers trained, bytype of training & cadre of provider

� Number of staff trained in finance,MIS, strategic planning, financialplanning

� Number of managers trained, bytype of training

� Number of monthly staff newslettersproduced

� Percent of trainees (providers) withknowledge in skill area (meetnational standard)

� Level of staff motivation� Percentage of senior staff with

continuing education opportunities

Individual/Community

� Average level of educationattained in the district

� Mean income level� Proportion of adults whose

partner recently died in centralhospital

� Community leadership (typeand quality)

� Number of health committees whomeet regularly and take action

� Percent of dispensary budgetsupported with community-basedfunding

� Level of community cohesiveness� Community experience negotiating

with district health office

� Proportion of non-users who desireto use contraception in the future

� Level of participation in communityhealth committees

� Number of health action plans

� Community needs presented todistrict health office on regularbasis

� Proportion who knows anemiaprevention practices

� Level of community mobilizationand empowerment

� Community support formaintaining new well


Table 5: Examples of Performance Indicators in Current Use in Health ProgramsLevelHealth system � Average time/distance to the nearest reproductive health facility offering a specific service

� Percent of facilities where percent of clients receive the service that meets the expected standards� Number/percent of trainees deployed to an appropriate service delivery point and job assignment� Percent of facilities that experience a stockout at any point during a given time period� Percent of health facilities providing STI services with adequate drug supply� Contraceptive prevalence rate (CPR)� Disability adjusted life years (DALY)� Disability adjusted life expectancy (DALE)� System responsiveness to clients� Index of equality of child survival� Total health expenditure as a percent of GDP� Public expenditure on health as a percent of total public expenditure� Out of pocket expenditure as a percent of total health expenditure

Organization � Percent of mothers examined every 30 minutes during the first two hours after delivery� Percent of data elements reported accurately in MIS reports� Family planning continuation rates in catchment population� Percent of annual revenue generated from diverse sources� Percent of target population that received DPT 3 immunization� Cost of one month’s supply of contraceptives as a percent of monthly wages

Health Personnel � Percent of deliveries in which a partograph is correctly used� Percent of newborns receiving immediate care according to MOH guidelines� Percent of pregnant women counseled and tested for HIV� Percent of STI patients appropriately diagnosed and treated

Individual/Community

� Percent of communities with active health center management committee� Percent of target population that received DPT 3 immunization� Percent of non-users who intend to adopt a certain practice in the future� Percent of infants 0 - < 6 months of age who are exclusively breastfed� Percent using condoms at last higher-risk sex

54

Determining cause and effect are not easilydone with capacity-building M&E, eventhough a capacity map might clearly stateassumptions about relationships among vari-ables. The multiplicity of capacity variablesand the frequent improvement and decline incapacity make it difficult to draw definiteconclusions from a complex situation. It is notsurprising, therefore, that some evaluatorshave found linear evaluation frameworks andthe strict use of indicators too inflexible andmechanical to be used effectively in moni-toring and evaluating capacity (Morgan,1997; Earl et al., 2001). For these and otherreasons, Morgan cautions evaluators not torely too heavily on indicators to provide com-plete insights into capacity development. Inspite of the growing list of capacity measures,“indicators used in monitoring and evaluationof capacity do not explain why complex sys-tems work the way they do” (Morgan, 1997).

In light of these challenges, the way in whichindicators are developed, measured and usedbecomes a critical determinant of the credi-bility and usefulness of monitoring andevaluation of capacity building. Many ofthese constraints can be addressed with care-ful indicator development and the use of arange of data-collection instruments that aresensitive to the intangible nature of what isbeing measured in capacity-building evalua-tion. At the same time, the use of linearevaluation frameworks also requires carefulmanagement. Evaluators need to focus oncritical process aspects of capacity building,and use maps to guide but not restrict M&E.


STEP 5 Identify AppropriateMethodologicalApproach and Sourcesof Data

The fifth step in developing a capacity-building M&E plan involves defining themethodological approach, identifying sourcesof data, and choosing (or developing) datacollection tools. Evaluators should ask thefollowing questions:� Which methodological approach is appro-

priate?� What sources of data are necessary for

measuring the indicators defined in Step4?

� Are there any existing tools for measuringcapacity that are appropriate for my pur-poses?

Methodological Approaches and ChallengesAs discussed throughout this guide, monitor-ing and evaluation require different meth-odological approaches and have different dataneeds. The choice of methods and datasources relates mainly to the purpose of theevaluation (see Step 1).� Is the purpose to monitor the implementa-

tion of a capacity-building intervention,assess its effectiveness, or both?

� Will the results be used mainly for inter-nal improvements or external reporting?

Clearly, all capacity-building programs needto be monitored to ensure they are workingwell (i.e. to track changes in inputs, processes,outputs and outcomes). However, the evalua-tion of program effectiveness happens lessfrequently and only for selected interventionsdue to cost and complexity. In the case ofcapacity-building evaluation, it can be par-ticularly difficult to conduct evaluations thatlook for an association between capacity-building intervention and changes in capacityor performance. These changes can occur fora number of reasons in addition to the capac-

ity-building intervention itself (e.g. contextualinfluence). Since capacity measures are noteasily quantified, and identifying similar or-ganizations or systems to facilitate compari-son (as in a case-control study) is difficult,experimental designs are not feasible or prac-tical for capacity measurement. As James(2001) notes about capacity-building evalua-tion, “precise measurement and attribution ofcause and effect is rarely possible and nevercost effective. The best we can hope for isplausible association.”

Evaluators are therefore advised to recognizethe challenges to capacity-building M&E andset realistic aims for evaluation. Many ofthese challenges have been discussed previ-ously in this guide. Some of them relate to theinherent nature of capacity (capacity and ca-pacity building are dynamic and multidimen-sional; contextual), while others are a functionof the early stage of development of capacitymeasurement. Four of the main challenges aredetailed below.

Capacity develops in stagesCapacity measurement tools should be able tocapture different stages of development ofcommunities, health personnel, organizations,or health systems. The “MSH organizationalprofile” used in the Management and Organ-izational Sustainability Tool (MOST), forexample, has identified different benchmarksaccording to an organization’s stage of devel-opment (nascent, emerging, mature). Capac-ity measurement must be able to capture indi-vidual elements of capacity and combinationsof elements, and relate them to the stage ofdevelopment of the entity being assessed.

Changes in capacity need to be measuredover timeRepeated measures are needed to capture theinterim steps in capacity-building processes aswell as trends in outcomes. While there areexamples of repeated application of capacity

56

measurement tools (INTRAH, SFPS, andPASCA), to date, only limited reports offindings from longitudinal evaluations areavailable (PASCA). Better techniques areneeded to capture the effects of capacitybuilding over time and elaborate the link be-tween capacity development and performanceimprovement.

Internal versus external validityCapacity building should be a self-motivatedand self-led process of change. Evaluationstrategies that use self-assessment techniquesand locally determined benchmarks of prog-ress inspire ownership of capacity develop-ment and increase the likelihood that evalua-tion results will be used. Nevertheless, therecan be a cost to this approach in terms of theperceived validity of findings. Externalstakeholders often prefer to measure progressagainst performance standards (of either na-tional or local origin) using standardized indi-cators to allow comparisons or a reference toother similar types of capacity-building pro-grams. Self-reported measures of capacitymay not meet the reporting expectations ofexternal stakeholders even if they supportbetter capacity development strategies. Box12 describes the experience of one project inusing the two different approaches.

Lag time between changes in capacity andchanges in performanceIt is very common to experience considerablelag time between a capacity-building inter-vention and changes in capacity, as well asbetween changes in capacity and changes inperformance. Timing of capacity or perform-ance measurement should take into consid-eration these delays and consider interimmeasures of change or longer timeframes forM&E.

Tackling Methodological ChallengesMany of the tools and methods reviewed forthis guide were able to tackle challenges tocapacity-building measurement. Others pro-vided useful lessons on how to move capac-ity-building M&E forward. Advice to evalu-ators follows:� use multiple data-collection instruments,

reflecting the multidimensional nature ofcapacity. Multiple data-collection instru-ments are useful to get a comprehensivepicture of capacity or to assess capacityfrom different perspectives (e.g., assessingthe views of managers and health workersor assessing internal perspectives andthose of external examiners).

Box 12: PASCA: From Self-Assessment to External Assessment

PASCA is a USAID-funded project focusing on capacity building of nongovernmental organiza-tions (NGOs) that provide HIV/AIDS services in Central America. During the first year of theproject (1996), PASCA conducted a self-administered needs assessment study among the NGOsreceiving support. Although the needs assessment provided useful information for planning, theresearchers felt that the self-administered methodology exaggerated the programmatic, ad-ministrative and managerial capacity of the NGOs. Thus, managers decided to conduct an exter-nally administered Validation Study in 1997 using mixed methods to determine the validity ofthe self-reported data, and provide an in-depth assessment of the management and program-matic needs of each NGO. When compared to the Needs Assessment survey, capacity scoresfrom the Validation Study were markedly different. The Validation Study, in which self-reported answers were validated with document observation, provided data that more accu-rately reflected the capacity of the NGOs (MEASURE Evaluation, 1998).


� combine qualitative and quantitativemethods, such as focus groups, individualinterviews (with both closed- and open-ended questions), surveys, and documentreviews.

� address more than one level. Capacityoften occurs at several levels simultane-ously. New measurement tools are neededto capture capacity building at a singlelevel and address the relationship betweenlevels.

� include self-assessment techniques incombination with external or standardizedmethods. (See Box 13 for a discussion ofself-assessment and external assessment.)Evaluators are urged to strike a balancebetween meeting the need for evaluationdata that different stakeholders will deem“objective” or credible, and promotingperformance improvement through moni-toring and evaluation.

� triangulate methods and data sources.Triangulation examines results from a va-riety of data-collection instruments andsources, strengthening the findings of ca-pacity-building monitoring and evalua-tion. If all data lead to the same conclu-sion, then there is some confidence the re-sult actually will reflect changes. Wherethere is discordance in the results, it isnecessary to examine possible sources ofthe differences. Looking at other sourcesof data on similar topics can help under-stand findings as well.

� use data interpretation workshops to ob-tain input from a range of stakeholders in-volved in the program (both internal andexternal).

Sources of DataA number of data sources are available formonitoring and evaluating capacity building.Since capacity measurement often includesthe use of multiple indicators, monitoring andevaluation usually requires multiple datasources. Indicator design should take intoaccount the potential availability of data par-ticularly from existing sources. Organizationsand systems often have records and reportsthat provide insights into different aspects ofcapacity. Some examples of existing datasources are presented below.

In many cases, however, it will be necessaryto collect new data to operationalize the indi-cators selected. As noted above, issues suchas data sensitivity (with respect to its effect onvalidity), the purpose of monitoring andevaluation, and the cost in terms of time andresources required should guide evaluators indetermining what data will be collected andhow they will be collected.

Sources of data by level of capacity in-clude:

System: national health policy records, na-tional data-collection efforts (census, vitalstatistics, national /regional surveys), interna-tional surveys (e.g., FPPE, API, DHS).9 MOHpolicies, financial reports, legal or regulatorystatements (bills, acts, recommendations,white papers, etc.).

Organization: routine health service recordsand reports, budget and expenditure records,financial statements, personnel records, pro-gram and donor reports, constitutional docu-mentation, strategic and annual plans, meetingminutes, evaluations and audits, organiza-tional networking analysis, organizationalassessments. 9 FPPE (Family Planning Program Effort Score); API(AIDS Program Effort Index); DHS (Demographic andHealth Survey).

58

Health personnel: personnel records (job de-scriptions, performance evaluations, back-ground checks, training summaries), supervi-sion reports, self-evaluations.

Individual/Community: community-based andsocial marketing surveys, community healthworker reports, meeting minutes, maps, focusgroups, and participatory appraisals.

In planning for data collection, it is oftenhelpful to develop a data chart that spells outthe key questions to be addressed, the indica-tor that links to the question, and the datasources needed to answer the question. Anexample of a data chart is found in Table 6.

Tools for Measuring Capacity at DifferentLevelsA number of data-collection instruments andtools have been developed and used to meas-ure capacity at the four levels. (See Table 7for a list of tools and their key characteris-tics). In most cases, these tools have beenused for capacity assessment rather than formonitoring and evaluation. In addition, mostof the tools identified are designed to assessorganizational capacities, although many of

them assess the capacity of health programpersonnel because of their central role in or-ganizational functions and performance. Weidentified a more limited number of tools tomeasure the health system and individ-ual/community level capacity. However, thefield of capacity measurement is changingquickly and several agencies are currentlydeveloping approaches to understandingchanges in performance at the system level(Partnerships for Health Reform, 1997;Murray and Frenk, 1999).

The tools listed in Table 7 are provided forreference only. To determine if a tool mightbe useful for a particular capacity develop-ment intervention, evaluators should addressthe following questions:

� At what level(s) do I want to assess ca-pacity?

� Do any of the existing instruments meas-ure the dimensions or indicators I haveidentified through mapping?

� How could I adapt one of these instru-ments for my needs?


Box 13: Advantages and Disadvantages of Self-Assessment andExternal Assessment Techniques

While practitioners value the role of self-assessment tools in stimulating interest in capacitybuilding and launching a change process, for monitoring and evaluation purposes it is importantto consider the potential advantages and disadvantages of both internal and external ap-proaches.

Advantages of self-assessment tools:� Greater involvement of those whose capacities are being assessed (e.g., staff of an or-

ganization), which can lead to greater ownership of the results and, ultimately, greaterlikelihood that capacity improvements (based on results of the assessment) will takeplace

� Non-threatening way to raise awareness of the importance of capacity improvementamong those involved in the assessment process

But self-assessment tools� Require an external facilitator� Rely on perceptions and may be less reliable when used repeatedly and are prone to vari-

ous biases (e.g., optimistic bias)� Become less useful with high staff turn-over (which results in changing the ‘self’ in ‘self-

assessment)� In many cases are interventions in and of themselves

Advantages of external assessments tools:� Often considered more objective

But external assessment tools� May be more costly due to the cost of external consultants; self-assessments, particu-

larly those that require intensive facilitation, can also be demanding in terms of time� May not reflect internal views accurately

Recommendation:� Use a mixture of methods that combine subjective and objective measurement.

60

Table 6: Example of a Table of Data Sources for an Organizational AssessmentM&E Question(s) Objective(s) Indicator Method(s) Data Sources1. Did financial and human

resource inputs changeover time?

2. Did the source offinancial resourceschange over time?

1. Determine whethercapacity-buildinginterventions increasedbudgetary resources of theorganization and thenumber of trainedpersonnel.

2. Determine whetherchange in reliance ondonor/NGO funding hasdecreased.

1. Amount of budgetaryresources by source overtime

2. Number of managementand staff positions filledover time

1. Records review (organizationand donor)

2. Record review of personnelresources

3. Interviews with seniormanagement in organizationand donors/NGOs

1. Accounts, budgets, annualreports

2. Personnel records, annualreports

3. Finance manager,accountant, donor/NGOrepresentative

1. Did the organizationestablish newrelationships or improvelinks with otherorganizations thatcontributed to achievingperformance objectives?

1. Determine the extent ofnetworking and its effecton organizationalbehavior.

1. Number of joint activitieswith other organizations

2. Frequency of contact withhigher and lower levelorganizations within publicsector

3. Types and frequency ofoutcomes from links withother organizationsanalyzed by organizationtype (public or private)

1. Prospective recording of linksto other organizations

2. Interviews with managementand staff

3. Facility survey (observation,exit interviews, providerinterview, inventory)

1. Record forms

2. Questionnaire and focusgroups

3. Survey data

4. Organizationalnetworking analysis

1. Did staff capacity toassess client needsimprove?

1. Determine theeffectiveness of trainingand mentoring.

1. Client satisfaction index

2. Provider satisfaction index

1. Facility survey (exitinterviews, provider interview)

2. Client focus groups

3. Provider focus groups

1. Survey data

2. Focus group data

1. Did staff capacity tomeet client needsimprove?

1. Determine theeffectiveness of trainingand mentoring.

1. Client satisfaction index

2. Provider satisfaction index

1. Facility survey (exitinterviews, provider interview)

2. Client focus groups

3. Provider focus groups

1. Survey data

2. Focus group data


Table 7: Capacity Measurement Tools

Tool Developed By Level MethodsSelf/

ExternalAssessment

Single/Multiple

toolsShort description

Enhancing OrganizationalPerformance: A Toolbox forSelf Assessmenthttp://www.idrc.ca

IDRC Organization Qualitative andquantitative

External andself-assessment

Multiple Measures the results of an organization’s programs, prod-ucts and services and then integrates these results with thetechniques of formative assessment in which the assessmentteam becomes involved in helping the organization meet itsgoals.

Outcome Mapping: A Methodfor Reporting on Resultshttp://www.idrc.ca/telecentre/evaluation/html/29_Out.html

IDRC SystemOrganization

Qualitative andquantitative

Self-assessment Multiple Outcome Mapping characterizes and assesses the contribu-tions a project or organization makes to significant andlasting changes (outcomes). In Outcome Mapping a pro-gram is assessed against its activities that contribute to adesired outcome, not against the outcome itself.

Integrated Health FacilityAssessment (IHFA)http://www.basics.org/publications/pubs/hfa/hfa_toc.htm

BASICS Organization Quantitativeassessment

External as-sessment

Multiple This manual outlines the key steps for planning and con-ducting an integrated health facility assessment at outpatienthealth facilities in developing countries. This assessment isdesigned for use by primary health care programs that areplanning to integrate child health care services.

Management and Organiza-tional Sustainability Tool(MOST)http://erc.msh.org/mainpage.cfm?file=95.40.htm&module=toolkit&language=English

Family PlanningManagementDevelopment(FPMD)/MSH

Organization Qualitative Self-assessment Single The Management and Organizational Sustainability Tool(MOST) is a package (instrument and user's guide) designedto facilitate management self-assessment and to supportmanagement improvement. MOST uses an instrument tohelp focus an organization on the actual characteristics oftheir management, identify directions and strategies forimprovement, and set priorities for the management devel-opment effort.

Management DevelopmentAssessment (MDA)http://erc.msh.org/mainpage.cfm?file=95.50.htm&module=toolkit&language=English

FPMD/MSH Organization Quantitative Self-assessment Single This tool includes four steps: 1) develop a preliminarymanagement map to guide assessment; 2) develop andadminister MDA questionnaire to collect information on themanagement capabilities of organization; 3) analyze surveyresults and develop a post-survey management map; and 4)develop action plan for making improvements.

The Child SurvivalSustainability Assessment(CSSA)http://www.childsurvival.com

Child SurvivalTechnicalSupport (CSTS)Project/ORCMACRO

System(local)OrganizationCommunity


Self and internalclient assess-

ment

Multiple Evaluation framework to systematically measure progresstoward sustainable health goals. Process that projects canuse to lead a participatory assessment with communities andlocal partners.

The Institutional StrengthsAssessment (ISA) Toolhttp://www.childsurvival.com/tools/project_planning.cfm

CSTSProject/ORCMACRO

System(local)Organization


Self and internalclient

assessment

Multiple This self-assessment tool is currently being pilot tested byCSTS.

62


ExternalAssessment

Single/Multiple


INTRAH/PRIMECapacity Building In TrainingQuestionnairehttp://www.prime2.org/prime2/techreport/home/50.html

INTRAH/PRIMEII

Organization Qualitative andquantitative

Self andinternal client

assessment

Multiple The framework and tool developed at the end of PRIME Ihas been used to aid program evaluation in differentcountries (e.g., Mexico, Ghana, India, and Bangladesh),when interventions have focused on the strengthening oftraining and service delivery institutions. The toolencourages organizations to discover root causes ofobstacles with a sustainable effort to build capacity in theorganization to recognize, address, analyze and prioritizeproblems.

Client-Oriented ProviderEfficient (COPE®)http://www.engenderhealth.org/ia/sfq/qcope.html

Note: COPE has now beenadapted for use with maternalhealth services and commu-nity partnershiphttp://www.engenderhealth.org/news/newsreleases/020516.html

Engender Health Organization Qualitative andquantitative

Self-assessment Multiple COPE encourages and enables service providers and otherstaff at a facility to assess the services they provide jointlywith their supervisors. Using various tools, they identifyproblems, find the root causes, and develop effectivesolutions.

Transformational Develop-ment Indicators Field Guidehttp://www.worldvision.orgNOTE: Tool not yet availableonline

World Vision Community Qualitative andquantitative


Multiple Provides technical guidance for measuring theTransformational Development Indicators. It includes 8volumes that cover indicator definitions and methods forcollecting, analyzing, and reporting on the indicators.

Communication for SocialChange: An Integrated Modelfor Measuring the Process andIts Outcomeshttp://164.109.175.24/Documents/540/socialchange.pdf

Center forCommunicationsPrograms(CCP)/JohnsHopkinsUniversity

Community Qualitative andquantitative


Multiple Presents model, process and outcome indicators, and somedata collection and analytical tools for use by communities.



ExternalAssessment

Single/Multiple


Assessing Institutional Ca-pacity in Health Communica-tion: A 5Cs ApproachWork in Progress.http://www.jhuccp.org

CCP/JohnsHopkinsUniversity

Organization Quantitative External andself-assessment

Multipleinstruments

Scores organizational competence, commitment, clout,coverage and continuity.

Management/FinancialSustainability Scale (MFSS)http://www.pasca.org

PASCA Organization Quantitative External andself-assessment

Singleinstrument

Tools are in Spanish only.

Systematic Approach Scale(SAS)http://www.pasca.org

PASCA Organization Quantitative External andself-assessment

Singleinstrument

Tools are in Spanish only.

Institutional AssessmentInstrument (IAI)http://www.worldlearning.orgorhttp://www.worldlearning.org/pidt/docs/wl_instcape.pdf

World LearningProject Inc.


Externalassessment

Multipleinstruments

Provides a framework for assessing the institutional needsof a single organization or a community of organizations.Pinpoints six key areas generally agreed to be the compo-nents of effective institutions.

Institutional DevelopmentAssessment (IDA)http://www.fha-sfps.org/documentsdownload/Institu-tional%20Development%20Assessments.PDF

SFPS Organization Qualitative andquantitative

Externalassessment

Multipleinstruments

Documents existing capacity and identifies potential areasof collaboration and capacity building in overall dimensionsof management, financial management and technical capac-ity.

Organizational CapacityAssessment Tool (OCAT)http://www.pactworld.org

Pact/Ethiopia Organization Quantitative Self-assessment Multipleinstruments

A methodology for organizational capacity assessment andstrengthening that helps organizations anticipate and over-come the greatest barriers to organizational change andgrowth. Through a guided self-assessment and planningprocess, organizations reflect upon their performance andselect the tools and strategies they need to build capacityand broaden impact. A four-staged process that includes:Participatory tool design; guided self-assessment; data-guided action planning; reassessment for continual learningthat allows organizations to monitor change, track the effec-tiveness of their capacity-building efforts, and integrate newlearning as their needs change and capabilities increase.

64


ExternalAssessment

Single/Multiple


Participatory, Results-Oriented, Self-Evaluation(PROSE)

SEE POET at:http://www.undp.org/csopp/poet.htm

EducationDevelopmentCenter and PACT


Self-assessment Singleinstrument

Participatory Organizational Evaluation Tool (POET) is anorganizational capacity assessment tool used to measure andprofile organizational capacities and consensus levels inseven critical areas and assess, over time, the impact ofthese activities on organizational capacity (benchmarking).POET is based on a methodology called PROSE.

PROSE stands for Participatory, Results-Oriented, Self-Evaluation, a new methodology for assessing and enhancingorganizational capacities. PROSE is designed for use byservice organizations, schools, and government units. It issuitable for assessing capacity and catalyzing organizationalchange in relation to such concerns as: practices related toexceeding customer expectations, organizational effective-ness in achieving mission, community participation, equity,decentralization, and managerial effectiveness.

National program effortindicesFamily Planning Effort Index(FPEI)http://www.agi-usa.org/pubs/journals/2711901.pdfThe AIDS Program EffortIndex (API)http://www.policyproject.com/pubs/countryreports/api.pdf

The FuturesGroup/PopulationCouncil

System(national)

Organization

Quantitative andqualitative

Externalassessment

Singleinstrument

Each index measures national level effort and identifiesstrengths and weaknesses of those efforts.


STEP 6 Develop anImplementation andDissemination Plan

The final step in planning for capacity-building M&E is to develop an implementa-tion plan to monitor and evaluate capacity. Ata minimum, the implementation plan shouldinclude a timetable for data gathering andreview of data, individual responsibilities, adissemination strategy, and a budget. In prac-tice, capacity measurement, as a reflection ofcapacity development, is likely to be an itera-tive process rather than a perfunctory “beforeand after” look at capacity. Experiencedevaluators (Horton et al, 2000; Lusthaus,1999; Earl et al., 2001; Morgan, 1997) rec-ommend regular review and discussion ofmonitoring results with stakeholders to guidethe process of capacity development and en-courage ownership of the monitoring process.Setting aside enough time to present the re-sults periodically and allow for discussion andfeedback from the stakeholders will greatlyenhance data interpretation and the impact ofthe evaluation itself. As Morgan (1997) notes,“Indicators by themselves provide few an-swers. The information they produce must be

screened through the mental models of theparticipants to acquire any diagnostic value.”

When developed before the evaluation begins,a dissemination strategy guides data collec-tion and analysis. Developing a format forpresentation of the results to the appropriateaudience identifies weaknesses and gaps inthe evaluation plan. It also helps to guide thedirection of the evaluation by emphasizingwhat is needed for addressing the needs of thedata users and raising awareness of possiblesensitivities. Gaps or excess data collectionbecomes obvious, and further refinement ofthe number or type of indicators being meas-ured is often necessary. In the process, evalu-ators identify all key stakeholders that shouldbe alerted to the results, if they are not di-rectly involved in the evaluation itself. Therecommended forum for disseminating resultsis one that promotes discussion and interac-tion among the key stakeholders and those ina position to influence the future direction ofthe capacity-building efforts. Sufficient fundsmust be set aside so that all those who make acredible contribution to the evaluation receiveat least summary results in a timely and rele-vant fashion.

Summary Checklist: Steps for Designing a Capacity-Building M&E Plan 67

Part 4 Summary Checklist: Steps for Designing a Capacity-BuildingM&E Plan

This guide is designed to assist manager andevaluators working in international health-sector capacity development to� gain a clear understanding of the con-

cepts of capacity and capacity building� critically evaluate the strengths and

limitations of current approaches to ca-pacity measurement

� design a capacity-building M&E planthat outlines a systematic approach tomeasuring capacity and assessing the re-sults of capacity-building interventionsin the health sector

The manual presents a discussion of theconcept of capacity and capacity building,and the influence of attributes of capacity onM&E approaches. It outlines a conceptualframework for understanding the role thatcapacity plays in enabling performance inthe health sector and suggests an approachto identifying key factors that influence ca-pacity and performance. Finally, it outlinessome basic steps for capacity-building M&Ethat result in a plan for evaluating a specificcapacity-building intervention. These stepsare summarized in the checklist that follows.

Checklist: Steps in Designing aCapacity-Building M&E PlanThe Guide recommends a six-step approachfor developing an M&E plan for capacitybuilding. The key components of each stepare outlined below.

Define the purpose of the evaluation(Step 1)� Evaluators and program planners should

work with key stakeholders to developan M&E plan during the design of a ca-

pacity-building or performance im-provement intervention.

� Capacity-building M&E can be usedinternally to improve capacity develop-ment interventions or to report results toexternal stakeholders. While these twopurposes are not mutually exclusive,managers should understand the benefitsand drawbacks of emphasizing one ob-jective at the expense of the other.

Define performance objectives (Step 2)� Capacity is a prerequisite for perform-

ance. Evaluators must clearly state theperformance objectives of a capacity-building intervention at the outset ofM&E planning and understand the pro-gram’s approach to improving perform-ance.

� Performance objectives can be expressedas variables or indicators that can bemeasured against international or na-tional standards or locally determinedexpectations. Normally, the definition ofperformance objectives reflects both ex-ternal and internal criteria.

Mapping capacity: Build a conceptualframework for the specific capacity-building intervention (Step 3)� Capacity mapping is a structured process

of “thinking through” the role capacityplays in ensuring performance by devel-oping a conceptual framework that isspecific to a particular capacity-buildingintervention. Mapping identifies keyfactors of capacity and assumptionsabout how they interact to influence ca-pacity and performance. If programplanning and M&E design are conductedsimultaneously, capacity mapping can

68

contribute to the choice of interventionstrategies and to the M&E strategy.

� The external or operational environmentmay have a considerable effect on thepace, process, outcome, and sustainabil-ity of capacity development. It is advis-able for program managers to track envi-ronmental changes periodically.

� Each type of mapping (single-level ormultiple-level) can be done in two orthree iterations. The first iteration of amap should attempt to provide a full listof capacity variables that may influencecapacity outcomes and performance. Itshould present capacity variables in ageneral way. The second or third itera-tion of a map should be more precise indepicting the variables to be monitoredover the course of the intervention.

� Capacity mapping is sometimes con-fused with Performance Improvement(PI). � For clarification, the reader isreferred to the definition of PI in theGlossary and the table in Annex D.

Identify capacity indicators (Step 4)� Well-defined indicators provide a refer-

ence framework for guiding allstakeholders toward the same goals. In-dicators also allow for standardizedmeasurement of change during imple-mentation, which enables evaluators tounderstand the process of capacity de-velopment over time and its relationshipto capacity-building intervention.

� Capacity indicators generally project anaspiration or a sought-after state or abil-ity. They capture the current “stock ofresources available” for various uses, oran individual or organizational behaviorthat puts those resources into action(Moore et al., 2001).

� When selecting capacity indicators it isadvisable to be clear about specific per-formance and capacity development ob-

jectives as well as particular capacity-building activities.

Identify appropriate methodological ap-proach and sources of data (Step 5)� All capacity-building programs need to

be monitored to ensure they are workingwell (i.e. to track changes in inputs, pro-cesses, outputs and outcomes). However,the evaluation of program effectivenesshappens less frequently and only for se-lected interventions due to cost andcomplexity.

� Impact evaluation is not advisable incapacity-building M&E since capacitymeasures are not easily quantified, andidentifying similar organizations or sys-tems to facilitate comparison (as in acase-control study) is difficult.

� Capacity measurement tools should beable to capture different stages of devel-opment of communities, health person-nel, organizations, or health systems.

� M&E tools are needed that allow forrepeated measures to capture the interimsteps in capacity-building processes aswell as trends in outcomes.

� Capacity building should be a self-motivated and self-led process ofchange. Evaluation strategies that useself-assessment techniques and locallydetermined benchmarks of progress in-spire ownership of capacity developmentand increase the likelihood that evalua-tion results will be used. However, therecan be a cost to this approach in terms ofthe perceived validity of findings.

� In the design of capacity-building M&Estrategies, evaluators are advised to usemultiple data-collection instruments,combine qualitative and quantitativemethods, address more than one level ofcapacity and relations between levels,include self-assessment techniques incombination with external or standard-

Summary Checklist: Steps for Designing a Capacity-Building M&E Plan 69

ized methods, triangulate methods anddata sources, and use data interpretationworkshops.

Develop an implementation and dissemi-nation plan (Step 6)� In disseminating results evaluators

should review findings regularly, anddiscuss them with stakeholders to guidecapacity development and encourageownership of the M&E process.

Example of Scoring Used for Measuring Capacity Building in Training, PRIME 1 71

Annex A Example of Scoring Used for Measuring Capacity Building in Training, PRIME I(Fort, 1999)

Dimensions Objectives Indicator ScoringI. Legal/Policy Support National FP/RH service guidelines

and training are official1. Existence of updated officialFP/RH service and trainingguidelines

0=Nonexistent guidelines (both serviceand training), to4=Complete/updated, disseminated,and official guidelines

Political support for traininginstitutionalization

2. Official (written) policysupporting institutional trainingcapacity - e.g., training units,cadre of master trainers, venues,etc. - for health providers

0=Nonexistent written policy to4=Written/updated, disseminated, andofficial

3. Favorable public statements onFP/RH training (for theimprovement of services) at leasttwice a year by senior officials

0=No mention, to4=Mentioned on several private and atleast twice on public occasions

II. Resources FinancialExistence of sufficient anddiversified training budget

4. </= 20% of training budgetcomes from external assistance

0=No in-country training budgets;funds are allocated on ad hoc basis, to4=20% or more of training budgetcomes from external assistance

5. Budget covers all aspects oftraining (including materials andequipment, travel and per diem byconsultants and staff, venue hireand maintenance, etc.)

0=Budget does not cover all aspects oftraining, to4=Budget covers all training costs

Venues/EquipmentAdequate venues

6. Accessible and available (own,rented) venues (at least one localvenue in each training area) ofstandard quality (continuouspower, food, lighting, acoustics,and sufficient capacity), accessibleto participants, and available whenneeded

0=Nonexistent venue, (incrementallyscoring coverage, capacity, and/orquality of venue), to4=Fully accessible, high-quality, andsufficient-capacity local venue fortraining events

72

Dimensions Objectives Indicator ScoringMaterials, equipment, andsupplies (MES)Appropriate and cost-efficientMES, (including AV equipmentand teaching aids)

7. MES are pertinent, updated,sufficient, and adapted to localculture (including locallyproduced)

0=MES are insufficient and/oroutdated, to4=MES of standard technical andmaterial quality and readability areavailable for each event participant

Systems are in place forreplacement and upgrading ofMES

8. Financial, printing and planningcapabilities exist for replacing andupgrading MES

0=There are no or insufficient meansfor replacing MES, to4=The means exist to produce, replaceand upgrade MES

HumanTrainers/preceptors formed haveupdated and standardizedtechnical and presentationknowledge and skills

9. Trainers/preceptors areconstantly formed (TOT) and doperiodic refresher courses andpass standard tests on FP/RHtechnical and presentationknowledge and skills

0=Trainers/preceptors not regularlyformed and/or do not update theirtechnical and presentation knowledgeand skills, to4=Trainers/preceptors constantlyformed and undergoing periodic (atleast once every two years) refreshercourses

III. Training Plans andCurriculum

Updated and periodicallyreviewed training plan

10. Training plan exists and isreviewed annually

0=No training plan performance(training conducted on ad hoc basis), to4=Training plans are drawnperiodically (at least annually) andreviewed

Updated curriculum is officialstandard for training institutions

11. Existence of a standard officialtraining curriculum guidingtraining institutions

0=No standard training curriculum orcurriculum is inadequate / outdated,different ones used by differentinstitutions, to4=There is a standard curriculum,reviewed periodically (at least onceevery 2 years) and used officially bytraining institutions

Example of Scoring Used for Measuring Capacity Building in Training, PRIME 1 73

Dimensions Objectives Indicator ScoringIV. Organization Leadership

Vision of training as a means toimprove services

12. Training plans are linked withquality of care and increasedservice access

0=Providers’ training plans are notcoupled with service and quality ofcare objectives, to4=Training plans form part of thequality of care and serviceimprovement strategies

Training is an integral part oforganization’s strategic planning

13. A training plan and activitiesare part of the organization’sstrategic plans

0=Training is not part of theorganization’s strategic plan, to4=Training is part of the organization’slong-term strategic plan (multiannual)

Promotion of public-privatecollaboration

14. Evidence of public-privatecollaboration

0=No evidence of public-privatecollaboration, to4=Evidence of public-privatecollaboration

InfrastructureExistence of decentralized trainingunits in all areas

15. Active training units exist atcentral and peripheral levels

0=No decentralized training units (evenif there is one at central level), to4= Active training units in central andperipheral levels

Human resource developmentHR training (TOT, formative andrefresher courses) is an integratedpart of a PerformanceImprovement system (e.g.,incentives, follow-up andsupervision, efficacy)

16. HR development is part of aperformance improvement (PI)strategy

0=Training is not coupled withproviders’ improvement objectives, to4=Training is part of HR developmentand performance

AdministrativeExistence of a reporting systemfor tracking number andcharacteristics of trainees andmaterials, according to needs

17. Existence and use of aTraining Needs Assessment

18. Existence of an MIS fortrainees and materials matchingTNA

0=No TNA customarily done, to4=TNA is integral and continuous partof training strategy

0=No MIS for tracking progress, to4=MIS for training

74

Dimensions Objectives Indicator ScoringTechnical capabilityTechnological transfer anddevelopment through networking,evaluation, and research

19. Contacts with other traininginstitutions and institution’evaluation and research feed intotraining improvement (e.g., traineeselection, training contents andformats)

0=No/little use of evaluation andresearch of information from othertraining institutions to improve, updatetraining capabilities, to4=Extensive use of internal andexternal data and resources forimprovement

Track recordProven capacity toconduct/replicate coursesautonomously

20. Replica/other courses carriedout independently (withinstitutional resources)

0=No replica or independent coursescarried out by the organization (or onlydone with foreign assistance), to4=Evidence of ongoingreplication/expansion of courses withinstitutional resources

V. Community Development-Participation

Community representatives areinvolved in planning andexecution of training activities, areaware of their rights, and/ordemand competent providerperformance

21. Evidence of communityinvolvement in providers’ trainingand/or performance assessment(e.g., quality of care circles)

0=No/little community involvement, to4=Extensive involvement /participation in provider training and/orperformance assessment; organizeddemand/petitions to improve services,etc.

Example of Results of PRIME Training Capacity Index 75

Annex B Example of Results of PRIME Training Capacity Index (Catotti, 1999)

Note: See Annex A for definitions of indicators.

El Salvador Capacity Building

1.0 1.0 1.0

3.0

1.5

3.7 3.72.8 2.8 2.8 3.2 3.5 4.0

2.32.0

3.7

2.53.2

2.52.0

3.3

2.0

56.7

1.51.71.71.7

1.0

1.81.3

2.32.0

1.52.2

35.8

2.02.8

1.82.2

1.7

3.7

2.2

1.0

10.0

100.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

1997-99 Scores for each of the 20 Indicators and Average Score (21) - Logarithmic Scale

Scor

e 19971999

Key Internet Resources for Monitoring and Evaluating Capacity-Building Interventions 77

Annex C Key Internet Resources for Monitoring and EvaluatingCapacity-Building Interventions

There is a wealth of information on capacity measurement and evaluation in general on theInternet. The list that follows describes Internet sites that focus specifically on capacity meas-urement; it also includes sites that provide general evaluation information and resources. Thedetails of many of the capacity measurement tools found on these sites are found in Table 7 inPart 3 of the Guide. Please note that inclusion on the list does not imply any judgment about anyitem listed or not listed.

Capacity Measurement Sites

1. The Manager’s Electronic Resource Center – Management Sciences for Health

http://erc.msh.org/http://www.msh.org/

The Health Manager's Toolkit is an electronic compendium of tools designed to assist healthprofessionals at all levels of an organization to provide accessible, high-quality, and sustainablehealth services. It is particularly useful for managers who lead others to produce results.

The Health Manager’s Toolkit includes spreadsheet templates, forms for gathering and analyzingdata, checklists, guidelines for improving organizational performance, and self-assessment toolsthat allow managers to evaluate the systems underlying their entire organization. The tools havebeen developed by organizations working throughout the world to improve delivery of healthservices.

For more information, contact Gail Price or Amanda Ip by e-mail ([email protected]).

2. INTRAH/Prime II

http://www.prime2.org/

The PRIME II Project is a partnership combining leading global health care organizations dedi-cated to improving the quality and accessibility of family planning and reproductive health careservices throughout the world. Funded by USAID and implemented by the University of NorthCarolina at Chapel Hill School of Medicine, PRIME II focuses on strengthening the performanceof primary care providers as they work to improve services in their communities. To accomplishits goals, PRIME II applies innovative training and learning and performance improvement ap-proaches in collaboration with host-country colleagues to support national reproductive healthgoals and priorities.

78

Since 1997, The PRIME Project has been committed to applying the guiding principles of per-formance improvement (PI) to real-world reproductive health contexts. Work in Yemen, BurkinaFaso, the Dominican Republic, and India indicates that PI users like the clear, highly participa-tory process and the focus on cost-effective interventions to address the most important problemareas.

This interactive Website, created by the PRIME II Project and INTRAH, presents a revised edi-tion of Performance Improvement Stages, Steps and Tools, first issued in print form in 2000.INTRAH/PRIME II published this site online in August 2002 (www.intrah.org/sst/).

For more information, please contact Marc Luoma by email ([email protected]).

3. JHPIEGO

http://www.jhpiego.org

Through advocacy, education and performance improvement, JHPIEGO helps host-country poli-cymakers, educators and trainers increase access and reduce barriers to quality health services,especially family planning and maternal and neonatal care, for all members of their society.JHPIEGO’s work is carried out in an environment that recognizes individual contributions andencourages innovative and practical solutions to meet identified needs in low-resource settingsthroughout Africa, Asia, and Latin American and the Caribbean.

TIMS is a computer-based tool to track and monitor training efforts. Each person’s skills, quali-fications, and location are stored, along with courses taken and taught, through a Microsoft Ac-cess 2000 database application that stores information about training course content, timing, par-ticipants, and trainers. In the standard form, TIMS tracks the following training results over aperiod of time:- Which providers from which service sites have been trained, and in what topic(s)- Which trainers have been conducting courses, and how many people they have trained- How many courses have been held, summarized by training center, district, or province

TIMS allows senior and mid-level program managers to monitor the variety of training activitiesand track results in a number of perspectives. TIMS is designed to be part of a country’s traininginformation system, replacing paper-based reporting and aggregation with a computer database.Ministries of Health, Planning and/or Finance can use TIMS to supplement service informationfor policy decisions on training, retraining, and provider deployment.

For additional information about TIMS, contact Catherine Schenck-Yglesias by e-mail([email protected]).


4. Child Survival Technical Support Program (CSTS)

http://www.childsurvival.com/

The Child Survival Technical Support Project (CSTS) assists PVOs funded through the Office ofPrivate and Voluntary Cooperation's Child Survival Grants Program. The technical supportCSTS provides to PVOs is targeted specifically towards increasing their capacity to achievesustainable service delivery in public health interventions.

The program’s goal is to help these organizations grow and to develop successful programs thatwill continue to serve mothers, children, and communities even when the PVO is no longer pres-ent in the area.

5. International Development Research Centre-Canada (IDRC)

http://www.idrc.ca/

The International Development Research Centre (IDRC) is a public corporation created in 1970to help developing countries find long-term solutions to the social, economic, and environmentalproblems they face. IDRC’s Evaluation Unit has been working in the area of organizational as-sessment for over 5 years and has developed a number of tools, including: Enhancing Organiza-tional Performance, a guidebook that presents an innovative and thoroughly tested model fororganizational self-assessment. The tools and tips presented in the guidebook go beyond meas-uring the impact of programs, products, and services to integrate techniques of formative as-sessment, in which the assessment team becomes involved in helping its organization becomemore effective in meeting its goals. The tools and techniques are flexible, and the model can beadapted to any type or size of organization. Worksheets and hands-on exercises are included.

Enhancing Organizational Performance will be useful to any organization that is initiating aprocess of self-assessment, internal change, or strategic planning. It will appeal particularly toheads and staff of research organizations, university administrators, staff of research-grantingagencies, and academics and professionals in organizational development and evaluation.

6. International Institute for Sustainable Development (IISD)

http://iisd1.iisd.ca/measure/

IISD has been working on measurements and indicators since 1995, with the aim of making sig-nificant local, national, and international contributions, and building the Institute into a worldcenter of expertise in this field. One of IISD’s strategic objectives is to develop robust sets ofindicators for public and private sector decision-makers to measure progress toward sustainabledevelopment and to build an international consensus to promote their use.

80

7. World Health Organization (WHO)

http://www.who.int/whr2001/2001/archives/2000/en/index.htm

World Health Report 2000. Health Systems: Improving PerformanceThe World Health Report 2000 aims to stimulate a vigorous debate about better ways of meas-uring health system performance and thus finding a successful new direction for health systemsto follow. By shedding new light on what makes health systems behave in certain ways, WHOalso hopes to help policymakers weigh the many complex issues involved, examine their options,and make wise choices.

8. USAID – Development Experience Clearinghouse (DEC)

http://www.dec.org/

The DEC includes Evaluation Publications such as the TIPS series, which provides guidance onusing the Results Framework, measuring institutional capacity and general quality of indicatorsand performance measures.

9. Pact

http://www.pactworld.org/services/oca/index_oca.htmhttp://www.pactworld.org/

Pact’s unique methodology for organizational capacity assessment and strengthening (OCA)helps organizations anticipate and overcome the greatest barriers to organizational change andgrowth. Through a guided self-assessment and planning process, organizations reflect upon theirperformance and select the tools and strategies they need to build capacity and broaden impact.

Pact's OCA is the product of ten years of research and field practice in partnership with the Edu-cation Development Center and USAID’s Office of Private & Voluntary Cooperation. Hundredsof local and international NGOs, private-sector corporations, and municipal governments aroundthe world have used this methodology.

OCA is a four-staged process that includes:

� Participatory tool design that empowers organizations to define the critical factors thatinfluence their performance and to identify relevant indicators for evaluating their com-petency.

� Guided self-assessment that leads employees, board members, and constituents throughstructured discussions followed by individual scoring on a series of rigorous performanceindicators.

� Data-guided action planning that provides organizations with an opportunity to interpretthe self-assessment data and set change strategies most appropriate to their environment.


� Reassessment for continual learning that allows organizations to monitor change, trackthe effectiveness of their capacity-building efforts, and integrate new learning as theirneeds change and capabilities increase.

For more information on Pact’s Organizational Assessment, please contact Betsy Kummer byemail ([email protected]).

Publications Available from Pactwww.pactpublications.org

From the Roots Up: Strengthening Organizational Capacity through Guided Self-Assessmentby World NeighborsPublisher: World NeighborsYear: 2000

Basic Guide to Evaluation for Development Workersby Frances RubinPublisher: OxfamISBN: 0-85598-275-6Year: 1995This book will help groups to plan for and carry out evaluations as an integral part of develop-ment activities. Easy to follow, it focuses on the principles underlying evaluation and dealsclearly and simply with the issues to be considered at the planning stage. It then examines thesteps involved in carrying out different types of evaluation, for specific purposes. The impor-tance of involving local people in evaluations is emphasized throughout.

Participatory Monitoring, Evaluation and Reporting: An Organisational Development Perspec-tive for South African NGOsby PactPublisher: Pact PublicationsYear: 1998This manual explains why participation is important and how to achieve effective stakeholderparticipation; the role of monitoring in sustaining progress toward better organizational effec-tiveness; how evaluation helps an organization to assess its capacity; and the critical role of re-porting to stakeholders. It then deals with applying the Organizational Capacity Assessment Tool(OCAT) in practice, together with examples. A step-by-step guide to designing and implement-ing a Participatory Monitoring, Evaluation and Reporting (PME&R) information system is in-cluded. Although it has been specifically adapted for use by South African NGOs, NGOs can useOCAT in other countries.

82

10. The International HIV/AIDS Alliance

www.aidsalliance.org/ngosupport

The AIDS Alliance has developed an HIV/AIDS NGO/CBO Support Toolkit that is available ontheir Website or by CD-Rom with over 500 downloadable resources and supporting information.

The toolkit includes practical information, tools and example documents to help those working toestablish or improve NGO/CBO support programs. The toolkit also describes key components ofNGO/CBO support programming, based on the Alliance's experience. It also includes resourcesfrom a wide range of other organizations to bring different perspectives and experiences to-gether.

The HIV/AIDS NGO/CBO Support toolkit has been developed for those wishing to establish orimprove NGO/CBO support programs. The toolkit will be useful both for NGO-led support pro-grams and for government-led or multi-sectoral programs, especially in the context of GlobalFund and World Bank financing for NGOs and CBOs working on AIDS. The toolkit will also beuseful to organizations that provide only funding or only training.

Order single or bulk copies of the CD-ROM and supporting publication free of charge from:[email protected]

11. International NGO Training and Research Centre (INTRAC)

http://www.intrac.org/

International NGO Training and Research Centre (INTRAC) provides support to organizationsinvolved in international development. Their goal is to improve the performance of NGOs byexploring relevant policy issues and by strengthening NGO management and organizational ef-fectiveness.

Documents can be ordered through their Website including:

Practical Guidelines for the Monitoring and Evaluation of Capacity-Building: Experiences fromAfricaISBN: 1 897748-64-7OPS No. 36, November 2001.

Capacity building and monitoring and evaluation have become two of the most important priori-ties of the development community during the last decade. Yet they have tended to operate inrelative isolation from each other. In particular, capacity-building programs have been consis-tently weak in monitoring the impact of their work. This publication aims to help NGOs and do-nors involved in capacity building to develop appropriate, cost-effective and practical systemsfor monitoring and evaluation. While not under-estimating the complexity of these tasks, thispublication puts forward some practical guidelines for designing monitoring and evaluation sys-tems based on experiences with three organizations in different parts of Africa.


12. Performance Improvement in Healthcare

http://www.picg.net/

This Website is designed to provide information, tools, and guidelines for planning, implement-ing, monitoring and evaluating performance improvement processes and activities in healthservices delivery organizations. The site is especially tailored for managers, leaders, providersand other employees working in international health organizations and institutions, whether theyare health ministries or health departments in the public sector or NGOs in the private non-profitsectors. The site is also for those working as partners with people in these institutions.

Performance Improvement (PI) is a process for enhancing employee and organizational perform-ance that employs an explicit set of methods and strategies. Results are achieved through a sys-tematic process that considers the institutional context; describes desired performance; identifiesgaps between desired and actual performance; identifies root causes; selects, designs and imple-ments interventions to fix the root causes; and measures changes in performance. PI is a continu-ously evolving process that uses the results of monitoring and feedback to determine whetherprogress has been made and to plan and implement additional appropriate changes.

The goal of PI is to solve performance problems or realize performance opportunities at the or-ganizational, process or systems and employee levels in order to achieve desired organizationalresults. The overall desired result in our field is the provision of high quality, sustainable healthservices.

The Website includes information on the performance improvement process and factors affectingworker performance, PI tools, and experiences using PI in different health care settings,

For more information or questions email [email protected].

13. Capacity.org

http://www.capacity.org/index_en.html

Capacity.org is a Website dedicated to advancing the policy and practice of capacity building ininternational development cooperation. Issue 14 of the web-based magazine Capacity.org pres-ents highlights of the UNDP initiative on capacity building and related information on the policyand practice of capacity building in international development cooperation (also see UNDP web-site at http://www.undp.org/dpa/publications/capacity.html).

14. ISNAR/CGIAR - Evaluating Capacity Development in Research & Development Or-ganizations:

http://www.isnar.cgiar.org/ecd/index.htm

This site promotes the use of evaluation as a tool to advance the development of organizationalcapacity and performance. Its main purpose is to support a group of managers and evaluators

84

who are evaluating capacity development efforts in their own organizations in Africa, Asia andLatin America. This site presents the work of a global project, "Evaluating Capacity Develop-ment Project (The ECD Project)." National and international research and development organi-zations are participating in the ECD Project, which is supported by five donor agencies and co-ordinated by ISNAR.

The site features the ECD Project's activities since 2000 and its result to date. It provides accessto project reports and events. Lists of useful concepts and terms, bibliographic references andInternet resources are also provided for use by capacity developers and evaluators

15. Reflect-Learn.org - The Organizational Self-Reflection (OSR) Project

http://www.reflect-learn.org/

The Organizational Self-Reflection (OSR) project aims to improve organizational learning byincreasing access to self-reflection tools. The process of reflection implies an organizational di-agnosis that will allow learning from experiences, styles of work and results in order to fosterstrategic vision, decision making, organizational change and capacity building. The organizationkeeps control over orientation of the process and use of results.

The project links a direct service, based on the Internet, and a research agenda designed to createknowledge about self-reflection and its contribution to organizational learning. The OSR projectseeks to engage diverse organizations in the use of self-reflection resources and also catalyzesthe development of a learning community that focuses on OSR, organizational learning, and theuse of the Internet for institutional strengthening. Several useful frameworks and tools for or-ganizational assessment are presented

16. UNDP United Nations Development Project

http://www.undp.org/dpa/publications/capacity.html

Developing Capacity through Technical Cooperation: Country Experiences provides some con-crete inputs to rethinking technical cooperation for today’s challenges based on six countrystudies – Bangladesh, Bolivia, Egypt, Kyrgyz Republic, Philippines and Uganda.

Capacity for Development: New Solutions to Old Problems, with prominent academics and de-velopment practitioners as contributors, proposes new approaches to developing lasting indige-nous capacities, with a focus on ownership, civic engagement and knowledge. It is a contributionto a process of debate and dialogue around the broader issue of improving effective capacity de-velopment.

Development Policy Journal is a new forum for presenting ideas on applied policies. The subjectof capacity for sustainable development is addressed in this first issue.


17. EngenderHealth

http://www.engenderhealth.org

EngenderHealth works worldwide to improve the lives of individuals by making reproductivehealth services safe, available, and sustainable. EngenderHealth provides technical assistance,training, and information, with a focus on practical solutions that improve services where re-sources are scarce in partnership with governments, institutions, and health care professionals.

EngenderHealth's trademarked COPE (client-oriented, provider-efficient services) is a set offlexible self-assessment tools that assist providers and supervisors to evaluate and improve thecare offered in clinic and hospital settings. Using self-assessment, client-interviews, client-flowanalysis and facilitated discussion, staff identify areas needing attention and develop their ownsolutions and action plans to address the issues. Originally developed for family planning serv-ices, COPE has been successfully applied in a variety of healthcare settings all over the world forover 10 years. With the growing popularity of COPE, healthcare providers from related disci-plines asked if the tools could be adapted to a wider range of health services. EngenderHealthhas answered the demand by creating these new products: COPE for Maternal Health Servicesand Community COPE: Building Partnership with the Community to Improve Health Services.

86

General Evaluation Sites

1. American Evaluation Association

http://www.eval.org

The American Evaluation Association, an international professional association of evaluators, isdevoted to the application and exploration of program evaluation, personnel evaluation, evalua-tion technology and other forms of evaluation.

The American Evaluation Association has a Collaborative, Participatory and EmpowermentEvaluation topical interest group that is dedicated to the exploration and refinement of collabo-rative, participatory and empowerment approaches to evaluation. You can find more informationabout them at: http://www.stanford.edu/~davidf/empowermentevaluation.html

2. Canadian Evaluation Association

http://www.evaluationcanada.ca/

The Canadian Evaluation Association is dedicated to the advancement of evaluation for itsmembers and the public. This site is also available in French.

3. The Evaluation Center at Western Michigan University

http://www.wmich.edu/evalctr/

The Evaluation Center, located at Western Michigan University, is a research and developmentunit that provides national and international leadership for advancing the theory and practice ofevaluation, as applied to education and human services.

4. Essentials of Survey Research and Analysis

http://freenet.tlh.fl.us/~polland/qbook.html

This site contains a complete manual entitled Essentials of Survey Research and Analysis: AWorkbook for Community Researchers, written by Ronald Jay Polland, Ph.D.,1998.

5. German Center for Evaluation (in German)

http://www.uni-koeln.de/ew-fak/Wiso/

This is the homepage for the German Center for Evaluation at the University of Cologne. It in-cludes the German translation of the Program Evaluation Standards of the American EvaluationSociety.


6. Government Performance Information Consultants

http://members.rogers.com/gpic/evalwebindex.htm

This site offers links to many Web resources on evaluation.

7. The Michigan Association for Evaluation

http://www.maeeval.org/

The Evaluation Promotion Committee has compiled a list of resources in an effort to provideMAE members and others interested in evaluation with sources for educational materials, tools,and other resources that may be interesting and helpful. For each resource, the site provides abrief description (generally from the resource itself) and where to find it.

8. Innovation Network, Inc. (InnoNet)

http://www.innonet.org/

Innovation Network, Inc. (InnoNet) is an Innovation Network, a national nonprofit dedicated tobuilding the evaluation capacity of nonprofits so they can better serve their communities. In-noNet has two services to meet this end: a search service to find model programs, and an evalua-tion service that guides agencies through a planning and evaluation process. Description of theirevaluation methodologies and documents available for ordering are listed on this site.

9. International & Cross-Cultural Evaluation Topical Interest Group (I&CCE)

http://home.wmis.net/~russon/icce/

International & Cross-Cultural Evaluation Topical Interest Group is an organization affiliatedwith the American Evaluation Association. The purpose of the I&CCE is to provide evaluationprofessionals who are interested in cross-cultural issues with an opportunity to share their expe-riences with one another.

10. MandE News

http://www.mande.co.uk/

MandE News is a news service focusing on developments in monitoring and evaluation methodsrelevant to development projects and programs with social development objectives. It is editedby Rick Davies in Cambridge, UK who can be contacted by email ([email protected]).

88

11. Sociometrics

http://www.socio.com/eval.htm

Sociometrics offers a wide variety of evaluation products and services to professionals across theworld. Their evaluation workshops and training services, technical publications, evaluation tools,and data sets are all designed to assist practitioners, administrators, evaluators, and funders ofsocial interventions to design and implement successful evaluation systems.

For additional information, contact Dr. Shobana Raghupathy by email ([email protected]) orby phone at 1.800.846.3475 x209.

12. Bill Trochim, Cornell University

http://trochim.human.cornell.edu/kb/conmap.htm

Bill Trochim is a faculty member at Cornell University; his work in applied social research andevaluation is described on this site. His published and unpublished papers, detailed examples ofcurrent research projects, useful tools for researchers, an extensive online textbook, a bulletinboard for discussions and links to other websites related to applied social research methods areincluded. Concept mapping is a general method that can be used to help individuals or groups todescribe their ideas about some topic in a pictorial form.

13. UNICEF

http://www.unicef.org/reseval/

This site lists some of the monitoring and evaluation tools recently developed by UNICEF andits partners, including the UNICEF Guide to Monitoring and Evaluation.

14. United Way

http://www.unitedway.org/outcomes/

The United Way’s Resource Network on Outcome Measurement offers a guide to resources formeasuring program outcomes for health, human service and youth- and family-serving agencies.Their manual, Measuring Program Outcomes: A Practical Approach, can be ordered at theWebsite.


15. National Science Foundation, Division of Research, Evaluation and Communication(REC)

http://www.nsf.gov/pubsys/ods/getpub.cfm?nsf97153

This site contains a complete manual, User-Friendly Handbook for Mixed Method Evaluations(August 1997), edited by Joy Frechtling and Laurie Sharp Westat, and developed with supportfrom the National Science Foundation, Division of Research, Evaluation and Communication.

Capacity Mapping and Performance Improvement Compared 91

Annex D Capacity Mapping and Performance Improvement Compared

Capacity Mapping Performance Improvement in RH

What is it? Tool for M&E planning (primarily) Tool for improving RH services

What is the purpose? Helps planners and evaluators decide: What M&E approach totake to determine whether this strategy succeeded in buildingcapacity (primary use)? What capacity-building strategy touse? (secondary use).

Helps managers decide: what PI strategy to use? Did perform-ance change as a result of the PI process?

Answers the question… What factors of capacity are required for performance? Howshould I measure these factors?

Is progress being made toward goals? Are appropriate actionsbeing undertaken to promote achieving those goals? What arethe problem areas?

What is the approach? Conceptual: Evaluators are encouraged to consider a widerange of factors that might influence capacity and performance.

Focused: Root causes of performance problems are linked tosix performance factors - job expectations; performance feed-back; workspace, equipment, and supplies; incentives; organ-izational support; and knowledge and skills.

Guides planners and evaluators in viewing capacity systemati-cally and identifying all areas that affect performance.

Guides organizations in viewing problems systematically andaddressing all areas that enhance performance.

Encourages understanding of capacity in the health sector as asystem that includes four interdependent levels: the system,organizations, health personnel, individuals and communities.

Encourages understanding of the organization as a system ofinterdependent functions and people.

When to use it? Can be used to organize and analyze information before orafter a capacity-building intervention is designed.

Used to organize and analyze information before deciding whatintervention is needed.

Focus of study/action Applies to systems, organizations, humans, and communities Applies to humans within organizational systems

Who is involved? Encourages stakeholder involvement Encourages stakeholder involvement

View of performance Performance is the result of capacity and context Human performance is a factor of knowledge, skills, capacityand motives, and context

Glossary 93

Glossary

Capacity is the ability to carry out stated objectives. It has also been described as the “stock ofresources” available to an organization or system as well as the actions that transform those re-sources into performance.

Capacity building (or capacity development) is a process that improves the ability ofa person, group, organization, or system to meet objectives or to perform better.

Capacity evaluation is normally more complex than monitoring, and is conducted to gain un-derstanding of the relationship between capacity-building interventions and capacity outcomes,or the links between capacity and performance variables.

Capacity mapping is a structured process of thinking through the role capacity plays in ensuringperformance by developing a conceptual framework that is specific to a particular capacity-building intervention. During capacity mapping, all the possible factors of capacity that influenceperformance and the relationships between them must be identified. Once the factors are all laidout, the program staff or evaluator can focus on those that are most essential for the evaluation.

Capacity monitoring normally would be used to understand the effectiveness and efficiency ofa capacity-building intervention during implementation (i.e., is capacity improving and at whatcost?) to contribute to strategic or operational decisions related to capacity building or enable aperiodic look at a program or system.

Cold chain: The system that ensures vaccine viability from manufacturing to delivery.

Contextual factors: external factors relating to the economic, social, cultural and political envi-ronment. Factors normally outside the control of most health sector actors.

Impact: Long-term results achieved through improved performance of the health system: sus-tainable health system and improved health status. Impact measures are not addressed in capac-ity-building M&E.

Impact evaluation: An evaluation that uses experimental or quasi-experimental study design toattribute changes in capacity or performance to program interventions. Impact evaluation is notappropriate or useful in the context of capacity-building M&E because of the difficulty of quanti-fying many elements of capacity and attributing capacity change to any single intervention oreven a range of them.

Input: Set of resources, including service personnel, financial resources, space, policy orienta-tion, and program service recipients, that are the raw materials that contribute to capacity at eachlevel (system, organization, health personnel, and individual/community).

Outcome: Set of results that represent capacity (an ability to carry out stated objectives), oftenexpected to change as a direct result of capacity-building intervention.

94

Output: Set of products anticipated through the execution of practices, activities, or functions.

Performance: Set of results that represent productivity and competence related to an establishedobjective, goal or standard. The four capacity levels together contribute to overall system-levelperformance.

Performance Improvement (PI): Performance Improvement (PI) is a process for enhancingemployee and organizational performance that employs an explicit set of methods and strategies.Results are achieved through a systematic process that considers the institutional context; de-scribes desired performance; identifies gaps between desired and actual performance; identifiesroot causes; selects, designs and implements interventions to fix the root causes; and measureschanges in performance. PI is a continuously evolving process that uses the results of monitoringand feedback to determine whether progress has been made and to plan and implement additionalappropriate changes.

Process: Set of activities, practices, or functions by which the resources are used in pursuit of theexpected results.

Theory of action: Part of a capacity-building plan that includes common objectives and sharedconcepts. A coherent theory of action agreed on by the key groups involved in the process stateshow activities are expected to produce intermediate and longer-term results and benefits. “With-out a theory of action, a capacity development effort could become a fragmented exercise inwishful thinking, rather than a coherent initiative with a high probability of success” (Horton,2001).

Triangulation: The use of multiple data sources or methods to validate findings, discover errorsor inconsistencies, and reduce bias.

Bibliography 95

Bibliography

Africa Bureau, Office of Sustainable Development USAID. 1999. Health and Family PlanningIndicators: Measuring Sustainability, Volume II. Washington: USAID.

Ampomah, K. 2000. PRIME’s Technical Report 20: An Assessment of the Impact of PRIME’sInterventions on the Training Capacity and Reproductive Health Service Delivery in Ghana.2000. Chapel Hill, NC: INTRAH.

Bertrand, J. and Escudaro, G. 2002. Compendium of Indicators for Evaluating ReproductiveHealth Programs, Volume 1. Chapel Hill: MEASURE Evaluation Project.

Brown, L., LaFond, A., Macintyre, K. 2001. Measuring Capacity Building. Chapel Hill:MEASURE Evaluation Project.

Catotti, D. 1999. PRIME’s Technical Report 13: Improving the Quality and Availability of Fam-ily Planning and Reproductive Health Services at the Primary Care Level: Institutional CapacityBuilding in the El Salvador Ministry of Health. Chapel Hill: INTRAH.

Development Resources Team, World Vision. 2002. Transformational Development IndicatorsField Guide. Washington, DC: World Vision.

Earl, S., Carden, F., and Smutylo, T. 2001. Outcome Mapping: Building Learning and Reflectioninto Development Programs. Ottawa: International Development Research Centre.

Eng, E. and Parker, E. 1994. Measuring Community Competence in the Mississippi Delta: TheInterface between Program Evaluation and Empowerment. Health Education Quarterly 21 (2):199-220.

Figueroa, M.E., Kincaid D.L., Pani, M. and Lewis, G. 2002. Communication for Social Change:An Integrated Model for Measuring the Process and Its Outcomes. Communication for SocialChange Working Paper Series, No. 1. Baltimore: Johns Hopkins Center for CommunicationsPrograms.

Fort, Alfredo. 1999. PRIME’s Technical Report 16: Capacity Building in Training: A Frame-work and Tool for Measuring Progress. Chapel Hill: INTRAH.

Franco, L.M., Bennett, S. and Kanfer, R. 2002. Health Sector Reform and Public Sector HealthWorker Motivation: A Conceptual Framework. Social Science and Medicine 54: 1255-1266.

Goodman, R.M., Speers, M.A., McLeroy, K., Fawcett, S., Kegler, M., Parker, E., et al. 1998.Identifying and Defining the Dimensions of Community Capacity to Provide a Basis for Meas-urement. Health Educ Behav 25 (3): 258-278.

Gubbles, P., Koss C. 2000. From the Roots Up: Strengthening Organizational Capacity throughGuided Self-Assessment. Oklahoma City: World Neighbors.

96

Horton, D. 2002. Capacity Development in Planning, Monitoring, and Evaluation: Results of anEvaluation. Briefing Paper No. 51. ISNAR.

Horton, D. 2002. Planning, Implementing and Evaluating Capacity Development. Briefing PaperNo. 50. ISNAR

Horton, D. (ed). 2001. Learning about Capacity Development through Evaluation Perspectivesand Observation from a Collaborative Network of National and International Organization andDonor Agencies. The Hague: International Service for National Agricultural Research.

Horton, D., Mackay, R., Andersen, A., and Dupleich, L. 2000. Evaluating Capacity Developmentin Planning, Monitoring, and Evaluation: A Case from Agricultural Research. Research Report17. The Hague: International Service for National Agricultural Research.

INTRAC. 1998. The Newsletter of the International NGO Training and Research Center. No. 11.

Israel, B., Schultz, A., Parket, E., and Becker, A. 1998. Review of Community-Based Research:Assessing Partnerships Approaches to Improve Public Health. Annual Review of Public Health19:173-202.

Israel, B.A., Checkoway, B., Schulz, A., Zimmerman, M., 1994. Health Education and Commu-nity Empowerment: Conceptualizing and Measuring Perceptions of Individual, Organizational,and Community Control. Health Education Quarterly 21(2): 149-170.

James, R. 2001. Practical Guidelines for the Monitoring and Evaluation of Capacity BuildingExperience from Africa. London: Intrac.

Kaul Shah, M., Degnan Kambou, S., Monahan, B. 1999. Embracing Participation in Develop-ment: Wisdom from the Field. Atlanta: CARE Health and Population Unit.

Knight, R.J., Tsui, A.O. 1997. Family Planning Sustainability at the Outcome and Program Lev-els: Constructing Indicators for USAID Strategic Planning. Chapel Hill: The EVALUATIONProject.

Kotellos, K.A., Amon J.J., Githens Benazerga, W.M. 1998. Field Experiences: Measuring Ca-pacity Building Efforts in HIV/AIDS Prevention Programs. AIDSCAP Family Health Interna-tional, AIDS 12 (suppl. 2): 109- S117.

LaFond, A. 1995. Sustaining Primary Health Care. London: Earthscan.

LaFond, A., Brown, L. and Macintyre, K. 2002. Mapping Capacity in the Health Sector: A Con-ceptual Framework. International Journal of Health Planning and Management. 17:3-22.

Bibliography 97

Lake, S., Daura, M., and Mabanddhala, M., et al. 2000. Analyzing the Process of HealthFinancing Reform in South Africa and Zambia. Zambia Country Reports. Major Applied Re-search Technical Paper 1. Bethesda: Partnerships for Health Reform Project.

Lande, R.E. 2002. Performance Improvement. Population Reports, Series J, No. 52, Baltimore:The Johns Hopkins Bloomberg School of Public Health, Population Information Program.

Luoma, M. 2000. PRIME’s Technical Report 19: Dominican Republic Performance Improve-ment Project Evaluation. Chapel Hill: INTRAH.

Lusthaus, C., Adrien, M., Andersen, G., and Carden, F. 1999. Enhancing Organizational Per-formance: A Toolbox for Self-Assessment. Ottawa: International Development Research Centre.

Mackay, R. and Horton, D. 2002. Capacity Development in Planning, Monitoring, and Evalua-tion: Results of an Evaluation. Briefing Paper No. 51. ISNAR.

Management Sciences for Health. 1996. Planning for Sustainability: Assessing the ManagementCapabilities of Your Organization. The Family Planning Manager. FPMD.

McCaffrey, J., Luoma, M., Newman, C., Rudy, S., Fort, A., Rosensweig, F. 1999. PI Stages,Steps and Tools, Chapel Hill: INTRAH.

MEASURE Evaluation. 1998. The Needs Assessment Validation Study and 1998 InstitutionalCapacity Assessment, PASCA Project. Chapel Hill: MEASURE Evaluation Project.

MEASURE Evaluation. 2001. Mapping Capacity in the Health Sector: Application of theMEASURE Conceptual Framework in Measuring Capacity Building for a Complex Non-governmental Organization. Draft.

Moore, M., Brown, L., and Honan, J. 2001. Toward a Public Value Framework for Accountabil-ity and Performance Management for International Non-Governmental Organizations. Presentedat Hauser Center/Keio University Workshop on Accountability for International Non-governmental Organizations, November 2 – 11, 2001.

Morgan, P. 1997. The Design and Use of Capacity Development Indicators. CIDA.

Murray, C.J.L., Frenk J. 1999. A WHO Framework for Health System Performance Assessment.Geneva: World Health Organization.

Oakley, P. 2001. Evaluating Empowerment: Reviewing the Concept and Practice. INTRACNGO Management and Policy Series No.13. London: INTRAC.

Partnerships for Health Reform. 1997. Measuring Results of Health Sector Reform for SystemPerformance: A Handbook of Indicators. Bethesda: Partnerships for Health Reform.

98

Plummer, J. 1999. Municipalities & Community Participation: A Sourcebook for CapacityBuilding. London and Sterling, VA: Earthscan Publications Ltd.

PRIME II. 2001. PRIME II Performance Monitoring Plan (PMP): General Guidelines for Moni-toring and Reporting on the 10 Key PMP Indicators. Version 1.01. Chapel Hill: INTRAH.

Pyle, D. and LaFond, A. 2001. Case Example: Measuring Capacity Building in Training -PRIME’s Evaluation, Documentation and Dissemination (EDD); draft.

Ross, J.A. and Mauldin, W.P. 1996. Family Planning Programs: Efforts and Results, 1972-1974.Studies in Family Planning 27 (3):137-147.

Sarriot, E. 2002a. The Child Survival Sustainability Assessment (CSSA), For SharedSustainability Evaluation Methodology in Child Survival Interventions. Maryland: CORE –CSTS.

Sarriot, Eric. 2002b. Sustaining Child Survival: Many Roads To Choose, but Do WeHave a Map? Maryland: ORC Macro.

Sullivan, T. and Bertrand, J. (eds). 2000. Monitoring Quality of Care in Family Planning by theQuick Investigation of Quality (QIQ): Country Reports. Technical Report Series, No. 5. ChapelHill: MEASURE Evaluation.

UNICEF. 1999. M&E of Capacity Building: Guidance and Tools. Working Draft.

World Health Organization. 2000.World Health Report 2000: Health Systems: Improving Per-formance. Geneva: World Health Organization.