Top Banner

of 215

Evaluating Development Programmes and Projects

Jun 01, 2018

Download

Documents

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • 8/9/2019 Evaluating Development Programmes and Projects

    1/215

  • 8/9/2019 Evaluating Development Programmes and Projects

    2/215

    EVALUATING D EVELOPMENTPROGRAMMES AND PROJECTS

  • 8/9/2019 Evaluating Development Programmes and Projects

    3/215

  • 8/9/2019 Evaluating Development Programmes and Projects

    4/215

    EVALUATING D EVELOPMENTPROGRAMMES AND PROJECTS

    SECOND EDITION

    Reidar Dale

    SAG E PUBLIC ATIONSNew Delhi Thousand Oaks London

  • 8/9/2019 Evaluating Development Programmes and Projects

    5/215

  • 8/9/2019 Evaluating Development Programmes and Projects

    6/215

    C ONTENTS

    List of Figures , Tables and Boxes 8 Foreword by Hellmut W. Eggers 9 Preface to the Second Edition 15 Preface to the First Edition 17

    P ART O NE : E VALUATION IN C ONTEXT

    1. GENERAL CONCEPTUAL AND ANALYTICAL

    FRAMEWORK 21Conceptualising Development and Development Work 21Evaluation in the Context of Development Work 24Underlying Conceptions of Rationality 26

    2. PURPOSES OF EVALUATION 31Changing Views on Evaluation 31Formative and Summative Evaluation 33Empowerment Evaluation 35Linking to Modes of Planning: Process Versus Blueprint 37Linking to the Concepts of Programme and Project 40

    3. EVALUATION VERSUS APPRAISAL AND MONITORING 44 Appraisal and Evaluation 44Monitoring and Evaluation 45

    A More Precise Definition of Evaluation 49

    4. LINKING TO PLANNING: MEANSENDS ANALYSIS 51Connecting Design Variables, Work Categories and

    Intended Achievements 51Linking Planned Outputs to Intended Benefits for People 53Two Examples 57

    5. LINKING TO PLANNING: SOCIETAL CONTEXT ANALYSIS 61

    The Basic Issue 61Opportunities, Constraints and Threats 62Moving Boundaries between the Internal and the External 65Stating Assumptions: One Example 68

  • 8/9/2019 Evaluating Development Programmes and Projects

    7/215

    6 Evaluating Development Programmes and Proj ects

    PART T WO : F OCUS , S COPE AND V ARIABLES OF E VALUATION

    06. A FRAMEWORK OF ANALYTICAL CATEGORIES 73The G eneral Perspective 73The Analytical Categories 76Some Examples 82

    07. ASSESSING ORGANISATIONAL ABILITY ANDPERFORMANCE 84Analysing Organisation in Programme/Project-Focused Evaluations 84Organisation-Focused Evaluations 86Main Organisational Variables 89

    08. EVALUATING CAPACITY-BUILDING 96The Concepts of Capacity-, Organisation- and Institution-Building 96Elaborating Organisation-Building 98Specific Concerns of Evaluation 101

    09. EVALUATING SOCIETAL CHANGE AND IMPACT 105H ighlighting Complexity and Uncertainty 105Analytical Scope: Societal Processes 107Assessing Empow erment 111

    PART T HREE : M ODES AND M EANS OF E VALUATION

    10. SCHEDULING OF EVALUATIONS ANDEVALUATION TASKS 117The Time Dimension in Evaluation 117

    A Range of Options 118Linking to Methods of G enerating Information 125

    11. GENERAL STUDY DESIGNS 127Sources of Information 127Qualitative and Quantitative Approaches 128Quantitative Designs 132Qualitative Designs 136

    12. METHODS OF INQUIRY 142Introduction 142An Overview of Methods 144A Further Elaboration of Some Topics 153A Note on Sampling 163

    13. ECONOMIC TOOLS OF ASSESSMENT 169Introduction 169BenefitCost Analysis 170Cost-Effectiveness Analysis 175

  • 8/9/2019 Evaluating Development Programmes and Projects

    8/215

    14. INDICATORS OF ACHIEVEMENT 177The Concept of Indicator and its Application 177Characteristics of G ood Indicators 181Examples of Indicators 183

    15. MANAGEMENT OF EVALUATIONS 187A Conventional Donor Approach 187Options for Improvement 190Two Alternative Scenarios 193The Evaluation Report 200

    References 207Index 211About the Author 214

    Contents 7

  • 8/9/2019 Evaluating Development Programmes and Projects

    9/215

    LIST OF FIGURES , T ABLES AND BOXES

    FIGURES

    02.1 Main Purposes of Evaluations 3404.1 MeansEnds Structure of H ealth Promotion Project 5804.2 MeansEnds Structure of Empowerment Programme 5906.1 Evaluation Perspective 1 7407.1 Evaluation Perspective 2 8507.2 Evaluation Perspective 3 8708.1 Evaluation Perspective 4 9909.1 Evaluation Perspective 5 11010.1 Steps, Timing and Time H orizons in Evaluation:

    Typical Scenarios 119

    TABLES

    05.1 Relations betw een M eansEnds C ategories and Assumptions 7006.1 Examples of Evaluation Variables 8311.1 Qualitative and Quantitative Approaches to Information

    G eneration and Management 130

    13.1 Income per Household from and BenefitCost Ratio ofAlternative Crops 171

    13.2 Earnings and Costs of an Income-G enerating Project 17213.3 Comparing the Profitability of Three Crops 17414.1 Assessed Quality of Selected Indicators 183

    BOXES

    02.1 Managing a Programme with Process Planning 3903.1 Two Cases of Participatory Assessment 4708.1 Building Local Community Organisations of Poor People 10309.1 A New View of Finance Programme Evaluation 10811.1 Seethas Story 13912.1 Example of Quest ionnaire Const ruct ion: Income Effect of Loans 15415.1 Evaluation Management : A Convent ional State Donor Approach 18715.2 Evaluation Management: An Action Research Scenario 19515.3 Evaluation Management: A Self-Assessment Scenario 19715.4 Structure of Evaluation Report on a C ommunity

    Institution-Building Programme 204

  • 8/9/2019 Evaluating Development Programmes and Projects

    10/215

    FOREWORD

    Evaluation is l a mode today. In public health policy, education,research and technology, criminal justice and, of course, develop-ment work, including international development cooperation, etc.,evaluation is playing an increasingly important role. Issues of evalu-ation methodology, execution and use have come under review bya growing number of evaluation societies that have sprung up inrecent decades all over the world, especially in the USA, Canada andEurope.

    Evaluation has, indeed, come a long way since its timid begin-nings.

    Let me cast a short look at my own evaluation history. In the latesixties I joined, as a recently appointed young official of the EuropeanC ommission, the staff of the Directorate G eneral for D evelopment ofwhat was then the European Economic Community (EEC) composedof the six founding nations (today, the European Union [EU] com-

    posed of 25 member states). I was responsible for agricultural devel-opment projects in Western Africa. It w as shortly after my appointmentthat we launched, as I still vividly recall, our very first evaluationmission. A colleague of mine, a professional architect, w as sent to oneof the countries of that region to ascertain what had become of aschool building project the European D evelopment Fund had financedthere a couple of years previously. I must have been much impressed

    otherwise I would have forgotten the whole story long agoby themain conclusion of my colleagues evaluation report: that the sizeof the ventilation openings under the roofs of the school buildings inquestion should have been of smaller dimensions, as the prevailingw inds at the beginning of the rainy season w ere liable to drive the raininto the schoolrooms.

    As I have said, evaluation of development cooperation has madesome progress since then: it went beyond the technical dimensionwhen discovering the economic one as well; it went on to include

  • 8/9/2019 Evaluating Development Programmes and Projects

    11/215

    10 Evaluating Development Programmes and Proj ects

    socio-cultural considerations; and finally, it included the evaluationfunction into project cycle management (PCM) as a whole, a meth-odological approach that the Evaluation Division of the DirectorateG eneral for D evelopment of the European Commission, of w hich I

    had become the head, conceived and introduced into operationalpractice in the mid-eighties.

    There seems to remain, however, one important (unresolved),problem area that has persisted into the twenty-first century: thepernicious tendency of evaluators of any school of thought, whatevermay be the subject matter of their research, in industrialised as wellas in developing countries, to keep themselves to themselves, that is,above all, out of the way of planners and implementers. Evaluatorsseem to be jealous of their particular field of competence and reluctantto recognise the almost inextricable links and relationships that (infact) bind them almost automatically to other disciplines. True, posi-tivism and the concern for objectivity have made ample room forconstructivist, post modern tendencies admitting to the obviousfact that the evaluator will necessarily be influenced by the evaluandas well as by the evaluated, and w ill thus be deprived of the virginityof independent judgement. There is even a school of thought that hasstriven to imbue with realism their approach to real-world problemsto such an extent that they call their method Realistic Evaluation.Yet, instead of concentrat ing on the projects and programmes asplanned by those in charge, they prefer re-inventing the programme

    theory so as to find out w hat w orks for w hom in w hat circumstances.Small wonder that evaluations have so often failed to make a differ-ence in real-world situations!

    I suspect, then, that my own view will not entirely coincide withthat of the majority of my fellow evaluators (I shall nonetheless dareto formulate it): the object ive of evaluators shoul d be ident ical w ith that of pl anners and implement ers: create sustainable benefits for t he

    target groups of projects, programmes and policies.This does not necessarily mean that evaluators should always goalong with the objectives pursued by planners, implementers andbeneficiaries, although, in the majority of projects and programmesthat will be the case. If not, then there should be a dialogue amongall of these parties in order to correct the objectives and as aconsequence to adapt the measures to obtain it. But whatever may

    be the circumstances of each individual case, the evaluator shouldalways be one of a team of actors working together in pursuit of a

  • 8/9/2019 Evaluating Development Programmes and Projects

    12/215

    Foreword 11

    common purpose. Planning, implementation and evaluation shouldbe part and parcel of one and the same project/programme cycle. Itis true that evaluators have long ago decided to leave Olympus. Itis now time for them to join the fray.

    I am not entirely sure whether or not Reidar Dale, the author ofthe book introduced by this foreword, shares in every respect theopinion voiced above. I am sure, however, that he would not evendream of separating planning from evaluation or evaluation fromplanning (as many evaluators tend to do).

    I have good reasons to believe that Dale favours such close amal-gamation: I was privileged to write the foreword not only for hispresent work on evaluation but also for his recently published bookD evelopment Planning: Concepts and Tool s for Planners, M anagers and Facilitators . I consider these two works twins, true companion volumes , and so, I am convinced, does he. I have yet to spot anotherauthor on evaluation who breaks more radically with the stand-alone stance on evaluation than Dale. And therefore I think that hisbook on the evaluation of development programmes and projects isthe most modern, the most up-to-date, that exists today. It appearsto me to be advancingin a unique way closely linked to the author ssolid professional backgroundthe practical science of evaluatingsocietal development w ork through its comprehensive coverage, holisticapproach, and by mirroring the hands-on wisdom that goes withyears of practical experience.

    Let us have a rapid look at the contents of that book to substantiatethis view:Right from the start, in Part One of the book, when setting out

    the general conceptual and ana lytical framework, D ale presents under-lying concepts of rationality that stick closely to real-life situations,echoing the resolutely action-oriented approach he follows in hisbook on planning. Linking evaluation to modes of planning and to

    the concepts of programme and project, Dale goes on to discussthe nature of evaluation as distinct from appraisal and monitoringand then turns to the dynamic aspects of means-ends analysis, show-ing how planning and evaluation are just different ways of lookingat the same issues. The first part of the book ends with an analysisof the societal context, and I am convinced that even the stoutestrealists among evaluators will admit that the author is thoroughly

    aware of the fact that benefit creation for intended beneficiarieswill only materialise when closely moulding applied mechanisms to

  • 8/9/2019 Evaluating Development Programmes and Projects

    13/215

    12 Evaluating Development Programmes and Proj ects

    the specific external, o r contextual, project/programme environmentbeing analysed, w hich w ill, of course, be found to be different in eachspecific case.

    In Part Tw o of the book, Dale sets out the general framework

    made up of the classical categories of relevance, impact, efficiency,effectiveness, sustainability and replicability, and then turns toorganisational questions and issues of capacity-building. I have notedwith particular interest an aspect that struck me also at the time ofreading the draft of Dales book on planning: the importance heplaces on people, and the way they can best cooperate for develop-ment through adequate organisation. Capacity-building is a keyto achieving this, being true for the individual and the organisations/he belongs to . Again, one is struck by the way D ale is able toamalgamate, on the basis of his ample practical experience, theoryand practice, especially, when he highlights the enormous complexity,and therefore the bewildering uncertainty, that besets all societalprocesses. The author is modest enough to admit his own limitations,and those that any professional approach to planning or to evaluationwill encounter when faced with the unfathomable intricacies andinevitable surprises, both good and bad, that real life always has instore.

    In Part Three, Dale turns to the nitty-gritties of the evaluatorscraftsmanship. True, the limitations just mentioned exist, but thereis so much one can do in spite of these limitations if one has learned

    to apply the numerous instruments at the evaluators disposal thatwill, when correctly handled, lead to worthwhile results in termsof improved development results. Correct timing of evaluations,methods of data gathering, the optimal mix between quantitativeand qualitative design according to each concrete case, economictools, the choice of indicators of achievement, are all considered.The authors resolute orientation towards the solution of real-life

    problems seems to make him almost automatically wary of an ex-aggerated faith in quantitative or quantifiable judgements and muchmore open to qualitative inquiry and modes of evaluation. He insistson the advantages of managing the evaluation process bottom-up withand for the people concerned, while not ruling out entirely theconventional top-down donor approach, and he emphasises the needfor practical feedback of evaluation results allow ing for empow ering

    learning for all stakeholders and creating that we-can-do-it feelingso vital for lasting success. This is the philosophy that the reader

  • 8/9/2019 Evaluating Development Programmes and Projects

    14/215

    Foreword 13

    will have absorbed, or so I hope, after reading Reidar Dales bookon evaluation of development programmes and projects.

    What more is there to say? That this book will grow on the readeras s/he follow s Dales voyage from thought to action in a convincingly

    structured presentation? That Dale avoids jargon and sticks to acomprehensive, unpretentious, clear and reader-friendly style? Thatthe reader will feel more and more involved, as each argumentunfolds and gives rise to the following one, in what comes close toa dialogue with the author on each of the issues addressed? I have,however, already said all of this when writing the foreword tothe twin of the present publication: Dales recently publishedbook on planning. These statements are certainly true for this bookas well (and so it is worthwhile repeating them). I have also appre-ciated the comprehensive use of case material and various otherillustrations Dale uses to connect the issues discussed to real-lifecontexts.

    I would be happy to see this book in the hands of:

    teachers and students of development-focused academic coursesin universities;

    trainers and participants in other courses on development plan-ning and evaluation;

    development cooperation policy makers, planners and evalua-tors in governmental and non-governmental organisations, in

    both less developed and highly developed countries; international and national agencies involved in developmentcooperation, including UN agencies and the World Bank, na-tional governmental development agencies and NG Os.

    As my concluding remark, I think the best I can do is to repeat w hatI had concluded in my foreword to Dales book on planning:

    I have read the draft of Reidar Dales book on evaluation withgrowing interest. I have been privileged to be of assistance byproviding him with a certain number of observations, which my ownpractical experience during four decades in the area of internationaldevelopment cooperation has led me to formulate. I warmly recom-mend this work to all those who, like myself, believe that theintegration of planning, implementation and evaluation into a con-

    sistent approach to project/programme cycle management holds thepromise to lead to lasting improvements in the quality of organised

  • 8/9/2019 Evaluating Development Programmes and Projects

    15/215

    14 Evaluating Development Programmes and Proj ects

    development work, public and private, including international devel-opment cooperation.

    Brussels Hellmut W. Eggers2004 Formerly H ead

    Evaulation DivisionD irectorate G eneral for D evelopment

    of the European CommissionEmail: H ellmut .Eggers@skynet .be

  • 8/9/2019 Evaluating Development Programmes and Projects

    16/215

    PREFACE TO THE SECOND EDITION

    In 1998 I wrote Evaluation Frameworks for D evelopment Programmes

    and Projects published by SAG E. That book has sold w ell. The presentbook Evaluating D evelopment Programmes and Projects may be view edas a second edition of the previous one. It has been thoroughlyreworked, though, and is a largely new book. Still, its aims, focusand target groups remain basically the same, and the Preface that Ihad written for the first book is generally applicable to this one as

    well.

    July 2004 Reidar DaleEmail: reidar _ [email protected]

  • 8/9/2019 Evaluating Development Programmes and Projects

    17/215

  • 8/9/2019 Evaluating Development Programmes and Projects

    18/215

    PREFACE TO THE FIRST EDITION

    Evaluation is a broad and many-faceted concept. The literature onevaluation is equally diverse. It ranges from presentations of generalideas and perspectives, via prescriptive frameworks for practitioners,to advanced texts on methodology.

    This book deals with evaluation of organised development work ,referred to as development programmes and projects. Its frame ofreference is such w ork in developing countries; how ever, the concep-tual and methodological framework and most of the more specificideas and arguments are relevant more widely.

    The book covers the major topics of context , perspectives, methods and management of evaluations, and I have endeavoured to connectthese in a systematic, stepwise argument. To this end, I start out w itha brief account of main perspectives and types of planning (consti-tuting the basis of any organised development work); connect plan-ning, implementation and monitoring to evaluation; seek to clarify

    purposes and aims of evaluations; incorporate and discuss factors oforganisation and management; present a range of methods which maybe used in evaluations; and briefly address presentation and use ofevaluation results.

    This int erconnected analysis of the main facets of evaluation dis-tinguishes the book from most other evaluation literature.

    The book also differs from most literature on evaluation of devel-

    opment work in that the scope of analysis is extended from projects to encompass different kinds of programmes as well. In doing so, Ibuild on the conventional project matrix (the logical framework),but break its limitations by taking cognisance of the complex andprocessual nature of societal development and corresponding per-spectives on development work.

    I have conscientiously tried to make the book easy to comprehendfor a diverse group of potential readers. To this end, a highly scientificterminology has been avoided; terms that require clarification are

  • 8/9/2019 Evaluating Development Programmes and Projects

    19/215

    18 Evaluating Development Programmes and Proj ects

    explained in simple language; issues are exemplified in the text; andthe text is supplemented with numerous diagrams and textboxes, thelatter mainly containing more detailed examples.

    In order to facilitate a broad presentation and to maintain focus

    on interconnections between topics and factors, I do not explorespecific issues in great detail. The reader is, instead guided to morespecialised literature in various fields, through references in the maintext and in footnotes and through the Bibliography presented at theend of the book. This limitation also keeps the size of the book downto a number of pages that most readers should find appealing.

    While I hope that the book will attract a broad readership , theprimary target group is practitioners in development work. Theseinclude, of course, people who conduct evaluations, but in additiona range of planners, managers and administrators who ought to befamiliar with concepts and tools of evaluation. By directly linkingevaluation with planning and organisation, I hope to promote evalu-ation as a more integral part of development work than it has tendedto be seen and, consequently, as a concern for whole developmentorganisations.

    The book should also be of interest to teaching institutions andstudents in development planning and management as w ell as in otherfields of organisation and social science. Students may also find ituseful for its overview of concepts and perspectives of developmentwork and for its discussion of methods of study, which have relevance

    well beyond the field of evaluation.

    1998 Reidar Dale

  • 8/9/2019 Evaluating Development Programmes and Projects

    20/215

  • 8/9/2019 Evaluating Development Programmes and Projects

    21/215

    P ART O NE

    EVALUATION IN CONTEXT

  • 8/9/2019 Evaluating Development Programmes and Projects

    22/215

    Chapter 1

    GENERAL C ONCEPTUAL ANDANALYTICAL FRAMEWORK

    CONCEPTUALISING DEVELOPMENT ANDDEVELOPMENT WORK

    The word development may be used in numerous contexts. In allthese contexts, it denotes some kind of change. It may also expressa state that has been attained through some noticeable change, forinstance, in the construct level of development. Beyond that, theterm carries a variety of more specific meanings. It may be utilisednormatively (expressing a desirable process or state) or empirically(expressing a substantiated process or state without any explicit valueconnotation).

    In the present book, the concept is used normatively in the contextof human societ ies . In this sense, developmentwhich we may alsorefer to as societal developmentis conceived by people with ref-erence to something that they perceive as valuable. The underlyingvalues may be more or less overt or hidden, and they may reflectdifferent kinds of rationality (see later in this chapter). Thus, mostbriefly stated, development is viewed as a process of societal change

    that generates some perceived benefits for people, or as a state ofperceived quality of life attained through such a process (see alsoEggers, 2000a; 2000b; 2002; Dale, 2000).

    Moreover, when analysing development, we should normally beable to clarify who benefits from societal changes and, in cases ofdiverging opinions about benefits, who considers changes to be ben-eficial and who does not.

    Development being a people-focused concept, its contents in spe-cific situations must be clarified in relation to people-related problems .

  • 8/9/2019 Evaluating Development Programmes and Projects

    23/215

    22 Evaluat ing D evelopment Programmes and Proj ects

    We may broadly refer to human problems as poverty and depriva-tion (C hambers, 1983; 1995; D ale, 2000). Poverty is the more overtand specific concept of the two. It is usually taken to mean few andsimple assets, low income and low consumption, and sometimes also

    an unreliable supply of cash and food . D eprivation frequently includespoverty (in the mentioned sense), but may encompass several otherfeatures of human misery or suffering. Thus, Chambers (1983) con-siders deprived households as having the following typical features, invarying combinations: poverty, physical weakness (in terms of thestructure of the households and abilities of household members),isolation (physically and information-wise), vulnerability (little resis-tance to unexpected or occasionally occurring events), and powerless-ness. According to Chambers, many such features tend to be interrelated,locking households into w hat he calls a deprivation trap. D ale (2000)presents a somewhat more detailed typology of deprivation, in whichindividual-related problems (such as mental suffering, poor health andilliteracy) are also more explicitly emphasised.

    Normally, deprivations beyond overt manifestations of poverty aredifficult or even impossible to express by any objective measure;they will, therefore, largely have to be assessed through subjectivejudgement. Of course, this also applies to changes on the respectivedimensions.

    Connecting to such diverse manifestations of human deprivation,Dale (2000) suggests a general typology of dimensions of develop-

    ment. The dimensions briefly summarised, are:

    economic features: income and income-related characteristics, expressed throughphenomena such as G ross D omestic Product (G D P) per capita,income distribution, rate of employment, etc., at the macro orgroup level; and income, consumer assets, production assets, etc.,

    at the level of the household or, less frequently, the individual; social features:

    various aspects of social w ell-being, expressed through phenom-ena such as life-expectancy at birth, child mortality, schoolenrolment, etc., at the macro or group level; and health, levelof literacy, social security, etc., at the level of the household orthe individual;

    dependent versus independent posi t ion: degree of bondage or, oppositely, freedom in making ones own

  • 8/9/2019 Evaluating Development Programmes and Projects

    24/215

    General Conceptual and Analyt ical Framework 23

    choices about aspects of ones life or lives, expressed throughfeatures such as degree and terms of indebtedness, degree ofcompetition for scarce resources, degree of inequality or equalityof gender relations, etc.;

    marginal ised versus int egrated posi t ion: degree and terms of participation in markets, politics and sociallife, and type and strength of economic and social securitynetworks;

    degree of freedom from violence: the extent to which individuals and groups may lead their liveswithout deliberate mental or physical maltreatment or fear ofsuch maltreatment, within the family and in the wider society;

    degree of ment al sat isfact ion: degree of mental peace, and the ability to enrich ones lifethrough intangible stimuli;

    survival versus development -orient ed at t i tude: perception of ones position, abilities and opportunities in thesociety, at the level of the social group, household or individual.

    Obviously, these dimensions or indicators may be expressed or usedmore or less normatively or empirically. 1

    All the mentioned dimensions of development may be addressed inplanned development work of various kinds, referred to as develop-ment programmes and projects (to be better clarified in Chapter 2).

    The presented framework should also have substantiated that de-velopment is characterised by unique features in specific settings,great diversity and high complexity. Additionally, processes of devel-opment are often little predictable and may be changing, for a varietyof reasons. This pulls in the direction of primarily qualitative meansof analysing development, and poses strict limitations to the possi-bility of directly replicating development interventions.

    In addition to features under the above-mentioned categories, manyaspects of institutions and infrastructure may be viewed as develop-ment variables of a causal kind. Examples of the former may be thepossibility for people to form their own organisations and the effec-tiveness of public administration. Examples of the latter may be roaddensity and physical standards of various facilities. Of course, devel-opment programmes and projects commonly address such aspects.

    1 See Dale, 2004 (Chapter 1) for a further elaboration of this point.

  • 8/9/2019 Evaluating Development Programmes and Projects

    25/215

    24 Evaluat ing D evelopment Programmes and Proj ects

    The above has direct implications for the evaluation of develop-ment programmes and projects, in terms of the evaluations overallpurpose, focus, scope and methodology.

    EVALUATION IN THE CONTEXT OF DEVELOPMENT WORK

    Literally, evaluation means assessing the value of . Evaluations areundertaken in all spheres of life, informally or more formally, when-ever one wants to know and understand the consequences of someevent or action. Sometimes, one may w ant to better know how and w hy

    things have happened or are happening, or how and why things havebeen done or are being done. These general purposes are often moreor less related. Also, one usually uses the know ledge and understandingacquired from an evaluation to perform a similar activity better, or toplan some related action, in the future. In o ther instances, the emphasismay be more on generating effective responses to events or changesthat are beyond ones control. In either case, one tries to learn from

    ones assessment in order to improve ones performance.Our concern is evaluation of planned and organised development

    work.With planned w ork w e mean activities that have been premeditated

    and for which some guide has been worked out in advance. Inprogrammes and projects, this usually also means that one or moreplan documents have been worked out. As we shall clarify later, there

    may be various kinds of plans, and they may have been preparedinitially or during the course of the programme or project period.

    In systematically premeditated (i.e., planned) development work,it is common to distinguish between the main activity categories ofplanning, implementation and monitoring. Very briefly stated, plan-ning is a process of problem analysis, assessment of possible measuresto address clarified problems, and decision-making for problem-focused action. Implementation is the process of transforming allo-cated resources into outputs, considered necessary to attain theobjectives of the particular development thrust. Monitoring is acontinuous or frequent (usually regular) assessment of implementa-tion and its outcomes. These basic activities of problem- and solution-focused analysis, decision-taking, action and reflection are undertakenin all sensible development endeavourseven if they may at timesbe done too informally for the thrusts to be called programmes orprojects.

  • 8/9/2019 Evaluating Development Programmes and Projects

    26/215

    General Conceptual and Analyt ical Framework 25

    Evaluation is closely related to monitoring. Sometimes, it mayeven be difficult to distinguish between them and to decide which ofthe two terms one should use in particular cases of performanceassessment. In other cases, the distinction is clearer. In other words,

    these work categories may be partly overlapping, but are not identical.We shall soon discuss this matter more comprehensively and clarifydifferences better. For now, let us just say that an evaluation tends tobe a less frequent, more time-bound and more comprehensive effortthan monitoring, commonly a lso focusing more explicitly on benefitsto people and fulfilment of objectives of the evaluated scheme.

    A more specific activity category may be appraisal, and develop-ment work may also involve operation and maintenance of createdfacilities. We shall soon distinguish appraisal from evaluation, and w eshall later see how evaluations may connect to operation and main-tenance.

    Development programmes and projects are also organised thrusts.That is, the activities are undertaken by some organisat ion or organisa-tionswhich may be more or less formalisedand are intended to beperformed in accordance with certain organisation-based principles,rules and systems. This has two main implications for evaluations:these will then also be organised thrusts, and aspects of organisationof the evaluated scheme will (or should) normally be subjected toassessment.

    As already stated, development programmes and projects are (or

    should be) unambiguously people-focused , in the sense of promotingimprovements in some aspects of some peoples quality of life. Evalu-ations tend to be seen as the main tool for ascertaining such improve-ments for intended beneficiaries. We shall, in fact, consider consequencesof work that is done for people (intended beneficiaries and sometimesothers) to be the overriding concern of evaluat ions in the developmentrealm. Simultaneously, this may be linked to assessment of more

    specific and even additional aspects of programmes and projects, asshall be duly clarified later in the book.

    For analysing changes relating to people, one needs to view theprogramme or project under study in a broad societal context . Thatis, one cannot limit oneself to aspects internal to the developmentscheme, but must also address relations between the programme orproject and numerous aspects of its surroundings (environment).

    C ommonly, the interface betw een the scheme and its environment maynot be any clear-cut dividing line, but a porous and often changing

  • 8/9/2019 Evaluating Development Programmes and Projects

    27/215

    26 Evaluat ing D evelopment Programmes and Proj ects

    boundarya fact that may make such contextual analysis highlychallenging.

    In consequence of the above, thorough evaluations tend to berelatively comprehensive , in the sense that they cover many topics and

    issues. This may not allow for deep analysis of single factors. How-ever, for explanation of findings, specific economic, technical, cul-tural, institutional and other factors may have to be analysed, someeven in great detail.

    Evaluation of development work may be undertaken during theimplementation of a programme or project or after it has beencompleted. This will depend on the overriding purpose of the exer-cise, but it may be influenced by other factors as well. In terms ofoverall purpose, we shall soon make a broad distinction betweenformative and summative evaluation.

    One may, in principle, use a wide range of methods of evaluation,from the most rigorous statistical ones to purely qualitative assess-ments and personal interpretations. The range of applicable methodsdepends in practice on the nature of the development programmeor project to be evaluated, the issues that are being analysed, theevaluator and numerous features and conditions in the surroundingsociety. Usually, the focus on people and the complex and changingnature of societies makes most statistical methods little applicable.Thus, a point that we have already alluded to and shall substantiatelater (particularly in Part Three), insightful and informative evalua-

    tions of development work will normally require predominantlyqualitative analysis, possibly supplemented with some quantitativedata that may exist or may be generated.

    UNDERLYING CONCEPTIONS OF RATIONALITY

    Normally, approaches to evaluation w ill be influenced by the ideology(the general logic) underpinning the development thrust to be as-sessed, as perceived by those involved in, or influenced by, tha t thrust.A scientific word for this perceived logical foundation is rationality.

    All development work should be rational, according to the normalunderstanding and usage of the word. However, one ought to distin-guish between various notions of rationality. In development work,this means that one should consider other conceptions of the termthan the most conventional one, that of instrumental or technical rationality. In its pure form, instrumental rationality is normally

  • 8/9/2019 Evaluating Development Programmes and Projects

    28/215

    General Conceptual and Analyt ical Framework 27

    understood as exclusive attention to and specification of means andways of attaining predetermined goals.

    With reference to Jurgen H abermas w ell know n and w idely quotedtypology of modes of reasoning (H abermas, 1975; 1984), 2 other main

    forms of rationality may be termed value o r substantive rationalityand lifeworld or communicative rationality respectively (Dixonand Sindall, 1994; Servaes, 1996; H ealey, 1997).

    In the perspective of development work, value rationality refers tothe values that are embedded in possible or intended achievementsof a development scheme, as perceived by the stakeholders in thatscheme. Thus, it relates to the top levels of a meansends structureof the scheme. It is incorporated in what we may refer to as nor-mative planning, being the logical opposite of functional planning. 3

    Lifeworld rationality is the kind of reason that tends to be appliedin day-to-day communication between people. It is contained in theworlds of taken for granted meanings and understandings, of every-day living and interaction, of emotions, traditions, myth, art, sexu-ality, religion, culture (D ixon and Sindall, 1994: 303). In suchreasoning, w e frequently do not separate facts from values ,[and] the whole process of reasoning and the giving of reasons, whatwe think is important and how we think we should express this andvalidate our reasoning claims [are] grounded in our cultural concep-tions of ourselves and our worlds (Healey, 1997: 51).

    The rationality or combinat ion of ra tiona lities that underpins deve-

    lopment work will directly influence the objectives of that work andtherewith also various related design features, such as the types andmagnitude of resources provided and different features of theorganisation of the programme or project.

    In development programmes and projects, the particular blendingof rationalities may materialise through a combination of factors, suchas: general political and administrative culture of the society; estab-

    lished policy of a donor agency; ideas of individuals involved inplanning and management; and the constellation of actors in ideageneration, plan formulation and decision-making.

    2 H abermas refers to these modes as instrumentaltechnical, mora l and emotiveaesthetic reasoning respectively.

    3 We shall address meansends structures and briefly clarify the concepts ofnormative and functional planning in Chapter 4. For comprehensive analysis, seeDale, 2004.

  • 8/9/2019 Evaluating Development Programmes and Projects

    29/215

    28 Evaluat ing D evelopment Programmes and Proj ects

    Normally, formulation of meansends structures (being a coreexercise in much planning) is easiest when the components may bespecified through basically instrumental reasoning. Equally normally,this is the case ( a ) when a clear and unambiguous goal has been set

    in advance, or such a goal is just assumed or at least not substantiallyexplored by the planners, and ( b ) when the planning is done byprofessionals in the field. This is the situation in much planning.For instance, the district unit of a national roads department willnormally not explore the needs of people and the usefulness forpeople of road building versus other development interventions. Theonly goal-related question it may normally address in planning isprioritisation betw een roads. The unit w ill mostly concentrate on how to build (or maintain) the respective roads and the resource inputsthat are needed for construction (or maintenance). Intended benefitsfor people are then wholly or largely assumed (rather than analysed)in planning.

    Evaluations may, o f course, still explore such benefits and may evenrelate them to assumptions about benefits that have been made, tothe extent the latter may be reasonably well clarified.

    However, most of what we refer to as development work embedsgoal analysisto a larger or lesser extent, and in some form or an-other. One example of schemes in which goal analysis is fundamentaland needs concerted and even continuous attention is relatively open-ended community development thrusts, depending on priorities and

    views of local stakeholders. In such cases, value rationality assumesprime importance. 4

    Consequently, most evaluations of schemes of this nature shouldalso pay prime attention to aspects of benefit generation.

    Value rationality tends to co-exist w ith lifeworld rationality: beyondthe most basic physical needs, wants and priorities are subjectivelyexperienced and culturally influenced, also meaning that reasoning

    about them will be lifew orld-based. C losely connected, the more inter-ested parties (stakeholders) there are, the more is the tendency forperceived needs and interests to differ. And, the more the respectivestakeholders are involved in assessing needs and interests, the moredifficult it may be to agree on the relative importance of the latter.The same argument goes for agreeing on objectives of corresponding

    4 Dale (2004) even defines development planning as an exercise that explicitlyinvolves goal analysis.

  • 8/9/2019 Evaluating Development Programmes and Projects

    30/215

    General Conceptual and Analyt ical Framework 29

    development schemes as well as ways and means of a tta ining w hateverobjectives may have been specified.

    G iven the need that w e have alluded to for clarifying meansendsrelations in development work (a point that will be further substan-

    tiated in Chapter 4), the question arises whether highly participatorydevelopment work may be at all rational. Dale (2004) argues that itmay and should be: if one has any purposeful organised thrust inmind, even many and diverse stakeholders must normally be assumedto share at least basic ideas about what should be achieved and whatneeds to be done to that end. Although they may be general, suchideas will usually represent important guidelines for ones work.Commonly, they may also constitute a sufficient rationale for initialallocation of at least some resources and for other initial commit-ments.

    Simultaneously, through iterative processes of purposeful thinking,decision-taking, action and reflectiontypical of schemes of thiskindconceptions of priorities, feasibilities, how to do things, etc.,may change. Decisions and actions at any time should still be basedon the participants best judgement at those points in time; that is,they should still be sound in the context of existing perceptions.Moreover, through a process of capacity buildingbeing normally amain thrust of such programmesthe judgements may become morefirmly based and sound over time. The perceptional framework forrational choice and action will then have changed.

    In general conclusion, our basic argument is that lifeworld ration-ality is nothing more mysterious than some combination of value andinstrumental rationality under perceptional frameworks that maydiffer from those of professionals (and normally expected to be usedin conventional planning contexts). Simultaneously, that distinctionis, in itself, a fundamental recognition in development practice.

    C onnecting to some reflections w e have already made on this, there

    are four general and interrelated im pli cat ions for evaluations of thequestion of rationality:

    by d irectly influencing the objectives of programmes and projects,the underlying rationality (or combination of rationalities) willalso influence the focus and scope of evaluations of the particu-lar schemes;

    a need exists for the evaluator to understand the perceived logic(the rationality) behind priorities and actions of planners and

  • 8/9/2019 Evaluating Development Programmes and Projects

    31/215

    30 Evaluat ing D evelopment Programmes and Proj ects

    managers, in order to make a relevant and fair judgement ofprogrammes and projects that they have planned and a re respon-sible for;

    different rationalities on the part of programme and project

    stakeholders may be a cause of collaboration problems or areason for less than intended benefits, and may therefore haveto be comprehensively explored; and

    the evaluators own ideas of what is important and what is lessso in development work may, if the evaluator is not very con-scious about it, influence the emphasis that he or she places ondifferent programme or project matters (i.e., what the evaluatoractually assesses or assesses thoroughly).

    Relations exist or may exist between rationality and main analytical categories of evaluat ion, to be addressed in Part Tw o. We shall exploresuch relations further in that context.

  • 8/9/2019 Evaluating Development Programmes and Projects

    32/215

    Chapter 2

    PURPOSES OF EVALUATION

    CHANGING VIEWS ON EVALUATION

    Systematic evaluation of development programmes and projects wasconceived and formalised from the 1960s onw ards by donor agencies,which wanted a true account of how well things had been done andw hat had been achieved. The idea w as to have objective assessments,done by independent persons using professional methods of datacollection and analysis (Mikkelsen, 1995; Rubin, 1995). These per-spectives were primarily promoted by engineers and economists,being the main professionals in donor agencies at these early stagesof development co-operation. But they were also shared by manysocial scientists, reflecting a general emphasis in most social sciences

    at that time on obtaining what was considered as objectively verifi-able knowledge and on concomitant quantification and statisticalanalysis.

    Many evaluations are still done largely according to these prin-ciples. But alternative approaches have emerged. This is to a largemeasure due to experiences gained by development o rganisat ions, butthe changes have also become more sanctioned by a new orientation

    of much social science. The core factor of this redirection has beenan increased recognition that societal changes tend to occur throughprocesses that are so complex that only fragments of a relevant andsignificant reality may be investigated objectively through standardisedprocedures. This calls for more case-specific, inductive and flexibleevaluation designs. In addition, purposes of evaluations have becomemore varied, with significant implications for approach.

    In general concord with Rubin (1995), we can clarify the mainfeatures of this revised perspective in somewhat more specific terms:

  • 8/9/2019 Evaluating Development Programmes and Projects

    33/215

    32 Evaluating Development Programmes and Projects

    a recognition of subjectivity in evaluation, i.e., that one canrarely give one true account of what has happened and why;

    a recognition that interpretations of various involved or affectedgroups of people (stakeholders) may differ, and a view that all

    these should be recorded and understood, as far as possible; more emphasis on using evaluations to provide feedback of

    knowledge and ideas to further planning and implementation ofthe studied schemes, not just to sum up and explain theirperformance and achievements;

    an understanding that evaluations may be empow ering pro-cesses, by which intended beneficiaries (and sometimes others)can themselves learn to analyse changes affecting their lives andtherewith propose and sometimes even implement corrective,additional or new measures;

    a recognition that quantitative measures rarely provide muchinsight into complex causeeffect relations and structures, andthat qualitative description and reflective verbal analysis arecrucial for understanding processes of change;

    a tendency to use less rigid and standardised methods of inquiry,such as semi-structured interviews and participatory assessment.

    Many of these points tend to be more or less related. In particular,the last-mentioned (use of less standardised methods) may be a conse-quence of one of more preceding points. Thus, it may be linked tothe recognised need fo r getting the interpretations of different people;it may be motivated by a wish to have beneficiaries participate inevaluations (which requires use of methods of analysis that suchpeople can understand and work with); and it may be because onefinds it impossible to adequately describe and explain complex so-cietal relations and processes by quantitative measures.

    Moreover, some authors 1 have argued for and even noted a ten-dency in many evaluations towards:

    more emphasis on programme and project organisation andmanagement.

    This has methodological implications as well, in the general directionmentioned above: most analysis of organisational performance has to

    1 See, for instance: Oakley et al., 1991; Dehar et al., 1993; Knox and Hughes,1994; and Dixon, 1995.

  • 8/9/2019 Evaluating Development Programmes and Projects

    34/215

    Purposes of Evaluat ion 33

    be basically qualitative, in order to properly account for and under-stand the complex relations and processes that are typical in organiseddevelopment work.

    Stronger emphasis on organisation and management may also be

    connected to a more learning orientated approach, as generallymentioned above, and may sometimes be more specifically based on:

    a w ish to use evaluations as learning exercises for responsibleand executing bodies, through their own analysis or at leastsubstantial involvement by such bodies in collaborative analysis.

    As will be further discussed in Chapter 3, such generative and parti-cipatory modes of assessment may lie in the borderzone betweenmonitoring and evaluation, and may often be referred to by eitherterm.

    FORMATIVE AND SUMMATIVE EVALUATION

    Depending on how the provided information is to be used, or pri-marily so, we may make a broad distinction between two main typesof evaluation: formative and summative .2

    Features and purposes of the two types are shown in Figure 2.1.Formative evaluations aim at improving the performance of the

    programmes or projects that are evaluated, through learning fromexperiences gained. For most programmes, the scope for evaluationsto induce changes of design and implementation may be substantial,since programmes tend to have a framework character and a greateror lesser amount of flexibility built into them. For typical projects,meaningful formative evaluations may usually only be done if theschemes are broken down into phases, each of which are precededby planning events, in which information that is generated throughassessment of previous phases may be used. 3

    Formative evaluations are commonly done more than once for apart icular scheme. They may then recurrently address the same matters

    2 These terms are also found in some other literature on research and evaluation,although they may not have been systematically paired for analytical purposes theway I do it here. See, for instance: Rubin, 1995; Dehar et al., 1993; and Rossi andFreeman, 1993.

    3 The concepts of programme and project in the development field will befurther clarified later in this chapter.

  • 8/9/2019 Evaluating Development Programmes and Projects

    35/215

    34 Evaluating Development Programmes and Projects

    and issues or different ones. Commonly, each exercise may be fairly

    limited. The evaluations may be done at set intervals or according toneeds, as assessed by the responsible agencies, in the course of pro-gramme or project implementation. They may be managed internallyor externally, or through some combination of internal and externalinvolvement.

    Summative evaluations are undertaken after the respective devel-opment schemes have been completed. Their general purpose is to

    judge the worth of programmes or projects as well as, normally, theirdesign and management. The findings may be used for learning in theplanning and implementation of other, similar, developmentendeavours. Commonly, however, the more immediate concern is toassess the accountability of programme or project responsible bodiesand/or funding agencies. In practice, summative evaluations havelargely been triggered by a need among foreign donor agencies to

    prove their accountability vis--vis their governments and/or othermoney providers as well as the general public in the donor country.

  • 8/9/2019 Evaluating Development Programmes and Projects

    36/215

    Purposes of Evaluat ion 35

    For this reason, summative evaluations have tended to be undertakenby persons who are considered to be independent of the responsibleprogramme or project organisations and the donor agencies.

    As briefly mentioned above, evaluations may also be conducted

    halfway through programmes and projects (commonly called mid-term evaluations) or betw een phases of them. While the main purposeof evaluations thus timed is usually to provide information for anyfuture ad justments of the same schemes, accountability considerationsmay be important here also.

    Mid-term and formative evaluations are sometimes referred to asreviews.

    There are direct relations between the purpose (or combination ofpurposes) of evaluations and the way they may be conducted. We shallexplore aspects of evaluation methodology in Part Three.

    EMPOWERMENT EVALUATION

    As we have already alluded to, evaluations can be empowering pro-cesses for those who undertake or participate in them. This idea hasgot expression through the term empow erment evaluation (Fettermanet al., 1996).

    Empowerment evaluations may mainly be done in the context ofcapacity building programmes and other programmes that emphasiseaugmentation of abilities of intended beneficiaries. They may also be

    done more generally of the performance of o rganisat ions, emphasisinglearning by members or employees. Commonly, in such evaluations,assessment of organisational and programme performance will beclosely intertwined.

    The evaluato rs may assess activities that are at least largely plannedand implemented by themselves, through their ow n organisat ions. Wemay then refer to the assessments as internal evaluations. In suchcontexts, the feedback that the involved people get by assessing theperformance and impact of what they have done or are doing maysubstantially enhance their understanding and capability in respectivefields.

    Evaluations with an empowerment agenda may also be done ascollaborative exercises between programme- or organisation-internalpersons and external persons or institutions, in w hich the involvementand views of the former are prominent in the ana lysis and conclusions.Even basically internal evaluations may often be fruitfully facilitated

  • 8/9/2019 Evaluating Development Programmes and Projects

    37/215

    36 Evaluating Development Programmes and Projects

    by an outsider. Such external facilitation may have a dual purpose:it may assist the participants in conceptualising and reflecting moreon perspectives and ideas than they may do solely from their ownexperience, and it may help structure the process of evaluation itself.

    Sometimes, intended beneficiaries may be empowered also byevaluating development w ork that is mainly or entirely done by someoutside organisation or organisationsalso schemes other than typi-cal capacity building programmes. That may be the case if they,through their involvement (by some collabo rat ive arrangement), cometo influence the programme or project to their ow n benefit, o r if theylearn things that augment their ability to perform better in some othercontext. Of course, the degree and kind of empowerment may alsobe influenced by many external factors over which the evaluators maynot have any or much control.

    Most genuine empowerment evaluations are formative , aiming atimproving the future performance of the evaluated programme ororganisation, through the evaluators efforts and to the evaluatorsbenefit. Sometimes, participatory summative evaluations may alsohave an empowerment effect, to the extent they help increase theparticipants ability to obtain benefits beyond the evaluated programme,as mentioned above.

    Fetterman et al. (1996) emphasise the internal and formative natureof most empowerment evaluations. Besides, according to them, suchevaluations are normally facilitated. In their own words,

    The assessment of a programs value and worth is not the end point of theevaluationas it often is in traditional evaluationbut part of an ongoingprocess of program improvement. This new context acknowledges a simplebut often overlooked truth: that merit and w orth are not static values. . . . Byinternalizing and institutionalizing self-evaluation processes and practices, adynamic and responsive approach to evaluation can be developed to accom-modate these shifts [of merit and worth]. Both value assessments and corre-sponding plans for program improvementdeveloped by the group with the

    assistance of a trained evaluatorare subject to a cyclical process of reflectionand self-evaluation. Program participants learn to continually assess theirprogress tow ard self-determined goa ls and to reshape their plans . . . accordingto this assessment. In the process, self-determination is fostered, illuminationgenerated, and liberation actualized (ibid.: 56).

    Empow erment, as perceived above, transcends relatively narrow tech-nical perceptions of capacity building, commonly held by develop-ment workers and organisations. It incorporates the augmentation ofdisadvantaged peoples self-confidence and influence over factors tha t

  • 8/9/2019 Evaluating Development Programmes and Projects

    38/215

    Purposes of Evaluat ion 37

    shape their lives, often also involving building of organisations andwider institutional qualities and abilities. 4

    In collaborative evaluations (referred to above), Steven E. Mayer(1996), suggests three mechanisms for allying evaluation of commu-

    nity development work with empowerment of local communities:

    help create a constructive environment of part icipation in theevaluation, by viewing outside experts and local people as co-discoverers; view ing negative findings of evaluations primarily as induce-

    ments for improvement rather than causes for punishment;

    promoting partnerships of various stakeholders; directly include the voices of the intended beneficiaries, by incorporat ing their view s of the purpose of the evaluation; incorporating their experience, w isdom and standards of good

    performance (connecting directly to the earlier discussed notionof lifeworld rationality);

    including the most marginalised people in the group of evalu-

    ators; help communities use evaluation findings to strengthen their

    responses, by using various media to spread informat ion and the lessons

    learnt; creating links betw een people w ho may use the information; helping community o rganisations to build on the experiences

    that they have gained.

    Empowerment evaluations of organised development work will haveto be conducted through collective or participatory methods of analy-sis, to be addressed in Part Three.

    LINKING TO MODES OF PLANNING:

    PROCESS VERSUS BLUEPRINTThe above presented no tion of formative versus summative evaluationconnects directly to modes of planning of the respective programmesand projects. One planning dimension is of particular relevance here,namely, that of process versus blueprint planning (Faludi, 1973; 1984;Korten, 1980; 1984; Dale, 2004).

    4 For definition and further discussion of capacity , organisation and insti-tution-building, see Chapter 8 and Dale, 2000; 2004.

  • 8/9/2019 Evaluating Development Programmes and Projects

    39/215

    38 Evaluating Development Programmes and Projects

    Process pl anning basically means that plans are not fully finalised orspecified prior to the start-up of implementation; that is, greater orlesser amounts of planning are done in the course of the implemen-tation period of the development scheme, interactively with imple-

    mentation and monitoring.Blueprint planning , in its extreme or pure form, means that one

    prepares detailed plans for all that one intends to do before imple-menting any of that work. Thereby, the implementers will know exactly w hat they are to do, in which sequence and at w hat cost, untilthe scheme is completed.

    Implicit in the above is that planning may be more or less processor more or less blueprint; that is, actual planning events will belocated somewhere along a continuum between extreme process andextreme blueprint.

    Uncertainty and uncertainty management are central notions inprocess planning. This planning mode is particularly appropriate incomplex environments, where no firm images may be created, orw hen the planners control over external factors is restricted for otherreasons. Korten (1980: 49899) also refers to process planning as alearning process approach, and he thinks that planning with peopleneeds to be done in this mode.

    With blueprint planning, all possible efforts must be made duringa single planning effort to remove uncertainties regarding implemen-tation and benefits to be generated. Ideally, then, blueprint planning

    is an approach whereby a planning agency operates a programmethought to attain its objectives with certainty (Faludi, 1973: 131).To that end, the planner must be able to manipulate relevant aspectsof the programme environment, leaving no room for the environ-ment or parts of it to act in other ways than those set by the planningagency (ibid.: 140).

    We see that Faludi uses the term programme for the set of activities

    that are planned and implemented. Korten (1980: 496), however,stresses that, in blueprint planning, it is the project [my emphasis]its identification, formulation, design, appraisal, selection, organiza-tion, implementation, supervision, terminat ion, and evaluation[that]is treated as the basic unit of development action. 5

    5 As will be clarified below, this complies with my perception of programme andproject.

  • 8/9/2019 Evaluating Development Programmes and Projects

    40/215

    Purposes of Evaluat ion 39

    The attentive reader may already have grasped that formativeevaluations may only be done of development schemes that areplanned and implemented in a process mode. Indeed, the moreprocess oriented the scheme is, the more meaningful and important

    may be formative evaluations. Commonly, that also means that theyare done more frequently, or that it would be advantageous withrelatively frequent assessments.

    Box 2.1 presents a programme in which a flexible system offormative evaluations (here called reviews) was designed and imple-mented, in order to help develop and improve the programme overa number of years.

    Box 2.1

    MANAGING A PROGRAMME WITH PROCESS PLANNING

    (Box 2.1 contd.)

  • 8/9/2019 Evaluating Development Programmes and Projects

    41/215

    40 Evaluating Development Programmes and Projects

    In programmes and projects that are planned and implemented in anoutright blueprint mode, evaluations may be only of a summativenature. In phased programmes or projects, with predominantly blue-print planning of each phase, evaluations between the phases may beboth summative (relating to the finalised phase) and formative (re-lating to any next phase).

    LINKING TO THE CONCEPTS OFPROGRAMME AND PROJECT

    In previous pages, I have already used the words programme andproject several times. That is because they tend to be the most

    frequently utilised terms for denot ing planned and organised w ork forsocietal development. In order to facilitate a focused and constructive

    (Box 2.1 contd.)

  • 8/9/2019 Evaluating Development Programmes and Projects

    42/215

    Purposes of Evaluat ion 41

    discourse on aspects of development workincluding evaluationwe should have their meanings better clarified.

    A programme is not easily defined. It is normally regarded as a lessspecified and/or less clearly bounded entity than a project, in terms of

    its focus, scope of activities, time horizon, etc. Many developmentprogrammes also tend to be relat ively broad-focusing and long-lasting.

    In terms of frequently used concepts of planning, a programmemay be formulated through strategic (overall, framework) planningonly or through both strategic and operational (detailed) planning. 6

    For instance, in a development programme, strategic aspects may beformulated in a separate document, which will usually be called theprogramme document. Parts of the development thrust may then befurther specified in one or more additional plan documents, whichare more appropriately referred to as project documents. Alterna-tively, strategic and operational planning may be undertaken as partsof one comprehensive exercise and formulated in the same document.In some instances, little or no operational planning may be under-taken by the planning agency, but is done informally by the users ofallocated resources. In such cases, we have unquestionably aprogramme, under which the mentioned users undertake their own,normally very small, schemes (w hich may be referred to as programmecomponents or projects).

    We may distinguish betw een tw o main types of developmentprogrammes by their scope: one-sector programmes and multi-sector

    programmes. Sector is normally defined by the general type ofactivity performed or the service rendered. A few examples may beeducation, primary health care, fisheries, irrigation, financial services,and social mobilisat ion. To the extent these activities are also concernsof governments, the mentioned categories normally coincide withpublic administrative responsibility as well.

    Both one-sector programmes and multi-sector programmes may be

    more or less flexible or rigid. That is, they may be planned in more orless of a process or a blueprint mode. Most development programmesdo require a substantial amount of process planning. In particular,this applies for relatively diversified programmes, programmes withbroad stakeholder participation, and programmes that aim at capacitybuilding.

    6 See Chapter 4 for a further, brief clarification of these terms. For a morecomprehensive analysis, see Dale, 2004.

  • 8/9/2019 Evaluating Development Programmes and Projects

    43/215

    42 Evaluating Development Programmes and Projects

    Multi-sector programmes, in particular, may also be more or lessintegrated or disjointed; that is, they may consist of components thatare functionally more or less connected or unrelated.

    There may be hierarchies of programmes as well. For instance, a

    development programme covering one district may contain severaldivisional programmes, each of which may contain local communityprogrammes. Within this hierarchy, projects may be planned and im-plemented at any level.

    Note that the word programme is also used with a differentmeaning in the phrase work programme. A work programme spellsout the implementation details of a project or parts of it. Linked tothis is programming, meaning detailed sequencing of activities.

    Definitions of the word project abound in business managementliterature and in literature on public planning and management.When signifying a formally organised endeavour, a project is normallystated to be a clearly delimited and relatively highly specified under-taking. A synthesis of typical definitions that have been presented maybe something like a planned intervention for achieving one or moreobjectives, encompassing a set of interrelated activities tha t a re under-taken during a delimited period of time, using specified human,financial and physical resources .

    The idea is that projects, like other endeavours that use resourcesthat must be accounted for, should be well specified before one maystart implementation, leaving as little uncertainty as possible about

    the quantity and quality of the outputs and their costs.The logical implication of such a conception is that a developmentintervention, to be termed a project, should be formulated in blue-print mode and specified through operational planning.

    In reality in the development sphere, the word project tends tobe used in a broader sense than this, encompassing endeavours thatought to be called programmes, according to most formal definitions

    of project and the argument above. In my view, a more restrictedand stringent usage of project than has been common (and a cor-responding more frequent use of programme) would be advanta-geous. This would help unify perceptions about characteristics ofvarious kinds of development thrusts and, therewith, facilitate com-munication about approaches in development work.

    Continuing from earlier statements, the intention of a project in

    the development sphere (i.e., a development project) should be togenerate specified benefits for people, and it should be substantiated

  • 8/9/2019 Evaluating Development Programmes and Projects

    44/215

    Purposes of Evaluat ion 43

    to do so. Usually, benefit-generation is explicitly addressed in theproject plan itself. In some cases, however, links to peoples well-being may be less explicitly formulated or even just assumed, nor-mally with reference to a wider context (for instance, a programme

    of which the project may be a limited part). This matter is furtherexplored in Chapter 4.

    As w ill be clarified later, the purpose, focus and mode of evaluationmay, to different degrees, depend on whether one evaluates a pro-gramme or a project.

  • 8/9/2019 Evaluating Development Programmes and Projects

    45/215

    Chapter 3

    EVALUATION VERSUS APPRAISALAND MONITORING

    APPRAISAL AND EVALUATION

    Three common words of the development vocabulary are partlyrelated. They are appraisal, monitoring and evaluation.

    In the context of development work, appraisal is usually taken tomean a critical examination of a proposal of a programme or project,normally before the latter is approved for funding and implementa-tion. From what we have already said about evaluation, the readershould then immediately see that there is only a limited sphere ofoverlap between appraisal and evaluation.

    The relationship may be further clarified with reference to thedefinition by Rossi and Freeman (1993) of what they call evaluationresearch, namely, the systematic application of social research pro-cedures for assessing the conceptualization, design, implementationand utility of social intervention programs (ibid.: 5).

    The point of attention in the present context is assessment ofconceptualisat ion and design. We have clarified that evaluation ofdevelopment work should emphasise benefits for people and related

    aspects. Simultaneously, benefits need to be substantiated and ex-plained, and conceptualisation and design may certainly be factorsof substantiation and explanation. The point is that, for an exerciseto be called an evaluation, conceptualisation and design must beviewed in this wider context of achievements. An examination ofthese features alone would be better referred to as appraisal, whichmay be needed before any substantial development work is initiated.

    Additionally, evaluations of development work may address aspectsof efficiency, occasionally even in their ow n right (see Part Tw o). In

  • 8/9/2019 Evaluating Development Programmes and Projects

    46/215

    Evaluat ion versus Appraisal and M oni toring 45

    such cases, conceptualisation and design would be related to theimmediate outputs and the processes that generate these outputs.Thus, even in this narrower context, conceptualisation and designwould be assessed in relation to programme or project outcome.

    MONITORING AND EVALUATION

    In the development sphere, monitoring may be done for three mainpurposes: assessing the performance of a programme or project;analysing organisational performance; and examining features and

    processes in the environment of an organisat ion or scheme. The threepurposes may be more or less related. In all instances, for the assess-ment to be referred to as monitoring, it should be of relatively currentor frequent nature.

    Monitoring of programmes and projects is usually taken to meanrelatively current or frequent assessment of planned work and theresults of that work. More precisely, we can define it as frequent andlargely routine generation of and reporting on information about theperformance of a programme or project, comparison of this with theprogramme or project plans and, commonly, suggesting correctivemeasures, if any.

    Thus, monitoring aims at meeting the information needs of currentprogramme and project management. This presupposes a clarifiedpurpose of, and a reasonably systematic approach to, informationcollection and use.

    Usually, monitoring of development work mainly covers aspects offinance, the quantity and quality of inputs and outputs, as well asactors and time use in implementation. It may also (and commonlyshould) encompass some regular assessment o f relat ively direct changesthat are brought about by the scheme, and may even address other

    matterswhich may possibly be analysed more thoroughly in someadditional evaluation.This kind of monitoring is usually done, entirely or primarily, by

    the organisation or organisations that are responsible for theprogramme or project. There may be several persons involved, oftenat different levels in the organisat ion. In some instances, these personsmay also be the intended beneficiaries of the work that is done. In

    other cases, intended beneficiaries may be involved in monitoringeven if they do not have any formal implementation role. In yet other

  • 8/9/2019 Evaluating Development Programmes and Projects

    47/215

    46 Evaluat ing D evelopment Programmes and Proj ects

    cases, some outside body may be engaged to play a supporting rolein designing a monitoring system or even in implementing it.

    Most organisations, also in the development sphere, will in addi-tion have their internal systems for assessment of organisational

    performance more generally, i.e., not directly or primarily relating tospecific programmes and projects that the organisations undertake. 1

    Occasionally, such general monitoring of the performance of one ormore organisations may be done by other organisations. Examplesmay be supervision of local government bodies by some centralgovernment department (which may be more or less direct andcomprehensive), and continued gathering of information about thework of built organisations by the organisation-building body.

    In addition, we may refer to a third, rather different, activity asmonitoring. This is relatively current assessment of external (envi - ronment al ) factors relating to and influencing the performance of aprogramme or project or an organisation, coupled with considerationof responses to any changes in relevant parts of the environment.

    This is well recognised as a crucial activity for success in thebusiness world. It may be equally important in the developmentsphere, particularly in long-term programmes that are planned in theprocess mode, the more so the more open and complex they are. Acouple of suitable terms for this activity are strategic monitoring andenvironmental scanning. 2

    We have already clarified that we consider evaluation to be a more

    specific and time-bound kind of activity than those presented above,and that it tends to focus more on changes relating to intendedbeneficiaries (and sometimes other people). We have also stated thatthere may be grey zones between what is unquestionably monitoringand evaluation respectively. For instance, this may often be the casein programmes where the assessment is undertaken by the intendedbeneficiaries themselves, partly or in full. These may be programmes

    that are also planned and implemented by peoples ow n organisations,or they may be programmes or projects that are supported or under-taken by others.

    We shall illustrate this issue through two cases of relatively continu-ous or frequent participatory assessment. Such assessments have been

    1 Love (1991) addresses such assessments comprehensively, under the heading of

    internal evaluation. Dale (2000) presents and discusses dimensions of organisationthat may need to be subjected to more or less frequent assessment.2 See Dale, 2000 for further elaboration and discussion.

  • 8/9/2019 Evaluating Development Programmes and Projects

    48/215

  • 8/9/2019 Evaluating Development Programmes and Projects

    49/215

    48 Evaluat ing D evelopment Programmes and Proj ects

    (Box 3.1 contd.)

    (Box 3.1 contd.)

  • 8/9/2019 Evaluating Development Programmes and Projects

    50/215

    Evaluat ion versus Appraisal and M oni toring 49

    The first caseinvolvement by intended beneficiaries in the fo llow -up of infrastructure building by outside professional bodies (govern-ment departments and contractors engaged by them)is clearly anexample of monito ring, by our definition of this term. The follow -upis done fairly continuously, and the focus is on aspects of implemen-tation.

    The specified activities of the second case, as well, may be referredto as monitoring. However, one may also refer to them as evaluation.First, they are done at pre-specified points in time and less frequentlythan monitoring would normally be done. Second, in any round ofstudy, the examination is done of systematically sampled units (goodsand shops) only, being a typical feature of most evaluations. Third,the assessment is also intended to address benefits for people, in termsof changes in their nutrition status (although the people themselvesmay be able to indicate such changes only very indirectly).

    In the second case, the activities that are analysed are those of onesown organisation. One may then refer to them as self-assessment,internal monitoring or internal evaluation (Love, 1991). If oneprefers to use the term evaluation (rather than monitoring), onecould add process, mentioned above, making this kind of assessmentread internal process evaluation.

    A MORE PRECISE DEFINITION OF EVALUATION

    We shall round off this chapter by going a step further in definingevaluation, bridging the previous overview with the more detailedexamination of aspects of evaluation that will be addressed in PartsTw o and Three of the book.

    The contact sphere between appraisal and evaluation is, we havestated, very limited, and the relation between these two conceptsshould have been well clarified.

    (Box 3.1 contd.)

  • 8/9/2019 Evaluating Development Programmes and Projects

    51/215

    50 Evaluat ing D evelopment Programmes and Proj ects

    Monitoring and evaluation are much more broadly and closelyrelated. They are both undertaken to find out how a programme orproject performs or has performed, including reasons for aspects ofperformance, whether positive or negative. Thus, a more specific

    definition of evaluation will primarily need to clarify this concept inrelation to monitoring, in situations where this is clearly warranted.

    Based on the above, we shall define evaluation, in the context ofdevelopment work, as mostly a more thorough examination than moni tori ng, at specified point s in t ime, of programmes, proj ects or organisational performance, usually with emphasis on impact for peopl e and commonly also relevance, effectiveness, efficiency, sustain- abil i ty and repli cabil it y .

    The last-mentioned terms w ill be defined in Part Tw o, as part ofan examination of the focus and scope of various kinds of evaluations.

    Evaluators may draw on information that has been provided throughmonitoring as well as additional information that they generate di-rectly, from primary sources, or indirectly, from other secondarysources. Information may be gathered through a range of methods,depending on the evaluations purpose and context. Aspects ofmethodology will be examined in Part Three.

    Evaluations may be done by programme or project responsiblepersonnel, independent persons, beneficiaries, or any combination ofsuch people. This matter and additional aspects of management willbe addressed in Part Three as well.

  • 8/9/2019 Evaluating Development Programmes and Projects

    52/215

    Chapter 4

    LINKING TO PLANNING: MEANSENDS ANALYSIS

    CONNECTING DESIGN VARIABLES, WORK CATEGORIESAND INTENDED ACHIEVEMENTS

    In the preceding chapters, we have discussed development work, andevaluation in the context of such w ork, in quite general terms. Weshall now connect design features, work tasks and activities of

    programmes and projects in a more systematic manner, focusing onthe essential thrust of exploring and substantiating meansends rela-tions. Most basically, in development work, resources are allocatedfor undertaking pieces of work that are intended to generate certainbenefits for certain people. Clarifying relations between resource use,work efforts and benefits may be readily proclaimed as a core taskand the main challenge in development programmes and projects. To

    that end one needs, in planning, to conceptualise and specify mecha-nisms and steps by which resources and pieces of work are intendedto be converted into benefits. In evaluation, one will then explore andseek to document whether or to what extent intended relations havematerialised or are materialising. In both planning and evaluation,this will invariably also involve analysis of contextual factors (to bespecifically addressed in the next chapter).

    All organised development w ork needs to be planned, in some basicsense. At the very least, this will involve some substantiated idea aboutrelations between the above-mentioned categories. Sometimes, theconceptions may be very loose, and may even exist only in the mindsof people. More commonly, however, planning is a more comprehen-sive and substantial thrust, and what is planned may have to bespecified in writing. The basic points are ( a ) that no organisationinvolved in development work (which may be anything from a smalllocal group to a government department or an international agency)

  • 8/9/2019 Evaluating Development Programmes and Projects

    53/215

    52 Evaluat ing D evelopment Programmes and Proj ects

    1 See Dale, 2004 for a comprehensive analysis of strategic development planningand relations between strategic and operational planning.

    w ould use resources for something about the outcome of w hich it hasno idea, and ( b ) that the amount and type of planning that may beneeded to clarify and induce the above-mentioned relations will varyvastly between kinds of development schemes and their context.

    We have already made brief mention of the concepts of strategicand operational planning. Strategic planning is the most fundamentalexercise in development work, on which any other activity andfeature builds and to which they relate. It seeks to clarify and fittogether the main concerns and components of a development thrust(programme or project). This involves identifying relevant problemsfor people, making choices about the problem or problems to beaddressed, clarifying the availability of resources, and deciding onobjectives and general courses of actionconsidering opportunitiesand constraints in the environment of the involved organisation ororganisations and abilities of various kinds. O perat ional planning means further specification of components and processes that one hasdecided on during preceding strategic planning. A good operationalplan should be a firm, detailed and clear guide for implementation.A planning thrust (stra tegic and /or operat ional) may encompassanything from blueprint planning of an entire big project to theplanning of a small component of a process-based programme some-time in the course of programme implementation. 1

    What is planned is, of course, supposed to be implemented. Inother words, implementation is intended to be done in accordance

    with planned work taskswhich I shall refer to as i mplementat ion tasks and planned resource allocation for these taskswhich I shallrefer to as inputs . Beyond this, relations betw een planning and imple-mentation depend much on whether one applies a process or ablueprint approach (see Chapter 2).

    The direct (or relatively direct) outcome of the work that is doneis normally referred to as outputs . For certain kinds of schemes, the

    project managers should be able to guarantee the outputs, since theyought to be in good control of the resource inputs and the workthat directly produces them. However, for most kinds of develop-ment work, the matter is usually not so straightforward (see alsoChapter 5).

  • 8/9/2019 Evaluating Development Programmes and Projects

    54/215

    Linking to Planning: M eansEnds Analysis 53

    During implementation, one will monitor the resource use, workprocesses, outputs and, possibly, fairly direct changes caused by theoutputs. As clarified earlier, the purpose of monitoring is to know w hether or to w hat extent the programme or project proceeds accord-

    ing to the plans and creates what it is intended to create, and toprovide information that may be needed for any changesregardingplans, mode of implementation or outputs. The role of monitoringmay vary substantially between types of programmes and projects.This function, as well, tends to be particularly much related to theplanning dimension of processblueprint.

    The outputs are produced in order to create intended benefits forpeople. A major challenge for development planners is to analyse andformulate meansends relations towards goal-achievement, and anequally big challenge for evaluators is to explore and substantiatebenefits and how they are being or have been generated. We shalltherefore discuss such meansends relations more specifically.

    LINKING PLANNED OUTPUTS TO INTENDEDBENEFITS FOR PEOPLE

    As repeatedly stated, the very purpose of development work is toenhance the quality of life of one or more groups of people. Theoverall intended improvement in peoples life quality (the intendedultimate benefit) is normally referred to as the development object ive

    (a term which we will be using) or goal .A major challenge in development planning is, then, to analyse

    meansends relations converging on the development objective andto then formulate a logical and well substantiated structure of linkedvariables, particularly from the level of outputs to this overall objec-tive.

    H ow ever, some planning does not incorporate any explicit analysisof consequences of w hat is planned for peoples quality of life. Faludi(1973; 1984) and others