Presentation Outline

Country-led Joint EvaluationCountry-led Joint Evaluation Dutch ORET/MILIEV ProgrammeDutch ORET/MILIEV Programme in China in China

Chen Zhaoying NCSTENCSTE , ChinaHan Jun NCSTE, ChinaHans Slot IOB, The Netherlands

Shanghai International Program for Development Evaluation Training Oct. 22– Nov.2, 2007, Shanghai, China

2

Presentation OutlinePresentation Outline

■ A brief note on Country-led Joint Evaluation

■ The Evaluation

initiation, initiation, governance structure, governance structure, methodology, implementation and methodology, implementation and dissemination

■ Issues for discussion

3

Joint EvaluationJoint EvaluationDefinitionDAC Glossary defines joint evaluation as: An evaluation to which different donors and/or partners participate.

Four categories of joint evaluation

■ Donor + Donor

Donor + Partner Country

■ Multi-Donor + Multi-Partner

■ Partner + Partner

4

OverviewOverview

■Partnership in development leads to partnership in evaluation. Joint Evaluation has been on the international development agenda since the early 1990s.

■Country-led Joint Evaluation of ORET/MILIEV Programme in China is regarded by DAC Network as one of the important joint evaluation cases since 1990.

Joint EvaluationJoint Evaluation

5

Country-led EvaluationCountry-led Evaluation

■ Ownership in development leads to ownership of evaluation.

■ The World Bank, UNDP, the DAC and some donor governments have been developing approaches to promote CLEs

■ The majority of evaluations of development aid programmes are still donor evaluations

■ What is needed are evaluations from the perspective of the partner country. CLEs will provide that.

Overview of CLEOverview of CLE

6

The Country-led JointThe Country-led JointEvaluation in ChinaEvaluation in China

■ About the title of the evaluation

■ The recipient country leads the process, supported by the donor and both sides are jointly responsible for the evaluation

The evaluation covers the period 1991-2003. A total project portfolio of 84 projects were approved. 40 projects were finished and 44 were ongoing. Of these 44, 17 completed the transaction but still had to produce the final report. The grant amount of the transaction is about € 200 million, the total transaction amount of the evaluated portfolio € 470 million.

Dutch ORET/MILIEV Program in ChinaDutch ORET/MILIEV Program in China

7

How is the Evaluation How is the Evaluation Initiated?Initiated?

■■ After the experiences of joint evaluation of aid After the experiences of joint evaluation of aid development, NCSTE would like to have a more development, NCSTE would like to have a more pronounced role in the development evaluation pronounced role in the development evaluation and IOB would like to be involved in a recipient-and IOB would like to be involved in a recipient-led joint evaluation.led joint evaluation.

■■ The joint evaluation was co-initiated by IOB The joint evaluation was co-initiated by IOB and NCSTE.and NCSTE.


8

9

The Evaluation Aims toThe Evaluation Aims to

Assess to what extent the programme (through its Assess to what extent the programme (through its activities / projects) has fulfilled the policy objectives, activities / projects) has fulfilled the policy objectives, needs and priorities of the Netherlands and Chinaneeds and priorities of the Netherlands and China

Assess to what extent the programme (through its Assess to what extent the programme (through its activities / projects) has fulfilled the policy objectives, activities / projects) has fulfilled the policy objectives, needs and priorities of the Netherlands and Chinaneeds and priorities of the Netherlands and China

Verify whether the funds have been Verify whether the funds have been appropriately and efficiently usedappropriately and efficiently used Verify whether the funds have been Verify whether the funds have been appropriately and efficiently usedappropriately and efficiently used

Provide information for both the Provide information for both the Chinese and the Dutch that could be Chinese and the Dutch that could be used to improve the programme, as used to improve the programme, as well as for policy formulationwell as for policy formulation

Provide information for both the Provide information for both the Chinese and the Dutch that could be Chinese and the Dutch that could be used to improve the programme, as used to improve the programme, as well as for policy formulationwell as for policy formulation


10

Evaluation CriteriaEvaluation Criteria

■■ The relevance of the ProgrammeThe relevance of the Programme

■■ The efficiency of the ProgrammeThe efficiency of the Programme

■ ■ The effectiveness of the ProgrammeThe effectiveness of the Programme

■■ The impact of the projects implemented under The impact of the projects implemented under the Programmethe Programme..


11

How is the evaluation How is the evaluation organized?organized?

Governance StructureGovernance Structure

An equal governance structure was established An equal governance structure was established for this particular type of evaluation.for this particular type of evaluation. ■■ The Steering Committee (SC)The Steering Committee (SC)

Make decisions on the evaluationMake decisions on the evaluation

■■ Team Leaders (TL)Team Leaders (TL)

Organise the fieldwork and other tasksOrganise the fieldwork and other tasks

■■ The Reference Group (RG)The Reference Group (RG)

Provide advice and supportProvide advice and support

12

How is the evaluation How is the evaluation conducted?conducted?

FFour steps of the joint evaluationour steps of the joint evaluation

■■ Design/TORDesign/TOR

■■ Setting up & organizationSetting up & organization

■■ Implementation & Final Draft ReportImplementation & Final Draft Report

■■ Report dissemination/Report dissemination/Following up Following up

13


Evaluation Design/TOREvaluation Design/TOR

TOR are even more important for the success of the joint evaluation

■ ■ TOR describe the overall evaluation and establish the initial agreements prior to the work plan

■■ TOR process establishes the basic guidelines so everyone understands the context of the evaluation

■■ Whether TOR is the product of collaborative process is an important indicator for assessing the degree of cooperation in joint evaluation.

■■ IOB and NCSTE jointly draft the TOR.

14


Evaluation Design/TOREvaluation Design/TOR

Identifying the Key Issues in Writing TOR

■■Focus on key issues to be addressed by the evaluation

■■Avoid too many issues. It is better to have an evaluation that examines a few issues in-depth

15


Key Issues In TORKey Issues In TOR

Policy RelevancePolicy Relevance

■In 1999 the Chinese government launched the Western Development Strategy. Has the number of projects approved in this region changed since 1999?

■Are the projects listed as priority projects in the local development plans?

■Are the projects in line with the Netherlands development policy?

■To what degree are government officials, on-lending banks, suppliers, window companies and end users well informed about the objectives of the programme?

16



EfficiencyEfficiency■ Have the projects achieved their expected outputs on Have the projects achieved their expected outputs on schedule and within budget? schedule and within budget?

■ Have the projects experienced delays during Have the projects experienced delays during implementation? If so, what were the main causes of these implementation? If so, what were the main causes of these delays? delays?

■ Has the condition that 60% or more of the transaction Has the condition that 60% or more of the transaction should be sourced in the Netherlands resulted in higher should be sourced in the Netherlands resulted in higher prices? prices?

■ Are the procedures for project application, appraisal and Are the procedures for project application, appraisal and approval considered reasonable by the end users, the approval considered reasonable by the end users, the organisations that receive ORET/MILIEV funds?organisations that receive ORET/MILIEV funds?

17



EffectivenessEffectiveness■ Has the programme facilitated access for Dutch Has the programme facilitated access for Dutch suppliers to China’s local and/or national markets?suppliers to China’s local and/or national markets?

■ Have the projects created skilled job opportunities in Have the projects created skilled job opportunities in China? China?

■ Is the equipment still operational and maintained after Is the equipment still operational and maintained after the completion of the projects? the completion of the projects?

■ In the opinion of the end users, are the projects In the opinion of the end users, are the projects sustainable in terms of maintenance, after-sales services sustainable in terms of maintenance, after-sales services and the availability of spare parts?and the availability of spare parts?

18


Key Issues In TORKey Issues In TORImpactImpact

■ Are the projects being replicated elsewhere? Are the projects being replicated elsewhere?

■ Have the projects had a negative impact on the Have the projects had a negative impact on the environment, the poverty situation and/or on the situation environment, the poverty situation and/or on the situation of women?of women?

■ Has the programme resulted in an increase in trade Has the programme resulted in an increase in trade between China and the Netherlands? between China and the Netherlands?

■ In the opinion of the end users, what have been the In the opinion of the end users, what have been the impacts of the project activities?impacts of the project activities?

19

How Is the Evaluation How Is the Evaluation Conducted?Conducted?Evaluation MEvaluation Methodologyethodology■ Desk Study: Cover all the 84 projects

■ Field Visits: 35 projects were chosen, 19 finished and 16 ongoing projects

■ Questionnaires: 68 were returned from end users and 60 from suppliers

■ Stakeholder Workshops: 17 provincial-level workshops were held during the field visits. More than 300 stakeholders were invited to participate in the workshops

■ Cross-verified & In-depth Analysis: The information in the fact sheets and the evidence collected from the field visits were cross-verified to support in-depth analysis; the database of initial evidence for the 35 visited projects has been built

20

21

The Responses from The Responses from StakeholdersStakeholders

■ The responses from Chinese stakeholders

■ The responses from Dutch stakeholders

■ The responses from other development agencies

■ The policy reaction in China and the Netherlands

22

The Dissemination of The Dissemination of the Evaluationthe Evaluation

■ The dissemination has been given attention from the beginning.

■The final evaluation reports published in both Chinese and English

■ The dissemination workshop in Beijing China, March 29, 2007

■ Communicate this evaluation case at Sixth Meeting of the DAC Network on Development Evaluation, 27-28 June 2007, Paris.

23

24

IIssues for Dssues for Discussioniscussion

■ The key benefits of the evaluation

■ The limitation of the evaluation

■ Potential challenges of the country-led joint evaluation

■ Making the decision to undertake a joint evaluation

25

The key benefits ofThe key benefits of the the Country-led Joint Country-led Joint EvaluationEvaluation

■ Offer great opportunities for mutual capacity development

■ Increase ownership, the evaluation reflects better the voice of the partner countries

■ More relevant to the needs of both sides

■ Sharing the burden of work of evaluation

26

The limitation ofThe limitation of the the Country-led Joint Country-led Joint EvaluationEvaluation

Institutional limitation in China

■ A national evaluation system has not been set up

■ No one in-charge of the official policy reaction

■ Lack of special budget for evaluation

27

The Potential challengesThe Potential challenges of the of the Country-led Joint Country-led Joint EvaluationEvaluation

■ The biggest challenge comes from different evaluation cultures and evaluation systems

■ Ensure that the evaluation meets the needs of both countries

■ Build consensus to design and conduct evaluation (such as: Identifying the key Issues in writing TOR, choice of field study cases)

■ Evaluate the impact of the program

28

Environmental issues in Dutch ORET/MILIEV evaluation

1-Introduction:

■ -The objective of the Dutch ORET/MILIEV Programme: development and environment related, but export transactions are mostly focused and stressed (equipment export; mix-credit + grant).

■ -Before the approval, at the appraisal stage, all projects are requested to go through environmental impact assessment with the basic principle of Not to harm the environment - positive or neutral would be OK.

The result of environmental ex-anti test:

almost half for positive; half for neutral.

■ -Among the portfolio, only 18 projects are focused on environment issues or environment orientated (21% of 84 total projects). Therefore, ORET/MILIEV programme is not so strong on environment issues.

29


1-Introduction:

■ -Among the 18 projects, 7 environment orientated projects were selected as cases, and visited on site during the field missions.

These 7 projects are related to waster water treatment, solid waste treatment, environmental master planning, integrated mangrove management, banknote destroying system etc.

■ -Difference of outcome or impact evaluation between the environment focused projects and other projects-direct or indirect, intended or unintended impact effect.

Because of the characteristics and objectives, the environment orientated projects for sure will have, to some extent, direct and intended environmental contribution indicators as well as the impact. While other projects are not necessary having influence on environment. Their impact, if there are, might be in an indirect and unintended way, but it is more difficult to measure the impact of these kinds of projects.

30


2-Limitations and difficulties to assess environmental impact:

■ -The environmental issues are so complicated that it’s rather difficult to measure their outcome and impact than normal projects. Environmental impact evaluation needs appropriate and useful approaches.

Although some environmental questions are discussed and analysed, especially in relevance, effectiveness and impact, environment sustainability are not related or concerned in this evaluation. At TOR and evaluation procedure, no specific indicators were designed for environmental issues. We use the same evaluation criteria for all projects.

■ -A measurable impact needs available information and comprehensive statistic data- due to the absence of suitable project-level data and related information, it has been impossible to assess the environmental issues at sector/programme level in this evaluation.

Especially, it’s not easy to get environmental statistic data in China, national wide and locally. Also it’s time and money consuming if you want to collect these complicated data. On the other hands, Chinese monitoring system is rather weak.

31



■ -Quantitative and qualitative evaluation-it is easier to make a judgment of positive, neutral or negative impact for a project in a qualitative way, but it is difficult to measure impact in a quantitative way.

At present, some countries like the Netherlands and some international organizations like World Bank have undertaken several research and impact evaluation on poverty reduction. It seems more quantitative indicators are included. But how about the environmental impact evaluation? Shall we have more quantitative impact indicators?

32



■ -Attributions - improvements in water and air quality can be the output or outcome of a project, but are there other factors that contribute to the improvement? How many people have been affected by such changes? And to what extent?

Even among the 7 environmental orientated projects, it’s easier to assess waste water treatment project because its goal and outcome indicators are clear and designed at the very beginning. For example, we can get these kinds of data like how many cubic meters of waste water treated daily? How many tons of sewage reduced when discharging into nearby rivers? How many local people benefited from this project? But others are difficult to analyze their contributions to environment.

33

Environmental issues in Dutch

ORET/MILIEV evaluation

3- Some findings related to 7 environment orientated projects:

■ -Overview - The 7 environment orientated projects have achieved (or highly achieved) the objectives and having positive impact on environment. For the whole project portfolio, only one project was found having negative impact on environment.

Negative impact -The Refined Potato Starch Processing Project in Hebei Province. According to the contract, the end user must build a waste water treating facility, but the factory failed to do so and discharged the waste water directly into a reservoir. Except for water pollution, this project have very positive impact on social, economic and poverty alleviation.

Debate: is it a good project if the end user will install the necessary waste water treatment equipment in future?

34




■ -Demonstration - 4 of the 7 projects were assessed having demonstration effects.

Demonstration usually presents good practice or successful experience. For example, the demonstration of the advanced environmental technology-Dutch Carrousel-2000 technique in waste water treatment plant. Over 300 people visited Beijing Shunyi Waste Water Treatment Plant (WWTP) for learning and training.

And also, the management of mangrove forest in Guangdong Province demonstrated to local farmers community on how to protect the forest and how to manage biodiversity issues, etc.

35




■ -Replication - Chinese stakeholders as well as the evaluation team hold the strong opinion that ‘replication’ should be an important indicator when measuring impact; 3 pilot projects were found having been replicated, either in other part of China or in Asian countries. When destroying old or worn banknote, in order to avoid air pollution by burning and water pollution by re-processing, The People’s Bank of China introduce 19 sets of banknote disintegration and briqueting machines. Later on, the Bank ordered additional destroying systems from the same Dutch company - Kusters BV and replicated this in several provinces. Moreover, the small family-owned Dutch company Kusters expanded its business rapidly in Indonesia, Philippines and Pakistan, and more projects were replicated in these counties. Sometimes replication may combine with demonstration: Beijing Shunyi WWTP also received over 30 potential Chinese end users for consultancy. As a result, the Dutch DHV company have replicated about 20 WWTPs in other provinces, using same Dutch Carrousel -2000 technology.

36




■ - Seed Money - After the implementation of the Environmental Master Plan of Yunnan Plateau Lakes, Japanese government used the result of the plan and invested $60 million in this province to enhance the protection of lakes. Also, World Bank has been involved in other environmental projects thereafter.

37

Making the decision to undertakeMaking the decision to undertake a a Country-led Joint Country-led Joint EvaluationEvaluation

■ Evaluation Culture. Understand the differences in the evaluation cultures and systems in both sides

■ Audiences. Make it clear that who will be the main audiences of the evaluation in both sides

■ Jointly draft the TOR. TOR is even more important for the success of the joint evaluation.

■ Timeframe. If one is in a great hurry to get an evaluation completed, the first time of joint work may be disadvantageous.

38

Listen toListen to

your Opinions and Adviceyour Opinions and Advice

■ What do you think of this joint evaluation? What are the advantages and weaknesses?

■ How to promote the country-led joint evaluation as a successful evaluation model?

■ What are the major barriers to What are the major barriers to the country-led joint evaluation ? Lack of demand or weak ? Lack of demand or weak evaluation capacity ? evaluation capacity ?

■ What is Donor’s role in CLEs, particularly in the “first generation” of CLEs?

39

Thank youThank youAny Questions, Opinions & AdvicesAny Questions, Opinions & Advices… …… …

Presentation Outline

Documents

evaluation ownership

ownership of evaluation

evaluation initiation

particular type of evaluation

evaluation team leaders

donor evaluations

development leads

recipient country