Evaluation Approach Country Assistance Program Evaluation ... · Evaluation Approach . Country Assistance Program Evaluation for Papua New Guinea . May 2014 . ... 1 The western half

Evaluation Approach Country Assistance Program Evaluation for Papua New Guinea May 2014 Team Leader: Toshiyuki Yokota, Senior Evaluation Specialist (email: [email protected]) Contact: [email protected] A. Introduction 1. The proposed country assistance program evaluation (CAPE) will be the second assessment of the country strategies and assistance programs of the Asian Development Bank (ADB) in Papua New Guinea (PNG). The timing of the CAPE is opportune given that the first CAPE was undertaken 10 years ago in 2003, covering the period 1986–2002, and the current country partnership strategy (CPS) will be completed by the end of 2015. The CAPE will cover the period 2001–2014, across three country strategies—country assistance plan, 2001–2003; country strategy and program (CSP), 2006–2010; and the CPS, 2011–2015. Projects which were approved before 2001 but were implemented in the evaluation period will be covered in the CAPE. The CAPE aims to (i) draw forward-looking lessons and make recommendations for the next CPS of PNG, and (ii) provide an objective assessment and rating of the performance of ADB support to the country and identify factors affecting such performance. B. Background

1. The Country’s Development Context and Binding Constraints 2. PNG is a country in South Pacific located 160 kilometers north of Australia. PNG comprises five large islands—New Britain, New Ireland, Bougainville, Manus, and the eastern half of New Guinea,1 and more than 600 islands with a combined land area of 452,860 square kilometers. PNG is situated on the so called “Pacific ring of fire” and has a number of active volcanoes. The country is at the point of collision of several tectonic plates which makes it vulnerable to earthquakes and volcanic eruptions. PNG experiences a moderate tropical climate and has a wet season from November to April and a dry season from May to October. Landslides, inland and coastal flooding, and droughts are common natural hazards. 3. Economy. PNG is richly endowed with natural resources, including mineral and renewable resources and the economy depends heavily on mineral and resource extraction. Mineral-related exports account for more than 70% of exports on average, mainly gold making up half of the exports. The country is also the second largest exporter of tropical hardwood logs in the world. PNG’s economy has performed well over the past decade. The International Monetary Fund noted in its 2011 report that PNG has enjoyed 10 years of sound economic growth owing to greater political stability, sound macroeconomic policies and improved public finances.2 PNG’s gross domestic product grew at an average of more than 8% during the period 2009–2012 (Figure 1) peaking at more than 11% in 2011.3 Gross domestic product per capita growth averaged 5% for the same period 2009–2012.

1 The western half is the Indonesian provinces of Papua and West Papua. 2 International Monetary Fund. 2011. Papua New Guinea: 2011 Article IV Report. Washington, DC. 3 ADB. 2013. Key Indicators for Asia and the Pacific 2013. Manila.

Asian Development Bank. 6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines Tel +63 2 632 4444; Fax +63 2 636 2163; [email protected]; www.adb.org/evaluation

2

4. In the past, strong economic growth has been driven by high commodity prices. Construction of PNG liquefied natural gas (LNG) project, various spin-off investments, and broader investments in the domestic service sectors continued to be key drivers. After a decade of strong growth, the expected commencement of LNG will bring a further increase in mineral output and financial revenues over the near medium term. Offsetting these factors, PNG now faces a sharp slowdown in the non-mineral sector as construction winds down on the large LNG project.4 The current challenges that PNG faces include ensuring that the high economic growth rates of the past can be sustained and that economic growth leads to a more inclusive development including converting mineral wealth into inclusive growth.5

5. Poverty and social indicators. As of 2012, PNG’s total population was estimated to be 7.17 million people, with about 13% living in urban centers. A large majority of the population (87%) live in the highland valleys which are isolated villages and lead subsistence lives based on a barter economy. Despite a rich resource base, PNG’s social indicators lie below those of other countries with similar levels of per capita income. Many people live in extreme poverty, with about 40% of the population living below the national poverty line.6 Between 1996 and 2010, material poverty rates are seen to be largely unchanged. Poverty is seen as more prevalent in rural areas in PNG—with the Highlands and rural Momase. The World Bank, drawing from the 2009–2010 PNG Household Income Expenditure Survey (HIES), determined that health and welfare have a long way to go in terms of improvement.7 The HIES data indicate that approximately 40% of the population consumed less than the minimum number of daily calories (2,200) and essential non-food items. More than 40% of children below 5 years of age are also estimated to suffer from moderate to severe chronic malnutrition.

4 International Monetary Fund. 2013. Staff Report for the 2013 Article IV Consultation. Washington, DC. 5 The critical constraints on future growth include the following: (i) weaknesses in governance and institutions,

particularly in regard to delivery of public services, maintaining law and order, controlling corruption, and managing land and land titles; (ii) poor infrastructure and infrastructure services particularly in the case of transport, electricity, and water supply; (iii) shortages of skilled human capital and poor and unequal access to affordable and quality education; and (iv) lack of and unequal access to affordable and quality health services. ADB. 2013. Papua New Guinea Critical Development Constraints. Manila.

6 World Bank. World Development Indicators database. http://databank.worldbank.org (accessed February 2014). 7 International Monetary Fund. December 2013. Staff Report for the 2013 Article IV Consultation. Washington, DC.

Box 1, as quoted from data compiled by the World Bank, p. 5.

Figure 1: Papua New Guinea's Gross Domestic Product Growth Rate, 2000–2012 (%)

Source: ADB. 2013 Key Indicators for Asia and the Pacific 2013. Manila.

-2.50

0.00

2.00

4.40

0.60

3.90

2.30

7.20 6.60 6.10 7.60

11.30

9.20

-4

-2

0

2

4

6

8

10

12

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

3

6. Notwithstanding its strong economic growth performance, PNG faces many development challenges. The country has considered that the global Millennium Development Goals (MDGs) would not be achievable and thus set its own national targets corresponding to the first seven global MDGs.8 Of these national targets, two are on track (Goals 1 and 4) and three are off track (Goals 5, 6 and 7), with mixed results for the remaining two (Goals 2 and 3).9 7. Fragility and governance. PNG was identified as a fragile and conflict-affected situations (FCAS) country since 2007 when ADB prepared the Achieving Development Effectiveness in Weakly Performing Country.10 The paper proposed that innovative approaches and flexible business processes be applied to FCAS countries. ADB adopted the harmonized country policy and institutional assessment 11 under ADB’s 2013 Fragile Conflict Affected Situation Operational Plan. 12 Since then, PNG has been no longer identified as an FCAS country.13 However, weak performance and fragility still exist because a country shows a small improvement in performance scores that is slightly above a cut-off point. In contrary, PNG is a member of the g7+ group of fragile countries, a voluntary association of countries that are or have been affected by conflict and are now in transition to the next stage of development.14 The weak governance on policies for social inclusion and equity, public sector management and institution, and enforcement of the rule of law and order is the main reason of fragility in PNG. 8. A survey of firms undertaken by the Institute of National Affairs found more than 80% of the respondents ranked provision of government services as “very poor,” “poor,” or “fairly poor.” To improve public service provision and the business and investment climate, the government needs to strengthen reforms that are monitored by suitable governance indicators. Financial sustainability remains a key development challenge for the government. 9. The weakness in governance arrangements lies in the country’s social structure and culture where there is extreme ethnolinguistic fragmentation of the society. Due to this fragmentation, there are no strong national political parties. The political system interacts with the underlying society’s social and cultural traditions. The boundary between politics and administration is blurred, and the interventions of legislators weaken the ability of the administration to act cohesively.15 Weak governance and institutions undermine all economic activity and the delivery of public services. The lack of fit between formal institutions and the underlying society became one of important generic obstacles to donor-supported governance reform and project implementation especially land acquisition. 10. Large portions of its population live in remote areas. This has made unit costs of service delivery high. Much of the rural forested hinterland where the local population resides is beyond

8 Footnote 7, p. 4. 9 Footnote 7, p. 34. 10 ADB. 2007. Achieving Development Effectiveness in Weakly Performing Countries. Manila. 11 The average of World Bank CPIA and ADB country performance assessment. 12 ADB. 2013. Operational Plan for Enhancing ADB’s Effectiveness in Fragile and Conflict-Affected Situations.

Manila. 13 Criteria for determining an FCAS country is quantitative CPIA cutoff of 3.2 or less, or the presence of a United

Nation and/or regional peace-keeping or peace-building mission during the past 3 years. PNG (CPIA = 3.3) is therefore not in the harmonized list of fragile situations of the World Bank.

14 http://www.g7plus.org/ 15 International Political Economy, Johns Hopkins School, USA. 2007. Although this view is consistent with the

findings of the reconnaissance mission carried out from 10 to 14 March 2014, IED received the other view that while political instability has certainly been an issue during the 1980s and 1990s, from the early 2000's until now the National Alliance Party and the People's National Congress have both become very dominant political parties in PNG. Together they have continued to rule PNG for more than 13 years. This issue will be further explored during the evaluation study.

4

the outreach of government and public services.16 The Organic Law legislated in 1995 had transferred many functions to the provinces and local governments without equipping them with the means to undertake these functions. However, it was commonly understood that whatever capacity the government had was further eroded and weakened by the law. The government was aware that the Organic Law has failed to achieve its primary role of effective service delivery to the people. This has led to calls by politicians and their constituents to repeal the law and develop a much simpler form of decentralization.17

2. Government Development Strategies and Plans 11. The government’s Medium Term Development Strategy (MTDS), 2003–2007 addressed the theme of recovery and development and focused on the objectives of good governance, export-driven economic growth, rural development, and poverty reduction. It was accompanied by a long-term poverty reduction strategy. The government updated the MTDS, 2003–2007 in 2005. In general, the objectives of the MTDS, 2005–2010 were the same as the previous MTDS, although it covered the social sector, improving basic education, and preventative health programs, addressing key social problems, and combating HIV/AIDS. The MTDS, 2005–2010 defined an indicator for achieving broad-based economic growth of 5% per annum. The MTDS, 2005–2010 highlighted a medium-term fiscal strategy and a medium-term debt strategy for the first time and prioritized prudent use of budgetary resources and sovereign debt as key objectives under the development program.18

12. The government has prepared a long-term 40-year development scenario—Vision 2050 which focuses on achieving a wealthy, healthy, and educated society by 2050. Vision 2050 defines PNG’s national development framework which aims to transform the current economy that is driven by the mining and energy sectors into one that can generate more inclusive economic growth. Under the long-term vision, the government has approved the PNG Development Strategic Plan (DSP), 2010–2030, which confirmed the national development priorities in Vision 2050 and elaborates the strategic framework for development planning. The DSP provides overarching guidance for all development agendas and sector strategies in PNG. The current and updated Medium Term Development Plan (MTDP), 2011–2015 is the first of four rolling plans that will implement the DSP.19 The MTDP, 2011–2015 is based on a result-oriented approach, supported by specific targets and financial estimates. The MTDP, 2011–2015 focuses on several key sectors that lay the foundations for PNG’s prosperity including unlocking land for development; normalizing law and order; and investing in infrastructure in primary, secondary, higher and technical education, health, utilities and transport sectors.

3. Asian Development Bank’s Country Strategies 13. Country strategies. The program for 2001–2003 focused on improved governance, private sector development, and improved social indicators. Three loans were approved during the period 2001–2003, amounting to almost $81 million, with the bulk of the loan ($70 million) allocated for PNG’s Public Service Program (Loan 1875). This program was complemented by 14 technical assistance (TA) projects, amounting to almost $8 million, where the majority of

16 The country is made up of four regions, 20 provinces which are the primary administrative divisions of the country, 89 districts, and 313 local level government areas.

17 Government of Papua New Guinea. 2011. Papua New Guinea Vision 2050. Port Moresby. 18 From Department of National Planning and Monitoring Secretary Joseph Lalang’s Message on “Introducing the

Medium Term Development Plan [2011–2015]”. 19 The National Research Institute Papua New Guinea. 2010. Papua New Guinea’s Development Performance 1975–

2008. NRI Monograph No. 41 (September). Port Moresby.

5

these funds were targeted at the health and public sector management sectors. The country assistance plan was followed by three CSP updates—2003–2006, 2004–2006, and 2005–2006. The CSP, 2006–2010 was designed to support the MTDS. ADB’s focus was: (i) public financial management, (ii) transport, (iii) private sector development, and (iv) health combating HIV/AIDS. A total of 12 loans were approved amounting to almost $400 million. A total of 18 TA projects were approved during the same period, amounting to almost $10 million. The strategic direction of the CPS, 2011–2015 is aimed at helping PNG plan and implement a successful transition through the conversion of its resource wealth into inclusive economic growth. Measures included the provision of support for infrastructure, private sector development, and regional cooperation (Attachment 1).20 14. Cross sectoral objectives. The common cross sectoral objectives in three ADB strategies and three updates are (i) economic growth, (ii) improvement of governance, (iii) private sector development, and (iv) improved social indicators. 15. Portfolio during the evaluation period. During the evaluation period, ADB support covered 31 projects and programs amounting to $1,424.62 million, and 48 TA projects totalling $34.43 million, including projects which approved before 2001 but completed after 2001. Over 94% or $1,350.62million of the allocation for projects and programs was directed to the public sector, while nonsovereign operations only accounted for $74.00 million. Sovereign funding is largely a blend of ADB’s ordinary capital resources and Asian Development Fund (ADF), about 55% of ordinary capital resources and 40% of ADF. The balance was co-financed by Australia, the Japan Fund for Poverty Reduction, and other sources (Attachments 2 and 3).

16. Figure 2 shows sectoral distribution of the sovereign programs, projects and technical assistance. The transport sector and information and communication technology (ICT) comprised a substantial share at 69% (covering 12 projects). The other major sectors in the ADB support program are energy (9% for three projects), health and social protection (6% for five projects), and public sector management (6% for one project). Twelve of the 31 projects have been completed and were rated as follows: four successful, seven partly successful, and one unsuccessful. Nine of these rated projects were subsequently validated and/or audited as follows: three successful, five partly or less than successful, and one unsuccessful. 17. For TA projects, transport and ICT received 27%, while public sector management as well as health and social protection interventions received over 22% and 19% respectively. Energy was allocated 16%, industry and trade with under 8%, agriculture and natural resources with more than 5%, education with under 2%, and finance under 1%. Twenty-four of the TA projects have been completed and rated as follows: 16 successful, 7 partly successful, and 1 unsuccessful.

20 ADB introduced some significant changes in ADB operations based on the mid-term review in 2009 for the 2006 CSP. ADB expanded operational areas to include the power sector and civil aviation, but little attention to the main social and human development issues was paid.

6

4. Other Key Development Partners’ Programs

18. For the period 2001–2012, PNG received $6,168 million in assistance from bilateral and multilateral development partners (Attachment 4). Australia is the largest donor—accounting for over 63% of support—with ADB coming in second and contributing 15%. Other significant partners were Japan, the European Union institutions, World Bank and New Zealand. Australia has provided aid to PNG since 1975, and it has been acknowledged the lead donor in the delivery of development assistance to the country. Australia, New Zealand, Japan, United States, World Bank and ADB formed a Pacific Region Infrastructure Facility in 2008 and demarcate areas of priority and cofinancing in transport, energy, and urban sectors. 19. An Australian share in development assistance to PNG for 2001–2012 is the largest, i.e., 63%. Australia has straddled several sectors: health, education, economic development, governance, humanitarian aid, and general development support. Public sector reform and management is one of the key thrusts for the country, with the other development partners undertaking specific subsector initiatives. The World Bank—which chairs the Consultative Group for PNG—oversees the economic development and macroeconomic stability component. Under the public sector reform program, the World Bank leaded initiatives for improving governance in the use of public funds and delivery of public services. The United Nations Development Programme has supported the World Bank in the governance sphere as well as taking the lead on promoting the environment agenda. The European Union institutions have invested heavily in water supply, governance (covering rule of law, and economic and institutional reforms), trade and regional integration, social cohesion and employment, skills development, disaster risk reduction, and agriculture and rural development. In the latter area, the European Union’s efforts are focused on improving technologies to mitigate climate change risks to food production and smallholder farming communities.

Figure 2: Sector Shares of ADB Support ($ million)

Projects and Programs Technical Assistance

ADB = Asian Development Bank, ANR = agriculture and natural resources, ICT = information and communication technology, PSM = Public Sector Management, WMS = water and other municipal infrastructure and services. Source: Asian Development Bank.

ANR 25.71 2%

Education 25.49 2%

Energy 129.00

9%

Finance 54.40 4%

Health & Social

Protection 95.50 6%

Industry & Trade 25.00 2%

Public Sector Mgt

70.00 5%

Transport & ICT

984.18 69%

WMS 15.35 1%

ANR 1.75 5% Education

0.60 2%

Energy 5.60 16%

Finance 0.23 1% Health &

Social Protection

6.48 19%

Industry & Trade 2.80 8%

Public Sector Mgt 7.78 22%

Transport & ICT 9.19 27%

7

5. Findings of the previous Country Assistance Program Evaluation 20. The first CAPE for PNG was completed in 2003, covering the period 1986–2002. Three issues were found to be central to PNG's development: (i) reforming the machinery of government, (ii) managing the economy which is dualistic and highly dependent on international commodity prices, and (iii) delivering public services. Overall, the initial CAPE noted that ADB supported the right development areas, but issues related to the machinery of government could have been recognized and incorporated into the program at an earlier stage. Sector priorities were consistent with goals. However, most of the project activities were undertaken within a narrow sectoral context and had only weak linkages. It was not clear how sectors would contribute to an overall development goal. This weakness resulted in the lack of a structured and sequenced program, which lost potential synergy effects and only generated isolated and short-lived outcomes. 21. To take the country forward, the CAPE noted that government commitment is essential for any improvements in economic performance, poverty reduction, or the management of public resources and sectors. Without government commitment, external assistance would remain ineffective, as illustrated by the many development efforts that continued to lack sustained results. Government and development partners' efforts needed to concentrate on determining and implementing a development agenda that addressed the urgent need to create jobs to stem poverty. The CAPE also recommended ADB’s different sectors to fuse together into a well-knit PNG strategy. C. Evaluation Methodology 1. Scope and Coverage 22. The CAPE will cover the period from 2001 to June 2014. The CAPE will be guided by the Revised Guidelines for the Preparation of Country Assistance Program Evaluations and Country Partnership Strategy Final Review Validations.21 The common cross sectoral objectives of (i) economic growth, (ii) improvement of governance, (iii) private sector development, and (iv) improved social indicators (para. 17) will be assessed under the development impact criteria. Selected issues related to capacity development, governance, gender, and poverty reduction will be reviewed and assessed. 23. Four sector assessments covering energy, health, public sector management, and transport will provide inputs to the CAPE. Assessment of a project which covers several sectors and has been categorized as multisector will be included in infrastructure sectors by distributing its various components to the appropriate sectors. The sector assessments will identify issues and lessons from projects and policy initiatives. ADB performance in the sectors will be assessed, the reasons for their performance will be identified to draw lessons and provide recommendations for improving performance of future ADB support in different sectors. The CAPE will carry out perception surveys across government, donors, stakeholders, and focus group discussions will be conducted with beneficiaries in the field.

21 ADB. 2010. Revised Guidelines for the Preparation of Country Assistance Program Evaluations. Manila.

8

2. Key Issues and Questions 24. In addition to undertaking the standard assessments according the CAPE guidelines, the CAPE will specifically assess the role and nature of ADB support in light of the rapidly changing country context (Attachment 5). 25. Cross-sectoral objectives and development impact. Questions to be addressed are as follows:

(i) Were CPS objectives appropriate (whether ADB chose the right things to do)? (ii) What were the achievements on ADB’s contribution to development in terms of

achieving the CPS cross-sectoral objectives? (iii) What were the primary factors/constraints on the achievement of the cross-

sectoral objectives? (iv) How did ADB cooperate with other donors, especially Australia to achieve the

CPS cross-sectoral objectives? 26. Inclusion. The country’s robust macroeconomic performance is not being translated into reduction of poverty and inequality.22 ADB support for inclusive growth in PNG is centered on infrastructure, but the impact of these infrastructure interventions on inclusive growth is yet to be demonstrated. 23 Women continue to face severe inequalities in access to essential social services. The prevalence of gender-based violence restricts women's security and their opportunities to participate in economic and political systems. Questions to be addressed are as follows:

(i) Why were MDGs in PNG estimated not to be met by 2015? How did ADB support PNG to achieve MDGs?

(ii) What are lessons learned on inclusive growth in ADB’s operations for PNG? (iii) Was ADB’s support to the gender mainstreaming in projects targeted to promote

women's empowerment responsive and effective? Should ADB provide more support to tackle pervasive gender inequality?

27. Fragility and governance. To explore modalities and instruments of assistance, and take flexible business processes in a holistic manner, ADB prepared the Achieving Development Effectiveness in Weakly Preforming Country and Operational Plan for ADB’s Effectiveness in Fragile and Conflict-Affected Situations for FCAS. In addition, ADB’s governance agenda has evolved to support the country in terms of strengthening governance and public sector management including integrity, transparency, and accountability. Questions to be answered in this area are as follows:

(i) What approaches for FCAS did ADB take in PNG and how those were effective? (ii) Although PNG is not an FCAS country now, weak performance and fragility still

exist in the country. What are the best methods to mainstream fragility-sensitive approaches in ADB’s country strategies and operations?

(iii) How has ADB’s governance agenda evolved to address the issue of weak absorptive capacity of government and the constraints revolving around the country’s social structure and culture?

22 The Gini index for per capita consumption increased from 44% in 1996 to 54% in 2009–2010. 23 IED. 2014. Thematic Evaluation Study: ADB’s Support for Inclusive Growth. Manila: ADB.

9

(iv) What lessons and good practices have emerged from ADB support that should be considered in the next CPS?

28. Climate change. PNG belongs to the nine countries and two regions (Caribbean region and Pacific) that were selected to participate in the pilot program for climate resilience. The program is part of the strategic climate fund, a multi-donor trust fund within the Climate Investment Funds. The program provides incentives for scaled-up action and transformational change to integrate climate resilience into national development plans in a way that is consistent with poverty reduction and sustainable development goals. Climate resilience is a pressing challenge for the country as the economy depends on natural resources such as agriculture, forestry, fisheries, tourism, and minerals. Questions to be addressed in this area are as follows:

(i) What are the approaches of the government, ADB, and other donors in PNG to adapt to the risks of natural disaster management and climate change?

(ii) How should strategies, policies and programs for climate change resilience be integrated in the next ADB CPS?

29. Land acquisition. About 97% of the total land area is held under a customary land tenure system which underpins the kinship ties and social institutions. The difficulty acquiring title to use customary land for business activities and development projects has been widely reported as a major constraint to the development of infrastructure, commercial projects, and urban real estate. The key questions to be addressed in this area are as follows:

(i) What kind of assistance has been provided by ADB and other donors to help resolve land acquisition problems?

(ii) How has the government ensured transparency to the processes and procedures for acquiring and determining compensation of land?

(iii) How has ADB supported the government to comply with the safeguard policy?

3. Limitations 30. Independent evaluation missions to PNG will involve field visits and consultations and/or meetings with the relevant in-country stakeholders (executing agency, implementing agencies, project beneficiaries, etc.). These missions will be subject to field security and guidance from ADB and the United Nations.

D. Implementation Arrangements 31. The CAPE will be prepared by a team comprising a senior evaluation specialist/team leader (T. Yokota), supported by a senior evaluation specialist (J. Asquith), an evaluation officer (L. Guevara), and an associate evaluation analyst (E. Li-Mancenido). Sector assessments will be carried out by the team led by T. Yokota for transport and public sector management, J. Asquith for health, and L. Guevara for energy. To support the team, consultants will be engaged individually following ADB’s Guidelines on the Use of Consultants (2010, as updated from time to time). The ongoing IED studies such as governance and role of technical assistance will be inputs to the CAPE. To ensure quality, internal and external peer reviews will be implemented. 32. It is proposed the CAPE activities be carried out in accordance with the following preliminary schedule:

10

Activity Tentative Schedule Approval of the EAP II May 2014 Evaluation mission (sector level) I August 2014 First draft of sector assessments II September 2014 Evaluation mission (country level) II September 2014 One stop review meeting II December 2014 Interdepartmental circulation II January 2015 Government review I February 2015 CAPE consultation mission III February 2015 Director General-level meeting I March 2015 Editing IV March 2015 IED Director General approval I April 2015

E. Dissemination 33. The final CAPE with management response and chair’s summary of a Development Effectiveness Committee meeting, and two-page learning curves of the CAPE, will be disseminated after the Development Effectiveness Committee meeting. Attachments: 1. Major Thrusts of Country Strategies and Programs in Papua New Guinea 2. Approved ADB Loans and Grants to Papua New Guinea, 2001–2013 3. Approved ADB Technical Assistance to PNG, 2001–2013 4. Development Assistance to Papua New Guinea by Commitment, 2001–2012 5. Evaluation Framework for the Country Assistance Program

MAJOR THRUSTS OF COUNTRY STRATEGIES AND PROGRAMS IN PAPUA NEW GUINEA

CAP 2001–2003 CSPUs 2003–2005,

2004–2006, 2005–2006 CSP 2006–2010 CPS 2011–2015 Overall Objectives Improve governance

Promote private sector development

Improve social indicators

Reduce poverty by: - improvements to

quality and efficiency of public sector management;

- development of an enabling environment for PSD, especially support for SMEs, and

- improvement in social conditions

Support an environment conducive for the private sector

Enhance supply of and demand for quality basic social services

Promote effective development processes

Help transition economy through the conversion of its resource wealth into inclusive economic growth

Create livelihood opportunities Improve access to basic services

Agriculture Strengthen support services for smallholders

Pursue agro-industry interventions through partnerships with private sector

Assist government in developing strong regulatory agencies, especially for natural resource exploitation

Carry over and build on initiatives through rural development and coastal area management project/s

Subsume under PSD Extend support where needed

Transport Infrastructure Assist government in rehabilitating and maintaining roads through: - physical

infrastructure investment;

- private sector participation in infrastructure financing, and

- institutional structures anchored on cost-recovery and road maintenance management systems

Restore maritime sector through:

- sustainable maintenance system/s;

Carry over and build on initiatives with: - establishment of

road authority and key projects in Highland and Southern Regions (for road transport) and

- development of local water transport facilities, shipping routes, and establishment of institutional framework (for water transport)

Carry over and build on institutional reforms, especially effective financing and management arrangements for road maintenance and coastal shipping services

Scale up support for infrastructure development and management—covering road transport, maritime transport, civil aviation and power—including finding alternative mechanisms for maintenance and financing of operating costs (e.g., proposed infrastructure fund)

12 Attachment 1

CAP 2001–2003 CSPUs 2003–2005,

2004–2006, 2005–2006 CSP 2006–2010 CPS 2011–2015 - long-term financing

mechanisms, and - appropriate policy

and regulatory environment

Finance Support development of microfinance services for viable enterprises and savings services for general population

Carry over initiatives Subsume under PSD Subsume under PSD

Social Infrastructure Health and HIV/AIDS

Health and Education

Assist government in improving outreach and quality of health services along with implementing policy reforms

Provide support for vocational and technical education, and skills development for employment

Progress WSS projects

Help address environmental protection institutional framework

Focus assistance to health-related MDG goals

Collaborate with other development partners on broad human development strategy

Carry over support for adult and informal education, and skills development

Carry over WSS projects

Include explicit gender orientation in projects and sector work

Explore alternative approaches to strengthening rural health service delivery

Expand support to primary health services to rural populations

Develop human resource capacities directly linked to infrastructure projects in specific sectors

Governance Public Financial Management

Work with other development partners in progressing public service reforms, focusing on financial management (e.g., Financial Management Project, Public Service Program) as well as provincial and local management capacities

Carry over initiatives, including capacity building and institutional development initiatives at sector levels

Strengthen financial controls and compliance as a driver of broader public sector reforms: - performance of

public enterprises; - revenue

management; - functioning

relationships between national, provincial and local governments, and

- participatory budget preparation and monitoring processes

Continue to strengthen PFM: - government-wide

framework for enhancing fiscal management;

- governance standards for infrastructure projects, statutory authorities, SOEs and executing agencies

Support development planning in key line agencies

Attachment 1 13

CAP 2001–2003 CSPUs 2003–2005,

2004–2006, 2005–2006 CSP 2006–2010 CPS 2011–2015 Private Sector Development

Help establish an enabling environment supportive of: - market access; - skills development; - microfinance, and - alleviation of barriers

to informal sector/s Foster PPPs, especially in infrastructure development and financial sector

Carry over initiatives Progress diagnostic work on business environment: - legal framework

for business; - role and

performance of SOEs;

- PPPs, and - private

participation in infrastructure development (particularly transport); energy; financial services, and commercial agricultural production and extension (building on nucleus agro-enterprises model)

Support government in implementing PPP policy to facilitate private participation in infrastructure, especially in major grid development

Help introduce competition to SOE sectors through a national CSO framework

Improve the enabling environment for the private sector

Develop financial solutions through innovative contractual and financial structures to encourage private company participation, enhance management expertise, and improve corporate governance

Encourage private participation in financial services sector:

Mainstream “access to finance” through innovative solutions: - secured

transactions reform to allow use of moveable assets for collateral, and

- mobile banking Regional Cooperation Regional

Cooperation and Integration

Promote the country’s participation in: - intra and inter-

regional trade relations;

- cooperative arrangements in aviation and shipping, and

- coordinated approaches to

Strengthen regional and subregional cooperation and integration: - improved road

connectivity between PNG and Indonesia, and

- Pacific regional cooperation

Promote growth corridor approach between PNG and Indonesia

14 Attachment 1

CAP 2001–2003 CSPUs 2003–2005,

2004–2006, 2005–2006 CSP 2006–2010 CPS 2011–2015 fisheries management

Environment Integrate across

sector projects Climate- and hazard-proof infrastructure projects

Prioritize clean energy solutions

Explore market mechanisms that can bring in additional carbon revenue (e.g., CDM)

Gender Integrate across

sector initiatives Mainstream in all sector projects

CAP = country assistance plan, CDM = Clean Development Mechanism, CSO = community service obligation, CSP = country strategy and program, CSPU = country strategy and program update, CPS = country partnership strategy, MDG = Millennium Development Goals, PFM = public financial management, PNG = Papua New Guinea, PPP = public private partnership, PSD = private sector development, SME = small and medium-sized enterprise, SOE = state-owned enterprise, WSS = water supply and sanitation. Sources: Asian Development Bank.

APPROVED ADB LOANS AND GRANTS TO PAPUA NEW GUINEA, 2001–2013 Loan/ Grant No. Project Name

Fund Source

Amount ($ million)

Date Approved

Closing Date

PCR Rating

PVR/ PPER/ PPAR Rating

Agriculture and Natural Resources L1925 Coastal Fisheries Management and

Development ADF 5.7 24-Oct-02 18-Sep-08 PS PS

L1889 Nucleus Agro-Enterprises ADF 5.9 18-Dec-01 21-Nov-07 PS PS L1656 Fisheries Development OCR 6.5 11-Dec-98 12-Dec-03 S L1652 Smallholder Support Services Pilot

Project OCR 7.6 10-Dec-98 30-Apr-09 S S/S

Total 11.6 Education G3632 Employment-Oriented Skills

Development Other

s (AUS)

5.5 20-Feb-01 25-Sep-08a

L1706 Employment-Oriented Skills Development

ADF 20.0 28-Oct-99 25-Sep-08 PS LS

Total 25.5 Energy

2713 Town Electrification Investment Program - Tranche 1

OCR 40.9 06-Dec-10 01-Jun-16

2714 Town Electrification Investment Program - Tranche 1

ADF 16.4 06-Dec-10 01-Jun-16

G0288 Improved Energy Access for Rural Communities

Others (NZ)

2.5 19-Apr-12 30-Jun-15

G9163 Improved Energy Access for Rural Communities

JFPR 2.5 19-Apr-12

L2998 Port Moresby Power Grid Development

OCR 51.7 26-Apr-13 31-Jul-17

L2999 Port Moresby Power Grid Development

ADF 15.0 26-Apr-13 31-Jul-17

Total 129.0 Finance L2686 Microfinance Expansion ADF 13.0 28-Oct-10 30-Jun-18 G0226

Microfinance Expansion

Others

(AUS)

6.0 28-Oct-10

L1768 Microfinance and Employment ADF 9.6 19-Oct-00 18-Nov-10 S S L1703 Financial Management OCR 25.8 21-Oct-99 30-Sep-08 U U

Total 54.4 Health and Social Protection G0042 HIV/AIDS Prevention and Control in

Rural Development ADF 22.0 25-Apr-06 31-Dec-14

L2785 Rural Primary Health Services Delivery

ADF 20.0 09-Sep-11

G0259 Rural Primary Health Services Delivery

Others

(AUS)

40.0 30-Sep-11

G9113 Lae Port Livelihood and Social Improvement

JFPR 1.5 18-Dec-07 11-Jun-14

L9130 Extending the Socioeconomic Benefits of an Improved Road Network to Roadside Communities

JFPR 2.0 02-Mar-09 02-Sep-13

L1518 Health and Sector Investment Program

ADF 10.0 20-Mar-97 13-Feb-04 PS

Total 95.5 Industry and Trade L2591 Pilot Border Trade and Investment ADF 25.0 27-Nov-09 30-Jun-18

16 Attachment 2

Loan/ Grant No. Project Name

Fund Source

Amount ($ million)

Date Approved

Closing Date

PCR Rating

PVR/ PPER/ PPAR Rating

Development Total 25.0 Public Sector Management L1875 Public Service Program OCR 70.0 12-Dec-01 31-Mar-05 PS PS

Total 70.0 Transport and ICT L1709 Road Maintenance and Upgrading

(Sector) OCR 63.0 16-Nov-99 19-Nov-10 PS

L1754 Rehabilitation of the Maritime Navigation Aids System

OCR 19.8 12-Sep-00 23-Oct-08 S S

L2079 Community Water Transport ADF 19.0 25-Mar-04 30-Jun-13 L2242 Road Maintenance and Upgrading

(Sector) (Supplementary Loans) OCR 35.0 29-Jun-06 07-Feb-13 PS

L2243 Road Maintenance and Upgrading (Sector) (Supplementary Loans)

ADF 18.0 29-Jun-06 07-Feb-13

L2398 Lae Port Development Project OCR 60.0 18-Dec-07 30-Jun-14 L2399 Lae Port Development Project ADF 40.0 18-Dec-07 30-Jun-14 L8237 Lae Port Development Project OPEC 6.0 G0102 Lae Port Development Other

s (HIV/ AIDS)

0.8 18-Dec-07

L2496 Highlands Region Road Improvement Investment Program - Tranche 1

ADF 71.2 22-Dec-08 29-Jun-15

L2497 Highlands Region Road Improvement Investment Program - Tranche 1

ADF 28.8 22-Dec-08 29-Jun-15

7302/ 2576

Digicel (PNG) Limited OCR 25.0 16-Nov-09 23-Nov-09

L2588 Civil Aviation Development Investment Program - Project 1

OCR 25.0 01-Dec-09 30-Jun-14

L2589 Civil Aviation Development Investment Program - Project 1

ADF 50.0 01-Dec-09 30-Jun-14

L2590 Civil Aviation Development Investment Program - Project 1

ADF 20.0 01-Dec-09 30-Jun-14

7328/ 2745

Bemobile Limited OCR 49.0 25-Mar-11 31_Dec-11

L2784 Bridge Replacement for Improved Rural Access Sector

ADF 50.0 28-Sep-11 30-Jun-17

L2783 Bridge Replacement for Improved Rural Access Sector

OCR 40.0 28-Sep-11 30-Jun-17

L2803 Lae Port Development (Additional Financing)

OCR 85.0 10-Nov-11 31-Dec-14

L2804 Lae Port Development (Additional Financing)

ADF 4.1 10-Nov-11 31-Dec-14

L2978 Maritime and Waterways Safety Project (SF)

ADF 41.5 18-Dec-12 30-Sep-18

L3069 Civil Aviation Development Investment Program-Tranche 2

OCR 115.0 25-Nov-13 30-Sep-17

L3070 Civil Aviation Development Investment Program-Tranche 2 (SF)

ADF 15.0 25-Nov-13 30-Sep-17

L3076 Highlands Region Road Improvement Investment Program-Tranche 2 (SF)

ADF 40.0 04-Dec-13 30-Jun-18

L3077 Highlands Region Road Improvement Investment Program-Tranche 2

OCR 69.0 04-Dec-13 30-Jun-18

Total 984.2

Attachment 2 17

Loan/ Grant No. Project Name

Fund Source

Amount ($ million)

Date Approved

Closing Date

PCR Rating

PVR/ PPER/ PPAR Rating

Water and other Municipal Infrastructure and Services L1812 Provincial Towns Water Supply and

Sanitation ADF 15.4 14-Dec-00 12-Nov-08 PS LS

Total 15.4 Grand Total 1,424.62 ADB = Asian Development Bank, ADF = Asian Development Fund, AUS = Australia, G = grant, JFPR = Japan Fund for Poverty Reduction, L = loan, LS = less than successful, No. = number, NZ = New Zealand, OCR = ordinary capital resources, OPEC = Organization of Petroleum Exporting Countries, PCR = project completion report, PPAR = Project/Program Performance Audit Report, PPER = project/program performance evaluation report, PS = partly successful, PVR = project validation report, S = successful. a Closing date for the grant is not available but we alternatively used information from the accompanying loan. Sources: ADB Loans, Technical Assistance, and Equity Approvals database.

APPROVED ADB TECHNICAL ASSISTANCE TO PAPUA NEW GUINEA, 2001–2013

TA No. TA Name Type TASF JSF Others Other

Sources Total Date

Approved Closing

Date TCR

Rating TPER Rating

Agriculture and Rural Development 4055 Agriculture and Rural Development PP 1,000,000 - 1,000,000 19-Dec-02 31-Dec-05 S 8362 Strategic Program for Climate Resilience

Implementation PP 750,000 SCF-

PPCR 750,000 02-May-13 19-Jun-14

Total 1,000,000 750,000 1,750,000

Education 4109 Literacy is for Everyone PP 600,000 - 600,000 09-May-03 31-Dec-06 PS

Total 600,000 600,000 Energy

8264 Implementation of the Electricity Industry Policy

PA 1,000,000 - 1,000,000 10-Dec-12

7783 Port Moresby Power Grid Development PP 1,200,000 - 1,200,000 28-Feb-11 31-Mar-13 7113 Power Sector Development PP 1,200,000 - 1,200,000 08-Aug-08 31-Mar-12 4932 Power Sector Development Plan AD 500,000 - 500,000 25-May-07 30-Jun-09 S 4710 PNG Gas Project PP 1,000,000 - 1,000,000 02-Dec-05 31-Oct-08 PS 3736 Gas Pipeline Development AD 700,000 - 700,000 10-Oct-01 31-Dec-03 PS

Total 3,900,000 1,700,000 5,600,000 Finance 8515 Supporting Anti-Money Laundering and

Combatting the Financing of Terrorism in Papua New Guinea

PA 225,000 - 225,000 14-Nov-13 30-Jun-15

Total 225,000 225,000 Health and Social Protection

7400 Strengthening Rural Primary Health Services Delivery (Supplementary)

PP 90,000 NZ 90,000 04-May-10 31-Mar-12

7400 Strengthening Rural Primary Health Services Delivery (2nd Supplementary)

PP 75,000 - 75,000 22-Nov-10 31-Mar-12

7400 Strengthening Rural Primary Health Services Delivery

PP 600,000 - 600,000 04-Dec-09 31-Mar-12

4798 Demographic and Health Survey in Papua New Guinea (Supplementary)

AD 213,523 UNICEF 213,523 30-Aug-07 31-Dec-09

4798 Demographic and Health Survey in Papua New Guinea

AD 250,000 1,982,000 AUS/NZ 2,232,000 07-Jul-06 31-Dec-09 S

4798 Demographic and Health Survey in Papua New Guinea (Supplementary)

AD 107,000 UNFPA 107,000 24-Oct-06 31-Dec-09

4882 Health Sector Support AD 850,000 - 850,000 04-Dec-06 31-May-10 S

4583 HIV/AIDS Prevention and Control in Rural Development Enclaves

PP 150,000 - 150,000 28-Apr-05

4208 Establishment of Pilot HIV/AIDS Care AD 450,000 - 450,000 30-Oct-03 31-Dec-08 PS

Attachm

ent 3 19

TA No. TA Name Type TASF JSF Others Other

Sources Total Date

Approved Closing

Date TCR

Rating TPER Rating

Centers 3660 Health Policy Support (Supplementary) AD 242,000 AUS 242,000 21-May-02 31-Dec-03 4057 Health Sector Development Program PP 500,000 - 500,000 19-Dec-02 30-Nov-03 S 3660 Health Policy Support AD 220,000 - 220,000 30-May-01 31-Dec-03 S 3762 Health Sector Review AD 250,000 - 250,000 06-Nov-01 31-Dec-03 S 3827 Gender and Population Project PP 500,000 - 500,000 21-Dec-01 30-Jun-03 U

Total 895,000 2,950,000 2,634,523 6,479,523 Industry and Trade 7393 Trade and Investment Institutional

Strengthening CD 900,000 - 900,000 27-Nov-09 31-Dec-14

7125 Pilot Border Trade and Investment Development

PP 900,000 - 900,000 11-Sep-08 31-Dec-09

4722 Policies for Private Sector Development AD 1,000,000 - 1,000,000 09-Dec-05 31-Dec-09 S Total 900,000 1,900,000 2,800,000

Public Sector Management 8213 Supporting Public Financial Management,

Phase 2 CD 1,000,000 - 1,000,000 12-Nov-12

7782 Facilitating Public-Private Partnerships CD 800,000 JFPR 800,000 23-Feb-11 30-Apr-14 7427 Supporting Public Financial Management CD 1,000,000 - 1,000,000 10-Dec-09 31-Dec-12 S 7205 Support for Development Planning AD 225,000 - 225,000 11-Dec-08 30-Sep-10 S 4947 Support for Public Expenditure Review

and Rationalization AD 500,000 - 500,000 29-Jun-07 30-Apr-09 PS

4829 Preparing Supplementary Financing for the Financial Management Project

PP 150,000 - 150,000 23-Aug-06 31-Mar-07

4388 Strengthening the Capacity of the Parliamentary Accounts Committee

AD 300,000 200,000 AUS 500,000 10-Sep-04 Terminated/Cancelled

3946 Improving Economic and Social Statistics AD 800,000 - 800,000 24-Oct-02 31-Dec-09 S 3667 Poverty Analyses for Socioeconomic

Development Strategies AD 650,000 - 650,000 06-Jun-01 31-Mar-04 PS

3812 Strengthening Public Sector Management AD 850,000 - 850,000 18-Dec-01 31-Mar-05 S 3444 Review of Mining and Hydrocarbon Tax

Regimes (previously listed as Fiscal Reform Studies)

AD 325,000

325,000 21-Dec-00 31-Dec-03 S

3280 Governance and Public Sector Reform PP 600,000 600,000 16-Oct-99 31-Mar-05 3280 Governance and Public Sector Reform

(supplementary financing) PP

381,000 AUS 381,000 26-Oct-00 31-Mar-05

Total 4,050,000 2,350,000 1,381,000 7,781,000 Transport 8123 Regulating and Sustaining Road

Transport PA 1,000,000 - 1,000,000 25-Jul-12 30-Mar-15

7874 Major Bridges Study PA 800,000 - 800,000 28-Sep-11 30-Jun-14

20

Attachm

ent 3

TA No. TA Name Type TASF JSF Others Other

Sources Total Date

Approved Closing

Date TCR

Rating TPER Rating

7869 Maritime and Waterways Safety PP 600,000 JFPR 600,000 20-Sep-11 31-Dec-12 7594 Bridge Replacement for Improved Rural

Access PP 800,000 - 800,000 03-Sep-10 31-Jan-12

7755 Second Community Water Transport Sector

PP 225,000 - 225,000 22-Dec-10 15-Jul-11 S

7420 Improving Road User Charges and Private Sector Participation in Road Development

PA 800,000 - 800,000 08-Dec-09 30-Nov-13

7263 Civil Aviation Development Investment Program

PP 225,000 - 225,000 08-Apr-09 31-Dec-10

7214 National Transport Development Plan (2011-2020)

AD 0 700,000 - 700,000 16-Dec-08 S

4793 Lae Port Development-Tidal Basin Phase I (Supplementary)

PP 160,000 - 160,000 08-Jun-07 31-Oct-08

4945 Highlands Highway (Southern Highlands and Enga Provinces Network) Rehabilitation

PP 500,000 - 500,000 27-Jun-07

4793 Lae Port Development-Tidal Basin Phase I

PP 900,000 - 900,000 30-May-06 31-Oct-08

3716 Road Authority Development PP 700,000 - 700,000 11-Sep-01 30-Jun-09 S 3717 Southern Road Maintenance and

Upgrading PP 400,000 - 400,000 11-Sep-01 30-Jun-03 PS

3615 Community Water-Transport Project (formerly listed as Remote Water Transport Improvement Project)

PP 580,000 580,000 21-Dec-00 31-Jan-03

3619 Rehabilitation of the Maritime Navigation Aids System (grant cofinancing to be provided by AusAID as part of Loan 1754-PNG: Rehabilitation of the Maritime

AD 800,000 AUS 800,000 19-Dec-00 30-Jun-05

Total 3,210,000 4,580,000 1,400,000 0 9,190,000 Grand Total

3,180,000 15,080,000 6,165,523 34,425,523

AD = advisory, ADB = Asian Development Bank, AUS = Australia, CAREC = Central Asia Regional Economic Cooperation, CD = capacity development, EAKPF = e-Asia and Knowledge Partnership Fund, JFPR = Japan Fund for Poverty Reduction, JSF = Japan Special Fund, HS = highly successful, PA = policy and advisory, PP = project preparatory, PS = partly successful, S = successful, SCF-PPCR = Strategic Climate Fund - Pilot Program for Climate Resilience, SS= small-scale, TA = technical assistance, TASF = Technical Assistance Special Fund, TCR = technical assistance completion report, TPER = technical assistance performance evaluation report, U = unsatisfactory, UK = United Kingdom, UNICEF = United Nations Children’s Fund, UNFPA = United Nations Population Fund. Note: Changes in grouping and sector descriptions are done, with some projects re-classed from original ADB classification. These groupings will be used for portfolio and sector assessments. Re-classed are: the agriculture and natural resources sector is renamed as agriculture and rural development; there is a disaster management classification to include all flood related projects; and education, health and social protection are placed under the social sector. Sources: ADB loans, TA and equity approvals database; Independent Evaluation Department.

Attachment 4 21

DEVELOPMENT ASSISTANCE TO PAPUA NEW GUINEA BY COMMITMENT, 2001–2012

($ million) Country/Organization Amount % Share 1. Australia 3,858 62.5 2. Asian Development Bank 972 15.8 3. Japan 312 5.1 4. European Union Institutions 257 4.2 5. World Bank 208 3.4 6. New Zealand 182 3.0 7. Global Fund 143 2.3 8. United Nations Agencies 78 1.3 9. Germany 33 0.5 10. United Kingdom 22 0.0

Subtotal (Top 10) 6,066 98.3 Rest of the World 102 1.7 Total (World) 6,168 100.0

Sources: Asian Development Bank Loan, Technical Assistance, Grant and Equity Approvals Database (accessed 16 February 2014); Organisation for Economic Co-operation and Development. Stat. http//stats.oecd.org (accessed 19 February 2014).

EVALUATION FRAMEWORK FOR THE COUNTRY ASSISTANCE PROGRAM EVALUATION

Evaluation Criteria

Areas of Evaluation and Subcriteria

Major Indicators and Areas of

Consideration Sources of Information

Evaluation Process

Strategic Positioning

Were sector/country strategies consistent with the sector/country’s development needs?

Did the ADB Country Strategies support the country’s National strategies?

Does ADB's sector strategy support the government’s sector development plan?

In the ADB’s strategies, how well did the documents explain (i) the policy issues targeted for support, (ii) the capacity-building efforts linked with policy issues, and (iii) how loans and TA projects supported the reconstruction and rehabilitation of the country?

What were the strategies to (i) address the capacity gap, (ii) to support emerging government institutions, and (iii) to support the consolidation of peace and stability.

Were the ADB’s strategies in response to Mid-Term Strategy, MTS II, and Strategy 2020?

Does the country strategy recognize and suitably adjust to constraints on the government’s absorptive capacity?

Does the sector strategy adequately recognize and adapt to ADB’s comparative advantage (or lack thereof) in the sector?

Is there an appropriate strategy for concentrating or diffusing ADB's efforts geographically?

Is there an appropriate strategy for continuing or

Macroeconomic and social indicators for PNG.

Economic reform milestones.

Evolution of PNG’s development plans.

Evolution of ADB’s CSPs.

Coordination of activities and strategies of other development partners.

Government’s strategies.

Country economic reports.

CSP and CPS.

Desk review of all relevant documents.

Discussions with ADB regional department, government officials, development partners, and other stakeholders.

Attachment 5 23

Evaluation Criteria

Areas of Evaluation and Subcriteria

Major Indicators and Areas of

Consideration Sources of Information

Evaluation Process

dropping sub-sector programs, and linking TA and loans?

For coordination with development partners, were there appropriate strategies for participating in sector meetings or events, formal partnerships, co-financing, etc.?

Do the sector results in the country results framework have relevant indicators, benchmarks, deadlines, and targets? Is the program clearly and realistically linked to the outcomes that ADB wants to influence?

Relevance Are ADB’s sectoral programs in PNG generally consistent with ADB’s sector and country strategies, or did they deviate significantly?

Does the program show a pattern of good, adequate, or poor design?

Does the program show a consistency of adequate or inadequate DMFs? Do the DMFs have relevant indicators with baselines, targets and deadlines? Are project level DMFs consistent with Sector Results Frameworks?

Assessment of policies, economic and political conditions and financial management of the transport sector.

Changes in ADB strategy areas of intervention.

Lessons and findings identified in PCRs and PPERs and TPERs.

DMFs.

CSPs, CSPUs, CPS, COBP, RRPs, TA reports, PCRs, PPERs, TPERs, and ADB databases.

Discussions with development partners, government, and other stakeholders.

Desk review of all relevant documents.

Discussions with ADB Regional Department, government officials, and development partners.

Efficiency What is the percentage of projects with EIRRs<12% and >18%? What is the weighted average EIRR?

How well was the portfolio administered? Did ADB’s bureaucratic processes cause systematic delays or other problems? Have ADB’s bureaucratic processes improved?

Implementation delays and cost overruns, disbursement issues if any.

Timeliness and frequency of data and collection, including timeliness.

Quality of data collected.

Timeliness and

PCRs, PPERs, PPRs Executing agency completion reports.

Back-to-office reports.

Discussion with government agencies and ADB regional departments.

Desk review of relevant documents.

Independent evaluation mission (IEM) and field observations and discussions.

24 Attachment 5

Evaluation Criteria

Areas of Evaluation and Subcriteria

Major Indicators and Areas of

Consideration Sources of Information

Evaluation Process

adequacy of counterpart funding.

Procurement issues including timeliness and transparency.

Effectiveness Has the program consistently delivered the outputs and produced the outcomes expected in project DMFs? Did/will the program achieve its outcomes as expected in the SRF?

Did the program systematically contribute to or detract from other important outcomes that were not represented in the DMF or SRF? or results framework?

What policy changes occurred that were supported by ADB?

What capacity development efforts or organizational change occurred that was supported by ADB?

Fulfillment of stated objectives for completed projects.

Feasibility of achievement of objectives based on current projects.

Performance of ADB program in PNG.

Findings of PCRs, PPERs, TPERs, and PVRs.

ADB status reports, back-to-office reports and other ADB reports.

Literature reviews.

Desk review of relevant documents.

IEM field observations.

Key informant interviews with government staff, observers, development partners, and other stakeholders.

Sustainability For financial security of project entities (as applicable), what is the percentage of projects with positive FIRRs-WACC (or positive FNPVs), percentage of projects with negative FIRRs-WACC ( or negative FNPVs) and the weighted average differential between FIRR and WACC or weighted average FNPV respectively?

Does the program have a secure basis of planning and budgeting for recurrent costs?

Will the program be supported by adequate O&M?

Do pertinent organizations have adequate capabilities to

Availability of other funding and resources.

Availability and adequacy of funding allocated by project owners for operations and maintenance.

Financial internal rates of returns.

Assessment of political will of the government to support reforms.

Government ownership and commitment to projects and reforms.

PCRs, PPERs.

Macroeconomic indicators such as government’s fiscal position and budget.

Literature reviews.

Discussions with government officials and other development partners.

Desk review of relevant documents.

IEM and field observations.

Attachment 5 25

Evaluation Criteria

Areas of Evaluation and Subcriteria

Major Indicators and Areas of

Consideration Sources of Information

Evaluation Process

support the program?

How strongly is the government committed to capacity building, organizational changes, or policy reforms supported by the program?

Have there been any positive or negative policy changes that affected the program?

Did RRPs recognize risks to sustainability; efforts to mitigate risks; appropriate project design?

Development Impacts

For project-level impacts, have the projects/ programs consistently delivered the impacts expected in project DMFs?

For sector-level impacts, did /will the program achieve its impacts as expected in the SRF?

Were there any notable positive or negative impacts of the program that were not represented in DMFs or the SRF?

Are there any notable innovations in ADB projects, including unplanned and unadvertised ones that contributed to development results?

Are there any other notable contributions to poverty reduction or other country-wide development goals?

Contribution of ADB transport sector operations to sustainable economic growth, gender issues, poverty reduction, environment sustainability, and governance reforms.

RRPs and PPERs

Time series of macroeconomic indicators from government sources.

Discussions with government agencies and other stake holders.

Desk reviews of relevant documents

IEM and field interviews.

ADB = Asian Development Bank, CAP = country assistance plan, COBP = country operational business plan, COS = country operational strategy, CSP = country strategy and program, DMF = design and monitoring framework, EIRR = economic internal rate of return, FIRR = financial internal rate of return, MTS II = medium-term strategy II, O&M = operation and maintenance, PCR = project completion report, PPER = project performance evaluation report, PVR = project completion report validation report, RRP = report and recommendation of the President, SRF = sector results framework, TA = technical assistance, TPER = technical assistance evaluation report, WACC = weighted average cost of capital. Source: Independent Evaluation Department.