1. BASIC INFORMATION a. Basic project data Project title: Export Marketing and Quality Awareness Project (EMQAP) Project code: P-GH-AA0-027 Instrument number(s): 2100150010245 Project type: Lending Sector: Agriculture & Agro-industry Country: Ghana Environmental categorization (1-3):2 Processing Milestones Key Events Disbursement and Closing date Date approved: 13 July 2005 Cancelled amount: NIL Original disbursement deadline: 31 March 2012 Date signed: 29 July 2005 Supplementary financing: NIL Original closing date: 31 December 2011 Date of entry into force: 24 March 2006 Restructuring: Revised disbursement deadline: 31 March 2014 Date effective for 1st disbursement: 31 May 2006 Extensions (specify dates): Once (from 31 December 2011 to 30 December 2013 Revised closing date: 30 December, 2013 Date of actual 1 st : 8 December 2006 b. Financing sources Financing source/ instrument (MUA) Approved amount (MUA) : Disbursed amount (MUA) : Percentage disbursed (%): Loan: 17.00 14.98 88.14 Government: 1.84 1.07 58.21 Other (ex. Co-financiers): TOTAL: 18.84 16.06 85.22 Co-financiers and other external partners: Execution and implementation agencies: c. Responsible Bank staff Position At approval At completion Regional Director Sector Director Sector Manager Task Manager Alternate Task Manager PCR Team Leader PCR Team Members d. Report data PCR Date: 31 May 2014 PCR Mission Date: From: 7 April 2014 To:22 April 2014 PCR-EN Date: Evaluator/consultant: Tesfaye Teklu Peer Reviewer/Task Manager: PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS
22
Embed
PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS · of the Government of Ghana namely, the Ghana Poverty Reduction Strategy (GPRS for XX ), the Ghana Shared Growth and Development
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1. BASIC INFORMATION
a. Basic project data
Project title: Export Marketing and Quality Awareness Project (EMQAP)
The Bank was responsive to the Borrower (Government of Ghana) request for ADF loan to support
growth in horticulture production and export marketing. The project identification, preparation and
appraisal involved consultations with relevant stakeholders including prospective beneficiaries. The
Bank also factored lessons from its previous experiences in the country. However, the quality at entry
was not satisfactory as evident in late and slow start-up of implementation due to delays in meeting
some of the conditions for effectiveness.
The Bank enforced fiduciary requirements and safe guards. The procurement processes were mostly
done within acceptable timeframe. The exceptions were the longer period for recruitment of consultant
for the design and supervision of residue laboratory, demonstration centre and pack house. The Bank
supported development of staff capacity for improved delivery of services to beneficiaries. It also
supported establishment of the project’s M&E system and this was instrumental in enabling the
Borrower to meet its reporting requirements on schedule. ADF loan disbursement was slow at the
beginning but attained 80% at project closure.
The Bank facilitated the implementation of the project through supervision missions. As per the country
PCR report, the Bank fielded fifteen (15) implementation support missions. These missions reviewed
status of project implementation, identified constraints facing the project and proposed solutions, and
agreed on prioritized list of activities that should be implemented for the attainment of project
objectives. The Bank showed some flexibility in addressing problems/issues as evident in modifications
of the scope of the project following the mid-term review. The establishment of the AfDB Field Office
in Ghana assisted in more speedy resolution of problems encountered by the project. However, there
were few implementation challenges that arose mainly from frequent change in task managers (14 Task
Managers during the project implementation period).
Overall, the performance of the Bank in quality-at-entry and implementation effectiveness is rated 3
(satisfactory)
b. Borrower performance:
The Borrower participated in the formulation of the project was in line with its long-term development
strategy and policy. It signed on to long-term loan agreement and committed to fulfill loan conditions
including provision of counterpart funds. However, the design of the projects was not adequately
developed in terms of aligning the size of the project to its readiness for implementation. The quality at
entry was not satisfactory as evident in late and slow start-up of project implementation due to delays in
meeting some of the conditions for effectiveness.
The Borrower performed reasonably well in complying generally with the Bank’s fiduciary
requirements by preparing and submitting annual audited project financial statements and adhering to
the Bank’s disbursement regulations. It ensured project’s compliance with environmental safeguards. In
addition, the Borrower monitored the performance of the project (i.e. input/output performance,
procurement performance, and financial performance including disbursement) and reported on schedule
to the Bank. In addition, the Borrower was responsive to supervision recommendations as evident in
adjustments following the mid-term review. However, the Borrower was not in compliance with
meeting its commitment to the level of counterpart funding. Moreover, there was no adequate exit
strategy for financial sustainability after project closure.
Overall, the performance of the borrowers in design and readiness as well as implementation was
satisfactory (rated 3).
c. Performance of other stakeholders:
Among the key other stakeholders were the various consultants and contractors. The performance of
these providers was assed in the country PCR in terms of quality of work performed, timeliness of
execution of contracts, and quality of management and financial soundness. Performance rating of the
various providers varied but the majority scored satisfactory. These were few contracts terminated due
to non-performance or inability to rectify identified defects.
The PCR rates the performance of other stakeholders satisfactory (3) in line with the PCR.
5. SUMMARY OF OVERALL PROJECT PERFORMANCE
a. Overall assessment:
The project is rated satisfactory in its relevance. It achieved most of its output targets. Few beneficiary-
focused surveys whose methodology could not be verified indicate improvements in household
horticulture productivity, quality and income in the catchment areas. There was reasonable expectation
that the project would achieve its development objectives. There was some weakness in the project’s
economic worthiness over the course of the project implementation, which might be related to weak
operational efficiency and underutilization of key assets of the project. There was no adequate exit
strategy for financial sustainability, but there was notable progress towards institutional sustainability
and strengthening capacity, and environmental sustainability is satisfactory.
The PCREN rates satisfactory (3) the overall performance of the project in design, implementation and
results.
b. Design, implementation and utilization of the M&E (appreciation of the evaluator):
Often the M&E system designed and implemented for monitoring project performance is basic; it
focuses on collecting, analyzing and reporting performance in project activities and outputs (I/O
monitoring), procurement, and financial management including audit reports. Missing is systematically
monitoring project’s outcomes, which are necessary to assess performance towards achieving the
project’s objectives.
The performance of the basic M&E system was satisfactory. The project met all the scheduled reporting
requirements by producing 27 quarterly reports and seven annual report. In addition, the Project
submitted audit reports to the Bank within scheduled time frames. Missing was properly designed
monitoring of the project’s outcomes such as change in profitability of horticulture production and
exports to the beneficiaries.
PCREN rates 3 (satisfactory) in line with the PCR.
6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS
a. Lessons learned:
Inadequate time and resources devoted to project appraisal, which should be thorough, ensuring very wide
consultation and ownership.
Involvement of key government departments -- retention of the Department of Feeder Roads to supervise the
work of contractors was useful in ensuring quality and timeliness
b. Recommendations:
None for the PCR to adopt
7. COMMENTS ON PCR QUALITY AND TIMELINESS
The PCR was complete in coverage of the dimensions of evaluation (relevance, efficacy in output and outcome,
efficiency, sustainability, Borrower and Bank performance). Access to data/evidence was limited. Quality of
analysis in terms of rigor and objectivity was poor. There were notable inconsistencies between rating and
explanation.
Overall, the performance of the PCR is 2 (unsatisfactory).
8. SUMMARY OF THE EVALUATION This is a summary of both the PCR and IDEV ratings with justification for deviations/comments. Appropriate section of the PCR
Evaluation should be indicated in the last column in order to avoid detailed comments. The evaluator must provide a reasonable
explanation for each criterion the PCR rating is not validated by IDEV. Consequently, the overall rating of the project could be “equally
satisfactory”.
Criteria PCR PCREN Reason for disagreement/ Comments
RELEVANCE 4
Relevance of project development objective 4 4
Relevance of project design 4 3 The discussion in the PCR was not about the
design of the project, i.e. rationale for its
choice, its internal logic, the extent the
design was informed, assessment of its
realism, and adjustment in scope over the
whole project life cycle, and relevance ex
post. The PCR rating is unrelated to the text.
EFFECTIVENESS
Development objective (DO) 3 3
EFFICIENCY
Timeliness 2 2
Resource use efficiency 4 3
Cost-benefit analysis 3 3
Implementation progress (IP) 3 3 The PCR rates on a single metrics of time
overrun. The PCREN had limited access to
the relevant documents, and hence the IP
rating was based on some (not all) the
suggested IP criteria.
SUSTAINABILITY 3
Financial sustainability 3 2 The PCR is not addressing the issues of the
recognition of the extent of threat
Institutional sustainability and strengthening
of capacities
4 3 Considerable disconnect between the rating
and the text in the PCR
Ownership and sustainability of partnership 2 2
Environmental and social sustainability 3 3
OVERALL PROJECT COMPLETION
RATING
3.25
Bank performance: 3 3 The rating in the PCR was based on a
narrow set of issues as compared to the ones
suggested in the PCR guidelines. The
PCREN was not able to access the Bank’s
supervision reports and IPR.
Borrower performance: 3 3 The rating in the PCR was based on a
narrow set of issues as compared to the ones
suggested in the PCR guidelines. The
PCREN was not able to access the Bank’s
supervision reports and IPR.
Performance of other shareholders: 3 3 Referring mainly to consultants and
contractors
Overall PCR quality: 2
9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR
PERFORMANCE EVALUTION REPORT, IMPACT EVALUTION,
COUNTRY/SECTOR REVIEWS OR THEMATIC EVALUATION STUDIES:
- Project is part of a series and suitable for cluster evaluation
- Project is a success story
- High priority for impact evaluation
- Performance evaluation is required to sector/country review
- High priority for thematic or special evaluation studies (Country)
- PPER is required because of incomplete validation rating
Major areas of focus for future evaluation work:
a) Performance evaluation is required for sector/ country review
b) Cluster evaluation (institutional support)
c) Sector evaluation (budgetary support or public finance management reforms)
Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to
facilitate a), b) and/or c).
Division Manager clearance Director signing off
Data source for validation:
Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or email)
Documents/ Database reports
Attachment:
PCR evaluation note validation sheet of performance ratings
List of references
Appendice 1
PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings
PCR rating scale:
Score Description 4 Very Good – Fully achieved with no shortcomings 3 Good – Mostly achieved despite a few shortcomings 2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced 1 Poor – very limited achievement with extensive shortcomings
UTS Unable to score/rate NA Non Applicable
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
RELEVANCE Relevance of the project
development objective
(DO) during
implementation
4 4
Relevance of project
design (from approval to
completion)
4 3
OVERALL RELEVANCE SCORE
EFFECTIVENESS* Effectiveness in delivering outcomes
Sector goal -- increase
in export earning at
project closure, 15%
Modest gain recorded at national level
Increase in household
income of horticulture
farmers
Recorded substantial improvements
Effectiveness in delivering output
Established 4
demonstration Centres.
100%
Published and
distributed 43,000
guides and GAP
manual.
143%
Production &
Dissemination of
improved seeds &
planting material,
variable by crop
104.6%
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
One study on local and
regional markets for
horticultural crops
100%
181.45 km of feeder
roads, 181.45 km
44.5% The rate was XXX relative to the revised target level
One of the four pack
houses built
25.0%
4 demonstration centers
were established
100%
30,00 farmers sensitized
and mobilized into
groups
37.5% The activity was discontinued to avoid duplications
with another Government of Ghana project
(MiDA) 21,134 beneficiaries
trained in good
agricultural practices
260%
1471 staff trained 122%
130 export traders
trained
11%
Development objective (DO)
Development objective
rating
3
Beneficiaries
Beneficiary1
Beneficiary2
Unanticipated outcomes (positive or negative not considered in the project logical
framework) and their level of impact on the project (high, moderate, low)
Institutional
development
Gender
Environment & climate
change
Poverty reduction
Private sector
development
Regional integration
Other (specify)
EFFECTIVENESS OVERALL SCORE
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
EFFICIENCY Timeliness (based on the
initial closing date)
Resource used
efficiency
Cost-benefit analysis
Implementation progress
(from the IPR)
Other (specify)
OVERALL EFFICIENCY SCORE
SUSTAINABILITY Financial sustainability
Institutional
sustainability and
strengthening of
capacities
Ownership and
sustainability of
partnerships
Environmental and
social sustainability
Criteria Sub-criteria PCR
Work
score
IDEV
review Reasons for deviation/comments
BANK
PERFORMANCE Proactive identification and
resolution of problems at different
stage of the project cycle
3
Use of previous lessons learned
from previous operations during
design and implementation
3
Promotion of stakeholder
participation to strengthen
ownership
2
Enforcement of safeguard and
fiduciary requirements
3
Design and implementation of
Monitoring & Evaluation system
3
Quality of Bank supervision (mix
of skills in supervisory teams, etc)
3
Timeliness of responses to
requests
3
OVERALL BANK PERFORMANCE SCORE
BORROWER
PERFORMANCE
Quality of preparation and
implementation
3
Compliance with covenants,
agreements and safeguards
3
Criteria Sub-criteria PCR
work
score
IDEV
review Reasons for deviation/comments
Provision of timely counterpart
funding 2
Responsiveness to supervision
recommendations
3
Measures taken to establish basis
for project sustainability
2
Timeliness of preparing requests 3
OVERALL BORROWER PERFORMANCE SCORE 3
PERFORMANCE
OF OTHER
STAKEHOLDERS
Timeliness of disbursements by
co-financiers NA
Functioning of collaborative
agreements
NA
Quality of policy dialogue with
co-financiers (for PBOs only) NA
Quality of work by service
providers 3
Responsiveness to client demands ?
OVERALL PERFORMANCE OF OTHER
STAKEHOLDERS
DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING AND
EVALUATION (M&E)
Criteria Sub-criteria IDEV
Score Comments
M&E DESIGN M&E system is in place, clear,
appropriate and realistic
Monitoring indicators and
monitoring plan were duly
approved
Existence of disaggregated gender
indicator
Baseline data were available or
collected during the design
Other, specify
OVERALL M&E DESIGN SCORE
M&E
IMPLEMENTA-
TION
The M&E function is adequately
equipped and staffed
OVERALL M&E IMPLEMENTATION SCORE
M&E
UTILIZATION The borrower used the tracking
information for decision
OVERALL M&E UTILIZATION SCORE
OVERALL M&E PERFORMANCE SCORE
PCR QUALITY EVALUATION
Criteria PCR-EVN
(1-4) Comments
QUALITY OF PCR
1. Extent of quality and completeness of the PCR
evidence and analysis to substantiate the ratings of
the various sections
2 Covered all the evaluation dimensions, but often
incomplete in factoring required evaluation criteria
for rating.
2. Extent of objectivity of PCR assessment score 2 Empirical based for some of the evaluation criteria
such as project’s outputs and measures of
operational efficiency. Otherwise, limited
objectivity.
3. Extent of internal consistency of PCR assessment
ratings; inaccuracies; inconsistencies; (in various
sections; between text and ratings; consistency of
overall rating with individual component ratings)
2 Often notable divergence between text and ratings.
4. Extent of identification and assessment of key
factors (internal and exogenous) and unintended
effects (positive or negative) affecting design and
implementation
2 No such frame of analysis to uncover influential
factors affecting performance or result
5. Adequacy of treatment of safeguards, fiduciary
issues, and alignment and harmonization
3
6. Extent of soundness of data generating and
analysis process (including rates of returns) in
support of PCR assessment
2 Apparently, the PCR had limited access to
data/evidence necessary for sound technical
analysis as evident from the low-objectivity and
consistency in PCR ratings.
7. Overall adequacy of the accessible evidence (from
PCR including annexure and other data provided)
2 Limited and problematic for the PCREN
validation
8. Extent to which lessons learned (and
recommendations) are clear and based on the PCR
assessment (evidence & analysis)
1 Too many lessons that are not necessarily learned
from the project experience
9. Extent of overall clarity and completeness of the
PCR
2
Other (specify)
PCR QUALITY SCORE 2
PCR compliance with guidelines (PCR/OM ; IDEV)
1. PCR Timeliness (On time = 4; Late= 1) 4
2. Extent of participation of borrower, Co-financiers
& field offices in PCR preparation
3
3. Other aspect(s) (specify)
PCR COMPLIANCE SCORE 3.5
*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)
References
African Development Bank Group. 2014. Project completion report for public sector operations (PCR) --
Export Marketing and Quality Awareness Project (EMQAP).
African Development Bank Group. 2012. Staff guidance on project completion reporting and rating.
Quality Assurance Department (ORQR).
African Development Bank Group. 2011. Staff guidance on implementation progress and results (IPR) for
public sector operations. Quality Assurance Department (ORQR).
African Development Fund. 2005. Republic of Ghana: Export Marketing and Quality Awareness Project
(EMQAP). Appraisal Report. Agriculture and Rural Development, OCAR.
Government of the Republic of Ghana, Ministry of Food and Agriculture. Undated. Ghana: Project
completion report for the export marketing and quality awareness project.
African Development Bank. Undated. EMQAP – Project Results Assessment. IDEV
References
Note the following documents were not available to the PCREN: EMQAP Mid-Term
Review (September 2010) and the project’s IPR (April, 2014).