-
QUALITY OF SUPERVISION IN FY01/02 (QSA5)
ANNEXES
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Administrator26311Vol. 2
Administrator
Administrator
-
Quality of Supervision in FY01/02 (QSA5) 35
ANNEX 1: LIST OF QSA5 OPERATIONS
Bangladesh Dhaka Urban TransportBangladesh Export
DiversificationBangladesh Silk Development Pilot ProjectIndia AP
DPIPIndia Madras Water Supply IIIndia Malaria ControlNepal Basic
& Prim. Ed. IISri Lanka Mahaweli Restructuring
South Asia Region
Benin Labor Force Development China Chongqing Urban
EnvironmentBenin Private Sector China Fujian II HighwayCameroon
Public/Private Partnership, Growth China Qinba Mountains Poverty
ReductionCameroon SAC III Indonesia Indonesia - IIDPCameroon
Transport Sector Project Indonesia Kecamatan Dev FundCAR Fiscal
Consolidation Credit Indonesia Sulawesi Basic Ed.Cote d'Ivoire
Distance Learning - LIL Indonesia Sumatra Basic EducationEritrea
Ports Indonesia WATSALGabon Pilot Com. Infra. Lao, PDR Dist Upland
DevGhana Community Water II Mongolia Financial Sector Adjustment
CreditGuinea Capacity Building for Service Delivery Philippines
First National RoadsGuinea Village Community Support Program
Philippines LGU Finance & Development ProjectKenya Nairobi
Mombasa Road Philippines Mindanao Rural DevLesotho Ag. Pol &
Cap Bldg Philippines Trans Grid ReinforceLesotho Highland Water IB
Vietnam Banking System ModernizationMadagascar Micro Finance
Vietnam Coastal Wetl/Prot DevMozambique Education Sector Strategic
Program Vietnam Rural Transport IINigeria Second Primary Education
ProjectSenegal Ag. Export PromotionSenegal Agr.Srcves & Prod.
Orgs Argentina B.A.Urb.Tsp.Senegal Endemic Diseases Argentina Flood
ProtectionSenegal Public Serv. Info-Systems Modernization Argentina
Forestry/DVSenegal Urban Mobility Improvement Program Argentina
Water Sector ReformSierra Leone Transport Sector Pro Bolivia
Hydrocarbon Sector Social & Envir (LIL)Sierra Leone Urban Water
Supply Bolivia Integrated Child DevlTanzania Agric Research
(TARPII) Colombia Financial Markets DevelopmentTanzania PSAC 1
Colombia SANTAFE I (Water/Supply)Zambia Environmental Support
Program Colombia Urban Infrastructure
El Salvador Education ReformGuatemala Land Fund (APL)
Albania Power Trnsm & Dist Guatemala Priv Prtcptn Infr.
TAArmenia Elec Trnsm & Distr Honduras GEF Biodiversity
ConservationBulgaria Trade & Trans Facil in SE Europe Paraguay
Asuncion SewerageGeorgia Judicial Reform St. Lucia Poverty
Reduction FundLithuania Health Venezuela Millennium Science
InitiativeMoldova General Education Venezuela Pub Sec Leg&Adm
ModMoldova PSD 2Romania Petrol Sector RehabRomania Schools
Rehabilitation Algeria Privatization Assistance (LIL)Russian Fed.
Bureau of Econ Pol Algeria Rural EmploymentRussian Fed. Cap Mrkt
Dev Djibouti Intl. Road CorridorRussian Fed. Coal IAP Djibouti
Public Works/Soc DevRussian Fed. Fin Inst. Egypt, Arab Rep Prvt
Sect.& Ag. Dev.Russian Fed. Sust Forestry Pilot Egypt, Arab Rep
Pvt. Sect Tourism Inf & EnvTurkey Commodities Mkt Dev Iran,
Islamic Rep 2nd Primary Health Care & NutritionTurkey Marmara
Earthquake Emer Recon Lebanon General EducationUzbekistan
Enterprise Institution Building Tunisia Greater Tunis
SewerageUzbekistan Fin Inst Bldg West Bank and G Elec. Sec. Inv
& MgmtUzbekistan Tashkent Solid Waste West Bank and G Housing
Fiannce
Middle East & North Africa Region
East Asia & Pacific RegionAfrica Region
Europe & Central Asia Region
Latin America & Caribbean Region
-
36 A QAG Assessment
ANNEX 2: LIST OF MAIN CONTRIBUTORS
Staff Director Arun BanerjeeModerators Arun Banerjee, P. Hari
Prasad, Amnon Golan, Xavier Legrain, Saeed Rana
Logistics & Administration Brónagh Murphy, Ann Walters,
Leila Ghosh, Susan Crisostomo, Vinod Ghosh
Agueh, Florent Cortes, Maria Hume, Ian Mukherji, S. Socknat,
JamesAiyer, Sri-Ram Cuadra, Ernesto Iyer, P. Nawaz, Tawhid Sood,
DaveAli-Khan, Mohsin Dakolia, Maria Iyer, Vijay Nyman, Kari
Spencer, RichardAnson, Richard de St. Antoine, Jean Kamkwalala, J.
Okidegbe, Nwanze Steeds, DavidBabbar, Suman Deininger, Klaus
Knudsen, Odin Pannenbourg, Ok Stottman, WalterBajpai, Jit Del Mar
Pernia, Joseph Konigshofer, F. Pellegrini, Anthony Sundberg,
MarkBanerji, Bob Dethier, Jean-Jacques Legrain, Xavier Prouty,
Robert Swaroop, VinayaBerryman, Sue Douglas, James Lockheed,
Marlaine Rana, Saeed Swartzendruber, FredBhargouti, Shawki
Drysdale, Robert Loos, Jane Rasmusson, Lars Tharakan, GeorgeBlanc,
Marc Fardoust, Shahrokh Ludwig, Peter Rees, Colin Tuck, LauraBond,
James Feder, Gershon Lundebye, Stein Ridker, Ron Varma,
KeshavBromhead, Marjory-Anne Galli, Fabio Marino, Manuel Ringskog,
Klas Wam, PerBurnett, Nick Golan, Amnon Marquez, Patricio
Rosenblatt, David Westin, RichardCackler, Mark Goldberg, Joseph
McCarthy, Eugene Sara, Jennifer Wilson, MarkCarruthers, Robin
Halperin, Ricardo Mehta, V. Shah, Jitu Wu, Kin BingChanne, J.
Hammam, Sonia Meyerman, Gerald Shetty, Y.T. Xenakis, IreneChaudhry,
Y. Hari Prasad, P. Mitric, Slobodan Shimizu, Masaharu Zampaglione,
GuiseppeChopra, Ram Hasan, Parvez Mohadjer, Gisu Sinha, Tray
Zearley, TomChowohury, Yusuf Hayward, John Motte, Edouard Skelton,
Rama Zjip, WillemChurchill, Anthony Hecht, Robert Mukherjee, Hena
Skolnik, Richard
Environment Procurement M&EAbu-Haija, Ayman Graham, Douglas
I. Baratz, Bernard Dhingra, Subhash Mosse, RobertoAgossou, Hugues
Janeiro, Jose Mercier, Jean Roger Ginnz, Jean-Louis Sartorius,
RolfAlbert-Loth, Agnes Morel, Xiomara Rahill, Bilal Larrieu, Jorge
White, ElizabethBoyce, Daniel Ndiaye, Moustapha Rees, Colin Sethi,
HarbakshChen, Nancy Nyaga, John Singh, Hardyal Social
DevelopmentDaoud, Edward Olowo-Okere, Edward Vijayaverl, Vijay
Cruz, Maria
Schwarz, Luis Harris, BruceRennie, Keith
Alexieva, Nevena (ECCU5)
Bhatnagar, Bhuvan (EASES)
Mba-Kalu, Roseleen (MNCA3)
Sennhauser, Ethel (SASRD)
Vickers, Piers (SASEI)
Balachander, Jayshree (EASHD)
Broemmel, Michael (AFTQK)
Nobakht, Hoveida (OPCPD) Sta. Ines, Noel (EAPCO)
Villaluz, Maya (EASES)
Bapna, Manish (SASRD)
Dundar, Halil (ECSHD)
Nucifora, Antonio (YPP)
Tarade, Ljiljana (ECCHR)
Wasfy, Mona (OPCIL)
Bellew, Romemary (EASHD)
Haddad, Afef (MNACS)
Pahm, Mirriam (EASES)
Tumaliwan-Belizario, Ma. Bella (EACPF)
Benavides, Livia (LCSHE) Khan, Irshad (SASRD)
Quinones, Marie-Theresa (EASRD)
Umali, Dina (SASRD)
Beschorner, Natasha (EACNQ)
Lantin, Manuel (CGIAR)
Sanchez, Pedro (SASEI)
Upadhyay, Jagdish (EASHD)
Observers
Main Contributors
Specialized ReviewersFinancial Management
Panelists
-
Quality of Supervision in FY01/02 (QSA5) 37
ANNEX 3. FIFTH QUALITY OF SUPERVISION ASSESSMENT (QSA5)
APPROACH PAPER (updated December 9, 2002)
1. The Quality Assurance Group (QAG) hereby announces the
commencement of the Quality of Supervision Assessment (QSA5)
beginning on July 31, 2002. The following outlines the objectives,
scope and methodology that will be employed to carry out the
assessment.
Objectives
2. The objective of the QSA5 is as follows:
• Provide senior management with real-time feedback on the
overall efficiency and effectiveness of the Bank’s supervision
work.
• Evaluate the Bank’s experience in identifying and resolving
problems in the riskier
parts of its portfolio.
• Improve learning in key areas of supervision quality and
processes. Special Areas of Focus
3. Based on the experience and conclusions of the previous
assessments, QSA5 will highlight some special areas which have
particular significance in improving the emphasis on outcomes of
supervision and accountability. These are as follows:
• As before, QSA5 will include specialized reviews in four areas
– financial management, social development, environment, and
procurement. Previous assessments also identified monitoring and
evaluation as another area with systemic weaknesses and it has,
therefore, been decided to revisit this issue more systematically.
QSA5 will include a more in-depth review of the monitoring and
evaluation (M&E) aspects of supervision, using the RSA3
framework.
• Both QSA4 and the QSR identified some weaknesses in the
contribution of managers
in the supervision process. These include the lack of timely
guidance to supervision teams and insufficie nt focus on task team
skill mix, task team continuity and mentoring. QSA5 will review the
progress made in this area.
• Realism of PSRs has been a perennial issue and panelists will
be asked to review the
PSR ratings of the projects in the sample and provide their own
ratings and explain when there is a difference of opinion with the
task team.
• Borrower ownership and capacity building are major ingredients
for a projects
readiness to enter its operational phase in a sustainable
fashion and achieve its development objectives. Panelists will be
asked to examine the extent to which these factors are receiving
adequate attention during supervision.
-
38 A QAG Assessment
• Borrowers (project agency/Ministry) and other key
stakeholders(co-financiers, others with direct role in the project)
can provide important feedback to improve the quality of Bank
supervision and panels will, on a selective basis, contact them to
receive client feedback on the Bank’s performance in supervision.
Task teams selected for the QSA5 will be requested to provide the
names, phone numbers and e-mail addresses of Borrowers/stakeholders
who might be contacted.
Sample Selection and Coverage
4. QSA5 would assess Bank performance in supervision of its
portfolio during FY01 and FY02. To avoid potential overlap with OED
evaluations, the universe will exclude projects expected to be
closed by December 31, 2002. Also, to ensure comparability of
results across the sample, projects approved during FY01-02 will
not be included. With these exclusions, adjustment lending is
unlikely to be a part of the sampling universe. The resultant total
universe from which the sample would be drawn would be about 950
projects. QSA5 will sample 100 of these projects. To reduce the
burden on task managers of multiple assessments, QAG will excuse
task managers who have been assessed during the Quality at Entry
Assessment for FY02 (QEA5), the ESW assessment for FY02 and the
QSR.
5. In order to be able to draw conclusions about the differences
in the quality of supervision efforts between the relatively
healthier and riskier parts of the portfolio, the sample of 100
projects will be drawn on a random basis from three strata: 35
projects from the problem projects stratum, 35 projects from those
with two or more flags, and 30 from those with one or no flag (as
of July 1, 2002). QSA5 results will be statistically robust at the
Bank-wide level with less than 5% margin of error at 95% confidence
level. The margin of error for the three strata will be about 10%
at 90% confidence level. However, due to the limitation in sample
size, conclusions at the regional or network level will not be
sufficiently robust.
Methodology 6. First Stage Assessments. The assessment
methodology remains unchanged. Except for areas of special
emphasis, for which the guidance questionnaire is being suitably
modified, the major determinants of supervision quality as measured
by QSA5 would be:
• Focus on development impact. How well is supervision assessing
project impact and what is the relevance, timeliness and
effectiveness of Bank responses to issues?
• Attention to fiduciary and safeguard aspects. How adequately
is the Bank supervising financial management, environment and
social aspects, performance and progress monitoring, and
procurement and disbursements?
• Appropriateness of supervision inputs and processes. What is
the quality of the Bank’s supervision missions, relations with
other partners, and management oversight?
• Realism of project ratings and reporting . How realistic and
candid are project ratings, including ratings for achieving
development objectives and justifications for upgrading project
status?
7. For the selected projects, QAG will extract from the Bank’s
information systems copies of the SAR/PAD, legal documents,
supervision budgets (planned and actual), supervision TORs, PSRs,
BTOR, aide-memoires, follow-up letters and status of covenants.
However, the
-
Quality of Supervision in in FY01/02 (QSA5) 39
task manager/task team will retain the responsibility for
ensuring that packages obtained from these systems are
complete.
8. Panels for the First Stage Assessments will work in two or
three person teams, where the third panelist will normally be from
outside the Bank. Specialized experts in financial management,
environment, social development, procurement, and monitoring and
evaluation will assist the panels.
9. The QAG panelists will review the written documentation and
rate the operations using the Guidance Questionnaire as the basic
framework. The ratings would be based on a four point scale :
highly satisfactory, satisfactory, marginal, and unsatisfactory. In
using the questionnaire, panelists would be expected to explain the
basis for a rating of either best practice or less than
satisfactory. Specialized reviewers will contact the responsible
members of the task teams in their areas of interest, if necessary,
to seek clarification. Following the desk review, the panelists
would prepare a list of questions to be discussed during the task
team interview. The immediate supervisors and Country Directors may
be interviewed if considered appropriate. If the project is
supervised from a country office, interviews by teleconference or
videoconference can be arranged. Following the interview, QAG will
provide the task team and immediate managers with a copy of the
completed questionnaire summarizing the rationale for its rating
and requesting their comments on the assessment. If the assessment
is satisfactory, QAG will finalize it taking into account the
comments received from the task team and/or its manager. If the
assessment is less than satisfactory, the region would have the
option of requesting a second stage in-depth review, on the basis
of which QAG would prepare the final assessment.
10. Second Stage Review. Whenever a region requests a second
stage review, a new panel will be established to take a fresh look
at the project. While the new panel would have access to the first
stage report, it will conduct its own comprehensive review and
interview managers and staff with knowledge about the country and
the sector. As needed, the panel might ask for information
additional to what was furnished for the first stage review and may
have more than one meeting with the Task Manager or particular task
team member. At the conclusion of the review, panels will revise
the guidance questionnaire to reflect their ratings, and will
provide a report giving details of their findings. The region would
have the option to add a rejoinder.
Reporting
11. A synthesis report summarizing the assessments and providing
portfolio level information and analysis of factors underlying
supervision efficiency and effectiveness will be prepared by
end-November, 2002. From the institutional perspective, the report
will synthesize the findings on generic issues, compare them with
results of previous assessments, revisit recommendations of the
QSA4 and QSR reports, and assess progress made to date to improve
supervision quality. The QSA5 draft report will be discussed with
the regions and networks before being submitted to senior
management by mid-January, 2003.
Costs 12. Estimated cost of QSA5 will be about $750,000. These
costs do not include approximately 4 staff days per project for
task teams assessed in QSA5, which are being funded at the regional
level. The budget also assumes that about half of the time of the
specialist reviewers will be funded by their respective Sector
Boards.
-
40 A QAG Assessment
Schedule
13. The schedule of key events of QSA5 is as follows:
July 31 QSA5 Announcement to managers/Country Offices August 1
QSA5 sample selection
August 2 Announcement to task managers/teams included in QSA5
sample
August 15-31 At regional request, QAG will provide orientation
sessions for
task managers/teams whose projects are included in the QSA5
sample
August 16 Deadline for submission of materials for the briefing
books when TL is at HQ (staff on leave may request additional
time)
August 20 Orientation for Specialized Reviewers
August 22 Orientation for panelists
August 23 Deadline for submission of materials for the briefing
books when TL in the field (staff on leave may request additional
time)
August 26 Specialized Reviewers begin reviews
Sept. 3 – Oct. 31 QAG Panels conduct reviews for QSA5 sample
January 17 First working draft of QSA5 synthesis report
February 14 Draft QSA5 report sent to regions/networks for
comment
February 28 Final QSA5 report sent to MDs, regions and
networks
-
ANNEX 4. DETAILED RATING TABLES
RESULTS BY REGION A. Overall Assessment by Region
B. Quality dimension by Region – Highly Satisfactory and
Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
AFR 52 30 28 8 13 11 46 70 71 40 17 18 6 0 0EAP 37 24 17 14 17 0
51 79 94 30 4 4 5 0 2ECA 37 25 19 16 12 27 62 84 59 22 4 14 0 0
0LCR 50 26 17 8 12 13 56 85 85 30 4 2 6 0 0MNA 37 23 11 8 17 6 49
83 84 27 0 10 16 0 0SAR 37 22 8 8 18 30 65 59 63 24 23 7 3 0
0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4
Region
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
AFR 54 83 82 58 87 88 79 90 85 63 77 84 50 53 32EAP 65 96 94 68
96 98 86 88 90 57 92 94 62 83 65ECA 78 96 86 78 100 86 86 76 98 81
100 100 62 72 67LCR 64 96 98 68 100 98 86 92 82 62 100 98 68 65
45MNA 57 100 90 65 96 90 62 96 96 56 100 90 73 74 38SAR 73 77 93 68
77 93 81 86 100 76 82 100 68 73 93Bank-wide 63 92 90 66 94 92 82 87
90 65 91 94 61 69 53
Region
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
Quality of Supervision in F
Y01/02
41
-
RESULTS BY NETWORK
A. Overall Assessment by Network
B. Quality Dimension by Network – Highly Satisfactory and
Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
42
A
QA
G A
ssessment
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
ESD 61 43 22 8 14 2 45 75 91 42 11 5 5 0 2FSE 12 6 6 24 40 0 39
60 92 20 0 8 17 0 0HDN 59 38 19 11 15 27 49 77 57 36 8 17 3 0 0PREM
15 7 8 7 0 28 53 71 60 32 29 11 7 0 0PSI 103 56 45 8 13 13 61 82 77
25 4 9 5 0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0
0.4
Network
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
ESD 53 89 94 56 92 98 78 90 85 54 94 96 52 51 46FSE 63 100 92 58
100 92 71 60 100 70 100 100 45 100 78HDN 60 92 83 61 90 83 87 89
100 71 89 93 76 68 56PREM 60 71 89 75 86 89 80 71 89 58 71 89 60 71
78PSI 70 96 91 72 98 93 82 90 88 67 93 93 59 78 50Bank-wide 63 92
90 66 94 92 82 87 90 65 91 94 61 69 53
Network
-
Quality of Supervision in FY01/02 (QSA5) 43
RESULTS BY SECTOR
A. Overall Assessment by Sector
B. Quality Dimension by Sector – Highly Satisfactory and
Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
ED 27 17 13 15 12 44 47 69 50 36 19 6 2 0 0EMT 29 13 7 10 30 0
48 70 100 27 0 0 14 0 0ENV 11 17 5 5 19 0 42 69 81 47 12 10 5 0 9EP
1 2 2 0 0 83 100 50 17 0 50 0 0 0 0FSP 12 6 6 24 40 0 39 60 92 20 0
8 17 0 0GIC 5 1 2 22 0 0 29 100 100 49 0 0 0 0 0HE 25 12 4 7 5 0 50
95 49 37 0 51 6 0 0PS 14 5 6 8 0 0 50 80 83 34 20 17 8 0 0PSD 9 10
8 0 24 27 64 71 51 36 6 22 0 0 0RDV 49 25 17 9 11 3 46 79 94 40 11
4 5 0 0SDV 1 1 0 0 0 - 0 100 - 100 0 - 0 0 -SP 7 9 2 10 33 0 55 67
100 35 0 0 0 0 0TR 28 16 12 3 7 21 63 86 67 30 7 12 3 0 0UD 16 10 6
11 0 14 76 100 86 13 0 0 0 0 0WSS 16 6 10 15 0 6 78 90 85 7 10 9 0
0 0Bank-wide 250 150 100 10 14 14 53 78 76 32 8 10 5 0 0.4
Sector
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
ED 62 81 94 64 81 94 90 81 100 73 73 89 82 77 92EMT 59 100 100
61 100 100 68 87 100 52 83 100 57 65 10ENV 47 88 81 47 94 91 88 88
81 54 94 91 28 56 17EP 100 50 100 100 50 100 100 50 100 100 50 100
100 50 83FSP 63 100 92 58 100 92 71 60 100 70 100 100 45 100 78GIC
51 100 100 51 100 100 51 100 37 78 100 100 78 100 63HE 57 100 49 57
95 49 83 100 100 65 100 100 72 52 0PS 58 80 83 73 100 83 79 80 83
56 80 83 58 80 75PSD 64 94 78 64 94 84 82 100 73 55 88 78 49 94
78RDV 55 89 96 60 89 100 76 92 86 56 95 96 57 45 52SDV 0 100 - 0
100 - 100 100 - 0 100 - 0 100 -SP 65 100 100 65 100 100 86 87 100
86 100 100 65 73 0TR 66 93 88 73 100 88 90 89 95 82 100 90 47 68
73UD 87 100 100 90 100 100 87 100 91 68 100 100 64 82 43WSS 93 90
90 88 90 97 100 60 85 77 90 97 85 100 32Bank-wide 63 92 90 66 94 92
82 87 90 65 91 94 61 69 53
Sector
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
-
RESULTS BY “AT RISK” CATEGORY*
A. Overall Assessment by “At Risk” Category
B. Quality Dimension by “At Risk” Category – Highly Satisfactory
and Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
Non-Risky 2/ 179 129 78 13 13 16 57 82 76 26 5 8 4 0 0.5
Risky 3/ 71 21 22 2 23 3 43 49 77 47 29 21 9 0 0Bankwide 250 150
100 10 14 14 53 78 76 32 8 10 5 0 0.4
At Risk Status
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
Non-Risky 2/ 70 95 92 72 95 94 86 91 91 70 93 95 62 68 56
Risky 3/ 45 71 79 48 83 85 70 63 84 52 83 84 58 71 40Bankwide 63
92 90 66 94 92 82 87 90 65 91 94 61 69 53
At Risk Status
Notes:1/ The PSR risk rating for RSA1 was taken at the end of
FY97, for QSA4 at the beginning of FY00,
and for QSA5 at the end of FY01.2/ Non-risky ratings include low
risk, 2 flags, and golden flags.3/ Risky group includes actual and
potential problem projects.
44
A
QA
G A
ssessment
-
RESULTS BY PROJECT AGE*
B. Overall Assessment by Project Age
B. Quality Dimension by Project Age – Highly Satisfactory and
Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
* Project Age calculated from Board date as follows: for RSA1 -
to end of FY97; for QSA4 - to end of FY00; and, for QSA5 - to end
of FY02.
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
5 yrs 57 49 30 11 13 3 46 81 87 32 6 10 10 0 0Bank-wide 250 150
100 10 14 14 53 78 76 32 8 10 5 0 0.4
Project Age
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
5 yrs 58 94 90 64 95 96 78 86 88 56 93 96 56 60 58Bank-wide 63
92 90 66 94 92 82 87 90 65 91 94 61 69 53
Project Age
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
Quality of Supervision in F
Y01/Y02
45
-
RESULTS BY LOAN SIZE
A. Overall Assessment by Loan Size
B. Quality Dimension by Loan Size – Highly Satisfactory and
Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
$50 M 122 62 39 12 15 22 57 75 73 25 10 6 6 0 0Bank-wide 250 150
100 10 14 14 53 78 76 32 8 10 5 0 0.4
Loan Size
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
$50 M 69 90 94 69 92 96 84 89 96 69 88 97 67 67 65Bank-wide 63
92 90 66 94 92 82 87 90 65 91 94 61 69 53
Loan Size
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
46
A
QA
G A
ssessment
-
RESULTS BY SOURCE OF FUNDS
A. Overall Assessment by Source of Funds
B. Quality Dimension by Source of Funds – Highly Satisfactory
and Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
IBRD 136 60 51 10 17 16 57 81 76 29 2 8 4 0 0IDA 114 70 46 10 10
13 48 79 73 34 10 13 7 0 1GEF 0 9 1 - 24 0 - 53 100 - 24 0 - 0
0Other 0 11 2 - 18 0 - 64 100 - 18 0 - 0 0Bank-wide 250 150 100 10
14 14 53 78 76 32 8 10 5 0 0.4
Source of Funds
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
IBRD 68 98 92 71 100 93 81 86 89 65 100 97 68 81 59IDA 58 90 86
60 91 90 82 92 90 65 85 88 54 60 54GEF - 76 100 - 88 100 - 76 100 -
88 100 - 65 0Other - 82 100 - 91 100 - 88 100 - 91 100 - 58
0Bank-wide 63 92 90 66 94 92 82 87 90 65 91 94 61 69 53
Source of Funds
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
Quality of Supervision in F
Y01/02
47
-
RESULTS BY LOCATION OF TASK LEADERS (TL)
A. Overall Assessment by TL Location
B. Quality Dimension by TL Location – Highly Satisfactory and
Satisfactory (%)
OA = Overall Assessment
R1 = Focus on Development Effectiveness
R2 = Supervision of Fiduciary/Safeguard Aspects
R3 = Adequacy of Supervision Inputs and Processes
R4 = Realism of Project Performance Ratings
HS = Highly Satisfactory M = Marginal
S = Satisfactory U = Unsatisfactory
Total project (#) HS (%) S (%) M (%) U (%)RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5
Field-based 34 31 20 17 16 18 47 76 62 36 8 21 0 0 0HQ-based 216
119 80 9 14 13 54 78 79 31 8 7 6 0 0.5Bank-wide 250 150 100 10 14
14 53 78 76 32 8 10 5 0 0.4
Location of TTL
OA R1 R2 R3 R4RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1 QSA4 QSA5 RSA1
QSA4 QSA5 RSA1 QSA4 QSA5
Field-based 64 92 79 70 92 79 83 96 100 75 98 92 63 73
73HQ-based 63 92 92 65 94 95 81 85 88 63 90 94 60 68 49Bank-wide 63
92 90 66 94 92 82 87 90 65 91 94 61 69 53
Location of TTL
48
A
QA
G A
ssessment
-
Quality of Supervision in FY01/02 (QSA5) 49
ANNEX 5. GUIDANCE QUESTIONNAIRE RESULTS FOR RSA/QSA
SUMMARY ASSESSMENT SHEET
Supervision Assessment 1. Focus on Development Impact 2.
Supervision of Fiduciary/Safeguard Aspects 3. Adequacy of
Supervision Inputs and Processes 4. Quality and Realism of
Reporting
Overall Assessment List the two or three factors critical to the
success of this project: The numbers in the boxes represent the
percentage of projects rated satisfactory or better. The overall
assessment is not an average of the assessments of the constituent
elements of supervision. Instead, the reviewer should use his/her
judgment in weighing the relative importance of each given the
country and the project context. In making its assessment, the
reviewer should consider the importance of each category, and
within each category, the various questions, to supervision
quality.
Assessment Rating
1=Highly Satisfactory 2=Satisfactory 3=Marginally Satisfactory
4=Unsatisfactory NA=Not Applicable
RSA1 QSA4 QSA5
66 94 92
82 87 90
94 65 91
61 69 53
63 92 90
-
50 A QAG Assessment
Context
A. With the experience implementing and supervising the project
during the past year, as far as can be readily determined:
RSA1 QSA4 QSA5
(a) Was the project design sound? (Yes/No) 68 80 70
(b) Was the project ready for implementation at approval?
(Yes/No) 60 71 57
(c) Was overall implementation performance satisfactory prior to
FY01 (Yes/No) N/A N/A 53 (d) Could the problems encountered during
implementation have been i dentified
at entry (Yes/No) N/A N/A 74
(e) Any major changes prior to FY01 (through restructuring
and/or MTR)? (Yes/No)
N/A N/A 22
B. Compliance with safeguard policies?
(a) Applicability and compliance with safeguard issues:
Policy Applicability
Environmental Assessment (OD 4.01) 73 Natural Habitats (OP 4.04)
9 Forestry (OP 4.36) 8 Pest Management (OP 4.09) 8 Cultural
Property (OPN 11.03) 5 Indigenous Peoples (OD 4.20) 8 Involuntary
Resettlement (OP 4.30) 13 Safety of Dams (OP 4.37) 4 Projects on
International Waterways (OP 7.50) 1
Projects in Disputed Areas (OP 7.60) 1
(b) In case of compliance issues, including those occurring
prior to FY01, what were there nature
and extent? (Count)
(c) Were appropriate measure taken to mitigate safeguard aspects
that could have had adverse
impacts? (Yes/No)
1= Highly Satisfactory 2= Satisfactory 3= Marginally
Satisfactory 4= Unsatisfactory NA=Not Applicable
14
76
-
Quality of Supervision in FY01/02 (QSA5) 51
1. Assessment – Focus on Development Impact
With the experience of supervising the project during FY01-02,
assess the quality of Bank performance for:
RSA1 QSA4 QSA5
1.1 Identification and Assessment of Problems
69 93 92
(a) Timely Identification of Implementation Problems
80 97 92
(b) Regard for Development Impact
64 85 86
1.2 Focus on Sustainability
N/A 89 93
(a) Borrower and stakeholder ownership
N/A 90 91
(b) Technical assistance, training, and capacity building
N/A 85 91
(c) Readiness for operational phase
N/A 84 82
1.3 Actions Taken and Follow-up
64 91 82
(a) Appropriateness of Advice and Proposed Solutions to the
Borrower (including Action Plan)
74 94 91
(b) Appropriateness and Speed of Bank follow-up (including
cancellations, suspensions, etc)
69 85 73
(c) Impact and Effectiveness of Bank actions (including risk
management)
58 83 79
(d) Quality of Mid-term Review (if undertaken during
FY01-02)
N/A N/A 84
(e) Quality of Restructuring Plan (if undertaken during
FY01-02)
N/A N/A 77
1.4 Effective use of CPPR (or other venues for portfolio review
with the borrower) to resolve problems affecting the project?
61 84 84
Assessment Rating
1= Highly Satisfactory 2= Satisfactory 3= Marginally
Satisfactory 4= Unsatisfactory NA=Not Applicable
-
52 A QAG Assessment
2. Assessment – Supervision of Fiduciary Aspects
With the experience of supervising the project during FY01-02,
assess the adequacy of supervision of:
RSA1 QSA4 QSA5
2.1 Procurement Compliance and promptness
88 92 91
2.2 Financial Management * Accounting/auditing and financial
information, SOE
77 84 78
2.3 Legal Aspects Compliance; relevance/clarity/timeliness of
advice
82 90 92
2.4 Environmental Aspects e.g. Category” A” compliance
N/A 85 72
2.5 Social Aspects e.g. Social impact
68 69 82
2.6 Performance and Progress Monitoring use of key indicators,
progress reporting, attention to results
N/A 79 65
* Includes Disbursement
Assessment Rating
1= Highly Satisfactory 2= Satisfactory 3= Marginally
Satisfactory 4= Unsatisfactory NA=Not Applicable
-
Quality of Supervision in FY01/02 (QSA5) 53
3. Assessment – Adequacy of Supervision Inputs and Processes
With the experience of supervising the project during FY01-02,
assess the quality of Bank inputs and processes:
RSA1
QSA4
QSA5
3.1 Staffing
81 93 97
(a) Staff Continuity 92 94 93
(b) Supervision Skill Mix 72 80 88
(c) Degree of Country Office Involvement and its Contribution 72
91 93
3.2 Supervision Activities
75 92 96
(a) Mission Preparation/TORs Frequency and Time Spent in the
Field 77 95 90
(b) Quality of Interaction with Borrower outside of formal
Missions? N/A N/A 97
3.3 Quality of Supervision Documentation and Follow-up N/A N/A
78
(a) Aide-memoire and Follow-up Letters (well organized, focused
on key issues and solutions, readable)
73 87 87
(b) Quality of supervision documentation received by the panel
N/A N/A 78 3.4 Relationships
85 97 93
(a) Relations with the Borrower 82 95 92
(b) Relations with Donors and other stakeholders 84 98 97
3.5 Management Inputs
62 86 79
(a) Adequacy and Speed of Management Attention and Actions 60 79
79
i) Mission’s strategic focus and problem solving N/A N/A 81
ii) Mission’s debriefing N/A N/A 92
iii) PSR and post-mission follow-up N/A N/A 59 (b) Adequacy of
Supervision Budget (Too Little, To Much or About Enough) 67 74 80
(c) Effectiveness of Budget Use N/A 89 92
Assessment Rating
1= Highly Satisfactory 2= Satisfactory 3= Marginally
Satisfactory 4= Unsatisfactory NA=Not Applicable
-
54 A QAG Assessment
4. Assessment - Realism and Quality of Project Performance
(FY01-02)
RSA1 QSA4 QSA5
4.1 Accuracy, Timeliness and consistency with Subsidiary Ratings
in PSR
61 69 50
4.2
Appropriateness of Risk Ratings in PSR N/A N/A 62
4.3
Adequate Justification for any change of Ratings 62 79 70
4.4
Adequate Explanation of DO and IP Ratings 55 64 52
4.5 Appropriate Use of Golden Flags, if applicable
N/A N/A 56
Assessment Rating
1= Highly Satisfactory 2= Satisfactory 3= Marginally
Satisfactory 4= Unsatisfactory NA=Not Applicable
-
Quality of Supervision in FY01/02 (QSA5) 55
ANNEX 6. SUPERVISION OF POVERTY AND SOCIAL ASPECTS Overview 1.
Social developme nt (SD) issues in the sample of 100 operations in
QSA5 were assessed by specialized social reviewers. SD issues were
found to be relevant in 86 of the 100 projects. The key areas in
which social issues were assessed were:
• Extent to which SD and poverty issues have been monitored
during supervision;
• Extent to which supervision has assessed and taken account of
views of key stakeholders or attempted to strengthen mechanisms for
participation in implementation and/or monitoring;
• Where relevant, compliance with Bank social safeguards
policies concerning cultural property, indigenous peoples and
resettlement and land acquisition;
• Effectiveness of supervision of SD and poverty issues;
and,
• Level of effort devoted to supervision of SD and poverty
issues.
Sample Profile 2. Of the 100 operations in the QSA5 sample,
• 69% of operations took SD issues into account during
preparation (64% in
QSA4);
• Projects with components that explicitly address gender issues
rose from 34% in QSA4 to 44% in QSA5; and,
• 30% were supervised by social development specialists, same as
QSA4. Overall Results 3. Of the 86 projects from the QSA5 sample in
which SD issues were considered relevant, 82% were rated
satisfactory. However, this performance masks substantial
differences in the performance of supervision by sub-categories.
Monitoring and evaluation (M&E) of SD issues, risk and outcomes
continues to be an area of weakness, with 55% of operations in QSA5
rated satisfactory on this dimension, a slight increase on the 49%
rating in QSA4. The level of effort and participation, with
satisfactory ratings of 79% are about the same levels for QSA4.
-
56 A QAG Assessment
4. The below chart illustrates trends for overall ratings and on
key sub-questions over the past three supervision reviews.
0.00%
20.00%
40.00%
60.00%
80.00%
SDV / POV Effort Effectiveness SD Monitoring Participation
Satisfactory percentage by social sub-ratings
RSA3
QSA4
QSA5
Supervision of social safeguards
5. In QSA5 the number of operations in which safeguards issues
were relevant dropped in comparison to QSA4. Safeguards policies
applied to 22% of the operations, as opposed to 25% in QSA4. By
category, 12% of operations had indigenous peoples issues, 15% had
resettlement issues and 6% had cultural property issues (with
overlap on a number of operations). Of these, all those operations
with indigenous peoples issues and cultural property issues were
rated satisfactory with regard to addressing safeguards issues; 81%
with resettlement issues were rated satisfactory. Though the
overall proportion of operations with safeguards issues declined
slightly, the proportion rated satisfactory on those issues
increased for both indigenous peoples and cultural property, and
remained essentially stable for resettlement.
0%
20%
40%
60%
80%
100%
Indigenous Peoples Resettlement Cultural Property Overall
Safeguards
Quality of Social Safeguards Supervision
RSA3 QSA4 QSA5
-
Quality of Supervision in FY01/02 (QSA5) 57
Performance by Region 6. While the performance by regions in
QSA4 had reached general parity, the results of QSA5 indicate that
two regions have continued to improve (notably EAP and SAR), two
have remained generally at the same level as QSA4 (LCR and MNA) and
the remaining two have dropped off (AFR and ECA). These figures
should, however, be treated with caution, as in several cases the
number of projects per region are too few for statistical
significance. Rating by Networks 7. Most networks have remained
fairly consistent on overall ratings from QSA4 to QSA5. For both
FSE and PREM the number of operations was statistically too small
to draw robust conclusions (only 2 FSE projects and 5 PREM
operations). PSI operations accounted for 46% of the total, and
this network maintained the level of QSA4. The following chart
illustrates trends by network over the past three reviews.
Performance on Gender Dimensions 8. Operations in the QSA5
sample performed better on gender issues than operations in past
supervision reviews. Of the 100 projects in QSA5, 42% had some form
of gender analysis (22% in QSA4), 44% of operations had project
components that addressed gender issues (34% in QSA4), and gender
issues were supervised in 36% of the operations (21% in QSA4). It
appears that gender considerations are increasing in operations,
and importantly, the gap between identification of gender issues
and supervision of those issues is narrowing. In QSA4, one-third of
the operations which identified gender issues as significant did
not supervise them. 9. The correlation between supervision of
gender issues and poverty/SD ratings continues to be high. For
example, of the 41 operations (out of 100) that identified gender
issues and supervised them, 95% received satisfactory ratings on
supervision of social development and poverty issues, compared to
75% satisfactory rating for projects without gender components.
Satisfactory Percentage by Networks
0%
20%
40%
60%
80%
100%
120%
ESSD HDN PSI PREM FSE Overall
RSA3 QSA4 QSA5
-
58 A QAG Assessment
Lessons: Elements of Supervision Good Practice
10. Projects rated highly satisfactory tend to focus on
achievement of social development outcomes. Those operations which
identify social development, poverty reduction, and/or gender
outcomes up front as project objectives subsequently pay greater
attention to such outcomes and score higher on supervision than do
those projects which do not identify such outcomes. The QSA5 review
highlighted the following elements of good practice, many of which
echo those identifie d in the QSA4 review.
• Results-based management through early identification of SD
outcomes.
Those operations which identify SD outcomes during preparation
generally focus supervision activities much more directly on
behavioral changes – changes in the “rules of the game” – than do
those operations which focus only on outputs or inputs. Examples
include India AP DPIP, Nepal Basic and Primary Education and
Algeria Rural Employment. Though not always successful, operations
which clearly identify the changes being sought in the service of
SD objectives tend to be much more effective in adjusting their
style of implementation based on actual outcomes on the ground.
• Systematic social analysis manifested through the life of the
operation. Good practice entails consistent monitoring and
evaluation of the progress of the operation with regard to social
development outcomes throughout the course of implementation. For
example, Russia Coal entails the use of TA in association with
SECALs to provide strong support for monitoring, and responding to,
the social changes and impacts resulting from restructuring of the
coal sector. Ongoing SAs and social impact studies provide support
for improvement in the design and implementation of community
development programs and pilot activities of community support and
employment. The Indonesia KDP has utilized a variety of studies and
analyses during implementation (assessments of the effectiveness of
local governments in reflecting the needs of local populations,
studies on corruption, conflict studies, etc.) which are then “fed
back” into the project to improve implementation and the likelihood
of achievement of SD outcomes.
• Sustained participation by intended beneficiaries during
supervision. The most critical mechanism for determining success or
failure in achieving SD outcomes is feedback from the intended
beneficiaries themselves. Those operations which include
participatory M&E as a part of supervision tend to be more
flexible, lead to greater ownership among in tended beneficiaries,
and appear to hold promise of greater sustainability of project
outcomes.
• Use of Innovative M&E strategies, particularly in CDD
operations. Ongoing supervision of SD outcomes, with the active
participation of intended beneficiaries, can be a time and
effort-consuming activity. The best approaches have used innovative
approaches to conduct such monitoring. For example, Indonesia KDP
utilizes independent journalists as both monitoring and evaluation
mechanisms, and engages civil society in similar roles to achieve
what is effectively continuous supervision. Supervision from field
offices, as opposed to formal missions from headquarters, also
result in more continuous M&E (India
-
Quality of Supervision in FY01/02 (QSA5) 59
AP DPIP, Indonesia KDP). The use of independent consultants for
M&E also seems to improve the transparency of supervision and
provides a valuable independent check on progress (e.g., Algeria
Rural Employment).
11. Good practice therefore entails (a) focus on social
development outcomes enabling results-based management; (b) ongoing
social analysis for adaptive learning, improved targeting, gender
equity, and understanding of informal institutional behavior; (c)
sustained participation enhancing stakeholder ownership and
empowerment, with feedback mechanisms to improve project
implementation, and (d) use of innovative and field-based M&E
strategies. Of course, application of these best practice elements
remains uneven. The challenge lies in mainstreaming these practices
across the portfolio.
-
60 A QAG Assessment
ANNEX 7. SUPERVISION OF ENVIRONMENTAL ASPECTS Overview 1. A team
of environment specialists assisted QSA panels in assessing the
quality of the Bank’s supervision of environmental planning and
management in QSA5. Environmental safeguard and other environmental
issues were found to be relevant in 72 of the 100 projects in the
sample. This annex discusses key findings, strengths and weaknesses
of environmental supervision, issues requiring Management’s
attention and proposed actions to improve performance. The key
areas in which environmental issues were assessed were:
• Extent to which environmental issues have been monitored
during supervision;
• Extent to which supervision has assessed and taken into
account the views of key stakeholders or attempted to strengthen
mechanisms for participation in implementation and/or
monitoring;
• Compliance with Bank environmental safeguard policies
concerning environmental
assessment, pest management, forestry, natural habitats and,
where relevant, safety of dams;
• Effectiveness of supervision of environmental issues; and,
• Level of effort devoted to supervision of environmental
issues.
Sample Profile 2. Table 1 provides a profile of the
environmental categories of the QSA 5 sample. Twenty-eight
projects, all in category C or uncategorized, ranging from fiscal
consolidation to labor force development, posed no significant
environmental impacts relevant to implementation or supervision.
Accordingly, the remaining 72 projects were reviewed and the
quality of the Bank’s supervision was rated on the basis of
attention given to compliance with the Bank’s environmental
safeguard policies and the effective implementation of
environmental mitigation and other protective measures. These
projects include large-scale infrastructure, community-driven deve
lopment, natural resources management and intermediary
financing.
Table 1: QSA Sample
Environmental Impact Category
No. of Projects
A 12 B 47 C 38
Other 3 Total 100
-
Quality of Supervision in FY01/02 (QSA5) 61
Key Findings: 3. Overall, supervision of only 72% of the
projects was found to be satisfactory. Rating of satisfactory
projects was particularly low in AFR (24%), with SAR scoring
relatively low (60%), LCR 78% and all other three regions rated
above 90% satisfactory (EAP 92%, MNA 95% and ECA 97%). 4.
Triggering of Environmental Safeguard Policies. Environmental
safeguard policies were triggered a total of 104 times, with 54
projects triggering just one and 18 projects triggering two or more
policies (Table 2).
Table 2: Environmental Safeguards Triggered
Safeguard Policy Projects
Triggered Environmental Assessment (OP/BP 4.01) 73 Natural
Habitats (OP 4.04) 9 Forestry (OP 4.36) 8 Pest Management (OP 4.09)
8 Dam Safety (OP 4.37) 4 International Waters (O P 7.50) 1 Disputed
Areas (OP 7.60) 1 Total 104
Level of Compliance 5. There were 14 instances of non-compliance
with environmental safeguard policies; nine in the case of OP 4.01
- Environmental Assessment (four projects in AFR, three in LCR and
one each in MNA and SAR), and five in the case of OP 4.09 - Pest
Management (two projects in AFR and one each in MNA, SAR and EAP).
During the panel reviews, Task Teams reported that they would take
appropriate remedial measures to bring these projects into full
compliance. 6. Eleven projects (mostly education and health) were
inappropriately assigned a “Category C” at the time of board
approval and have given rise to environmental problems during
implementation. The significance of these issues was discussed
during panel meetings and corrective measures and monitoring
arrangements were suggested to ensure compliance with the Bank’s
environmental safeguard policies. Implementation of Environmental
Mitigation and other Protective Measures. 7. Bank supervision of
the implementation of environmental mitigation and other protective
measures had a 72% satisfactory rating, lower than that of all
previous QSA exercises (Figure 1). Ratings for level of effort and
effectiveness of supervision of environmental issues and risks were
rated 71% and 68%, respectively. Explanations for the decline may
be found in inadequate budgets, unavailability of skilled staff,
inappropriate category rating, and the continuing marginal
importance assigned to environmental supervision. On a more
positive note, EMPs were prepared and are being implemented for 83%
of projects sampled. However, their implementation was found to be
satisfactory in only 69% of the cases (see para. 10). The views of
the stakeholders was satisfactorily taken into account 75% of the
time.
-
62 A QAG Assessment
Figure 1: Quality of Environmental Supervision (%) RSA3 to
QSA5
82
101
13 15
0
21
7
72
78
64
0102030405060708090
HS S M U
%
RSA3 QSA4 QSA5
8. Performance by region (Table 3A) and network (Table 3B)
demonstrates differences noted in previous assessments. AFR has the
lowest rating (24%), SAR (60%) and LCR (78%) relatively low and the
other three regions exceeding 90%. Ratings by network suggest that
ESD is low (56%) while the other three networks are above 75% or
above.
Table 3A: QSA5 Ratings by Region for Supervision of
Environmental Aspects*
HS S HS or S M U Region No. of Projects % No. % No % No. % No. %
No.
AFR 19 - 0 24 8 24 8 59 9 16 2 EAP 11 24 3 68 6 92 9 3 1 5 1 ECA
11 16 1 81 9 97 10 - 0 3 1 LCR 13 - 0 78 10 78 10 22 3 - 0 MNA 10 -
0 95 9 95 9 5 1 - 0 SAR 8 3 1 57 5 60 6 20 1 20 1
Bank-wide 72 8 5 64 47 72 52 21 15 7 5
Table 3B: QSA5 Ratings by Network for Supervision of
Environmental Aspects*
HS S HS or S M U Network
No. of Projects % No % No. % No. % No % No.
ESD 21 3 1 52 12 56 13 24 4 21 3 FSE 0 - 0 - 0 - 0 - 0 - 0 HDN
12 18 1 66 10 84 11 16 1 - 0
PREM 2 15 1 85 1 100 2 - 0 - 0 PSI 38 6 2 69 24 75 26 22 10 3
2
Bank-wide 72 8 5 64 47 72 52 21 15 7 5
* Note: • Out of 100 projects, 28 posed no significant
environmental impacts relevant to supervision • The percentages are
weighted averages (as referred to in the main body of the
report)
-
Quality of Supervision in FY01/02 (QSA5) 63
Strengths and Weaknesses of Supervision 9. A clear strength
included the use of Environment Specialists from Resident Missions
that undertook regular, on-site visits and faithfully reported
their findings to the TL. In some cases, this was ma de easier by
proactive actions taken by the Borrower and the direct involvement
of government environment agency staff or NGOs. Although of varying
quality, EMPs are now more regularly included in preparation
documents, sometimes being cited in annexes to the PAD. 10.
Weaknesses are apparent in EMP implementation, exacerbated by the
ill-defined roles of the PIUs, lack of or insufficient coverage of
environmental mitigation in operational manuals and major lapses in
the quality of reporting in PSRs and Aides-memoire. There are also
continuing concerns over the marginal use of Environment
Specialists during supervision, the treatment of FIs and CDD
projects, and environmental category misclassification. However,
most worrying is the high level of non-complia nce with OPs 4.01
and 4.09 – some 19% of the projects reviewed – well above that
found in previous QAG exercises. Issues Requiring Further Attention
and Proposed Actions 11. Based upon QSA5 results, many aspects of
environmental supervision require immediate attention and follow-up
actions. Most of these have been noted in previous QAG reports and
Management should review how the Regions (with the support of the
Environment Department) propose to take concerted action to avoid
repetition in the future. Environmental Management Plans (EMPs) 12.
Although of variable quality, EMPs or variants were available to
guide the implementation of mitigation measures in 83% of the
classified projects sampled – an improvement over previous years.
However, monitoring their implementation and ensuring satisfactory
performance continues to be a major shortcoming. Frequency of site
visits by Bank staff and discussions with the Borrower to review
needed actions were often insufficient and there is little evidence
in project documents of diligent follow-up, including
correspondence with the Borrower on actions needed to assure
continued compliance with the Bank’s safeguards. Part of the
problem lies with the lack of, or poorly defined, environmental
supervision arrangements in project supervision plans. Equally,
unlike procurement and financial requirements (that are regularly
detailed in annexes to the PAD), environmental requirements are
rarely stated in commensurate detail. Another failure concerns the
varying quality of environmental performance indicators and output
– an issue noted in last year’s QSR report. 13. It was also found
that changes in project design and/or reallocation of funds to new
components were rarely assessed on environmental grounds. Such
changes often pose major safeguard risks, and the Regions should
alert Task Teams to potential risks (e.g. in one project in the
sample, funds were reallocated from a road component to a municipal
solid waste component without any environmental and social
review).
-
64 A QAG Assessment
Recommended Actions
• Most Regions, as part of cross-support, are providing guidance
on the preparation and implementation of EMPs, including examples
of good practice. However, effective EMP implementation requires
increased use of Environment Specialists as part of field
supervision.
• Regional Environmental Safeguard Specialists should assess
changes in project design/reallocation of funds to new components
for potential environmental impacts and, as appropriate, a new
project review exercise should be conducted.
• Project supervision plans should be more explicit about
environmental supervision requirements.
Application of Environmental Safeguard Policies 14. There are 14
instances of non-compliance with safeguards, 9 concerning OP 4.01 -
Environmental Assessment and 5 concerning OP 4.09 - Pest
Management. Some Task Teams failed to recognize the application of
the OP during preparation, some others failed to take action during
implementation, sometimes in the face of a compliance violation
being indicated by independent study. During panel discussions,
TTLs for the relevant projects stated that they were taking
appropriate actions to employ appropriate specialists to assure
compliance with the OP. This should be followed up by the Regional
Environmental Safeguard Units. Recommended Actions
• Ensure that non-compliant projects are brought into full
compliance with the relevant safeguard policies.
• Continue staff training under the General Safeguard Policies
Training Course. This course should emphasize that safeguards may
be triggered even during project implementation and measures need
to be developed regardless of any up-front work.
• The Environment Department should convene a meeting with the
Regions to agree on a program to assist and ensure that Task Teams
comply with OP 4.09. This might include staff from the FAO, as
their failure to apply good practice was evident in those projects
where they were handling pest management components.
Financial Intermediary and Community Driven Development Projects
15. Environmental supervision of these projects is especially weak
and deserves urgent attention. Environmental guidance for FIs was
often available (criteria, institutional arrangements, etc.), but
Task Teams rarely chose to supervise its application or to conduct
spot checks/audits during implementation. Consequently, the
adequacy of environmental protection measure for sub-projects could
not be determined. Fortunately, in some instances, the borrower
applied national regulations and reported progress or otherwise in
quarterly reports to the Bank. The CDD projects face similar
problems.
-
Quality of Supervision in FY01/02 (QSA5) 65
Recommended Actions
• Regional environment units should conduct workshops to assist
Task Teams to prepare and supervise FI and CDD projects more
effectively. Guidance should be pragmatic but flexible enough to
accommodate the evolving nature of these instruments as recognized
in the recent Environment Update on FIs. The Environment Department
is currently developing guidance on the handling of CDD projects;
this should be made available to the Regions through workshops and
guidance materials.
Operational Manual and the Project Implementation Unit (PIU) 16.
The inclusion of environmental management requirements in the
Operational Manual is crucial to ensuring environmental due
diligence by the project team. However, many manuals included vague
statements giving no details of institutional responsibilities,
scheduling or budgeting; some were without any mention of the need
for environmental management. As an exception, road sector projects
were very instructive in providing detailed guidance (especially in
AFR), including performance indicators and outputs, and gave Task
Teams a rigorous framework in which to supervise and monitor
progress. On an allied issue, many Task Teams took the view that
environment issues were “taken care of” if the Project
Implementation Unit (PIU) hired an environment officer and chose
not to evaluate performance. In addition, supervision reporting
often adopted the environmental supervision findings of the client
(PIU or other unit) without adequate evaluation. Recommended
Actions
• Task Teams should ensure that environmental management
requirements for projects are satisfactorily reflected in
Operational Manuals. Regional Environment Units should assist as
required.
• When PIUs have their own environmental staff, Task Teams
should assess the capacity of PIUs environmental staff, review the
effectiveness of PIU performance, ensuring adequate review of the
implementation and reporting of environmental protection measures,
including compliance with Bank safeguards.
Quality of Reporting 17. Reporting in PSRs continues to be
perfunctory on compliance with safeguard policies and environmental
protection performance in general. This is perhaps a reflection of
the limitations of the PSR format and the minor importance accorded
to environmental aspects by managers. However, aides-memoire were
often equally perfunctory in reporting environmental due diligence,
especially on problems requiring management’s attention. These are
valuable means of recording issues, identifying agreed actions and
measures for follow-up. Some of the Regional Environment Units (AFR
& MNA) are conducting environmental supervision reviews and
these are proving instructive to Task Teams in demonstrating the
value of comprehensive reporting on environmental issues during
implementation; they also point out instances of
non-compliance.
-
66 A QAG Assessment
Recommended Actions
• The PSR format should be modified to accommodate better
reporting of compliance and implementation of environmental
mitigation and other protective measures.
• Aides-memoires should be used more effectively to report
environmental issues and recommended actions.
• The practice of environmental supervision reviews adopted by
some Regions should be considered for possible Bank- wide
application.
Role of Environment Specialists 18. The failure to use
environment specialists during supervision continues, particularly
in relation to Category B projects, some of which are particularly
complex/ris ky. During panel interviews, it was learned that TTLs
often accepted the need to include such specialists, but were
constrained by insufficient budget.1 As in the past, some chose to
partially offset this constraint by staff doubling as environment
specialists – sometimes with disastrous results. The practice of
using Resident Mission staff again served to assure quality of
supervision, largely because of regular (sometimes surprise) field
visits and immediate follow-up. The value of including environme
ntal agency staff in field missions to improve project performance
and help mainstream environmental supervision is being recognized
in a number of regions. Recommended Actions
• Environment Specialists should be included in project
supervision of category B projects at least at mid-term review, and
for the most complex ones, every year, and country environment
agency staff should be encouraged to join supervision missions on a
selective basis.
• Supervision review by environment specialists could be
enhanced by their offering recommendations on linking the project
with other geographically proximal Bank or donor assisted projects
having similar objectives (e.g.. natural resources management/rural
development and water management projects) to gain efficiencies and
in dealing with the systemic issues.
• Opportunities for giving presentations to government/project
staff during supervision should be explored, e.g.
safeguards/EMPs.
1 The panel heard that some TTLs used local consultants to
overcome this constraint; others availed themselves of funds from
regional safeguard units.
-
Quality of Supervision in FY01/02 (QSA5) 67
Environmental Category Misclassification 19. This persists as an
issue affecting most Regions. In 11 projects (mostly education and
health) a “C” Category classification was inappropriate despite
evidence to the contrary at preparation and subsequent
identification of significant adverse impacts during
implementation. Some education projects reviewed included the
building of hundreds of schools and associated sanitation
facilities; some health projects involved use of chemicals to
control vectors. However, the panel learned that in all cases the
Task Teams have introduced or are introducing appropriate
mitigation measures. Recommended Actions
• The region’s environment units should undertake a series of
workshops with education and health sector Task Teams to
demonstrate the process of EA classification of education and
health projects and disseminate the existing guidance documents,
identifying mitigation measures and their implementation, including
supervision.
-
68 A QAG Assessment
ANNEX 8. SUPERVISION OF FINANCIAL MANAGEMENT ASPECTS
Overview 1. A team of financial management specialists (FMSs)
assessed the quality of the Bank’s supervision of financial
management (FM).1 The reviewers were guided by specific criteria
and their ratings were based primarily on the Bank’s timeliness in
identifying and addressing issues, the frequency and quality of FMS
inputs, coverage of FM issues in project documentation, the extent
that management support was provided to resolve FM issues (if
required), and the degree of support provided by the Bank to the
borrower in resolving problems or otherwise improving project FM
performance.
Table 1 – QSA5 Financial Management Ratings Number of
Projects
Type of Operation
Highly Satisfactory Satisfactory
Marginally Satisfactory Unsatisfactory
Total
Investment 15 61 19 0 95 Adjustment 0 5 0 0 5
2. The level of satisfactory supervision of financial management
was 78%. This is somewhat lower than the levels in QSA4 (84%) and
higher than QSR (73%). In addition, the results confirm that the
quality of FM supervision remains below that of FM Quality at
Entry. 2 3. Key findings were that:
• Approaches to FM supervision are inconsistent. Guidance to
staff is needed to better define and support good practices.
• Good FM supervision is a team effort – involving effective
collaboration within
the regional FM team, the FMS, the TL and Country Office Staff.
Strong supervision of FM resulted when the task team worked
together to not only identify issues but also follow up to resolve
them.
• Supervision of FM was often highly dependent on the TL’s
perception of FM. • Weak FM supervision was in many cases due to
insufficient human resources
(number of FM specialists) in the regional FM teams. •
FM-related supervision often is not well documented, although there
are some
good practice examples worth noting.
1 The review also covered disbursement aspects to a limited
extent: SOE reviews and the resolution of Special Account problems.
2 The three most recent Quality at Entry (QEA) exercises, which
focus on new projects, have judged the quality of compliance with
Bank FM policies to be 88% (QEA5), 95% (QEA4), 89% (QEA3).
-
Quality of Supervision in FY01/02 (QSA5) 69
Table 2 – Sub-data on FM Supervision
Number of Projects
Issue Yes No N.A. Total % Satisfactory
Adequate communication with the borrower on FM issues outside of
missions 65 19 16 100 77
SOE reviews were carried out and results were
documented 47 39 14 100 55 Special Accounts issues were dealt
with in a timely,
decisive, and effective manner 34 7 58 99 83 Audit reports
reviewed timely, by a qualified person 72 17 11 100 81 Appropriate
and timely response sent to the borrower
regarding review of audit report 63 23 13 99 73 FM issues
identified, and followed up appropriately 71 18 9 98 80 Task team
assisted the borrower in working out
problems and/or improving FM arrangements 74 13 13 100 85
Appropriate level of FM-certified staff in supervision 75 22 3 100
77 Documented inputs addressing relevant FM areas 81 16 3 100
84
Key Findings: Strengths and Weaknesses of Financial Management
Supervision 4. As shown in Table 2, the reviewers rated particular
aspects of FM supervision. The yes/no ratings of these questions
are generally consistent, within projects and overall, with the FM
ratings assigned to the projects. Of the main FM and disbursement
activities of supervision, the most significant outlier is SOE
reviews, which, where applicable, were only carried out in 55% of
the projects over the two year span. In addition, the nature and
documentation of SOE reviews was highly inconsistent, ranging from
a sample of a few transactions and brief mention in an
aide-memoire, to a region-wide, sophisticated review across
projects. Regarding adjustment loans, while these were all rated
satisfactory (Table 1), the FM reviewers had considerable
difficulty in defining appropriate FM supervision of these
operations. Key general findings are:
a) Supervision is generally adequate, but there are substantial
weaknesses that should be addressed. With 78% of projects rated
satisfactory, FM supervision while not weak, has room for
improvement. There are clear steps that can be taken by the Bank to
improve FM supervision, which are detailed in the Recommendations
section below.
b) There is a lack of clarity in the Bank as to what constitutes
good FM supervision.
Other than the QAG criteria, no explanation has been provided to
staff as to what FM supervision should entail. Therefore
disagreements between staff, and the combination of “old habits”
and new but unclear mandates, have continued to confuse staff as to
what is required. Some examples of this are that:
-
70 A QAG Assessment
• Financial analysis is confused with financial management. In
many projects,
financial analysts carried out the valuable role of analyzing
the financial performance of entities, but their work was
incorrectly considered to be addressing financial management
aspects of supervision.
• Financial management supervision is often seen as a reactive
rather than
proactive process. Some teams continue to focus on reviewing
audit reports, supervising the FM aspects of the project only to
the extent that the audits indicate FM issues.
• Documentation varies widely. (see below).
c) Highly satisfactory supervision of FM requires a collective
effort. The best FM
supervision occurred where the FMS was able to closely monitor
the project, while being integrated into the task team. Since most
FMSs are in field offices, the importance of linkages to
headquarters staff is also important. Strong supervision was found
in some large country offices, where the FM team was strong, and
had a balance of national and international expertise. More
generally, country office FM staff were effective when integrated
into task teams, which is particularly important in projects
involving complex implementation arrangements. In one
community-oriented project, for example, the reviewer noted that:
“The decentralized, community driven development approach…poses
challenges to financial management, to which the Bank team has been
responsive. FM is well integrated into the team’s overall approach.
Continuity in FM team composition has helped consistency in
supervision…. Overall, the team is to be commended for its support
to capacity building in a decentralized financial management
system.”
d) TTLs often determined the quantity, and influenced the
quality, of FM supervision.
Supervision of FM was ultimately the responsibility of the TTL.
Therefore the intensity of supervision was driven by TTL judgments,
or in many cases, budgetary concerns. In projects where FM was well
supervised, the TL was not only open to FMS involvement, but
engaged in FM issues. Poor supervision resulted when there was an
FMS/TTL disconnect; in some country offices, local FM staff often
did not carry out supervision if not requested by the TTL. In
others, TTLs were not sufficiently aware of the FM work being
carried out.
e) FM-related supervision often is not well documented.
Supervision of FM is not
documented or archived consistently. Documentation of mission
work is often separate from that of the team, informal supervision
is verbal or in e-mails, and required records are not in one place.
While it is understandable, and in many cases even desirable, for
FM staff to visit projects independently, documentation of missions
as well as other FM supervision activities should be more
consistently prepared, be readily available, and be integrated with
other supervision documents. The problem may be especially relevant
in field offices, where FM staff may have frequent informal
exchanges with clients that are never formally documented. While
part of the problem is a general one, the FM teams can do
better.
-
Quality of Supervision in FY01/02 (QSA5) 71
f) Weak FM supervision was sometimes due to shortfalls in human
and budgetary resources. In some cases, especially in EAP and LCR,
TTLs cited the unavailability of FM staff as a reason for
insufficient FM supervision. Budget constraints also were blamed,
especially during FY01 when task budgets were constrained and TTLs
were faced with choosing between competing priorities.
Recommendations for the Financial Management Sector 5. The
findings of the QSA5 suggest certain actions that the FM Sector
Board should undertake to improve project supervision:
• First, as a general measure to improve the quality of FM work,
regional FM teams should more fully adopt quality arrangements.
These arrangements should include a process of internal reviews,
feedback to (especially junior) staff as to their performance, a
system of electronic documentation of FM supervision and its
review, and improved identification of, and attention to, projects
with the highest FM risks and opportunities.
6. In three other areas, actions are already underway and should
be completed. The FM Board should:
• Manage the shift of responsibilities for FM supervision from
TTLs (demand driven) to the regional FM teams (risk and opportunity
based). In most regions, the Bank’s internal budgetary and
accountability practices have been revised to allow the FM team to
more easily monitor FM aspects of projects as it sees fit, in
collaboration with its internal clients. This will allow for a more
appropriate allocation of staff and budgetary resources to FM
supervision than in the past. While the TTL still has ultimate
responsibility for supervision in a given project, the FM Board
should work closely with its internal clients to agree on roles and
responsibilities, recognizing the TTL’s role as head of the task
team but also the FM team’s responsibility to optimize FM
supervision across the region.
• Prepare FM Supervision Guidelines, as per the Board’s FY03
work program.
These guidelines will help to standardize the supervision
process and to more clearly define what is “good FM supervision”
and how it should be documented. Such guidelines would complement
the FM Assessment Guidelines, which have served as an important
tool for documenting the FM reviews of new projects.
• Determine the proper role of SOE reviews. This is currently
being discussed as part of reforms to the Bank’s audit and
disbursement policies and practices.
• Clarify the role of FM supervision in adjustment operations.
This is expected to be addressed by the in-process guidelines to
staff being prepared by the Financial Management Sector Board.
-
72 A QAG Assessment
Improving the QSA Process for FM 7. New approaches to the QAG
process, applied during QSA5, should be continued in the future.
Process improvements of this review included training the reviewers
prior to the beginning of the exercise, improving communication
between reviewers and the FM coordinator, copying regional FM
Managers on draft reviews sent out by QAG, and involving the FM
Board Quality Committee in both reviewing the criteria prior to the
exercise and providing feedback once the reviews begin. In the
future, the following additional steps could help to improve the
reviews of supervision:
a) Focusing more on the PSR rating and how it was determined.
This rating, introduced only in March 2002, requires the team to
make a determination as to the quality of FM arrangements on the
project. While it was only applicable to a small part of the review
period, reviewers found the responses to the question of “How was
the FM rating determined?” to be a good indicator of the team’s
approach to FM supervision.
b) Incorporating more explicitly the extent to which interim
reports (especially Financial Monitoring Reports) are being
reviewed and utilized to monitor the project. Following the
simplifications to progress reporting requirements in the “fixing
LACI” exercise, QAG reviews should consider the impact of reporting
and the use of reports to monitoring the financial activity,
outputs, and procurement plan of the project.
c) Aligning FM supervision with revised business processes and
objectives. The FM Sector Board is currently studying how the
sector can shift its focus more heavily towards impact while still
paying appropriate attention to compliance. This study will assist
in re-defining “quality” FM work, and this new definition should be
reflected in the FM Supervision Guidelines and future QAG
supervision reviews. Some initial ideas would be for FM, and
perhaps QAG, to carry out reviews of the performance of supervision
on a country, or even regional, basis, in order to more closely
look at progress regarding some of the items (e.g. risk and
opportunity based approaches, regional quality arrangements for FM)
mentioned above.
-
Quality of Supervision in FY01/02 (QSA5) 73
ANNEX 9. SUPERVISION OF PROCUREMENT ASPECTS
Overview 1. The assessment of procurement supervision was done
by specialized procurement reviewers. The reviewers conducted
in-depth assessments of various aspects of procurement including
planning, implementation, progress monitoring, addressing
procurement issues and problems during project implementation, post
reviews, compliance with loan agreements and Bank policies, and
development of institutional procurement capacity. Results for the
QSA sample indicate that 91% of the projects benefited from
satisfactory supervision of procurement aspects (compared to 92% in
QSA4). Key Findings 2. Key findings were as follows:
• Percentage of projects with highly satisfactory procurement
supervision has remained same at 10% between QSA4 and QSA5
• Percentage of projects with satisfactory procurement
supervision has also remained same (92% in QSA4 and 91% in
QSA5).
• Procurement supervision of projects in HDN network (Health,
Education and Social Protection) has improved slightly in QSA5 to
86% (83% in QSA4).
• Although procurement plans were prepared at entry for most of
the projects, they were often not adhered to or used for monitoring
progress during implementation.
• Supervision reporting, documentation by task teams on
procurement status, were found to be the weakest aspect of
procurement supervision.
Table 1-A: QSA5 Ratings by Region for Procurement
Supervision
HS S HS or S M U Region
No. of Projects % No. % No % No. % No. % No.
AFR 25 7 3 74 17 81 20 17 4 2 1 EAP 15 16 2 80 12 96 14 5 1 - 0
ECA 19 2 1 96 17 98 18 2 1 - 0 LCR 16 22 2 62 11 84 13 16 3 - 0 MNA
11 - 0 100 11 100 11 - 0 - 0 SAR 8 10 1 83 5 93 6 7 2 - 0 Bank-wide
94 10 9 81 73 91 82 9 11 0.4 1
Note: Out of the sample of 100 projects, there were 6 projects
without any procurement.
-
74 A QAG Assessment
Table 1-B: QSA5 Ratings by Network for Procurement
Supervision
HS S HS or S M U Network
No. of Projects % No % No. % No. % No % No.
ESD 21 4 2 81 15 84 17 16 4 - 0 FSE 5 - 0 91 4 91 4 9 1 - 0 HDN
19 15 2 71 15 86 17 14 2 - 0 PREM 5 9 1 82 3 91 4 9 1 - 0 PSI 44 12
4 84 36 96 40 3 3 1 1 Bank-wide 94 10 9 81 73 91 82 9 11 0.4 1
Note: Out of the sample of 100 projects, there were 6 projects
without any procurement. Please note that when the Projects are
divided by Regions or Sectors, the number of projects in some
Regions and in some Sectors are too few to be statistically
significant.
Strengths and Weaknesses of Procurement Supervision
3. There was not much variation in the quality of procurement
supervision. There were some common features among the well
supervised projects, which are:
• Prompt responses provided to procurement issues raised by the
Borrower.
• TORs outline d procurement supervision tasks.
• BTORs and aide-memoires addressed procurement implementation
issues.
• Supervision team include d a Procurement Specialist (PS) or a
Procurement Accredited Staff (PAS).
• Project procurement plans are updated periodically. 4. The
general weaknesses in the supervision were:
• Lack of written reports on post-review of procurement
contracts below the prior-review threshold.
• Action plan/ recommendation for capacity building of
implementing agencies not followed through.
• Even projects with specialized procurement needs are designed
with the traditional approach and do not design specific
procurement arrangements for such purposes. (for fertilizers,
pesticides, live stocks, crop plants etc).
• Quality of supervision by PS and PAS is uneven. The PAS looks
after mainly the technical and overall (as TTL) aspects of the
project and then does not provide the necessary support and/or
supervision of procurement function for the project.
-
Quality of Supervision in FY01/02 (QSA5) 75
Issues Requiring Further Attention 5. There are two issues,
which may require further attention:
(i) Overall quality of procurement supervision by staff who are
PAS, compared with Procurement specialist have been found inferior.
Therefore, this issue needs to be followed up by the procurement
sector board.
(ii) Proper documentation on procurement supervision continues
to be a problem. Panels encountered difficulties retrieving in a
timely manner the key correspondence and progress reports related
to procurement issues. Where documented information was missing,
panels relied on verbal information provided by the task team.
Conclusions
6. The following points will be helpful for effective
procurement supervision:
• A procurement plan is essential. The procurement plan at
appraisal should be the “base” and should be updated/revised as
necessary or at least annually.
• Procurement plans should be monitored at least every quarter
and reviewed regularly during supervision. The PS/PAS should help
the implementing agency design a Procurement Monitoring Table that
includes details such as the bid/contract number, nature of
bid/contract, procurement method, name of supplier, contractor or
consultant, key procurement dates, value, origin of bidder, and
other relevant information. The Financial Management Report has a
section for procurement, which covers this aspect for contracts not
subject to prior review.
• The PS/PAS responsible for the project should communicate
regularly with the implementing agency and should be available to
provide prompt, quality responses to procurement issues raised by
the Borrowers. (Decentralization facilitates this, however,
evidence shows that a satisfactory level of service can also be
provided from HQ).
• The supervision mission’s TOR should clearly state the
supervision tasks related to procurement. During this mission,
project staff should:
(i) review the procurement progress against the plan and revise
as necessary; identify and address procurement issues affecting (or
having the potential to affect) the implementation of the
project;
(ii) identify and address pending contractual issues from
various parties (PIU, contractor, and consultant);
(iii) conduct post reviews of the procurement decisions below
the prior-review threshold and also review documents relating to
contract variations, price adjustments, etc. for prior-review
contracts;
(iv) review the procurement capacity of the implementing agency
and address training needs of the implementing agency staff,
including conducting workshops, and
(v) visit project sites and conduct actual physical
inspection.
-
76 A QAG Assessment
• The procurement rating in the PSR must be justified by the
write up in the implementation status and the issues sections.
Whenever a PS/PAS participates in a mission, an annex on
procurement progress should be attached to the aide-memoire or a
separate report should be written. The report should summarize
procurement activities, such as current status, planning of
remaining activities, problems/issues and actions/agreements with
the Borrower.
• Supervision missions follow-up letters should confirm the
mission’s findings and recommendations, summarizing the above
procurement problems/issues.
• The BTOR of the PS/PAS should include the post-review report
in the recommended format, and it should be sent to RPAs and
recorded in the regional monitoring system. If a post review
exercise detects non-compliance with the legal agreements, then the
report should include proposed remedial actions.
• The PS/PAS is responsible for the procurement supervision of
projects and, therefore, should be a party to any procurement
related communication with the Borrower.
-
Quality of Supervision in FY01/02 (QSA5) 77
ANNEX 10. SUPERVISION OF MONITORING AND EVALUATION ASPECTS
Overview 1. The assessment of monitoring and evaluation
(M&E) in supervision received special attention in QSA5, as a
team of specialized reviewers assisted the QSA5 panelists in
assessing 100 projects in the sample. The M&E ratings in the
main questionnaire were drawn from detailed comments and
sub-ratings which assessed (i) the appropriateness of the M&E
system to assess progress towards outcomes and implementation, (ii)
the use by the Bank of M&E information in supervision
(outcomes, outputs, implementation progress, and risks), and (iii)
the degree of recognition, follow up, and support to the borrower
when M&E issues arose (such as weak or non-functioning M&E
systems). 2. The level of satisfactory supervision for M&E was
65%, a decline from QSA4 (79%). It should be noted that in this
review a team could receive a satisfactory rating in M&E in
supervision without an appropriate M&E system in place (at
entry), if the Bank team made significant efforts to improve
project M&E. Many times, however, efforts were well intended,
but failed to be “technically” at par.
Key Findings 3. The key findings of the QSA5 M&E review
are:
• There are major shortcomings in the quality of the M&E
systems in place.
Only 42% of the projects had appropriate M&E systems, which
includes clear project development objective, measurable indicators
to track progress towards outcomes and consideration of borrower
capacity.
• There is an increased recognition that there are problems with
the quality of
the M&E system in place. However, the teams either do not
have suffic ient support to rectify the system (financial or human)
or they are overconfident that the team can restructure/redefine
the M&E system themselves. Few projects were able to address
identified weaknesses in M&E suff