Public Disclosure Authorized Quality Results€¦ · Guidance \ Questionnaire: Assessing Quality at Entry = target audience: Operations and OED staff levels 22+ w :\qag\guide\at_entry.

$Page 1: Public Disclosure Authorized Quality Results€¦ · Guidance \ Questionnaire: Assessing Quality at Entry = target audience: Operations and OED staff levels 22+ w :\qag\guide\at_entry.$
Quality Results(I publication of the W.1orclc B3atkls

Quality Asssur(lance Group

Quality is Everyone's BusinessThe message has beenz hard to miss lately: quality is everyone's business. With that commitment in fullforce, we see

strategic management playing a critical role in improving the health of the portfolio. Quality assessments are also gettilngstaff and managers thinking about new ways-and tried and true ways-of improving project quality.M W ff ore than anything else, in late June or early July to assess the Countries Targeted

portfolio improvement is adequacy of improvements in project and lor Special Attentiona matter of managerial portfolio performance.attention-and managers AFR:

are paying attention. Led by Mr. BOARD WELCOMES APPROACH AngolaWolfensohn through his discussions Burundiwith borrowers, operational managers The December 5 Board discussion Cameroonare taking up portfolio issues at the of the Annual Report on Portfolio Guineahighest levels. In the best examples, Performance provided an occasion for Madagascarcountry teams are conducting full the Board to comment on the value of Mozambiqueportfolio reviews and taking action: the PIP approach. Without exception, Nigeriaproblem projects are supervised inten- the executive directors saw the PIP as an Rwandasively and resources have been commit- important instrument for managing the Senegalted to resolve crucial issues. portfolio health in a strategic fashion. Togo

While it is still too early to measure "The introduction of the 'project[s] at Ugandathe overall impact of the Portfolio risk' concept is a major contribution andImprovement Program (PIP) which was advancement in the Bank's rating EAP:launched last April, we are beginning to system," commented one board mem- Papua New Guineasee a tumaround in several portfolios ber. For the Board and senior manage-with high concentrations of projects at ment, the approach assists them in ECA:risk, including Bangladesh, Brazil, monitoring the health of the portfolio.D HungaryKenya, Mozambique, Poland and Russia. Romania

- ~~~~~~Russian FederationNEW TARGETS SET Turkey

Following a recent stocktaking of LAC:the portfolio, we updated lists of projects - A M Brazilat risk of not meeting their development ereuMll~~l Venezuelaobjectives. Twenty-five country portfo- -T, -' y

lios with at least one half of their projects MNA:at risk or with one third of commitments Algeriaat risk are targeted for special attention. IranThe 100 largest projects at risk were also Yemenidentified for intensive supervision. We

will revisit these projects and portfolios

42~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Page 2 Quality Results

Our Toolkit: Identifying Projects at RiskThe Qualit Assurance Group uses the projects at risk approachto target management attention on projects considered to have ahigh risk of not meeting their development objectives. Thefollowing is a brief introduction to the tool. Long Effectiveness Delays

What are Projects at Risk? Significant Disbursement Delays

Long History of Past Problems

Projects at risk are made up of two types of projects: In a Risky Countryactual problem projects and potential problem projects.

Actual problem projects are those that the Regions Poor Macroeconomic Settingrate in the latest Form 590 as "unsatisfactory" for eitherimplementation progress (IP) or for the likelihood of In a Risky Subsectorreaching their development objectives (DO). Potential Environment/Resettlement Problemsproblem projects are associated with at least three of theleading indicators known to be precursors of future Poor Compliance with Legal Covenantsproblems. (See box at right for a list of these indicators.)

The projects at risk approach was refined in Decem-ber, and tested to see how good it was in predicting Project Management Problemsoutcomes. We applied the methodology retroactively toprojects under implementation back to fiscal 1989, and Poor Financial Performancecorrelated this with eventual outcomes based on OED Procurement Problemsevaluations. The projects at risk concept identified 68percent of projects with unsatisfactory outcomes, com-pared to only 40 percent identified by DO/IP ratings. You can also find it in the "W" drive under

The projects at risk concept is more fully explained in w:\qag\atrisk\note.doc. We encourage all staff to review it,a note to operational advisors and managers of February 7. and welcome your feedback.E

A Historical Look at Projects at RiskGood News: Signs point Bad News: 32 percent ofto improved performance ongoing projects are still g .

An analysis of trends using atk

the projects at risk concept Yet with over 32shows a noticeable reduction percent of ongoing projects in riskiness over the past at risk, we cannot afford tocouple of years. It reflects be complacent. This meansimprovement in several that we have to continue tounderlying factors. For one, work both to get the mostthe number of projects in out of the current portfolioextended problem status has and assure strong quality at been reduced. We are also entry for new projects.E 0 i

seeing fewer serious effec- -

tiveness delays-this suggestssome improvements in NO. of Projects U Net Comtesquality at entry.

Quality Results Page 3

Our Toolkit: Measuring Realism and Proactivity

We cannot expect that all Bank-supported projects willmeet their development objectives. Development meanstaking risks-we tackle complex issues; we work in coun- 0I O m

tries with weak institutions; we seek innovation. e R 1Yet we do expect that Bank performance be C100

percent satisfactory every time. Key to improving our own gperformance is identifying problems early, and moving _ _C___'___--_swiftly with our clients to find solutions.

Two new indicators are being used Bankwide to trackour performance: the Realism Index and the ProactivityIndex.

Because supervision ratings (DO and IP) are not asreliable as they should be-although they are substantially ___:7better than they were a few years ago-the Realism Index 86 84was developed. This index represents the ratio of actual 61problem projects to the number of projects at risk. 51

The Proactivity Index is the percentage of projectsrated as problem projects twelve months earlier that havesince been restructured, upgraded or closed. This indicator A_

measures the actions taken to resolve critical portfolioproblems. MN S

The Bankwide target for fiscal 1997 is 75 percent orbetter for both indices. Bankwide, the Realism Index iscurrently 61 percent; for Proactivity, 54 percent.R

, _ _, _, _. _. _, _, _ _ _ _ _;_, , 7,

P1~~~~~$n~~~ ~ ~

~~~~~~~~~~~~~~~6~~~~~~~~~ ~~~~~~~~~~~~~85

56~n 52 A ~At~ N

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _& - Q fJ a fla W i4 0


Quality Assessments: What's in StoreIf your project is targeted for a quality assessment or if you past and present. Field missions are included for someare a future panelist, you may have a few questions about assessments to interview borrowers, beneficiaries, fieldwhat is in store for you. Here are a few answers to the staff and other stakeholders.'who,' 'what,' 'where,' 'why' and 'how long' questionsthat may be foremost on your mind. How do panels reach a judgment?

The panelists utilize a guidance questionnaire toWhy conduct quality assessments? target their thinking on quality dimensions in categories

Quality assessments are about accountability-they such as project design, economic and financial analysis,are credible, independent assessments of the quality of the and quality of supervision performance. They assess anBank's work. Quality assessments are also an important individual rating for those dimensions, and finally assessmeans of learning about systemic issues affecting the an overall rating for the project: best practice, fullyquality of Bank operations. We use our learning from the satisfactory, marginally satisfactory or unsatisfactory.assessments to promote change in Bank process, policies,and programs. How long does the assessment last?

The process is intense-it lasts about twelve days forWho conducts assessments? assessments that do not include field visits. After the

Assessments are carried out by panels of about four assessment, the panel is available to the task team tosenior Bank staff and well-respected "outside" develop- provide advice and TA.ment professionals, each selected so that complementaryfields of expertise are represented on the panel. Who gets the assessment report?

The report is given to the regional vice president;Who is involved in the assessment? regional managers; the managing directors, operations; the

The panels interview operations staff and managers- head of the sector board; and OED. The regional linetask teams, senior operations advisors, division chiefs, managers are responsible for any follow-up.-country directors, and others-associated with the project,

W~~~~~~~ f g

0 2 g g | g , g

~~~h Pr I2J: 1t0


Reactions to Quality AssessmentsFor panelists and those on the receiv- but had a few words of caution. He for quality assessments do gain someing end, quality assessments are remarked, "Hindsight is easy, "real time" TA.intensive reviews. Here are some of we... .need always to remember just Augusta Molnar, task manager forthe reactions from completed how difficult project design is ......" a project in LAC, incorporated some ofassessments. the panel's suggestions into that project's

Task Teams' Reactions: "Tough but design even though the Region wasPanel Reactions: "Best Training" Worth It" otherwise ready to negotiate.

"Out of a number of theirWithout exception, the panelists Although technical assistance recommendations, two or three were

have found the experience to be a isn't a prirnary objective of the assess- really useful .... We have strengthenedwelcome learning opportunity for ments, task teams for projects targeted the indicators, made some changes inthem personally, yet a rigorous exer- institutional arrangements. We neededcise. One panelist commented that it stronger institutional agreements...."was "the best training" available for For Teresa Ho, task manager insenior managers. Andrew Rogerson, ECA, the experience was "tough butCountry Director for Central Africa 1, worth it." She went on to say that "itcommented that the experience was one of the most valuable discus-"opened up my eyes to ways of sions I have had in my time at thelooking systematically at quality." Bank. If such frank and open discus-

Julian Schweitzer, a panelist for a sions can continue, without rancor,pre-approval quality at entry assess- that is good for the Bank." A

ment, is very supportive of the process ta i gdrh a.

Improving Quality: New Tools and Training

Assessing Project Quality: the Guidance Questionnaire In cooperation with the Learning and Leadership Center,

the Quality Assurance Group is promoting hands-on training forThe panelists for quality assessments have commented on the Bank staff for building quality in project preparation and

value of using the guidance questionnaire as an analytical tool for implementation. Contact Sukai Prom-Jackson at the Learningreviews of project quality at the Regional level. In response, we and Leadership Center (tel. 34752) for more information abouthave made the guidance questionnaires-for both assessing quality the following training programs:at entry and quality in supervision-available on the W drive.

* Project Preparation and Loan Processing: Quality at Entry\ Guidance Questionnaire: Assessing Quality at Entry = target audience: Operations and OED staff levels 22+

w :\qag\guide\at_entry. doc

* Assuring Project Quality-Middle Managers* Guidance Questionnaire: Assessing Supervision Quality X target audience: Operations middle managersw:\qag\guide\superv.doc

* Project Supervision and ImplementationAssessing Borrower Commitment/Institutional Capacity +PoetSpriinadIpeettoA=> target audience: task team leaders, resident mission staff,and senior staff who wish to update their skills

The Quality Assurance Group has recently launched aneffort in cooperation with the Networks and OED to developuser-friendly tools with verifiable, analytical indicators to assistBank staff in assessing borrower commiment and institutionalcapacity for project implementation. We will report on th is in afuture edition of Quality Results.

Palge 6 Quality Results

Findings: Quality Assessments Reveal Recurring Themes

With only seven projects assessed, it's still early days.However, some recurring themes have emerged from the * Risk and stakeholder assessments have been poor inreviews. many cases.

* We have seen some great examples of talented, * Project complexity continues to unnecessarily increasecommitted task teams. However, they haven't always project risks. The best practice cases use pilots andgotten the best out of the Bank's store of global phasing to manage risks.knowledge and experience about what works and whatdoesn't. * Task teams too often are flying solo-their managers

are not always active in resolving difficult policy* The peer review system is not working (see box issues. In the best projects, senior managers provide

below). The creation of the Networks provides an clear, timely guidance, and support the team by raisingopportunity to develop more effective ways of getting these issues with their counterparts when high levelhigh quality, high impact independent advice to task intervention is needed.teams.

Look for Findings in every editionof Quality Results


Managers Reflect on Poor Performance

When Bank performance is poor, it is first and foremaost a Most of the responses demonstrated that the manag-managerial accountability. Thirty-seven Bank managers ers concerned took ownership for the problems in Bankassociated with projects that were judged in a recent OED performance under their watch. There were a few quibblesreport to have had unsatisfactory performance throughout with the OED findings, and indeed many of the responsesthe project cycle were asked by the managing directors, emphasized that the ICRs themselves had provided candidoperations to reflect upon the reasons for poor Bank accounts of Bank performance.performance. Listed below are some of their comments.

Quality at entry suffered from... Supervision performance can be improved by...

* Over-ambitious objectives given the social, political and * Focusing on project start-up. Consider high supervisioninstitutional environment intensity and cost in the first year for smoother implementa-

* Inconsistency between project objectives and instruments tion later* Unrealistic financial covenants * Better monitoring and evaluation indicators* Inadequate technical appraisal * Greater recourse to restructuring when original project design

is no longer relevant or appropriate* Collaborative yet rigorous and predictable enforcement of

covenants

The 1987 reorganization had costs...

K It led to a loss of project ownership due to Bank staffdiscontinuities and uncertainties

Source: OED, Annual Review of EvaluationResults,1995ts

Source: OED, Annual Review of Evaluation Results, 1995


Staff and Managers Honored for Excellence

In a reception held January 24, 1997, the managing directors, operations recognized staff andmanagers associated with seven projects where Bank performance was evaluated by OED to be exceptional

throughout the project cycle. Each individual was singled out by their managers and colleagues ashaving had a major impact on project success.

The forty-eight individuals honored (see list below) include task team members, support staff,directors, division chiefs, lawyers, operations advisors, technical specialists, economists and

disbursement officers. Congratulations for a job well done.

Marianne Haug Valeriano GarciaNick Hope Edgardo FavaroPeter Scherer Geoffrey ShepherdJean Paul Pinard Cheikh KaneEugene McCarthy Izumi OhnoMihkel Sergo Janet EntwistleSalahuddin Khwaja Denise von Gersdorff

Rudolf Van PuymbroeckGabriela Huffman

Hans Binswanger Geoffrey ShepherdMichael Baxter Valeriano GarciaJorge Garcia Garcia Janet EntwistleJohn McIntire Izumi OhnoLouise Cord Rudolf Van PuymbroeckJaime Fernandez (IDB) Angelica SilveroRoberto LaverRudolf Van PuymbroeckMarcos SugarMichael Fowler Katherine MarshallMargarete Rodousakis Peter WatsonTeresa Roncal Leslie PeanNelly Lukas Christiane Eliopoulos

Joanne NickersonMax Pulgar-VidalFriedrich Peloschek

Basil Kavalsky Hovsep MelkonianWafik GraisGeoffrey FoxJonathan Walters Rory O'SullivanClaude Blanchi Trayambkeshwar SinhaKaren BrooksTjaart Schillhorn Van VeenRadwan Al-Jabri

SmQ ,~ ~ ~~~~~1M o--.-

Public Disclosure Authorized Quality Results€¦ · Guidance \ Questionnaire: Assessing Quality at Entry = target audience: Operations and OED staff levels 22+ w :\qag\guide\at_entry.

Documents