Document of The World Bank Report No: ICR2078 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-38670) ON A CREDIT IN THE AMOUNT OF SDR 61.0 MILLION (US$ 89.0 MILLION EQUIVALENT) TO THE REPUBLIC OF INDIA FOR A RAJASTHAN HEALTH SYSTEMS DEVELOPMENT PROJECT March 28, 2012 Human Development Sector Unit South Asia Region Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized
70
Embed
Document of The World Bank Report No: ICR2078 ......ANM Auxiliary Nurse-Midwife ASHA Accredited Social Health Activist ... TABLE OF CONTENTS ... However, the results framework was
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Document of
The World Bank
Report No: ICR2078
IMPLEMENTATION COMPLETION AND RESULTS REPORT
(IDA-38670)
ON A
CREDIT
IN THE AMOUNT OF SDR 61.0 MILLION
(US$ 89.0 MILLION EQUIVALENT)
TO THE
REPUBLIC OF INDIA
FOR A
RAJASTHAN HEALTH SYSTEMS DEVELOPMENT PROJECT
March 28, 2012
Human Development Sector Unit
South Asia Region
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
ii
CURRENCY EQUIVALENTS
(Exchange Rate Effective 09/30/2011)
Currency Unit = Indian Rupee (INR)
US$ 1.00 = INR 49.06
FISCAL YEAR
April 1 – March 31
ABBREVIATIONS AND ACRONYMS
A-M Aide-Memoire
ANM Auxiliary Nurse-Midwife
ASHA Accredited Social Health Activist
BCC Behavior Change Communication
BPL Below Poverty Line
CAG Comptroller and Auditor-General
CAS Country Assistance Strategy
CHC Community Health Center
CTF Common Treatment Facility
DH District Hospital
DPC District Project Coordinator
EmOC Emergency Obstetric Care
FM Financial Management
GOI Government of India
GOR Government of Rajasthan
HAF Hospital Activity Format
HCWM Health Care Waste Management
HMIS Health Management Information System
HR Human Resources
HSIT Health System Improvement Team
ICR Implementation Completion Report
IDA International Development Association
IEC Information and Education Campaign
INR Indian Rupees
ISR Implementation Status Report
JSY Jonani Surakshi Yojana
M&E Monitoring and Evaluation
MoHFW Ministry of Health and Family Welfare
MCH Maternal and Child Health
MTR Mid-Term Review
NFHS National Family Health Survey
iii
NGO Non-Governmental Organization
NRHM National Rural Health Mission
PAD Project Appraisal Document
PDO Project Development Objective
PMU Project Management Unit
PPP Public-Private Partnership
PPR Procurement Post Review
PSA Procurement Support Agency
QAG Quality Assurance Group
QEA Quality-at-Entry Assessment
QER Quality Enhancement Review
RCH Reproductive and Child Health
RHSDP Rajasthan Health Systems Development
Project
Rs Rupees
SC Scheduled Caste
SIHFW State Institute of Health and Family Welfare
ST Scheduled Tribe
TOR Terms of Reference
TTL Task Team Leader
Vice President: Isabel M. Guerrero
Country Director: N. Roberto Zagha
Sector Director: Amit Dar (Acting)
Sector Manager: Julie McLaughlin
Project Team Leader: Patrick M. Mullen
ICR Team Leader & Author: Caryn Bredenkamp
iv
INDIA
Rajasthan Health Systems Development Project
TABLE OF CONTENTS
A. Basic Information ....................................................................................................... v B. Key Dates ................................................................................................................... v
C. Ratings Summary ....................................................................................................... v D. Sector and Theme Codes .......................................................................................... vi E. Bank Staff .................................................................................................................. vi F. Results Framework Analysis .................................................................................... vii
G. Ratings of Project Performance in ISRs .................................................................... x H. Restructuring (if any) ................................................................................................ xi
I. Disbursement Profile ................................................................................................. xi
1. Project Context, Development Objectives and Design ............................................................... 1
2. Key Factors Affecting Implementation and Outcomes ............................................................... 6
3. Assessment of Outcomes ........................................................................................................... 15
4. Assessment of Risk to Development Outcome ......................................................................... 23
5. Assessment of Bank and Borrower Performance ...................................................................... 24
C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)
Bank Ratings Borrower Ratings
Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory
Quality of Supervision: Satisfactory Implementing
Agency/Agencies: Moderately Satisfactory
Overall Bank
Performance: Moderately Satisfactory
Overall Borrower
Performance: Moderately Satisfactory
vi
C.3 Quality at Entry and Implementation Performance Indicators
Implementation
Performance Indicators
QAG Assessments
(if any) Rating
Potential Problem Project
at any time (Yes/No): No
Quality at Entry
(QEA): Moderately Satisfactory
Problem Project at any
time (Yes/No): Yes
Quality of
Supervision (QSA): None
DO rating before
Closing/Inactive status:
Moderately
Satisfactory
D. Sector and Theme Codes
Original Actual
Sector Code (as % of total Bank financing)
Compulsory health finance 10 10
Health 60 60
Non-compulsory health finance 10 10
Other social services 10 10
Sub-national government administration 10 10
Theme Code (as % of total Bank financing)
Child health 20 20
Health system performance 40 40
Other communicable diseases 20 20
Population and reproductive health 20 20
E. Bank Staff
Positions At ICR At Approval
Vice President: Isabel M. Guerrero Praful C. Patel
Country Director: N. Roberto Zagha Michael F. Carter
Sector Manager: Julie McLaughlin Anabela Abreu
Project Team Leader: Patrick M. Mullen Sadia Afroze Chowdhury
ICR Team Leader: Caryn Bredenkamp
ICR Primary Author: Caryn Bredenkamp
vii
F. Results Framework Analysis
Project Development Objectives (from Project Appraisal Document) PDO1: Increase access to health care of poor (BPL) and underserved population by upgrading health
care facilities in the remote areas, promoting public private partnership and improving health care
seeking behavior through demand side interventions.
PDO2: Improve the effectiveness of health care through strengthened institutional framework for
policy development, program implementation and management capacity, and increase in the quality
of health care.
Revised Project Development Objectives (as approved by original approving authority)
The PDOs were not revised. However, the results framework was changed in April 2006 in order to
make the indicators measurable, as well as to reduce the overall number of indicators. Target values
were also introduced for the first time then, and baseline values were put in. Some of these target
values were formally revised in December 2009.
(a) PDO Indicator(s)
Indicator Baseline Value
Original Target
Values (from
approval
documents)
Formally
Revised
Target
Values
Actual Value Achieved
at Completion or
Target Years
Indicator 1 : Percentage of outpatients seen at project facilities who are from BPL households
Value
quantitative or
Qualitative)
8.7% 50% 17% 16.6%
Date achieved 01/31/2006 04/30/2006 12/30/2009 06/30/2011
Comments
(incl. %
achievement)
97.6% achieved
Source: Monthly hospital activity formats
Indicator 2 : Percentage of inpatients seen at project facilities who are from BPL households
Value
quantitative or
Qualitative)
8.5% 50% 13% 17%
Date achieved 01/31/2006 04/30/2006 12/30/2009 06/30/2011
Comments
(incl. %
achievement)
130.8% achieved
Source: Monthly hospital activity formats
Indicator 3 : Percentage of inpatients seen at 49 project facilities in 6 tribal districts who are from
scheduled tribe (ST) households
Value
quantitative or
Qualitative)
8.3% 50% 15% 24.9%
Date achieved 01/31/2006 04/30/2006 12/30/2009 06/30/2011
Comments
(incl. %
achievement)
55.3% achieved
Source: Monthly hospital activity formats
Indicator 4 : Percentage of project community health center (CHC) facilities conducting more
than 10 deliveries in a month
Value 60% 90% 96.6%
viii
quantitative or
Qualitative)
Date achieved 01/31/2006 04/30/2006 06/30/2011
Comments
(incl. %
achievement)
107.3% achieved
Source: CHC MIS
Data include both CHCs and upgraded PHCs
Indicator 5 : Percentage of upgraded First Referral Units (FRUs) offering 24 hour CEmOC
Value
quantitative or
Qualitative)
0 25%
Date achieved 04/11/2004 04/11/2004
Comments
(incl. %
achievement)
This indicator was not monitored by the project since activities taken over by
NRHM. However, it was not formally dropped as a PDO indicator, either.
Indicator 6 : Percentage of clients (patients and non-patients) satisfied with the services received
at project facilities
Value
quantitative or
Qualitative)
92% were satisfied with
doctors; 85% were
satisfied with nurses
"Increasing
satisfaction"
94% were satisfied with
doctors; 72% were
satisfied with nurses
Date achieved 09/30/2008 04/30/2006 09/30/2011
Comments
(incl. %
achievement)
100% achieved for doctors; not achieved for nurses.
Indicator 7 : Percentage of the following categories staffed in project facilities: (i) Doctors, (ii)
Source: National Family Health Survey (NFHS) data, 1998/99.
4. Specific health system challenges at the time of appraisal included inadequate
institutional arrangements and weak program management; the low quality of services in both
public and private sectors; ineffective targeting of public funds to the poor; an inadequate
framework for engaging the private sector; low efficiency; and, limited financial resources.
Furthermore, the health system was characterized by insufficient integration of health, family
welfare and disease control programs, as well as inadequate coordination and integration at
2
different service delivery levels. Lack of adequate and trained manpower and weak management
of human resources was a statewide concern, especially in the tribal and hard-to-reach areas.
5. At the time of appraisal, the resources available to address the above-mentioned
challenges were constrained and investment in the health sector had been declining for decades.
The health share of the Rajasthan state budget fell from over 8% in 1980/81 to 6.6% in 1997/98
to 5.4% in 2000/01 to 4.5% in 2003/04 (Reserve Bank of India data). Even as a share of total
social expenditure, the health sector share had been declining (see Annex 7, Figure1).
Recipient’s policies, strategies, commitment and actions
6. The RHSDP was to be implemented in one of the Indian government‟s priority states. In
2001, the Ministry of Health and Family Welfare identified Rajasthan as one of the eight
“Empowered Action Group” (EAG) states that would benefit from targeted reforms and programs
(on the basis of its large population and its very poor health indicators). The state‟s “Health
Vision for Rajasthan 2025” articulated the following goals: reduce IMR to below 30 by 2025,
reduce MMR to below 100 by 2025, increase delivery by skilled attendants to 85% by 2010,
increase immunization coverage to 90% by 2010 and increase the tuberculosis treatment rate to
85% by 2010. The “Health Sector Policy Matrix”, prepared by the Government of Rajasthan to
more clearly articulate its health sector priorities and inform project preparation, emphasized
increasing financing and improving resource allocation in the health sector, strengthening
capacity for management, planning and coordination, enhancing participation of the private and
voluntary sectors, increasing access and equity of access to health care, cost-sharing and
sustainability, providing incentives to the workforce and redressing regional imbalances
(including disparities in tribal and remote districts). The government‟s commitment to these areas
is further evidenced by the reforms that were ongoing at the time of preparation, including
subsidies to BPL patients, contracting out specialized services, overcoming medical staff
shortages in rural areas, and creating an enabling environment for partnership with the private
sector.
Rationale for Bank involvement:
7. The 2002-2004 Country Assistance Strategy (CAS) provided the framework for project
design and engagement. The CAS supported the objective of developing more efficient and
effective health prevention and care systems at the state level that would better serve the needs of
the poor. Additional 2002 CAS objectives supported by the project included enhancing the role of
the private sector in achieving important public health goals, improving governance, and
enhancing community participation and empowerment.
Related projects
8. Within the health sector in India, there were a number of state-level “health systems
strengthening” projects under implementation when RHSDP was being prepared (see Annex 7
Table 1). The RHSDP was considered the first of the “second generation” of health system
strengthening projects, incorporating lessons learned from the preparation and implementation of
earlier projects in Andhra Pradesh, Maharashtra, Orissa and the multi-state project in Karnataka,
Punjab and Tamil Nadu that were approved in the late 1990s. Other state health projects followed
in Tamil Nadu, Karnataka and Uttar Pradesh. These health system strengthening projects were
complemented by support to the national programs and disease-specific initiatives. In other
sectors, the Bank portfolio in Rajasthan at the time of appraisal included primary education,
district poverty initiatives, water sector restructuring, and power sector restructuring (see Annex 7
3
Table 2). In addition, there was a modest program of technical assistance to the state, including in
the areas of strengthening governance, financial management, and regulatory reforms.
1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved)
Original PDOs:
9. The PDOs, as they appear in the Project Appraisal Document (PAD)1, are follows:
(i) Increase access of poor [i.e. below the poverty line (BPL)] and underserved
population to health care, and
(ii) Improve the effectiveness of health care through institutional development and
increase in the quality of care.
Original key outcome indicators:
For PDO 1:
(i) Increased utilization of government health services by poor (BPL) and Scheduled
Tribes (ST) households;
(ii) Increased proportion of BPL/Scheduled Tribe patients among those attended at
government health facilities at various levels;
(iii) Increased number of BPL/Scheduled Tribe patients exempted from user fees at
government health facilities;
(iv) Increased awareness of poor and tribal households of health services offered by
different levels of government health facilities.
For PDO 2:
(v) Constant or rising expenditure on primary and secondary levels of care;
(vi) Increased proportion of non-wage expenditure in total health expenditure;
(vii) Increased proportion of government health care facilities staffed according to
agreed norms;
(viii) Decreased irrational use of drugs in government health care facilities;
(ix) Increased patient and community satisfaction with primary and secondary levels
of health care services;
(x) Increased number of appropriate referrals at community health centers (CHCs),
district and sub-divisional hospitals from primary health centers (PHCs) and sub-
centers;
(xi) Increased health care utilization in terms of number of outpatient visits, hospital
admissions, and deliveries.
1 While the PDOs are framed slightly differently in the PAD and the Development Credit Agreement
(DCA), they are identical in spirit and scope. The PDOs, as stated in the DCA, are to assist Rajasthan in
improving the health status of its population, in particular the poor and underserved population through: (i)
providing such populations with equitable and greater access to healthcare; and (ii) improving the
effectiveness of health care through institutional development and increase in the quality of health care.
4
1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and
reasons/justification
10. The PDOs were not revised. However, the key results indicators were revised to a set of
seven PDO indicators and 11 intermediate outcome indicators in 2006 because the original
indicators were considered too numerous (11 outcome indicators, 32 intermediate outcome
indicators), many did not satisfy “SMART” measurement criteria2, and a number still lacked
baseline data (see section 2.3 for details) and targets. This was mutually agreed by the Bank and
Government and recorded in the Aide-Memoire of April 20063.
Revised indicators for PDO 1:
(i) Percentage of BPL populations among outpatients seen at all project facilities,
i.e. district (DH) and sub-divisional hospitals (SDH and CHC);
(ii) Percentage of BPL among inpatients seen at all project facilities;
(iii) Percentage of ST populations among inpatients seen at all 49 project facilities in
six tribal districts, i.e. at district (DH) and sub-divisional hospitals (SDH and
CHC) in six tribal districts.
Revised indicators for PDO 2:
(iv) Percentage of CHCs conducting >10 deliveries in a month;
(v) Percentage of upgraded first referral units (FRUs) offering 24 hr EmOC;
(vi) Percentage of clients (patient and non-patient) satisfied with the services received
at the project facilities;
(vii) Percentage of the following categories staffed in project facilities (a) Doctors, (b)
Nurses/ANMs and (c) Lab technicians.
11. These indicators remained in place for the remainder of the project. However, the targets
for the three PDO 1 indicators were subsequently revised downwards from their 2006 levels
during a November 2009 restructuring (see section 2.3 for details).
1.4 Main Beneficiaries
12. The beneficiaries included the entire state population that would access health care at
primary and, especially, secondary-level government health facilities, but with a special emphasis
on poor and tribal households. In fact, PDO 1 focused explicitly on the well-being of the “poor”
and “under-served” populations, concepts that the project operationalized as households below
the poverty line (BPL) or with scheduled tribe (ST) designation. 238 project facilities (including
at least one hospital or community health center per block) were identified for renovation and
refurbishment investments, but with additional facilities also benefiting from some of the “soft”
investments, such as training, health care waste management interventions and HMIS
improvement. Quantitatively, the project was expected to directly benefit an additional three
million outpatients and 34,000 in-patients annually through the expansion of services, as well as
an existing 10 million outpatients and 300,000 in-patients annually from improvements in the
2 A “SMART” indicator is Specific, Measurable, Achievable, Relevant and Time-bound.
3 At that time, a change in the outcome indicators did not require formal Board approval.
5
quality of services. Strengthening of government institutions involved in health policy and
planning was also envisaged.
1.5 Original Components (as approved)
13. Component One: Project management, policy development and capacity-building
(US$19.32 million appraised). This component was to support the establishment of a Strategic
Planning Cell to build institutional capacity for health policy development and planning. It was to
improve state capacity for public-private partnerships (PPPs), develop a regulatory framework for
the provision of quality health care in the private sector, build health management information
systems, and conduct clinical and managerial training. This component was to also support
overall project implementation through the establishment and operationalization of the following
bodies: a State Empowered Committee, Project Steering Committee and Project Management
Unit (PMU) at the state level, as well as Project Coordination and Monitoring Committees and
District Project Management Cells (DPMC) at the district level.
14. Component Two: Development of Primary and Secondary Health Care Services in the
Public Sector (US$50.41 million appraised). This component was to support the strengthening of
28 district hospitals, 23 sub-district hospitals, 185 community health centers, and 2 block-level
primary health centers. First, there were to be investments in infrastructure (including the
renovation of facilities and construction of specialized wards/units) and provision of equipment,
furniture, drugs, medical supplies, as well as HMIS and IEC materials. Second, there were to be
“soft” investments to improve the clinical quality of care through the development of clinical
protocols, strengthened drug prescribing practices, stronger M&E systems, and improved referral
mechanisms. This component was also to be responsible for the institutionalization of facility-
based health care waste management protocols and practices.
15. Component Three: Health Care Innovations for the Disadvantaged (US$31.92 million
appraised). This component was to improve the access (including geographical, financial and
social access) of disadvantaged populations (namely SC, ST and BPL populations) to secondary
health care. This was to be achieved through the targeted operationalization of an IEC strategy,
outreach camps, leveraging traditional systems of care for provision of essential medical services,
and piloting various initiatives designed to improve access to care.
1.6 Revised Components
16. The project components were not revised. However, existing activities were realigned at
Mid-Term Review in 2007 in order to respond to the changing health sector environment and,
during the two year extension phase, some new activities were added. This will be discussed in
section 2.2.
1.7 Other significant changes
17. The RHSDP was approved on March 11, 2004 (SDR 61 million, US$89 million) for five
years six months with an original closing date of Sept 30, 2009.
18. Extension and reallocation (Sept 2009): On September 4, 2009, the project was extended
by two years until September 30, 2011. The extension, which would utilize the US$29 million
unspent credit, was intended to continue funding for planned activities, but also support some
additional activities and inputs (see section 2.2). The large unspent credit was a result of several
factors, including a partial completion of planned activities (especially on Component 3),
6
strengthening of the dollar (from Rs 45.23/US$ in March 2004 to Rs 48.87/US$ in Sept 2009),
and savings on the procurement of goods, works and consultancies. Extension also involved a
reallocation of 13.8% of the credit across categories, towards civil works (mainly due to an
increase in the costs of steel, cement and other raw materials) and away from training and
workshops (since the government no longer wanted to support international training). See Table
for the change in expenditure across components and Table for the changes in allocations across
expenditure categories during implementation (Annex 1).
19. Level II restructuring – revision of PDO targets and deletion of covenant (Dec 2009): In
December 2009, the project was restructured in order to (i) revise the targets of the indicators of
PDO 1 to more realistic targets (see section 2.3) and (ii) delete the covenant on the “development
of a regulatory framework for the private sector” since the Clinical Establishment Act (eventually
passed in May 2010) was being drafted at the national level and would be reflected in state-level
legislation.
20. Level II restructuring – cancellation of credit (Sept 2011): Two weeks before project
closing, at 78.6% disbursement, the project underwent a Level II restructuring in order to cancel
SDR 6.875 million (US$11 million) from the credit, reducing the final credit amount to SDR
54.125 (US$84.5 million). The unspent credit mainly reflected partial completion of a number of
project activities planned for the extension phase, but also savings from exchange rate
fluctuations.
21. There were no other major changes to the project design, scope, scale, implementation
arrangements or schedule during the course of implementation that were not envisaged at project
design. Within the scope of the original PDO and design, however, there were some shifts in
emphasis during implementation that will be discussed in Section 2.
2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design and Quality at Entry
22. The scope for engagement was defined by the 2002 CAS and government priorities
(described in section 1.1) and the project was well-aligned with them. In particular, it dovetailed
well with the CAS‟s emphases on reaching the poor and engaging the private sector, as well as
the government‟s emphasis on addressing the needs of vulnerable groups (including BPL and
tribal populations), especially in Rajasthan which the national government considered a focus
state.
23. The development objectives were highly relevant and clearly-stated. The emphasis on
reaching the poor and reducing inequalities in access to care (PDO 1) was highly appropriate
given the high degree of inequality across castes and income groups (see Table 1) and the focus
on improving the quality of care (PDO 2) were essential due to years of underinvestment in health
care infrastructure and processes by the state, and the resultant health system challenges described
in section 1.1.
24. The design used to achieve these objectives was a combination of traditional input-based
investment (e,g. renovation, construction, provision of equipment, clinical training, PMU-
strengthening) and new generation health system strengthening interventions and innovations (e.g.
institutional capacity-building, health information systems, public-private partnerships,
managerial training, health care waste management, and innovations/ interventions to reach poor
and vulnerable households). The traditional inputs of the project were appropriate and sorely
7
needed, while the new generation activities served to maximize the returns to investment in these
inputs and strengthen health system institutions and processes, e.g. through Health System
Improvement Teams (HSITs), investment in health care waste management (HCWM) and
training. The project design was also appropriate to address the particular characteristics of
Rajasthan‟s health system (e.g. the large share of the private sector in health service delivery) and
its socioeconomic profile (e.g. the large share of ST households). On balance, the design was not
overly complex and could be feasibly implemented within the project period, given prevailing
capacity.
25. The project design also reflected lessons learned from the experience of previous
projects4, including the importance of improving sectoral planning and program management,
targeting the poor and vulnerable (including those living in remote areas), involving the private
sector, and enhancing the quality of care. To complement these lessons, many studies and
assessments were undertaken during project preparation in order to provide the analytical basis
for project design, as well as provide opportunities for participation in the design by the potential
beneficiaries5.
26. Good consideration was given in project design to both issues of sustainability and to risk
mitigation. Project expenditure was set at levels that were considered to be financially sustainable
(with the appropriate supporting calculations undertaken) and the design incorporated plans to
slowly internalize and institutionalize a number of the project activities, such as HCWM
processes and the Health Management Information System (HMIS) strengthening activities.
Attempts were made to mitigate the risk of slow procurement, and hence disbursement, by
extensive preparation of procurement plans and related documents6. Measures were also put in
place to help to ensure that there was no disruption in the flow of funds. One risk for which there
was no sound mitigation strategy, and about which the team should have been more concerned
given past project experience, was human resources – both at the facility level and in the PMU.
Indeed, staffing at PMU level was one of the major factors affecting project implementation (see
section 2.2).
4 A critical input was the findings of the 2002 Quality Enhancement Review (QER) of a number of
previous India state health systems development projects. It emphasized the importance of promoting
demand-side interventions, private sector engagement, institutional development, governance and policy
reforms, as well as adjusting state projects to fit state context and capacity. 5 These included a social assessment study (including a beneficiary needs assessment and an NGO
assessment), a tribal development plan based on consultations with communities and NGOs, a draft gender
strategy, a facility survey, an institutional assessment, a review of clinical service norms, an analysis of
community-based health insurance schemes elsewhere in India, a public-private partnership feasibility
study, a study of pilot programs for reaching the poor, and a study of existing larger-scale programs for
reaching the poor. 6 These included a revised draft of the health care waste management plan, a tribal development plan,
revised on the basis of consultations with community and NGOs, a draft gender strategy; a project
implementation plan for improved access for disadvantaged populations, a draft terms of reference
prepared for most studies and consultancies planned for the first year of the project, draft procurement
plans for the total project period, a draft TOR for external audit of the project by the Comptroller and
Auditor General of India (CAG), detailed drawings and the costing for 5 of the 27 works planned for Year
1, and a draft of the Health Sector Development Policy Matrix.
8
27. The preparation period was lengthy, even compared to other health projects in India, at
almost 27 months7. This was mainly due to the time needed to obtain government support for the
design (especially the “soft” health system strengthening interventions and Component 3) and to
set up implementation arrangements (including the detailed preparations for procurement
described above). A Quality Assurance Group (QAG) review, undertaken in 2005, rated overall
quality-at-entry as moderately satisfactory, highlighting as strong points the effort made to obtain
government buy-in, the targeting of the vulnerable, and the building of partnerships with
government, NGOs and the private sector. It also noted, as weak points, the quality of the results
framework and insufficient attention to the preparation of Component 3. This ICR concurs with
the first point, (see section 2.3), but considers analytical work undertaken to support Component
3 adequate, especially given the team‟s intention to move forward incrementally with those
interventions, many of which would operate on a small-scale and pilot basis. The project moved
from approval to effectiveness within four months.
2.2 Implementation
28. Overall implementation performance can be characterized as weak at the beginning,
strong in the middle period leading up to extension, and weak again during the extension phase.
Implementation performance mirrors performance on procurement and staffing/human resources
which were the chief factors affecting implementation. The project also made some explicit shifts
during implementation in order to be more complementary to new national programs, but
remained within the scope of the original PDOs and components. Factors affecting
implementation include:
29. Human resources: The project experienced five changes in leadership at the project
director level during its seven year period which was disruptive to project implementation,
especially during the extension phase when there was not only rapid turnover of management, but
the incumbent project directors held several other responsibilities in addition to RHSDP.
Vacancies were frequent, again especially during the extension phase when the project lost core
staff (in biomedical engineering, M&E, health care waste management, data analysis and IEC)
who were never replaced with the result that the responsibilities and workload of remaining staff
increased. The impact on project implementation was clear: processing time of the procurement
planned during extension increased and critical decision-making was delayed. By contrast, at the
district level, while it took a long time to fill the positions of District Project Coordinators (DPCs),
until April 2006, once filled, these positions had little turnover and implementation capacity
improved over time.
30. Procurement (civil works and equipment): Procurement barely moved during the first
few years of the project when procurement was managed by procurement support agencies
(PSAs)8 because of the low capacity of the PSAs (despite all the procurement materials that had
been prepared during project preparation) coupled with the non-availability in desert and tribal
areas of contractors who could fulfill the bid criteria. During this period, procurement was rated
as unsatisfactory twice, both in 2005 (due to the poor performance of the first PSA), but also,
since civil works and goods together constituted about half of the project funds, delayed
7 There was a project identification mission in Oct-Nov 2001, four intervening preparatory missions,
negotiations on Jan 20, 2004, Board approval on Mar 11, 2004 and effectiveness on July 21, 2004. 8 These were firms that were contracted to manage procurement for the project in the first few years so as to
avoid procurement difficulties, while simultaneously strengthening PMU capacity for procurement.
9
procurement was a major driver of slow disbursement, and also contributed to the less than
satisfactory performance on the second component in which most of the procurement was
concentrated. When the PMU and DPMUs took over responsibility for procurement (from 2006-
2008), most of the annual planned procurements were initiated within the year for which they
were planned, but typically with delay. During the extension phase, when procurement was the
sole responsibility of the PMU, procurement delays were primarily due to the PMU‟s difficulty in
managing large ICB procurements and a risk-averseness in making decisions related to the bids
for fear that there would be complaints that about the process. This was aggravated by turnover of
procurement staff and lengthy vacancy periods. Additional procurement issues are discussed in
section 2.4.
31. New government health programs and additional health financing: Subsequent to project
effectiveness, the Government of India (GOI) launched a number of large new health programs
which shared a number of the objectives of RHSDP and injected substantial additional resources
into the health system that were not anticipated during project preparation, necessitating a re-
alignment of RHSDP. The National Rural Health Mission (NRHM) was introduced in April 2005
as the Government‟s “flagship” health program, providing an overarching strategic direction and
umbrella structure to health system development. Moreover, Rajasthan was identified as a focus
state for NRHM, meaning that significant additional funding was made available to improve the
health outcomes of its population (especially the poor, vulnerable and rural groups). Subsequently,
in November 2006, the centrally sponsored Reproductive and Child Health II (RCH II) program
was launched under NRHM with the aim of strengthening health systems for improved maternal
and child health. It emphasized reaching the poor and vulnerable, continuity of care and evidence-
based programming. The Jonani Surakshi Yojana program (JSY), introduced in 2005 but with
implementation only gathering speed later, provided demand-side and supply-side financial
incentives to improve maternal health9. Partly as a result of, but also independent of, the programs
described above, both the state and central governments dramatically increased their spending on
health services in Rajasthan during project implementation. Government health expenditure grew
from approximately US$116 million in 2004-05 (US$1.91 or Rs 84 per capita) to US$443 million
in 2010-11 (US$6.46 or Rs 296 per capita) such that RHSDP eventually accounted for an annual
average of 3.5% of total government health spending rather than the 5% originally envisaged.
32. The project adapted well to this changing environment in the health sector. In particular,
at Mid-Term Review (July 2007), key decisions were taken to ensure that the project
appropriately aligned itself with the NRHM and that RHSDP resources were used in a
complementary fashion, inputs were synergized and there was no duplication in programming.
Specifically, this involved the realignment of Component 1 to focus on supporting NRHM
implementation, identifying where RHSDP and NRHM would respectively take the lead10
.
9 In Rajasthan, pregnant women who deliver in a government (and accredited private) institution are
rewarded with financial payments, with higher benefits paid in rural than in urban areas. Community health
workers – known as accredited social health activists (ASHA) – receive payments for institutional delivery,
postnatal visits and BCG vaccination. Private sector providers can be paid for performing emergency
caesarean sections. 10
RHSDP would (i) function as a management unit for training for NRHM, (ii) undertake the training and
monitoring related to HCWM, (iii) support the policy framework for outreach activities, working with RCH
II, (iv) support IEC for outreach and BCC for providers, (v) conduct feasibility studies and pilots for drug
logistics management and equipment repair and maintenance. NRHM would take the lead in (i) human
resources planning for the health sector, (ii) implementation of PPPs, (iii) integrated MIS for the health
sector, and (iv) strategic planning at the state level.
10
Project extension:
33. In light of the improved project performance subsequent to the Mid-Term Review (MTR),
combined with the fact that almost a quarter of the credit was still unspent, the project was
extended by two years in order to consolidate and maintain these gains by completing planned
activities, as well as in order to initiate new investments. However, most of the new investments
planned for the extension period were not initiated – including training programs and workshops,
a medical equipment management and maintenance workshop, drug logistics and supply chain
management system, special purpose fund for heavy duty repairs of infrastructure, pilots for
improving preventive care and health promotion among disadvantaged populations, and
strengthening of secondary level hospitals with equipment – while several other activities were
only partially completed – such as health care waste management training programs, introduction
of a hospital management information system, studies and evaluations, and social accountability
pilots. The issues with procurement and staffing/human resources that were the main causes of
these deficits were discussed earlier.
34. Altogether, over the seven years of implementation, most of the activities that were
envisaged during the initial project design were eventually completed, but only very few of the
activities planned for extension were completed. Annex 2 summarizes all project outputs by
component.
Disbursement delays
35. Due to the difficulties in procurement described above, actual and planned disbursement
deviated from each other early in the project. At MTR (July 2007), only 29.33% (US$27.5
million) of the credit was disbursed, triggering the disbursement delay flag which remained in
place for the duration of the second half of the original project period (from Oct 2007 through
April 2009). US$11 million was cancelled from the credit two weeks before closing and by the
end of the four month disbursement grace period the project had disbursed 81% of the original
and 92% of the revised credit allocation. The grace period was subsequently extended to March
31, 2012 because all utilization receipts had not yet been received by districts. At the time of
writing the project had disbursed 97% of the revised credit allocation.
Problem project status and other flags
36. RHSDP was flagged as problem project at the end of 2005 due to moderately
unsatisfactory implementation progress, driven by unsatisfactory procurement and the slow
implementation of Component 2. The country record flag was triggered for the period of May
2007 through April 2008 as a result of the Detailed Implementation Review (DIR) of five health
projects, conducted by the World Bank in 2006-2007, which found “significant indicators of
fraud and corruption in all five projects” (World Bank 2007, see Executive Summary). These
findings halted all projects under preparation and led to significant slow-downs in the
implementation of the centrally-sponsored schemes. The RHSDP, not being one of the five
projects examined by the DIR, and also a state-level project, was not as affected as many other
projects.
37. The team was proactive in responding to the DIR findings and recommendations. It held
in-depth discussions with government to assess whether the RHSDP was at risk of the
weaknesses that had been highlighted by the DIR and prepared a Joint Action Plan containing
mitigation measures. This plan was agreed to by the MoHFW and DEA and presented to the
Bank Board, and its implementation was monitored during subsequent missions.
11
2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization
Design
38. The results framework and indicators, as approved, were poor. Weaknesses included: (i)
too many indicators to feasibly track over time – 11 PDO indicators and 32 intermediate
indicators; (ii) most of the PDO indicators and a number of intermediate indicators were framed
in a way that was difficult to measure with available data; (iii) 9 of the 11 PDO indicators and all
of the 32 intermediate indicators lacked baseline data at the time of approval (with the exception
of those for which the baseline was zero); (iv) for a number of indicators, it was not realistic to
use the data sources identified in the PAD because the data sources either did not contain the
information needed (as was sometimes the case with the HMIS) or were not available with the
frequency needed (as was the case with large household surveys); and (v) all 11 PDO indicators
and 21 of the 32 intermediate indicators lacked explicit targets, including well into
implementation.
39. The evaluation design was insufficient. There were no plans for a project-specific
baseline survey or follow-up survey with which to measure the impact of RHSDP. There were
also no plans to use the large household surveys that are regularly implemented in India and
representative at the state level as baseline and follow-up data sources. Instead, the evaluations
planned for the project are better classified as operational research and assessments of different
interventions within the project, rather than evaluations of the overall project effect or outcome.
Implementation
40. During the first few years of the project, no proactive steps were taken to improve the
results framework despite the continued absence of information on key project indicators.
41. Revision of entire results framework, including PDO indicators and targets (2006): In
2006, the results framework was assessed and revised to address a number of the earlier
weaknesses. The number of indicators was reduced to seven PDO indicators and 11 intermediate
outcome indicators; existing indicators were replaced by indicators that were more measurable;
baseline data were obtained; and targets were put in place. Since this was prior to the
implementation of the revised restructuring guidelines (July 2006) which would have required
Board approval of any change in project indicators, no formal restructuring was needed. Rather,
the change was recorded in the aide-memoire of April 2006.
42. Despite these improvements, considerable weaknesses remained. First, targets for the
three PDO 1 indicators were over-ambitious and others do not appear sufficiently thought through.
For example, it was probably unrealistic to set a target for BPL utilization as a share of total
patients at 50% when BPL households constituted only 31% of rural and 11% of urban
households. Second, the earliest date for which most baseline values for the new indicators were
available was late 2005-200611
. Third, on the whole, indicators chosen for PDO 2 were not
convincing indicators of improving the effectiveness and quality of health care. For example, the
11 On the one hand, as noted, 2006 is too many years into project implementation to establish a project
baseline. However, in the case of the RHSDP, since implementation was so slow in the early years, it is not
unreasonable to consider values from 2006 as de facto baseline values for implementation.
12
percentage of staff in place is an input indicator, and does not reflect whether effectiveness and
quality have improved. Also, the indicator on the percentage of facilities conducting 24/7 EMoC
should have been dropped since, as acknowledged by the task team, it was NRHM and not
RHSDP that was undertaking those investments, and it was never reported on in any aide-
memoires. Fourth, despite the extensive revision, the causal chain between the PDO and
intermediate indicators was not very clear (especially with respect to PDO 1), making using the
results framework for project monitoring, feedback and change rather difficult.
43. Revision of targets (2009): When the project was extended in September 2009, the 2006
original project targets became the targets for the end of the extension period. In December 2009,
the PDO targets of the three indicators of PDO 1 were revised downwards to levels that were
more realistic through a Level II restructuring (as per the November 2009 project restructuring
guidelines). In September 2010, an aide-memoire included mutually agreed upward revisions to
the targets of the PDO indicators, but these were not formalized through a formal restructuring.
Consequently, the targets that have standing for the purposes of this ICR are the December 2009
targets.
44. Data collection for monitoring purposes: In order to obtain information on the indicators
in the revised results framework, RHSDP developed a separate hospital activity format (HAF)
which was filled out by facilities and submitted to the PMU which reviewed it for completeness
and accuracy, giving feedback to facilities where necessary. Since the MoHFW‟s regular HMIS
system did not contain the information needed to track progress towards project development
objectives, and in particular did not disaggregate utilization figures by poor and vulnerable groups,
the development of the HAF was the only way that the project could monitor its results. By
October 2007, the HAF had shifted from paper-based reporting to electronic reporting, achieving
good standards for timely and completeness in reporting (>90% for project facilities). On the
downside, the net result was the development of a parallel system of results monitoring, rather
than a strengthening of the existing HMIS system, adding to the reporting burden at the facility
level. In 2009, the project attempted to integrate the HAF and the existing HMIS reporting
systems, but this did not succeed. Relying on the HAF also meant that the values for the PDO
indicators were first only available from late 2005. However, the HAF did provide the first-ever
state-level regular (monthly) information on inequalities in health care utilization.
45. Evaluation: It is laudable that a number of assessments were carried out towards the end
of the project, and some at mid-line, including surveys of patient satisfaction, an assessment of
the patient counselor intervention, the quality and maintenance of civil works, and equipment
audits, among others. The studies were useful for understanding implementation issues, but their
design was not robust enough to make inferences about intervention/project effect. The quality of
the data from the patient satisfaction surveys, which were supposed to serve as a key data source
for PDO 2, was poor. There were also some missed opportunities to evaluate more of the pro-
poor innovations.
Utilization
46. While the PMU gave feedback to facilities on the completeness of their reporting, very
seldom did the PMU have a results-oriented discussion with facilities or other parts of the
administration about what the data had to say about performance. At the facilities themselves,
however, Health System Improvement Teams (established with project support) reviewed the
HAF results during their monthly meetings and clearly used the results to inform decision-making.
Some DPCs were also very proactive in analyzing findings and using the results to communicate
problems and improve service delivery. However, by and large, the multiple reporting formats,
13
staff time limitations, and insufficient interest within the administration meant that the substantial
information that was generated by the HAFs was not used to its full potential.
47. The Bank team strongly encouraged the use of monitoring and evaluation information.
Indeed, some assessments were used to inform project implementation, but others were conducted
too late to do so. Nevertheless, the project's focus on indicators and assessments/evaluations has
had an impact within the administration. It provided a good start for further improvements in the
use of information for decision-making and started to cultivate a culture of monitoring and
evaluation.
2.4 Safeguard and Fiduciary Compliance
Health care waste management
48. The project was classified as category “B” (partial environmental assessment),
necessitating the development of a HCWM Action Plan (Feb 2003). The plan provided the
blueprint for the environmentally sound management of hospital waste from healthcare facilities
in compliance with national biomedical rules. The implementation of the action plan was
included as a covenant and was complied with by its due date of Sept 2010.
49. Overall, the RHSDP made a substantial contribution to institutionalizing the systems and
protocols for collection, segregation, disinfection, storage and disposal of biomedical waste
generated from public health facilities. With a national policy in place only since 1998, the
project supported Rajasthan in an emergent area of public health where there was not much
capacity or action. The RHSDP provided training and refresher training for all secondary health
care personnel; facilitated the development of IEC materials; provided equipment and
consumables and equipment; connected hospitals to private sector common treatment facilities
(CTF) for waste collection and disposal and provided deep burial pits wherever CTF connectivity
was not possible; negotiated, and paid, fees for CTF connectivity; helped all project-supported
facilities to register with the State Pollution Control Board; ensured the removal of waste from
outreach camps; made sure that health care waste management was an agenda item for facility
HSITs; conducted two evaluations and helped to share best practices with a national audience.
Taken together, these activities constituted a system-strengthening intervention that linked the
multiple stakeholders (namely the State Pollution Board, MoHFW, private sector CTFs,
municipalities and facilities) involved in the health care waste management chain to each other.
50. Health care waste management was only once rated unsatisfactory (May 2007) due to
slower than planned progress on CTF connectivity, but this was resolved within six months and
the environmental assessment rating was upgraded to MS, and subsequently S. At project close, it
was the opinion of the Bank environmental specialist that, in the sphere of health care waste
management, the RHSDP performed better than any other recent or current state health system
reform project in India.
Indigenous peoples (safeguard policy OP 4.10)
51. A Tribal Development Plan, dated February 7, 2003, was prepared by project appraisal. It
was designed to address the specific health-related needs of tribal people, including their
inclusion within the health care delivery system, their willingness to access seek care in a timely
manner and financial barriers to accessing care. The completion of the implementation of the
Tribal Development Plan was included as a covenant and was complied with by its due date of
September 2010. Increasing the utilization of secondary care by tribal populations was also an
14
explicit development objective and, from 2006, was monitored at the facility and PMU level
using the specially-designed HAFs.
Procurement
52. Procurement performance can be divided into three phases: a period of poor performance
during the first two years when procurement was managed by PSAs (until end-2005), a
subsequent period of better performance (corresponding to the latter part of the original project
period) when the PMU and DPMU took over responsibility for procurement from the PSA (2006-
2008/09) and the extension period which was also characterized by weaker performance when all
procurement was centralized in the PMU (2009-2011). The reasons for this, as well as their effect
on implementation, have already been discussed in section 2.2.
From a fiduciary standpoint, there were some additional weaknesses. These were not major
shortcomings, however, and included some small misprocurements (mostly due to
misunderstanding of allowable expenditures), a number of (frequently unjustified) procurement
complaints by competing bidders, slow responses to complaints, failure to complete timely
procurement post reviews (PPRs), and an occasional failure to satisfactorily address PPR
recommendations. By project close, however, the project was able to satisfactorily close all
procurement-related complaints and issues.
Financial management (FM)
In terms of internal controls, the PMU maintained good financial records and submitted all annual
financial statements to the CAG on-time. On two occasions, accountability issues were identified,
but these were satisfactorily resolved. At the time of project close, there were no outstanding FM
issues. There were no difficulties in the flow of funds from the Special Account to the state or
from the state to the districts. Careful analyses of performance in financial management were
undertaken by Bank FM specialists on each mission and clear follow-up actions identified. Based
on these findings, no separate comprehensive FM assessments were deemed necessary.
Compliance with covenants
53. With the exception of the covenant on the implementation of different models of public-
private partnerships (PPPs) which was complied with after 1.5 years delay, all covenants were
complied with on-time. As already notes in section 1.7, the covenant on the design of a quality
regulatory framework for the private sector was deleted because of the development of national
legislation in this area.
2.5 Post-completion Operation/Next Phase
54. No follow-on operations are envisaged. However, since the integration of project
investments with the NRHM environment had been planned since MTR, a number of project
activities will be continued by the state. These include earmarked funds in the budget to provide
the equipment and staff to the facilities that were renovated by RHSDP (where these could not be
provided by the project before closing); continued training by SIHFW, on behalf of NRHM, using
the training modules that were developed by RHSDP; and continuation of the outreach camps
started by the project by the RCH II program, even though resource constraints (money,
medicines, staff) mean that the camps will not take place with the same frequency and intensity as
under RHSDP. However, reaping the full benefits of RHSDP investments will require some
additional investments that have not been budgeted for. For example, while the law requires
15
compliance with appropriate HCWM processes and the project helped facilities to obtain their
authorizations for CTF disposal, it remains unclear whether sufficient budgetary provision has
been made for the continued payment of CTF connectivity fees beyond the project period.
55. While there are no projects planned for the foreseeable future, there appear to be a
number of areas where technical support from the Bank would be useful. One is in the
development of a human resource strategy for health care workers to improve the incentive
structure facing doctors, the working conditions of nurses and the managerial competencies of
administrators; another could be the introduction of some hospital autonomy in all aspects of
service delivery (possibly in combination with performance-related incentives); a third could be
assistance in the pharmaceutical supply chain which while very much improved, still suffers from
frequent stockouts; and, finally, support to monitoring and evaluation.
3. Assessment of Outcomes
3.1 Relevance of Objectives, Design and Implementation
56. Ten years after the start of project preparation, the development objectives remain very
relevant. Access to care remains an issue, with average preventive and curative utilization rates in
Rajasthan lagging behind the India average, especially among poor and tribal households (see
Annex 7 Table 6). As the data in section 3.2 will show, there also remain large disparities in the
utilization of inpatient and outpatient care, and the quality of care needs improving.
57. The PDOs and project activities also remain well-aligned with government priorities,
through the project‟s active adaptation to the new health policy environment which will continue
to be defined by the NRHM. The project objectives and design also remain in sync with the
Bank‟s current strategy for engagement. At project close, engagement in India, and Rajasthan,
was governed by the 2009-2012 CAS which pledges to devote more resources to engaging with
India‟s seven low-income states including Rajasthan where it will focus on poverty reduction and
reaching the Millennium Development Goals (MDGs). Achieving these goals will necessarily
include a focus on strengthening secondary care such as that found in this project. The CAS also
emphasizes focusing on increasing accountability to citizens, decentralizing responsibilities, and
enhancing private sector participation in the delivery of these services.
Rating: Substantial
3.2 Achievement of Project Development Objectives
58. The RHSDP was conceived as a contribution to overall health sector development and, in
its latter stages, was conceived as an explicit complement to the NRHM. Consequently, its
activities, like in other state health projects in India, were a combination of discrete investments
and contributions to overall health system strengthening. Moreover, its financial contributions
were dwarfed by domestic spending. Consequently, it is neither possible nor appropriate to
attempt to attribute the observed outcomes in the results framework to the RHSDP. Indeed, one of
the strengths of the project lay in how closely it worked with government to complement the
initiatives that the government was undertaking, adjusting its activities when needed. The
performance that is captured by the results framework and other data sources used below is, thus,
taken to reflect the contribution of the project.
59. To assess overall project efficacy, we start by looking at performance against the targets
of the PDO indicators. Then, we attempt to validate that assessment by looking at other data
16
sources, internal and external to the project. Finally, we examine performance on intermediate
outcomes and activities to see whether the findings are supported by the results chain.
For PDO 1: Increase access to health care of poor (BPL) and underserved
Were PDO 1 targets reached?
60. Based on data from 238 project facilities, all three PDO 1 indicators of access to
secondary care services by disadvantaged groups show steady progress over the project
implementation period and exceed the formally revised (2009) targets (see Figure 1). As a
proportion of total outpatients, BPL patients rose from 8.7% in 2006 to 16.6% in June 2011,
reaching the target of 17%, and as a proportion of total inpatients, BPL patients rose from 8.5% in
2006 to 17% in June 2011, exceeding the target of 13%. As a proportion of total in-patients in
the 49 project-supported facilities in six districts with large tribal populations, the ST population
rose from 8.3% in 2006 to 24.9% in June 2011 (see Figure 3), exceeding the target of 15%. Thus,
the formally revised targets for all PDOs related to access to care for the poor and vulnerable
were clearly attained.
Figure 1 Below poverty line (BPL) and scheduled tribal (ST) patients as % of all patients in project facilities
Below-poverty-line (BPL) patients as % of all patients in 283 project facilities, 2006-11
Scheduled Tribe (ST) in-patients as % of all in-patients in 49 project facilities in six tribal districts, 2006-11
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
Jan
06
Ap
r 0
6
Jul 0
6
Oc
t 0
6
Jan
07
Ap
r 0
7
Jul 0
7
Oc
t 0
7
Jan
08
Ap
r 0
8
Jul 0
8
Oc
t 0
8
Jan
09
Ap
r 0
9
Jul 0
9
Oc
t 0
9
Jan
10
Ap
r 1
0
Jul 1
0
Oc
t 1
0
Jan
11
Ap
r 1
1
BP
L a
s %
of
all p
ati
en
ts i
n p
roje
ct
fac
ilit
ies
out-patients
in-patients
0%
5%
10%
15%
20%
25%
30%
35%Ja
n 0
6
Ma
y 0
6
Se
p 0
6
Jan
07
Ma
y 0
7
Se
p 0
7
Jan
08
Ma
y 0
8
Se
p 0
8
Jan
09
Ma
y 0
9
Se
p 0
9
Jan
10
Ma
y 1
0
Se
p 1
0
Jan
11
Ma
y 1
1
% o
f in
-pa
tie
nts
in
49
pro
jec
t fa
cilit
ies
in 6
tri
ba
l d
istr
icts
ST as % in-patients
Source: Aggregation of monthly hospital activity formats
61. While performance on PDO 1 would be rated satisfactory relative to the December 2009
targets, performance would be rated unsatisfactory relative to the April 2006 targets of 50%.
Weighting these ratings by disbursement at the time of the change of targets (according to the
17
formula in the ICR guidelines) would give an overall rating of moderately unsatisfactory12 for
PDO 1 (see Annex 7 Table 3). However, this ICR considers a rating of moderately satisfactory to
be a more appropriate assessment of performance on PDO 1. Why? First, as discussed in section
2.3, the 2006 targets of 50% utilization by the poor and scheduled tribe groups were not only
unrealistic, but demographically impossible: scheduled tribes constituted only 12.6% of
Rajasthan‟s population, and the BPL population constituted only 31% of the rural and 11% of the
urban population. Downgrading performance on this PDO to moderately unsatisfactory based on
a rigid calculation would be too heavy a penalty for what is essentially a failure to put in place a
realistic target. Second, measuring efficacy is not only about performance against project targets;
it is about achieving the overall project development objective, namely increasing access to health
care of the poor and underserved population. On that score, the data in Figure 1 make it clear that
the increase among these groups over the period of the project for which data are available was
quite tremendous, especially considering how difficult it is to bring secondary care within the
reach of these groups who live mainly in rural, often remote, areas.
Can the PDO 1 achievements be validated by other data sources or indicators?
62. External data sources, such as the National Family Health Survey (NFHS), District-Level
Household Survey (DLHS) and UNICEF household surveys, are informative, but in a limited
way. These limitations include timing, lack of information on key outcomes of interest such as
general care utilization (inpatient and outpatient), as well as lack of access to some of the original
data which (in some cases) is necessary to disaggregate results by economic status and caste. The
limitations are summarized in Annex 7, Table 4.
63. Because of the absence of indicators on general care utilization in these surveys, we rely
on trends in maternal and child health (MCH) indicators as a proxy. From 2002/03 to 2009 (see
Table 5, Annex 7), very large increases in immunization coverage, antenatal care utilization and
institutional delivery were observed. Also, when data are disaggregated by caste and
socioeconomic status (using the only original data to which we have access, namely the 1998/99
NFHS and the 2005/06 NFHS), we find improvements in most indicators for scheduled tribes and
for the populations in the lowest two quintiles (i.e. corresponding to BPL households) (see Table
6, Annex 7). In conclusion, patterns of MCH service utilization over time lend support to the
hypothesis that access to care among the poor and vulnerable has been increasing.
The logic of the causal chain for PDO 1
64. Further down the results chain of PDO 1, at the level of intermediate outcome indicators,
we find that targets for nursing in tribal areas were far exceeded but not for doctors or technicians,
100% of patient counselors were in place and 100% of payments from equity funds were made.
Unfortunately, there is no data relating IEC to increased utilization. Some of the activities
implemented under Component 3 (which focused on PDO 1) were very successful, while others
were less so (see Annex 2). For example, IEC activities were slow to start and a number of
activities were implemented on relatively small-scale; on the other hand, outreach camps were a
resounding success reaching tens of thousands of people in some of the most remote areas, and a
variety of innovative mechanisms to reach the poor were attempted. In addition, the improved
12 It is not possible to measure performance relative to objectives and targets for the period between
approval and March 2006 because those indicators lacked targets. However, since only 10% of the total
credit (including the US$4 million advance, or 6% excluding the US$4 million advanced) had been
disbursed by then, the effect on the overall efficacy rating would likely be inconsequential.
18
quality of service provision (see subsequent paragraphs) is likely to have contributed to improved
service utilization. While it cannot be conclusive, the logic of the results chain suggests a
moderately satisfactory rating.
Rating on PDO 1: Moderately satisfactory
For PDO 2: Improve effectiveness and quality of health care
Were PDO 2 targets reached?
65. There were strong achievements on some PDO 2 indicators and weaker results on others.
First, the proportion of the 185 project-supported Community Health Centers (CHC) that
undertook more than ten deliveries per month increased dramatically from a baseline of 60% in
2006 to 96.6% in June 2011, well exceeding the target of 90%. Second, the proportion of staff
positions filled in project facilities exceeded the 90% target for nurses/ANMs (at 117%), as well
as for lab technicians (91.5%), but not for doctors (64.3%). Third, surveys of patient perceptions
from 2008 and 2011 suggest relatively stable patient perceptions, with some deterioration in the
perception of nurses‟ performance, in contrast to the expectation that the project would improve
patient satisfaction13
. The fourth PDO indicator on the availability of 24/7 emergency obstetric
care was never tracked by the project because related activities were undertaken by NRHM and,
as discussed in section 2.3, should have been dropped. Taken together, then, one of the PDO 2
targets was exceeded, one was exceeded on two dimensions but not on a third, and one was not
met, suggesting that, based purely on the attainment of targets, the outcome rating for PDO 2
would probably lie on the line between moderately satisfactory and moderately unsatisfactory.
Data from other sources, which will provide more information on the performance on the overall
development objective, are thus employed to make the final decision.
Can the PDO 2 achievements be validated by other data sources or indicators?
66. The hospital activity formats (which were put in place by the project) contain additional
indicators that capture other elements of the effectiveness and quality of care. These include the
number of neonatal deaths, number of maternal deaths, referral from lower level facilities,
feedback to lower-level facilities, percentage of caesarean sections, number of laboratory tests,
number of radiological screens, and drug stockouts. While the merits of each individual indicator
as a measure of quality of care can be disputed, taken together they contribute to a more holistic
view of the trends in PDO 2 than are available in the results framework. In order to examine these
indicators, the ICR team manually reviewed individual facility monthly report data – for 238
facilities – between 2005 and 2011.
67. Between 2005 and 2011, the average number of maternal deaths per facility fell
dramatically, by about one third. This result is especially remarkable given the increases in
institutional delivery over the period, but could also reflect a declining share of high-risk
pregnancies in the overall institutional delivery numbers as institutional delivery became more
common. The average number of neonatal deaths reported increased, but this is likely due to the
13 In 2008, among approximately 1,000 patients at project-supported facilities, 92% were satisfied with the
behavior of doctors and 85% were satisfied with the behavior of nurses; in the 2011 survey, among 1,763
patients at project-supported facilities, 94% were satisfied with doctors‟ behavior but only 72% were
satisfied with nurses. As highlighted in section 2.3, there are also concerns with the quality of these surveys.
19
increase in institutional delivery over the same period14
. The number of laboratory tests increased,
as did the number of radiographic screenings, which is interpreted as an improvement in the
availability of diagnostic tests. The percentage of deliveries that were caesarean sections (which
is sometimes used as an indicator of quality because it indicates the availability of an anaesthetist,
surgeon and equipment) fell sharply, but its declining share is probably due to the declining share
of high-risk pregnancies and, in that regard, the fact that the share has declined provides some
reassurance that treatment protocols were being followed and caesarean sections were not overly
supplier-induced. Referral, taken to be an indicator of quality of care because it indicates that
patients at lower levels of care have access to more specialized care when needed, also showed
improvement and, moreover, there was an improvement in the amount of feedback on the
patient‟s condition given to the lower levels. Perhaps the most powerful indicator of quality of
care is the percentage of drug stockouts. The percentage of facilities that had a drug stockout in
the previous year declined dramatically from 67% to 37% between 2007 (the first date for which
data are available on this indicator) and 2011. In terms of the severity of the stockouts, the
number of the 15 essential drugs that were not available in those facilities with stockouts
averaged 6 in both years.
68. Overall, the hospital activity formats provide strong evidence of improvements in the
quality and effectiveness of care over the project period (for which data are available), lending
support to a rating that is firmly “above the line”, i.e. moderately satisfactory.
Table 2 Change in (alternative) indicators of quality of care, 2005 to 2011, in project facilities
Indicator 2005 2011 Comment
Average number of neonatal deaths per facility per
month
2.5 3.57 Likely due to sharp increase
in institutional delivery
Average number of maternal deaths per facility per
month in 42 days post-partum
0.82 0.27 Improvement
Percentage of institutional deliveries that are
caesarean sections
9.2 4.23 Not necessarily indicative of
falling quality; it may result
from higher number of
facility-based low risk births
Average number of referrals from lower facilities
per month
29.4 40.15 Improvement
Average number of feedback about referred
patients given to lower facilities per month
10.87 15.92 Improvement
Number of laboratory tests 874.7 1761.6 Improvement
Number of radiography screens 252.0 890.3 Improvement
Percentage of facilities which had a drug stockout
for at least one month (2007, not 2005)
66.8% 36.6% Improvement
Among those facilities reporting stockouts, number
of the 15 essential drugs not available, on average,
in any given month and facility (2007, not 2005)
5.77 6.13 Little change
Source: ICR team‟s calculations from hospital activity formats
Note: Information on drug stockouts is available only from 2007 onwards
14 In the HAF data, neonatal deaths are recorded only for children that were born at the facility, but include
those deaths that occurred both at the facility and at home.
20
The logic of the causal chain for PDO 2
69. At the level of the PDO outcome indicators, while patient satisfaction appears to be
unchanged, and the staffing of doctors remained below targets, staffing of nurses and technicians
exceeded targets, as did the improvement in institutional deliveries. Although the initial increased
demand for institutional deliveries could have been mainly attributable to the JSY program and
other health policy programs, sustained take-up would not be possible without significant supply-
side strengthening. Indeed, the intermediate project indicators confirm that the project improved
health system infrastructure and processes – targets on renovations, drug availability, health care
waste management and the functioning of health system improvement teams were all met. In
addition, most planned equipment was procured, quality-related guidelines were implemented and
much planned clinical training was undertaken (see activities included Annex 2). The good
performance on intermediate outcome indicators also likely explains why it is not only
institutional deliveries that improved, but also the alternative measures of health system quality
described in Table 2. The logic of the results chain confirms a rating of moderately satisfactory.
Rating on PDO 2: Moderately satisfactory
Overall efficacy
70. Overall efficacy is determined by a combination of performance on PDO 1 and PDO 2,
weighted by the resources allocated to each. Most of the activities related to PDO 1 were
concentrated in Component 3 which was worth US$29 million. Most of activities related to PDO
2 were concentrated in Component 2 which was worth US$48 million. Component 1 (worth
US$18 million) is assumed to contribute equally to both Components 2 and 3. Consequently, the
performance on PDO 2 should be weighted slightly more heavily than performance on PDO 1 in
the overall efficacy rating. Overall efficacy is, thus, rated as moderately satisfactory.
Rating: Moderately satisfactory
3.3 Efficiency
Increase in the efficiency of service delivery
71. Measuring the efficiency of service delivery is not easy, especially given the limited type
and quality of data available in Rajasthan. However, the HAFs developed under RHSDP contain
measures of average length of stay and the bed occupancy rate, both of which capture aspects of
the efficiency of service delivery15
. Average length of stay fell from 5.96 inpatient days in 2005
to 1.77 inpatient days in 2011; bed occupancy rate increased from 39.4% to 61%; and the
percentage of CHCs conducting at least 10 deliveries increased from 60% in 2006 to 97% in June
2011. These three indicators suggest better use of existing facilities (i.e. increased outputs per unit
input) and, thus, better efficiency in health care delivery. Another indicator of enhanced
efficiency is the increase in the number of referrals (since it is more efficient to treat illness at the
lowest level that it can be effectively treated, with referral for more complex cases), and as shown
in Table 2, the number of referrals increased.
15 Again, to obtain these data, the ICR team compiled and aggregated information from monthly reports
from 238 facilities to obtain the trends discussed. The quality of the reported data was not always good
(including a number of implausible values), but on the assumption that these errors were random and
uncorrelated with overall efficiency, the indicators provide a useful general picture of trends.
21
Good targeting
72. Targeting is an efficient way to improve health outcomes since it concentrates scarce
resources on those who are likely to have the greatest health deficits, and where resources can
make the most difference. This argument refers both to targeting vulnerable groups (BPL
households and scheduled tribes) and targeting vulnerable areas. Consequently, the fact that the
project was designed as a targeted intervention with PDOs focused on Rajasthan where health
indicators are lower than the India average and, moreover, on reaching vulnerable groups within
Rajasthan is indicative of an efficient design. Not only was targeting of the poor and vulnerable
envisioned during design, but the project also succeeded in increasing the utilization of the poor
and the vulnerable vis-à-vis other groups.
Efficiency of project execution:
73. The efficiency of project execution, as measured by the speed of disbursement, was poor.
As described in section 2.2, actual disbursement lagged behind planned both in the initial and
extension phases. Slow disbursement imposes opportunity costs by tying up resources that could
have been made available to other projects or activities. At least, the September 2011 credit
cancellation helped to keep the money in India‟s IDA allocation, minimizing the opportunity cost
of unspent resources at project close. Counterpart (GOI) funds were made available in a timely
fashion.
Sectoral and allocative efficiency:
74. Given how government financing for health grew from when the project was designed to
its close (see Annex 3), the marginal financial contribution of RHSDP to the health sector fell
over time. Over the seven years of implementation, the average annual contribution of the project
fell from a projected 5% of government health spending to only 3.5%. However, its funding was
highly complementary to the focus of the additional government expenditure and to the focus of
other Bank projects that were implemented over a similar period. While NRHM, for example,
focused more on primary care (and was also supported by the Bank‟s RCH II program), RHSDP
focused mainly on secondary facilities and the quality of care at those levels. Moreover, the
project made strategic investments in strengthening particular areas of health system development
(such as those related to policies, processes and training) so that with a given amount of resources,
government would be able to produce better outcomes, effectively pushing out the production
possibility frontier. This suggests that expenditure on RHSDP is unlikely to have (inefficiently)
distorted health program allocations.
Rating: Satisfactory
3.4 Justification of Overall Outcome Rating
75. Given substantial relevance, moderately satisfactory efficacy and satisfactory efficiency,
the overall outcome rating is moderately satisfactory.
Rating: Moderately Satisfactory
22
3.5 Overarching Themes, Other Outcomes and Impacts
(a) Poverty Impacts, Gender Aspects, and Social Development
76. This has been discussed considerably in the previous sections. To summarize:
Impact on poverty and social development
77. It is well-known that health shocks are a major cause of poverty. Loss of income (linked
to absence from work and farming) and the associated direct costs (consultations, medicines,
diagnostic care) and indirect costs (opportunity cost of family members spending time in care-
giving) push households into poverty and exacerbate poverty among the already-poor. By
focusing on improving the effectiveness and the quality of care (PDO 2) for all groups, but
especially the poor and vulnerable (PDO 1), the project‟s development objectives are focused on
reducing the poverty associated with illness shocks. However, this project went further than most
in a number of respects. First, it targeted a poor state and, within that state, some of the most
vulnerable groups. Second, it devoted an entire project component to innovations that work on
different dimension of access to care (e.g. geographic, financial and social) to enable health care
to better reach the poor and vulnerable (see Annex 2). Third, it revised the results framework and
set up a new data collection process specifically to enable outcomes among the poor and
vulnerable to be monitored – something that was not previously possible in the state and,
moreover, was not being done in many Bank projects – and succeeded in collecting the relevant
data. Finally, project indicators show that health care utilization among the poor and vulnerable
households improved both in absolute terms and relative to non-poor and non-tribal households.
Gender aspects
78. The health facilities targeted by the project were important for the quality of institutional
delivery in the state, especially those that function as referral hospitals for complex pregnancies,
with implications for maternal mortality. With the JSY program promoting institutional delivery
on the demand-side, the project was an important, and necessary, supply-side complement. Most
patient counselors were women, in part for purposes of gender representation and in part to help
ensure that female patients received the care that they needed.
(b) Institutional Change/Strengthening
Through its component on policy development and management, the project contributed to the
strengthening of institutional capacity at many levels. Most importantly, through working in such
an integrated fashion with NRHM, capacity-development within the project was simultaneously
strengthened within NRHM institutions and the Ministry of Health and Family Welfare. One
highlight was the development of documentation and models for public private partnership,
building MoHFW‟s capacity to contract with the private sector going forward; another was the
development and implementation of the hospital activity formats which will strengthen the
capacity of the MoHFW to monitor performance in the health sector; as well as extensive training
including in health administration, management, procurement of drugs/supplies. Many RHSDP
project staff will remain part of the MoHFW and the State Institute for Health and Family
Welfare (SIHFW) will take up all the training previously provided under RHSDP with the aid of
the RHSDP materials. The role of the project in institutional strengthening goes beyond the health
sector. For example, RHSDP worked with a variety of other public and private institutions, such
the State Pollution Board, private sector CTFs and municipalities to set up processes by which
they could work together for health care waste management.
23
(c) Other Unintended Outcomes and Impacts (positive or negative)
79. Unintended beneficiaries include the new state programs that were not envisaged when
the project was designed. NRHM, introduced in 2005, was an unintended beneficiary since it had
not come into being when the project was being prepared, but yet the project adjusted to support
it by sharing resources, capacities and defining complementary goals and activities. The SIFHW,
too, was an important beneficiary, since RHSDP was tasked with building its capacity and left it
with a mandate to undertake all training for the NRHM.
4. Assessment of Risk to Development Outcome
80. The project was designed to be financially and institutionally sustainable and, moreover,
when the external environment changed during implementation, the activities were modified
accordingly in an attempt to ensure continued convergence with government programs and, thus,
sustainability. At design, it was projected that the total recurrent cost of the project would be low
enough at project closing to be taken up by the government and, by the actual closing date, the
actual annual amount was even lower than originally planned. In addition, with an increasing
share of recurrent operational costs planned to be borne by the state as implementation proceeded,
the gradual adoption of certain on-going activities was planned from the start. Moreover, when
the environment changed with the launch of NRHM, there was careful discussion between the
task team and NRHM to re-align activities with NRHM priorities and in accordance with those
that could be taken up by NRHM post-project.
81. A number of project activities were, in fact, investments in future service delivery and,
thus, can be expected to contribute to the sustainability of development outcomes. These include,
among others, training in clinical and managerial skills, the development of HCWM protocols
and processes, public-private partnership development and HMIS-strengthening (see section
3.4b).
82. It is likely that the improvements in many of the project development outcomes will be
sustained, if not further enhanced, beyond the project period – even if actual project
activities/outputs are not taken up by the government to the extent that is hoped. This is largely
due to factors external to the project, including the increase in health financing in the state (see
Annex 3), the priority given to Rajasthan as a focus state in the NRHM, and the number of
demand-side interventions that remove financial barriers to accessing care. The JSY program
continues to provide free institutional delivery for all mothers, as well as additional demand-side
incentives to BPL women (to cover transport and opportunity cost) and the ASHAs who
encourage them to seek care; the Chief Minister‟s program provides free care for BPL patients;
and, from October 2011, a new program provides 400 essential medicines free to all BPL patients
in public facilities. In addition, the number of nurses/ANMs in tribal areas increased (although the
number of doctors fell) over the course of the project and the government has indicated its
intention to continue the outreach camps started by the project. While the system of patient
counselors has been discontinued, project studies shows that they were anyway not particularly
instrumental in improving the access of the poor. Taken together, these factors mean that the risk
to PDO 1 (i.e. access to care of poor and vulnerable populations) is low.
83. What is less clear is whether the increased utilization by these groups, and others, will
continue to be met by the appropriate supply-side investments in institutional strengthening and
capacity-building necessary to improve the quality and effectiveness of care (PDO 2). On the
positive side, resources are available and, with respect to project activities that will contribute to
sustained development outcomes, health care infrastructure has been improved by the renovations
24
to all project facilities (at least once), the SIHFW (which is tasked with all training under NRHM)
has taken over all training modules developed under RHSDP, HCWM processes have become
institutionalized (although this outcome is threatened by the fact that the state is unlikely to
commit funds to maintaining CTF connectivity), the HSITs use the project HAF to monitor and
improve service delivery and the government had promised to equip recently-renovated facilities.
However, more profound policy changes are still needed to improve service quality over the long-
term, e.g. better drug prescribing, and changes in the structure of incentives faced by doctors in
order to keep good doctors in remote areas and, possibly, more managerial autonomy at the
facility level to manage in a quality- and results-orientated fashion. Consequently, the outlook for
the sustainability of PDO 2 is less optimistic.
Rating: Moderate
5. Assessment of Bank and Borrower Performance
5.1 Bank Performance
(a) Bank Performance in Ensuring Quality at Entry
84. The Bank team performed quite well in delivering a project that was well-aligned with
government and Bank priorities, was informed by substantial analytical work and lessons learned
from previous Bank projects, could feasibly be implemented, carefully considered future
sustainability and took steps to mitigate the risk of slow procurement (see discussion in section
2.1 for details). The team is also to be commended for its tenacity in working with government to
obtain the buy-in needed to introduce “soft” elements into a state-level health system
strengthening project. Attention to safeguards was adequate and the project had numerous design
elements in place to better reach the poor.
85. The main weakness was the quality of results framework, as well as the monitoring and
evaluation plans (discussed at length in section 2.3). The team could also have given more
consideration to the implications of human resource (HR) constraints for project design and
implementation and whether related risk mitigation strategies were needed. The 2005 QAG
quality-at-entry review suggested that the implementation of Component 3 would have been
faster if more of the details of its interventions had been elaborated during preparation rather than
left for implementation. Finally, the PAD could have more substantially addressed the state‟s
socioeconomic situation and health sector challenges and it did not describe very well how all the
background analysis actually informed project design (something on which the QAG Panel also
commented).
86. The QAG panel rated overall quality at entry as moderately satisfactory (3 on scale of 1
to 6 where 1 is the highest rating), citing the overall project strengths as its partnership with
government and targeting, and its overall weaknesses as the monitoring and evaluation
arrangements, as well as the preparation of Component 3. This ICR concurs, but would like to
point out that more detailed elaboration of the Component 3 interventions beyond the analytical
work undertaken would have been difficult (and premature) while the team was still working on
getting government support for the inclusion of these types of activities.
Rating: Moderately satisfactory
25
(b) Quality of Supervision
87. The intensity of implementation support provided by the task team to the PMU was
exceptional and the staff of the PMU speak highly of the commitment of the task team that
supervised this project for most of this period. They cite, especially, the way that the task team
worked with the PMU to develop precise follow-up actions for each mission and weekly audio
calls during the difficult implementation period in 2008-09. The Bank was able to provide
relatively good continuity of staffing with only three changes of team leadership in ten years of
preparation and supervision, and the task team leader (TTL) who was in place for all but the last
few months of supervision was also part of the preparation team. However, the client was
discouraged by the fact that a new TTL joined just prior to project closing (when the existing
TTL left the unit) and that there was some uncertainty as to whether the project would again be
extended in 2011.
88. The task team did its due diligence in ensuring fiduciary compliance (financial
management and procurement) throughout the project, as well as monitoring safeguards.
Moreover, a special effort was made in the post-DIR period through the development and
implementation of an action plan to help to prevent fraud and corruption and address any
associated weaknesses. The task team assured the implementation of the tribal development plan
and performed well in bringing the HCWM plan to fruition.
89. The supervision period was marked by great proactivity and flexibility. Most notable was
the dynamic way in which the task team responded to the changing environment in the health
sector during implementation, modifying the design to one that was complementary to the new
programs (see section 2.1). The team also planned well for the post-project period when NRHM
and RHSDP activities would be converged and pushed hard to get the necessary transitional
arrangements in place. The team was also proactive in the just-in-time cancellation of the
remaining credit before project close to ensure these funds were returned to India‟s IDA pool.
90. Faced with a sharply declining supervision budget during the extension phase (with
annual expenditure on supervision during the extension phase less than half of that during the
original project period (see Annex 4), the team tried to maintain the intensity and breadth of
missions by reducing travel costs. The team also brought a fair amount of technical skills and
subject experts to meet the evolving needs of the project.
91. The team was less successful in working to improve the monitoring and evaluation in the
project. As has already been discussed, it did not revise the weak results framework early on
(despite this weakness being pointed out by a QER), realistic targets were available only fairly
late into implementation, and no overall project evaluation was planned. There also appear to
have been some missed opportunities to engage senior policymakers in larger health system
reform issues, as communication tended to focus mainly on lagging project implementation issues,
and less on broader policy dialogue. The final project aide-memoire is an exception in that regard;
it discusses at length the future of health system development in Rajasthan, implicitly also
highlighting possible areas of future engagement for the Bank. Finally, more could also have been
done to expose the client to international experience and practices, although (as already noted) the
government withdrew support for international training in the later years of the project.
Rating: Satisfactory
26
(c) Justification of Rating for Overall Bank Performance
The above discussion shows that the Bank delivered a well-designed project whose main
weakness was the quality of the monitoring and evaluation arrangements and, to a lesser extent,
adequate consideration of HR-related risks and preparation of the multiple small-scale pro-poor
interventions envisaged under the project. During supervision, while the team remained weak in
the area of M&E and also could have exploited more opportunities for policy dialogue, it was
remarkably committed, diligent and proactive in working with the government over a long (and
sometimes difficult) implementation period to overcome bottlenecks and flexibly adjust the
project to better contribute to health system development in a changed health sector environment.
With a rating of moderately satisfactory for preparation and a rating of satisfactory for
supervision, overall Bank performance is rated as moderately satisfactory.
Rating: Moderately satisfactory
5.2 Borrower Performance
(a) Government Performance
92. It took some time for the project to obtain government buy-in during preparation,
especially on the health system strengthening elements and Component 3, but overall government
commitment to the project during implementation was fairly good, especially in the middle years.
This is not surprising since the project‟s development objectives were closely aligned with those
of the government, as expressed through NRHM. Moreover, the government dedicated significant
resources to these goals and also hired more than 4,000 additional nurses/ANMs over the course
of the project in support of NRHM. At the same time, however, NRHM so much eclipsed
RHSDP in importance that the project sometimes suffered from a lack of attention.
93. Nevertheless, the appropriate enabling legislative and regulatory changes (e.g. the
legislative framework for PPPs and the regulations related to health care waste management)
were made in time to allow the project to proceed with related components. In addition, the
government clearly owned the results of RHSDP investments, (rightly) claiming credit for a
number of the accomplishments. Multiple arms of government have also worked together, where
needed, e.g. in implementation of HCWM processes. Fiduciary arrangements were adequate and
covenants were met. Finally, the government has shown some (limited) commitment to sustaining
project investments beyond RHSDP (e.g. through budgeting for equipment, planning for outreach
camps, and absorbing training materials).
94. The weakest area of government performance was its failure to effectively manage the
staffing needs of the project. Unfortunately, the effect of this failure on project performance was
profound. Key position remained vacant for extended periods and burdened staff with multiple
responsibilities (in addition to RHSDP). Consequently, current staff did not have sufficient time
to devote to implementation. This was especially problematic during the extension phase. There
was also frequent turnover of the project director on whose support and proactivity the
functioning of the rest of the PMU was highly dependent.
Rating: Moderately satisfactory
27
(b) Implementing Agency or Agencies Performance
95. The implementing agency was the Department of Medical, Health and Family Welfare of
Rajasthan. Within this department, there were a number of different actors responsible for project
implementation, including the state-empowered committee (overall policy guidance), project
steering committee (supervision of implementation and coordination), state project management
unit (implementation) and district project coordination and monitoring committees.
96. By and large, the core staff in the PMU were very committed to the implementation of
the project and, together were able to achieve much, especially in the latter part of the original
project period. During the extension phase, however, the performance of the PMU was
characterized by a high degree of risk-averseness, especially when it came to procurement, and
staff were over-burdened because of their multiple responsibilities. Moreover, the frequency of
monitoring of project implementation and site visits declined because key project staff did not
receive permission and resources to undertake supervision visits.
97. There were a number of highlights in the state PMU‟s performance, including the
execution of training activities, their successes in procurement when they first took over that
responsibility from PSAs, and their monitoring of the timeliness and completeness of HAF data.
Financial management was sound.
98. There was considerable variation in the performance of the district PMUs which was
reflected in quality of interventions supported by the projects in those districts. There was little
turn-over, however, capacity improved over time and, as seen on ICR field visits, in some cases
the additional interventions and project monitoring undertaken at the district level were quite
remarkable.
Rating: Moderately satisfactory
(c) Justification of Rating for Overall Borrower Performance
99. Overall, with a moderately satisfactory rating for government performance and a
moderately satisfactory rating for implementing agency performance, overall borrower
performance is rated as moderately satisfactory.
Rating: Moderately satisfactory
6. Lessons Learned
100. Capacity of procurement staff in PMUs and procurement support agencies needs to be
more rigorously assessed: Despite the fact that all procurement plans, bid documents, technical
drawings etc. needed for the first year of implementation were already prepared during project
preparation phase, and that procurement support agencies were contracted, the project still
experienced major procurement delays which resulted in slow disbursements. Teams need to
more carefully assess whether planned activities designed to overcome procurement bottlenecks
are likely to work, paying careful attention to the capacity of the project procurement staff and the
selection of the procurement support agencies.
101. The long-term payoffs of integration and harmonization with government programs need
to be balanced with the short-term needs of project implementation: Following government‟s
implementation of NRHM, RHSDP and government sought convergence and integration of the
28
two programs in order to promote complementarities and ensure sustainability of RHSDP
investments. While this process had the positive effect of having senior policy makers in charge
of both the program and the project, it did reduce the amount of time that the project director was
able to devote singularly to project implementation because he was also the head of NRHM.
102. Human resource constraints, both at project management and at facility level, need to be
explicitly considered in project preparation: The RHSDP did not give sufficient consideration to
the implications of human resource (HR) constraints for project design and whether alternative or
additional investments in HR were needed. While staffing decisions are typically beyond the
Bank‟s (and even the project‟s) influence, teams would do well to at least have a risk mitigation
strategy in place to deal with potential problems of vacancies and under-staffing. Complementary
analytical work, as well as policy dialogue on human resource issues, could also be considered. It
would be helpful if task teams working on other projects within the Bank‟s health, nutrition and
population sector portfolio were given more guidance in this regard.
103. Ensure that dealing with project implementation and processing does not crowd out
engagement in effective policy dialogue: Faced with implementation difficulties, teams tend to
focus on the project implementation issues that require resolution. However, they also need to
make a concerted effort to use the opportunities provided by an active project to engage in policy
dialogue with clients on overall health systems development which could potentially have a far
greater long term impact.
104. Monitoring and evaluation arrangements should receive more priority so that project
performance can be correctly measured: In particular, baselines should be in place early,
evaluations should be planned and changes to the results framework should be made as soon as
possible to reflect project restructuring and any changes to targets; waiting until late in project
implementation may negatively, and unnecessarily, affect project outcome ratings. This is an area
that may need greater management attention.
7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners
(a) Borrower/implementing agencies
105. All issues raised in the Borrower‟s ICR (see Annex 5) were taken into consideration in
the drafting of the Bank‟s ICR. In addition, any factual inaccuracies in the Bank‟s draft ICR that
were pointed about by the Borrower in the Borrower‟s comments on the Bank‟s draft ICR (also
see Annex 5), received March 26, 2012, were corrected in this version of the ICR.
(b) Cofinanciers
106. There were no other cofinanciers.
29
Annex 1. Project Costs and Financing Table 1: Project Cost by Component (in USD Million equivalent)
Components Appraisal Estimate
(USD millions)
Actual/Latest
Estimate (USD
millions)
Percentage of Appraisal
1.Policy Development and Project
Management 19.32 18.44
95.45%
2.Improving Service Quality at
Primary and Secondary Levels of the
Public Health Care Services
50.41 47.67
94.56%
3.Improving Health Care Services for
the Poor and Disadvantaged
Population
31.92 29.21
91.51%
Total Baseline Cost 101.65 95.32 93.77%
Physical Contingencies 1.28 0.00 0.0
Price Contingencies 3.05 0.00 0.0
Total Project Costs 105.98 95.32 89.94%
Total Financing Required 105.98 95.32 89.94%
Source: Client connections and FM reports, end-February 2012
Table 2: Financing, sources and amounts
Source of Funds Type of
Cofinancing
Appraisal
Estimate
(USD millions)
Actual/Latest
Estimate
(USD millions)
Percentage of
Appraisal
Borrower Joint 16.98 19.6 115%
International Development Association
(IDA) Joint 89.00 75.72 85%
Note: IDA financing was 97% of the revised credit (post-cancellation of US$6.875 million in Sept 2011).
Government share exceeds 100% because of exchange rate fluctuations.
Table 3: Reallocations, by disbursement category, in SDR
Total 0.166 0.267 0.604 0.660 0.138 0.236 0.362 0.414
Source: Author‟s own calculations. National Family Health Survey (NFHS) 1998/99 and 2005/06
PhalodiPhalodi
ShahabadBijolia
Nimbahera
Shivpuri
Rajgarh
Sikandra
Indergarh
Mandar
Mertacity
Ratangharh
Talchhapur
Nokha
Pallu
Ratwatsar
Chhatargarh
Hanumangarh
Ganganagar
Bikaner
ChuruJhunjhunun
SikarAlwar
Dausa
Sawaimadhopur
Bharatpur
Dholpur
Tonk
Bundi
Kota Baran
Jhalawar
Chittaurgarh
Pratapgarh
Bhilwara
Ajmer
Nagaur
Jodhpur
Pali
Rajsamand
Dungarpur
Sirohi
Jalor
Barmer
Jaisalmer
Banswara
Jaipur
NEW DELHI
GANGANAGAR
BIKANER
JHUNJHUNUN
SIKAR
JAIPUR
ALWAR
SAW
AIM
ADHO
PUR
BHARATPUR
BUNDI
KOTA
CHITTA
URG
ARH
BHILWARA
AJMER
NAGAUR
JODHPUR
PALI
DUNGARPUR
SIROHI
JALOR
BARMER
JAISALMER
BANSWARA
CHURU
DAUSA
DHOLPUR
TONK
BARA
N
JHAL
AWAR
RAJSAMAND
R A J A S T H A N
U T T A RP R A D E S H
D E L H I
M A D H Y AP R A D E S H
H A R Y A N A
P U N J A B
G U J A R A T
PA
KI S
TA
N
15
15
15
11
11
12
1111A
814
8
8
2
2
3
NATIONAL HIGHWAYS
STATE HIGHWAYS AND SELECTEDMAJOR DISTRICT ROADS
RAILROADS
SELECTED CITIES
DISTRICT CAPITALS
STATE CAPITAL
NATIONAL CAPITAL
DISTRICT BOUNDARIES
STATE BOUNDARIES
NATIONAL BOUNDARY
1
INDIA
RAJASTHAN HEALTH SYSTEMSDEVELOPMENT PROJECT
0 50 100 150
KILOMETERS
IBRD 39214
APRIL 2012
This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other informationshown on this map do not imply, on the part of The World BankGroup, any judgment on the legal status of any territory, or anyendorsement or acceptance of such boundaries.