Document of The World Bank Report No: ICR00003381 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-77370) ON A LOAN IN THE AMOUNT OF US$77.82 MILLION TO THE REPUBLIC OF INDONESIA FOR A HEALTH PROFESSIONAL EDUCATION QUALITY PROJECT (HPEQ) June 15, 2015 Health, Nutrition and Population Global Practice Indonesia Country Management Unit East Asia and Pacific Region Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Document of
The World Bank
Report No: ICR00003381
IMPLEMENTATION COMPLETION AND RESULTS REPORT
(IBRD-77370)
ON A LOAN
IN THE AMOUNT OF US$77.82 MILLION
TO THE
REPUBLIC OF INDONESIA
FOR A
HEALTH PROFESSIONAL EDUCATION QUALITY PROJECT (HPEQ)
June 15, 2015
Health, Nutrition and Population Global Practice
Indonesia Country Management Unit
East Asia and Pacific Region
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
i
CURRENCY EQUIVALENTS
(Exchange Rate Effective June 15, 2015)
Currency Unit = Indonesian Rupiah
US$1.00 = IDR 13,330.22
IDR1.00 = US$ 0.000075
FISCAL YEAR
January 1 – December 31
ABBREVIATIONS AND ACRONYMS
AIPKI Asosiasi Institusi Pendidikan
Kedokteran Indonesia (Association of
Indonesian Medical Schools)
BAN-PT Badan Akreditasi Nasional Perguruan
Tinggi (National Accreditation Body for
Higher Education)
BAPPENAS Badan Perencanaan Pembangunan
Nasional (Ministry of National
Development Planning)
BHE Board of Higher Education
CBC Competency-based curriculum
CBT Competence-based testing
CPCU Central Project Coordination Unit
CPS Country Partnership Strategy
DGHE Directorate General of Higher Education
FAP Financial Assistance Package
GDP Gross Domestic Product
GoI Government of Indonesia
HEI Higher Education Institutions
HELT Higher Education Long Term
HRH Human resources for health
HWS Health Work Force and Services Project
ICT Information, Communications and
Technology
IMC Indonesian Medical Council (see KKI)
IDR Indonesian Rupiah
IFLS Indonesia Family Life Survey
ISR Implementation Status and Results
KKI Konsil Kedokteran Indonesia (Indonesia
Medical Council)
KPI Key performance indicators
LAM-PTKes Lembaga Akreditasi Mandiri Pendidikan
Tinggi Kesehatan
LASA Directorate of Learning and Student
Affairs
LPUK-NAKES Lembaga Pengembangan Uji
Kompetensi Tenaga Kesegatan
MCQ Multiple Choice Questions
MDG Millennium Development Goal
MoF Ministry of Financing
MoEC Ministry of Education and Culture
(2011-2014)
MoH Ministry of Health
MoNE Ministry of National Education (before
2011)
MoRTHE Ministry of Research, Technology and
Higher Education (2014 – now)
NAA National Accreditation Agency (then
LAM-PTKES)
NACEHhealthPro National Agency of Competence
Examination for Health Professions
NCBE National Competency-based
Examination
OSCE Objective Structured Clinical
Examination
OOP Out-of-pocket
PAD Project Appraisal Document
PBL Problem Based Learning
PD-DiktiKes Database for Health Higher Education
PDO Project Development Objectives
PHC Primary health care
PUSKESMAS Pusat Kesehatan Masyarakat
(Community Health Center)
PUSDIKNAKES Pusat Pendidikan Tenaga Kesehatan
(Center for Health Workforce
Education)
SATKER Satuan Kerja (DGHE’s Working
Unit)
TOR Terms of References
UNCEN Cendrawasih University
UNDANA Nusa Cendana University
Vice President: Axel van Trotsenburg
Country Director: Rodrigo A. Chaves
HNP GP Director: Olusoji O. Adeyi
Practice Manager: Toomas Palu
Project Team Leader: Puti Marzoeki
ICR Team Leader/Primary Author: Edson Correia Araujo
ii
Republic of Indonesia
Health Professional Education Quality Project
CONTENTS
Data Sheet
A. Basic Information
B. Key Dates
C. Ratings Summary
D. Sector and Theme Codes
E. Bank Staff
F. Results Framework Analysis
G. Ratings of Project Performance in ISRs
H. Restructuring
I. Disbursement Graph
1. Project Context, Development Objectives and Design ................................................... 1
2. Key Factors Affecting Implementation and Outcomes .................................................. 8
3. Assessment of Outcomes .............................................................................................. 12
4. Assessment of Risk to Development Outcome ............................................................. 19
5. Assessment of Bank and Borrower Performance ......................................................... 20
6. Lessons Learned ......................................................................... 23
7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners............... 24
Annex 1. Project Costs and Financing .............................................................................. 25
Annex 2. Outputs by Component...................................................................................... 26
Annex 3. Economic and Financial Analysis ..................................................................... 29
Annex 4. Bank Lending and Implementation Support/Supervision Processes ................. 33
Annex 5. Beneficiary Survey Results ............................................................................... 35
Annex 6. Stakeholder Workshop Report and Results ....................................................... 36
Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ......................... 37
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................... 41
Annex 9. List of Supporting Documents .......................................................................... 42
MAP ................................................................................................................................. 43
iii
A. Basic Information
Country: Indonesia Project Name: Health Professional
Education Quality Project
Project ID: P113341 L/C/TF Number(s): IBRD-77370
ICR Date: 06/15/2015 ICR Type: Core ICR
Lending Instrument: SIL Borrower: REPUBLIC OF
INDONESIA
Original Total
Commitment: USD 77.82M Disbursed Amount: USD 67.18M
Revised Amount: USD 77.82M
Environmental Category: C
Implementing Agencies:
Ministry of National Education, Directorate General of Higher Education
Cofinanciers and Other External Partners:
B. Key Dates
Process Date Process Original Date Revised / Actual
Date(s)
Concept Review: 11/25/2008 Effectiveness: 12/30/2009 12/09/2009
Appraisal: 06/01/2009 Restructuring(s): 02/28/2013
09/16/2014
Approval: 09/24/2009 Mid-term Review: 10/29/2012 10/29/2012
Closing: 12/31/2014 12/31/2014
C. Ratings Summary
C.1 Performance Rating by ICR
Outcomes: Satisfactory
Risk to Development Outcome: Moderate
Bank Performance: Satisfactory
Borrower Performance: Satisfactory
C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)
Bank Ratings Borrower Ratings
Quality at Entry: Satisfactory Government: Satisfactory
Quality of Supervision: Satisfactory Implementing
Agency/Agencies: Satisfactory
Overall Bank
Performance: Satisfactory
Overall Borrower
Performance: Satisfactory
iv
C.3 Quality at Entry and Implementation Performance Indicators
Implementation Performance Indicators QAG Assessments (if any) Rating
Potential Problem Project at any
time (Yes/No): No Quality at Entry (QEA): None
Problem Project at any time
(Yes/No): Yes
Quality of Supervision
(QSA): None
DO rating before
Closing/Inactive status: Satisfactory
D. Sector and Theme Codes
Original Actual
Sector Code (as % of total Bank financing)
Health 50 50
Tertiary education 50 50
Theme Code (as % of total Bank financing)
Health system performance 33 33
Other human development 67 67
E. Bank Staff
Positions At ICR At Approval
Vice President: Axel van Trotsenburg James M. Adams
Country Director: Rodrigo A. Chaves Joachim von Amsberg
Practice Manager/Manager: Toomas Palu Juan Pablo Uribe
Project Team Leader: Puti Marzoeki Puti Marzoeki
ICR Team Leader: Edson Correia Araujo
ICR Primary Author: Edson Correia Araujo
F. Results Framework Analysis
Project Development Objectives (from Project Appraisal Document)
The specific Project Development Objectives are to strengthen quality assurance policies governing
the education of health professionals in Indonesia. This will be achieved by: 1) rationalizing and
assuring competency-focused accreditation of public and private health professional training
institutions; 2) developing national competency standards and testing procedures for certification
and licensing of health professionals; and 3) building institutional capacity to employ results-based
grants for encouraging the use of accreditation and certification standards in the development of
medical school quality.
v
Revised Project Development Objectives (as approved by original approving authority)
(a) PDO Indicator(s)
Indicator Baseline Value
Original Target
Values (from
approval
documents)
Formally
Revised Target
Values
Actual Value
Achieved at
Completion or
Target Years
Indicator 1 : The establishment of an independent National Accreditation Agency (NAA)
Value
quantitative or
Qualitative)
BAN-PT-IMC Joint
Commission
Establishment of the
Charter of the NAA Y
Date achieved 12/31/2009 12/31/2011 17/31/2014
Comments
(incl. %
achievement)
Achieved. NAA (created as LAMP-PTKes) was established and legally ratified by the
MoEC Decree in October 2014.
Indicator 2 : The establishment of an independent National Agency for Competency Examination of
Health Professionals (NACEHealthPro)
Value
quantitative or
Qualitative)
NA
Establishment of the
Charter of the
NACEHealthPro
Y
Date achieved 12/31/2009 12/31/2011 12/20/2013
Comments
(incl. %
achievement)
Achieved. The legal framework for the agency (created as LPUK-NAKES) was
established in 12/20/2013. The agency had exercised its function, as a task force under
the CPCU, to develop the methodology for competence examination since early
implementation.
Indicator 3 :
The percentage of health professional schools (medical, dentistry, nursing, and
midwifery) that have gone through the accreditation process and have publicized the
results
Value
quantitative or
Qualitative)
zero
medicine (100%),
dentistry (100%),
nursing (48%),
midwifery (18%).
medicine (29,
42%), dentistry
(10, 42%),
nursing (52,
10%),
midwifery (33,
10%). Total 124
schools
medicine (21),
dentistry (9), nursing
(82), midwifery (56).
Total 168 schools
Date achieved 12/31/2008 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. %
achievement)
Surpassed - the total number of schools accredited exceeded the target. The number of
accredited medical and dentistry schools was (slightly) lower than the target and
exceeded the target for nursing and midwifery. Baseline data only included percentages.
Indicator 4 : The percentage of graduates of health professional schools (medicine, dentistry, nursing,
and midwifery) passing national competency testing at the first attempt
Value
quantitative or
Qualitative)
medicine (71.67%),
dentistry (81.69%), nursing
(0%), midwifery (0%)
medicine (84%),
dentistry (90%),
nursing (65%),
midwifery (65%)
medicine (72.47%),
dentistry (92.31%),
nursing (57.81%),
midwifery (64.65%)
Date achieved 12/31/2009 12/31/2014 12/15/2014
vi
Comments
(incl. %
achievement)
Achieved for dentistry and marginally under the target for midwifery. Target not
achieved for medicine and nursing. However, for nursing and midwifery the baseline
values were zero.
Indicator 5 : The mean test score of graduates from the Financial Assistance Package (FAP) recipient
schools who have taken the National Competence Test
Value
quantitative or
Qualitative)
60.61 67.07
Date achieved 12/31/2010 12/31/2014
Comments
(incl. %
achievement)
There was no target defined. The mean test scores of non-FAP recipients was 56.03 at
baseline and 63.83 at project closing (14% increase against 11% increase among
grantees).
(b) Intermediate Outcome Indicator(s)
Indicator Baseline Value
Original Target
Values (from
approval
documents)
Formally
Revised Target
Values
Actual Value
Achieved at
Completion or
Target Years
Indicator 1 : Completion status of preparatory activities
for the establishment of the NAA
Value
(quantitative
or Qualitative)
NA
NAA has
independent and
adequate budget to
conduct
accreditation and
has access to
adequate numbers of
suitably trained
assessors
Y
Date achieved 12/31/2009 12/31/2014 12/15/2014
Comments
(incl. %
achievement)
Achieved.
Indicator 2 : Standard of Competencies and Standard of Education for the four health professions are
available.
Value
(quantitative
or Qualitative)
medicine (standards
released by the IMC in
2006), dentistry (standards
released by the IMC in
2006), nursing (draft of both
standards available),
midwifery (standard of
competencies released by
MoH in 2007. Draft
standard of education
available)
nursing (both
standards completed
and legalized),
midwifery (standard
of education
completed and
legalized)
added: three
professions
(nutrition,
pharmacy and
public health)
Y
Date achieved 12/31/2009 12/31/2011 12/31/2013 12/31/2013
vii
Comments
(incl. %
achievement)
Achieved. The standards of competencies were achieved for pharmacy and public health
disciplines. Both have conducted the national competency examination trial. Standards
not completed for nutrition, expected to conduct trial in 2015.
Indicator 3 : Accreditation instrument for the four health professional schools are ready for use.
Value
(quantitative
or Qualitative)
medicine (draft available)
medicine
(international peer
review of
instrument),
dentistry (piloting
and dissemination),
nursing (piloting and
dissemination),
midwifery (piloting
and dissemination)
Y
Date achieved 12/31/2009 12/31/2012 12/15/2014
Comments
(incl. %
achievement)
Achieved.
Indicator 4 : Number of trained assessors
Value
(quantitative
or Qualitative)
0
medicine (86),
dentistry (39),
nursing (105),
midwifery (105).
added:
40/profession
for nutrition,
public health
and pharmacy
medicine (132),
dentistry (34),
nursing (123),
midwifery (70),
public health (17),
nutrition (16),
pharmacy (41)
Date achieved 12/31/2009 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. %
achievement)
Surpassed for medicine, nursing and pharmacy. The target was not achieved for
dentistry (slightly under the target), midwifery, public health and nutrition.
Indicator 5 : Introduction of CBT and OSCE for NCE
Value
(quantitative
or Qualitative)
N CBT and OSCE for
all four professions.
CBT for all four
professions.
OSCE for
medicine and
dentistry, CBT
try out for the
added
professions,
OSCE
preparation only
for pharmacy.
CBT for medicine,
dentistry, and nurse.
Paper based test for
midwifery and
diploma of nursing
(DIII). OSCE for
medicine and
dentistry, and under
preparation for
pharmacy.
Date achieved 12/31/2009 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. %
achievement)
Achieved. OSCE was added formally to the medicine and dentistry NCE in 2013. CBT
had been implemented for medicine and dentistry and for nursing. Paper based test was
implemented for midwifery a diploma of nursing (DIII). Not achieved for nutrition.
Indicator 6 : Number of National OSCE trainers
Value
(quantitative 0 72/profession
72/profession
(medicine and
medicine (4,950),
dentistry (84)
viii
or Qualitative) dentistry only).
36/Pharmacy.
Date achieved 12/31/2009 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. %
achievement)
Surpassed. The total number of OSCE trainers exceeded the target for medical and
dentistry. Target no achieved yet for pharmacy.
Indicator 7 : Number of National MCQ Item Writers
Value
(quantitative
or Qualitative)
0 1,044/profession
added:
36/profession
for nutrition,
public health
and pharmacy
medicine (254),
dentistry (650),
nursing (828),
midwifery (675),
pharmacy (219),
public health (141)
and nutrition (144).
Date achieved 12/31/2009 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. %
achievement)
Target not achieved for the four initial professions (medicine, dentistry, nursing and
midwifery). Surpassed for the three additional professions (nutrition, public health and
pharmacy).
Indicator 8 : Number of National OSCE Item Writers
Value
(quantitative
or Qualitative)
0 72/profession
72/profession
(medicine and
dentistry),
36/pharmacy.
medicine (221), nurse
(48), dentistry (650),
pharmacy (39).
Date achieved 12/31/2009 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. %
achievement)
Surpassed. The total number of OSCE item writers exceeded the targets for all
professions.
Indicator 9 : Number of MCQ Item Writers and reviewers
Value
(quantitative
or Qualitative)
NA 1,044/profession
added:
36/profession
for nutrition,
public health
and pharmacy
medicine (216),
dentistry (650),
nursing (828),
midwifery (675),
pharmacy (78),
public health (84) and
nutrition (132).
Date achieved 12/31/2009 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. %
achievement)
Target not achieved for the four initial professions (medicine, dentistry, nursing and
midwifery). Surpassed for the three additional professions (nutrition, public health and
pharmacy).
Indicator 10 : Number of OSCE
Value
(quantitative
or Qualitative)
0 1,044/profession
1,044/professio
n (medicine
and dentistry
only),
36/pharmacy
medicine (86),
dentistry(63),
pharmacy (39).
Date achieved 12/31/2009 12/31/2014 12/31/2014 12/31/2014
Comments
(incl. % Target not achieved for the medicine and dentistry. Surpassed for pharmacy.
ix
achievement)
Indicator 11 : Number or percentage of medical schools receiving finance support to strengthen the
program
Value
(quantitative
or Qualitative)
0 43 43
Date achieved 12/31/2009 12/31/2014 12/15/2014
Comments
(incl. %
achievement)
Achieved.
G. Ratings of Project Performance in ISRs
No. Date ISR
Archived DO IP
Actual Disbursements
(USD millions)
1 04/12/2010 Satisfactory Satisfactory 3.50
2 12/27/2010 Satisfactory Satisfactory 3.50
3 06/04/2011 Satisfactory Satisfactory 11.95
4 03/28/2012 Satisfactory Satisfactory 27.00
5 01/08/2013 Moderately Unsatisfactory Moderately Satisfactory 38.44
6 07/24/2013 Moderately Unsatisfactory Moderately Unsatisfactory 43.87
7 02/21/2014 Moderately Unsatisfactory Moderately Unsatisfactory 59.94
8 09/08/2014 Moderately Satisfactory Moderately Satisfactory 65.16
9 12/24/2014 Satisfactory Moderately Satisfactory 65.87
H. Restructuring (if any)
Restructuring
Date(s)
Board
Approved PDO
Change
ISR Ratings at
Restructuring
Amount
Disbursed at
Restructuring
in USD millions
Reason for Restructuring & Key
Changes Made DO IP
02/28/2013 N MU MS 43.87
Upon request of the Government
of Indonesia, the following
changes were made: (i) expand the
activities to include three
additional professions (pharmacy,
nutrition and public health) and
two additional medical schools
(UNCEN and UNDANA); (ii)
reallocate the proceeds of the Loan
(redistribution of funds across
components to accommodate
changes); and (iii) update the
reference to the national regulation
on procurement for the
clarifications relating to National
Competitive Bidding Procedures
x
Restructuring
Date(s)
Board
Approved PDO
Change
ISR Ratings at
Restructuring
Amount
Disbursed at
Restructuring
in USD millions
Reason for Restructuring & Key
Changes Made DO IP
(to accommodate inclusion of two
universities – UNCEN and
UNDANA)
09/16/2014 N MS MS 65.16
Following the Government of
Indonesia request, the following
change was made: reallocation of
funds from component 3 (FAP) to
component 1 (training, workshops,
incremental operating costs,
research expenditures, consultants'
services and goods under
component 1) and component 4
(training, workshops, incremental
operating costs, research
expenditures, consultants' services
and goods under component 4)
I. Disbursement Profile
1
1. Project Context, Development Objectives and Design
1. The Health Professional Education Quality (HPEQ) was approved on September 24, 2009 in
the amount of US$77.82 million and became effective on December 09, 2009. Further financial
support in the amount of US$8.9 million was provided by the Government of Indonesia (GoI).
1.1 Context at Appraisal
2. Background. At the time of project preparation, Indonesia ranked as a lower-middle income
country, with a gross domestic product (GDP) per capita of US$2,272. Indonesia’s economic
growth was 4.6% while the percentage of the population living in poverty (less than US$2 a day)
was 17.8% (fall of 4.7% in the period of 2002-2009)1. Indonesia is a geographically dispersed
country, spread over 17,000 islands with 34 provinces and approximately 500 districts. It has
enormous variety within its borders, ranging from densely populated urban cities, in the island of
Java, to sparsely populated rural and remote islands of Nusa Tenggara Timur and Maluku. Since
2001, the provision of health care services was decentralized with regional governments having
full discretion over how the health services were distributed and budgets are allocated. The central
government role was restricted to the distribution of financing and regulation of the health care
sector.
3. Main issues in the health sector. In 2009, the total health spending accounted for only 2.8%
of the GDP, with a per capita health expenditure of US$64.2, which was significantly lower than
the average in the East Asia and Pacific countries (4.9%), excluding Japan, Korea, Singapore and
Australia. The private sector was the dominant source of health care financing with 49% of the
total health expenditure paid through household out-of-pocket (OOP) payments, higher than the
average of 37.8% in the region. The public health spending accounted for 36.1% of the total health
spending, much lower than the average of 51.9% in the neighboring countries (e.g. Malaysia,
59%).2
4. At the time of project appraisal there were three major health insurance schemes in the
country, namely, Jamkesmas, Jamsostek and Askes. Jamkesmas was a government financed
health insurance program for the poor and the near poor, covering about a third of the population.
The beneficiaries were identified by a combination of means testing and local government
eligibility criteria. The Jamkesmas faced some important challenges, particularly related to
beneficiary entitlement awareness (reports suggest coverage rates among the poor households of
only 47.6%) and leakages (evidence available suggests leakages of about 52.4 %,).3
5. Indonesia had, and still has, a mixed public-private provision of health care services. Basic
primary health care was provided by the public sector via a network of Puskesmas (Pusat
Kesehatan Masyarakat, or ‘health center at the sub-district level’), which serve a catchment area
at the sub-district level of about 25,000 to 30,000 individuals, and by the private sector via private
clinics and individual health professional private practices. Each Puskesmas was expected to
1 Data source: http://data.worldbank.org/indicator. 2 Data source: http://data.worldbank.org/indicator. 3 Harimurti et.al. (2013). Nuts and Bolts of Jamkesmas. Indonesia’s Government Financed Health Coverage Program. The World
Bank, Washington DC, January 2013.
2
provide outpatient care, health prevention and health promotion services, to function as a
gatekeeper to the health care system, and to ensure continuity of care. Nevertheless, a third of the
Puskesmas also provided inpatient services. Secondary care was provided by both public and
private health care providers. Approximately half of the secondary-care hospitals and a third of the
hospital beds (estimated 163,000 in 2008) were private.4 Despite these numbers, more than half of
the inpatient visits in 2009 occurred in the public sector (57%).
6. At the time of project appraisal the Indonesian health sector faced major human resources
(HRH) challenges. Although the total availability of health personnel was not low, there were
issues regarding HRH distribution, skill mix and quality of health care personnel. The production
of new physicians had grown steadily with a peak in the 2008/2010 period when the production
was approximately 9,000 new physicians/year.5 In terms of distribution of health professionals,
there were imbalances across provinces and across rural and urban areas.6 HRH quality was a
growing concern in Indonesia during project preparation. The main source of evidence was the
2007 Indonesia Family Life Survey (IFLS) vignettes, which measured the diagnostic and treatment
ability among doctors, nurses and midwives. The IFLS found a low percentage of correct responses
to vignette questions: 45% for antenatal care, 62% for child curative care, and 57% for adult
curative care.
7. The trends in the quality of the services provided by the health care professionals was
associated with the fast expansion in the number of private schools. At the time of project
preparation 57% of the medical schools in Indonesia were private and over half of 7,000 doctors
graduated from private schools. The expansion of schools, also observed in other professions,
notably in nursing and midwifery, was not accompanied by improvements in quality standards.
One-third of medical schools were not accredited and only a quarter received the highest
accreditation standard given by the Indonesia Directorate General of Higher Education (DGHE).
According to the Association of Indonesian Medical Schools (AIPKI), in 2007 only 50% of
students passed the national based test examination with a passing score of 45 out of 100.
8. At the time of project preparation there were already some initiatives that placed the
foundation for establishing quality assurance system for health professional education.7 The
Medical Practice Act (2004) supported the establishment of the Indonesia Medical Council (KKI)
which, in 2006, produced the standards of competencies for doctors and the standards for medical
education. These were the basis for the DGHE of the Ministry of National Education (MoNE) to
require all medical schools to implement a competency-based curriculum (CBC) along with the
adoption of problem-based learning and the integration of various medical disciplines. However,
given the different capacity of the schools, at the time of project development, the CBC was
implemented in an unstandardized manner across schools, which resulted in a large variation in
the quality of education across medical schools.
4 BDEHR (2010). HRH Registration 2010. MoH: Jakarta. 5 Data source: http://apps.who.int/gho/data/node.main.A1444?lang=en 6 Anderson et al. (2014). The production, distribution, and performance of physicians, nurses, and midwives in Indonesia: an update.
HNP GP Discussion Paper 91324, World Bank Group, Washington, DC. 7 The National Education System Act (2003), Medical Practice Act (2004) and the Lecturers Act (2006) provided the legal basis
for improving the quality of Indonesian doctors.
3
9. The Medical Practice Act (2004) also established the accreditation of medical and dentistry
schools as a mandate to measure quality of education. The accreditation function was under the
responsibility of the National Accreditation Body for Higher Education (BAN-PT) under MoNE.
BAN-PT faced challenges as an independent accreditation body given its financial dependence on
MoNE, and in terms of developing specific instruments and processes more consistent with the
characteristics of the medical and dentistry education. From June 2007 all graduates of medical
schools were required to take the National Competency-Based Examination (NCBE) before
obtaining their certification.
10. The regulatory framework to ensure quality of nurses and midwives education was much
less developed than for doctors and dentists at the time of project preparation. The
accreditation of nursing and midwifery schools was undertaken by both the Center for Health
Workforce Education of the Ministry of Health (PUSDIKNAKES, MoH) and by the BAN-PT
without a common approach nor criterion and there was no formal entity certifying the quality of
graduating midwives and nurses. After finalizing the undergraduate program, graduates were
allowed to practice without going through a nationally standardized competence testing process.
11. Government strategy. The project objectives and interventions were highly relevant as it
supported the strengthening of the quality assurance system of health professional education, one
of the priorities of the government's 2010-2014 health sector medium-term development plan.
They were also consistent with the Higher Education Long Term Strategy (HELTS) 2003-2010
stating that quality assurance should be internally driven and institutionalized; and that quality
improvement should aim at producing quality outputs and outcomes as part of public
accountability, while BAN-PT, professional associations and other independent agencies could
play a key role in conducting an objective external control based on standards.
12. Rationale for Bank’s assistance. The project was aligned with the World Bank Country
Partnership Strategy (CPS) 2009-2012, which supported the reform of the education sector from
early childhood education to higher education and teacher upgrading. The project was also aligned
with the CPS 2009-2012 objectives in the health sector which aimed to improve quality, coverage,
and utilization of health services, especially for the poorest 40%. The CPS identified quality and
access to health services determinants of the slow progress towards the attainment of key
Millennium Development Goals (MDGs), specifically maternal mortality rates (which was among
the highest in the region). Broadly, the project was aligned with the sectoral core engagement
component, which focused, among other things, on improving accountability, effectiveness of the
schools and on strengthening human resource capabilities through medical/health education.8
13. Since early 1990s the Bank provides financial assistance to build capacity of higher
education institutions in Indonesia. As noted in the PAD, these experiences have been successful
and the government allocated own budget to support and institutionalize them. HPEQ also
followed up the Health Work Force and Services Project (HWS), closed on December 2008. The
HWS included a sub-component on “enhancing the quality of medical education”, more
specifically it supported the MoNE to: i) increase its institutional capacity to organize and manage
8 World Bank (2008). Indonesia Country Strategy and Program 2009-2012.
4
medical education; ii) improve the quality of formal medical education; and iii) enhance the
learning and teaching environment for both undergraduate and postgraduate medical education.9
1.2 Original Project Development Objectives (PDOs) and Key Indicators
14. The specific PDO was to strengthen quality assurance policies governing the education
of health professionals in Indonesia. This was to be achieved by: i) rationalizing and assuring
competency-focused accreditation of public and private health professional training institutions;
ii) developing national competency standards and testing procedures for certification and licensing
of health professionals; and iii) building institutional capacity to employ results-based grants for
encouraging the use of accreditation and certification standards in the development of medical
school quality.
15. Progress towards achieving the PDOs were to be measured against the following key
performance indicators (KPIs):
The establishment of an independent National Accreditation Agency (NAA);
The establishment of an independent National Agency for Competency Examination of
Health Professionals (NACEHealthPro);
The percentage of health professional schools (medical dentistry, nursing, and midwifery)
that have gone through the accreditation process and have publicized the results;
The percentage of graduates of health professional schools (medical, dentistry, nursing,
and midwifery) passing national competency testing at the first attempt;
The mean test score of graduates from the Financial Assistance Package (FAP) recipient
schools who have taken the National Competence Test.
16. The progress of the project implementation were to be measured against the following
intermediate indicators:
Completion status of preparatory activities for the establishment of the NAA;
Standard of Competencies and Standard of Education for the four health professions are
available;
Accreditation instrument for the four health professional schools are ready for use;
Number of trained assessors;
Introduction of Competence-Based Testing (CBT) and Objective Structured Clinical
Examination (OSCE) for National Competency-Based Examination (NBCE);
Number of National OSCE trainers;
Number of National Multiple Choice Questions (MCQ) Item Writers;
Number of National OSCE Item Writers;
Number of MCQ Item Writers and reviewers;
9 HWS ICR, June 2009.
5
Number of OSCE Item Writers and reviewers;
Number or percentage of medical schools receiving finance support to strengthen the
program.
1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and
reasons/justification
17. The PDO was not modified during the project implementation period. Nevertheless, the results
indicators were revised to reflect the changes made during the 2013 project restructuring. This
revision led to the expansion of the project through the incorporation of three professions
(pharmacist, nutritionist and public health specialist) in components 1 and 2 and two medical
schools (UNCEN and UNDANA) in component 3.
1.4 Main Beneficiaries
18. The primary target groups were the Indonesian MoNE, the Indonesian Ministry of Health
(MoH), schools of the seven professions included in the project (medical, dentistry, nursing,
midwifery, nutrition, pharmacy, and public health) and the students of these programs. The
ultimate beneficiaries of the project were the population of Indonesia who will benefit from better
quality of health care that is now provided by certified health professionals trained by accredited
schools. Finally, the Indonesian higher education system as a whole is expected to benefit from
the quality assurance system developed under the project, which can serve as a model for other
professions.
1.5 Original Components
19. Component 1: Strengthening Policies and Procedures for School Accreditation (US$7.184
million). This component was designed to support the GoI in improving the accreditation system
of medical, dentistry, nursing, and midwifery schools and make it comparable to internationally
recognized systems. The main objective was to create an independent accreditation body (the
NAA) to conduct the accreditation of health higher education institutions in Indonesia. The sub-
components were: i) development of strategic framework, policies and procedures for
accreditation; ii) development of standards of health education programs and standards of
competencies; iii) development of accreditation instruments; iv) development of a pool of
assessors; v) establishment of an accountability system for accreditation of health higher education
institutions; and vi) data management to support the accreditation system. The component
expenditure categories included organizing workshops, trainings, benchmarking the accreditation
instruments, international and local technical assistance, conducting legal assistance studies and
surveys, acquiring office equipment, IT, audio visual equipment and furniture.
20. Component 2: Certification of Graduates Using a National Competency-based
Examination (US$12.899 million) by: i) establishing an independent national competence
examination agency (the NACEHhealthPro); ii) improving the methodology and management of
the national competency-based examination; and by iii) developing an item bank networking
system to support the national competence examination. The project aimed to support the
NACEHhealthPro in establishing CBT and OSCE facilities in 12 medical schools that were
expected to function as regional centers. The component expenditure categories included
6
information technology, and audiovisual, computer software, skills laboratory, office equipment
and furniture, contracting international and local technical assistance, international benchmarking
of competence standards, and organizing workshops and trainings.
21. Component 3: Results-based Financial Assistance Package (FAP) for Medical Schools
(US$61.4 million). This component aimed to support selected medical schools to obtain the
needed resources to improve medical education quality and capacity-building to achieve the
national accreditation standards. The key principles guiding the FAPs allocation were: i) results-
based allocation of resources; ii) fair competition among medical schools according to their
capacity; and iii) partnership between leading and less advanced medical schools to build the
capacity of the latter according to their specific needs. Medical schools were divided into three
FAP schemes: Scheme A - FAP to support leading medical schools to build their international
reputation and to strengthen Indonesia’s global competitiveness in the area; Scheme B - FAP to
support weak capacity new medical schools in achieving the medical education standards
mandated by the Indonesian Medical Council (KKI) through partnerships with a leading medical
school; and Scheme C - FAP to support moderate capacity medical schools in achieving the
medical education standards mandated by the KKI.
22. The Board of Higher Education (BHE) was in charge of the FAP competitive selection process.
This included establishing the guidelines for the FAP recipient selection and proposal approval
process. BHE was also responsible for overseeing the FAP implementation, including the
preparation of a FAP manual to guide the implementation of the grants. Only one selection process
was expected during the lifetime of the project which had to be conducted during the first year of
the project. Selected schools (grantees) had to implement the proposed program within three years.
23. FAP resources could be used to finance interventions in the following areas: i) improving the
implementation of the competence-based curriculum (CBC) (which included: establishing student-
centered learning, early significant clinical exposure, adjusting student evaluation to be consistent
with the CBC, and periodic review of the curriculum to ensure achievement of competencies); ii)
strengthening teaching, training and learning facilities (which included: modernizing and
strengthening libraries, computer centers to allow e-learning through e-libraries, and establishing
electronic connectivity and high-speed internet facilities to allow networking among the medical
schools); iii) development of the medical faculty (which included: support for recruitment systems
and the training of clinical instructors, the training of examination item writers, the training of
Problem Based Learning (PBL) problem writers, and the training of PBL tutors); iv) strengthening
the Medical Education Unit (which included: staff recruitment system and physical/office facilities
improvement, and staff capacity building through short- and long-term in-country and overseas
training); and v) establishing a data management capacity (which included: building capacity to
manage a database on medical education, data analysis and reporting for education planning and
development, institution decision making and accreditation).
24. Eligible expenditures under the FAP included: workshops, teaching and laboratory
equipment, degree and non-degree training, scholarships for poor students from underserved areas
(maximum 10 students per school), information technology, technical assistance, minor building
renovation and enhancing library collections. Medical schools under Scheme A could not allocate
more than 20% of the total package to purchase goods, while those under Scheme B and C, the
maximum allowed to purchase goods was set to 60%.
7
25. Component 4: Project Management (US$5.239 million). This was expected to fund the
establishment of the Central Project Coordination Unit (CPCU) at the DGHE. Project resources
were expected to finance incremental operating costs, project management consultant, office
equipment, furniture and project monitoring and evaluation.
1.6 Revised Components
26. On December 20, 2012, the GoI requested a Level II restructuring of the HPEQ. This aimed
to: i) expand the project activities to include three additional professions; and ii) include two
universities under the FAP scheme. This request was approved on February 28, 2013 and the
project components were changed as follows:
Components 1 and 2 were expanded to incorporate three new professions: pharmacist,
nutritionist, and public health specialist. This included financing the system for quality
assurance of these three professions through building of stakeholder capacity and commitment,
preparation of academic papers, formulation of standards and accreditation of instruments,
development of examination blue prints and other items, and piloting the system;
Component 3 was expanded to include two universities in the FAP scheme: Nusa Cendana
University (UNDANA) in East Nusa Tenggara, and Cendrawasih University (UNCEN) in
Papua. The two universities were not successful in the FAP competitive grant selection.
However, given the strategic roles both universities played in meeting the needs for basic
health services in Eastern Indonesia, the government requested a specially designed scheme
for providing FAP grants to both universities (this was called the ‘affirmative FAP’).
27. In the light of the above restructuring there was a reallocation of funds to accommodate the
inclusion of three professions and two universities listed above. There were also changes in the
procurement plan to accommodate the participation of the two new universities.
1.7 Other significant changes
28. The GoI requested two reallocations of funds across components, as follows:
The first, as described above, on February 28, 2013, aimed to accommodate the inclusion of
the three professions and the two universities. This request included reallocation of loan
proceeds from component 3 (reduction of 6.7%) to components 1 (26% increase) and 2 (16%
increase) – see Annex 1, section c;
The second, on September 16, 2014, aimed to: a) fund activities under component 1 in order
to meet the KPI for this component (number of accredited schools); and b) fund activities under
component 4 in order to complete the various evaluation and learning activities leading to
project closing in December 2014. These reallocations resulted in transfer of proceeds of the
loan from component 3 (3% reduction) to component 1 (9.5% increase) and to component 4
(17% increase) – see Annex 1, section c.
8
2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design and Quality at Entry
29. Project Preparation. The project design was in line with national priorities and the World
Bank CPS (2009-2012) for Indonesia. Project preparation was relatively short, it took only eight
months from the identification in late September 2008 to appraisal in mid June 2009. The project
was effective on December 9, 2009. Project preparation included analytical work, including a
stakeholder analysis to map key actors, their interests and potential course of actions. Preparation
also involved a quality enhancement review (QER), which took place in May 2009 and provided
recommendations to improve project’s design. The QER panel endorsed the project objectives and
design, with the following recommendations: i) simplify project design by reducing the number of
KPIs; ii) reconsideration of the feasibility of components 1 and 2 implementation during the
proposed timeline and caution with the use of KPIs that depends on policy changes through
legislative processes; iii) adjustments in the school selection and matching criteria under the FAP
component 3 (to allow schools to select their partners); and iv) reconsideration of the feasibility of
the proposed FAP grant in driving institutional changes in the light of the institutional incentives
and capacity to absorb the investment.
30. Design and Quality at Entry. The project built on previous successful experiences in
improving quality of higher education in Indonesia through Bank operations and on the priorities
identified under the HWS Project.10 PDOs were defined with precision and KPI and intermediate
indicators generally helped to assess project achievements. Major limitation in terms of design was
the fact that grants were allocated only to medicals schools, representing 71% of the project’s
funds, but PDOs aimed at quality assurance for seven health professions.
31. The emphasis on improving medical schools capacity to meet the accreditation standards
was justified because: i) medical schools had already an incipient system of accreditation prior to
the project approval, consequently it made sense to allocate more resources to strengthen this
system; ii) strengthening medical education through accreditation and competence-based
examination would incentivize other professions to follow medical profession standards given the
role and influence of the profession in the provision of health care services; iii) the investments
made by medical schools on equipment, computer stations, etc., to conduct examinations (OSCE
and CBT) would benefit other professions (by sharing mannequins for OSCE and using computer
stations for CBT).11 Additionally, important to point that the initial GoI request for a lending
program focused only on the improvement of medical education. Midwifery and nursing were
added during the identification mission, given their importance in achieving the national goals of
maternal mortality rate (MMR) and infant mortality rate (IMR) reduction. But due to the
anticipated challenge of conducting a competitive grant program involving large number of
schools (around 700 nursing and 700 midwifery), it was decided to limit the block grant component
(FAP) to medicine, to gain experience before expanding to other programs.
10 Previous Bank-funded operations include the Development of Undergraduate Education (closed in September 2002) and the
Quality of Undergraduate Education (closed in March 2004) Projects. The Health Workforce and Services (HWS) project was a
World Bank-financed project implemented by MoH and MoNE from 2003-2008. 11 Although HPEQ is funded through public resources, the GoI agreed to include the private medical schools in competitive
selection for the FAP grant. This was a recognition that medical graduates regardless of their school of origin, would provide
services to the entire population – indeed 46% of the original block grant recipient were private schools.
9
2.2 Implementation
32. The project was able to implement all the envisaged activities and achieved the large majority
of the output targets. The initial implementation was faced with some challenges at different levels,
which are explained below and in more details in Section 5.
33. At government level: The main implementation challenges were: i) the changing of the
ministry structure (from MoNE to MoEC); ii) the delays in the transfer of resources from Ministry
of Financing (MoF) to MoEC due to delays in the approval of government’s budget (this resulted
that few activities could be implemented between the 6th and 7th implementation support missions);
and iii) the lengthy process to finalize and build consensus around the legal framework for the
functioning of the two agencies (NAA and NACEHhealthPro). Additionally, changes in the legal
landscape during project implementation with the approval of new acts, the Higher Education Act
and the Medical Education Act, required additional efforts to establish NAA and
NACEHhealthPro. Due to these delays in the initial years of the project, during the mid-term
review the project implementation progress was downgraded from moderately satisfactory to
moderately unsatisfactory.
34. At implementing agency level: The main implementation challenge was the DGHE’s limited
capacity to supervise and implement activities due to scarce human resources. Despite this
limitation, the DGHE was able to implement and monitor the project satisfactorily (see M&E
section).
35. At university level: The capacity to implement the grants varied widely among schools. This
was reflected in the heterogeneous levels of achievements across schools as well as different
disbursement performance. Schools in A scheme, for example, generally had access to other
sources of funds, local and international, and added to the restrictions and time limitation to use
the FAP grant, some schools did not use resources available.
36. Despite these initial challenges, during the last 2 years the project implementation was
largely on track to achieve its PDOs. At the time of project closing, most of the envisaged
activities had been carried out and the majority of the project funds had been disbursed (86.26%).
These are considerable achievements given the initial implementation delays, the complexity of
the project (multiple stakeholders often with conflicting interests) and the several changes in the
ministry structure during implementation period. These achievements are attributable to the
constant DGHE oversight and actions to maintain stakeholder engagement and progress of
implementation (e.g., creation of task force to develop standards of education and implement
NCBE). The remaining balance (13.67%) was not used and was cancelled at project closing.
37. The project had two restructurings, as mentioned in paragraphs 26-28 above. Both included
reallocations of funds across components and only the first restructuring involved the revision of
the project’s components (inclusion of three professions).
2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization
38. M&E design is moderately satisfactory. The project was monitored using five KPIs and
eleven intermediate indicators. Overall, the selected KPIs and intermediate indicators were defined
10
with enough precision to allow the measurement of implementation progress to achieve the PDOs.
Specific issues related to the indicators are:
Most of the indicators did not have baseline data which is understandable as the agencies and
standards for accreditation and competence examination were not in place at the beginning of
the project;
Target values were provided in the PAD for most of the intermediate indicators and KPIs
except for KPI 5: “Mean test score of graduates from FAP recipient schools who take the
National Competence Test”. The ICR team understand this implies a comparison between
grantees and non-grantees schools, but there was no benchmark or target associated with this
indicator;
The feasibility of component 1 KPI “Percentage of health professional schools that have gone
through accreditation process and publicize results” was questioned during the project
preparation (QER) due to the complexity of the institutional changes necessary to implement
this component within the project timeline. And indeed there was a revision of the initial target
due to the delays in establishing the accreditation agency (LAM-PTKes);
Likewise, component 2 KPI “Establishment of an independent National Agency for
Competency Examination of Health Professionals (NACEHealthPro)” was also linked to
policy actions which depended on lengthy negotiations and bureaucratic process.
39. M&E implementation and utilization was satisfactory. Throughout the project the CPCU
was able to provide, during the Bank’s team implementation support missions, information about
the project implementation progress. This information was collected through several instruments
developed to assess project progress (see paragraph 40). This was sufficient to assess project’s
overall performance as documented in the Aide Memoires and Implementation Status and Results
(ISR) reports. The M&E framework in place prioritized actions to monitor identified risks and to
respond to complaints. These actions ranged from periodic reviews to continuous monitoring and
review depending on the nature of the risk.
40. Several monitoring instruments were developed to assess the progress of project
components. These included administrative data and field visits. For components 1 and 2 the
Database for Health Higher Education (PD-DiktiKes) was used to monitor the number of schools
going through the accreditation process and the number of students undertaking the CBT and their
passing scores. The CPCU also made use of Information, Communications and Technology (ICT)
to improve communications and coordination across different stakeholders and to monitor
complains and improve governance. For example, the CPCU made available a website
(www.hpeq.dikti.go.id) to record complaints and follow up. For component 3 (FAP), the CPCU
maintained two strategies: i) regular assessment of the quality of terms of references (TORs)
submitted by website from each recipient school; and ii) regularly sent teams of experts to monitor
the technical and managerial performance of the FAP recipient schools. These teams were
composed of one expert in higher education from the BHE and one medical education specialist
from the AIPKI, and one procurement and one financial management specialists.
11
2.4 Safeguard and Fiduciary Compliance
41. Environmental Safeguard. The project was classified into safeguard category C, endorsed by
the World Bank’s safeguard review team. The FAP manual included a provision that any civil
work to be undertaken through the FAP program was limited to the rehabilitation of existing
lecture rooms and laboratories.
42. Social and Indigenous People Safeguards. Project was classified into category C on
condition that the scholarship program under the FAP grant component was not going to actively
seek scholarship candidates. During implementation, some medical schools requested to have
some active components in their recruitment processes. In response, the Bank team included a
social safeguard specialist in the implementation support mission to conduct a specific review of
the scholarship program implementation and assess its compliance with Bank’s safeguard policies.
Based on this review an implementation framework for the scholarship program was developed to
ensure that the program implementation was in compliance with relevant safeguard policies. This
framework was then included in an updated version of the FAP implementation manual, which
was revised and approved by the Bank safeguard team. At the end, the scholarship program for the
poor under HPEQ was not extensively used because the DGHE launched a national fellowship
program targeting the poor (called bidik misi) with significant amount of funding and for longer
period of time than HPEQ scholarship.
43. Financial management. During the initial implementation support missions the financial
management was rated satisfactorily as the selection of the FAP recipients was completed on time.
During the 3rd implementation support mission it was detected that the capacity to implement the
FAP varied widely among the FAP recipients especially in terms of financial and procurement
management. As a result, the supervision team requested the CPCU to map existing
implementation capacity of the grantees and to provide support based on each specific need.
During the subsequent implementation support missions the rating was downgraded to moderately
satisfactory due to the following challenges: i) the recruitment of individual consultants for
financial management was delayed as the CPCU could not identify individuals fulfilling the
required qualifications; ii) the submission of financial management reports was delayed; iii) the
performance of the financial management was inadequate; iv) the audit follow-up was delayed;
and v) the disbursement discrepancy between the Bank and project records was uncorrected.
44. Procurement. Procurement ratings ranged from satisfactory at the onset of the project
implementation to moderately satisfactory throughout the project implementation. Initially, the
Bank team found that the capacity to implement the procurement packages varied widely across
FAP recipient schools and, as a result, there were delays in completing procurement packages.
Following the assessments done during the implementation support missions the following
specific recommendations were made: i) recruit individual consultant to replace the existing
project management consultant; and ii) improve the monitoring of procurement process in each
school and proactively communicate with the schools to expedite the procurement process.
45. Midway through the project implementation there were some improvements in the
management of procurement activities. The CPCU managed to recruit the procurement consultant,
the team completed the mapping of FAP recipient capacity and provided technical assistance to
12
weaker FAP recipients. The disbursement rate increased significantly indicating that the
universities had successfully overcome initial challenges. The final disbursement rate (86.26%)
did not negatively affect the achievement of project objectives as the disbursement gap was mainly
due to rupiah depreciation and efficiency gains in the procurement of goods. The mapping of the
school capacity to implement the FAP conducted by the CPCU and its proactive engagement in
managing the project were key in addressing the challenges occurring during the project
implementation.
2.5 Post-completion Operation/Next Phase
46. HPEQ established the pathway to strengthen quality of higher education through accreditation
and competence-based examination. The legal framework, processes, instruments, manuals,
training materials and modules required to implement accreditation and competence-base
examination can be of use for more health professions (HPEQ initially covered 7, but the MoH
has signaled there is scope to apply for 22 health related occupations) as well as for other sectors.
The MoRTHE has indicated that HPEQ approach will be used as a model for other professions,
started already with engineering.
47. HPEQ success has triggered challenges for the future. The main one being the need to align
health professionals’ roles and functions within service delivery with CBC. This from one side
will ensure employability of graduating health professionals, and from the other will improve
health service delivery by increase the supply of health professionals with enhanced competences
and skills. The MoH has introduced the PHC competences, though this has not yet been linked
with the new CBC for any of the seven professions included in HPEQ.12 Another challenge is the
expansion of quality assurance mechanisms to a wider pool of health professionals, beyond the
seven categories included. And even when considering only the professions included in HPEQ the
process to scale up the accreditation and examination for the total universe of schools and
graduates will require major efforts, as for example: i) develop more accreditation instruments,
expand the training of facilitators, assessors and validators, expand the item bank and the number
of item bank writers; ii) ensure financial and operational sustainability of the two agencies; iii)
keep close monitoring of the effects of the new systems on health professionals labor market in
Indonesia and the impacts on population’s access to care and health outcomes.
3. Assessment of Outcomes
3.1 Relevance of Objectives, Design and Implementation
48. Relevance of Objectives. The PDOs were highly relevant to the country’s sectoral need of
strengthening the quality assurance system of health professional education, one of the priorities
of the government's 2010-2014 health sector medium term development plan. The new quality
assurance system is particularly important in the context of rapid expansion of health
12 The competencies of medical doctors are based on the standards of competencies released by the IMC in 2006 and revised in
2012, while the standards of competencies for the other professions were developed by the respective professional association
during HPEQ.
13
professionals’ schools in Indonesia, particularly in the private sector. Newly established schools,
both public and private, face challenges to achieve and sustain standards. In 2010, one-third of
medical schools were not accredited and only one-quarter had received the highest accreditation
level. The project was also aligned with the World Bank Indonesia CPS 2009-2012, which
supported the reform of the education sector.13
49. Relevance of Design. Design of the HPEQ is considered substantial. The project built on
priorities identified from previous sector work and project preparation benefited from Bank’s
cross-sectoral expertise (health and education). Design was supported by rigorous analytical work,
which helped to identify risks, opportunities and mitigation actions as well as build partnerships
and foster ownership of project activities and objectives among key stakeholders. HPEQ design
also benefited from a QER, which included experts from health and education sectors as peer-
reviewers.
50. PDO’s were defined with precision and KPI and intermediate indicators generally helped to
assess project achievements. There were very few shortcomings in the design, namely: i) the
selection of the KPIs – KPIs 1 and 2 were linked to policy actions which depended on legislative
and lengthy bureaucratic process; and KPIs 4 and 5 were related to students CBT results which
will take longer maturation period to have more pronounced impacts; ii) the allocation of resources
across components – although PDOs aimed at 7 professions, FAP grants were allocated only to
medical schools and it accounted for 71% of project funds at appraisal; iii) the affirmative FAP
was not successful – both universities had low rates of disbursement and no significant progress
was achieved in terms of capacity building to improve quality of medical education in these
institutions. However, important to point out, the two universities were included after project
restructuring and not at appraisal.
51. Relevance of Implementation. Implementation is considered substantial. Most of project
activities were successfully completed at the project closing date. This include the achievement of
almost all PDOs and intermediate outcomes. Disbursement rates varied across components, but
overall disbursement was 86.26%. There were some issues with the financial and procurement
management capacity of FAP recipients, which caused delays in the initial years of the FAP
implementation. The most important aspect of the project implementation that led to the
achievement of almost all PDO targets was the strong commitment and sense of ownership
fostered among main stakeholders (government, professionals’ associations, schools associations,
students, etc.). This was crucial to build consensus around the accreditation and competence-based
examination systems and around the role and functioning of the two agencies created under HPEQ
(NAA and NACEHealthPro). This is an important aspect of HPEQ that needs to be highlighted as
accreditation and certification are highly controversial processes and HPEQ managed to overcome
conflicting views and interests to establish the new systems and institutions.
3.2 Achievement of Project Development Objectives
52. The ICR team rates the achievement of PDOs as substantial. The reasons for this rating are
explained below and are supported by the data provided in the ICR datasheet and in the Annex 2.
13 Anderson et al. (2014).
14
The PDO aimed to strengthen quality assurance policies governing the education of health professionals in Indonesia. This was to be achieved by: i) rationalizing and assuring competency-
focused accreditation of public and private health professional training institutions; ii) developing
national competency standards and testing procedures for certification and licensing of health
professionals; and iii) building institutional capacity to employ results-based grants for
encouraging the use of accreditation and certification standards in the development of medical
school quality. The achievements of the KPIs, for each intermediate PDO, are explained below:
i) Rationalizing and assuring competency-focused accreditation of public and private health
professional training institutions.
53. With regard to the first KPI - The establishment of an independent National Accreditation
Agency (NAA). The NAA was established in October 2014 and formally recognized as an
independent entity authorized to conduct the accreditation of health higher education institutions
in Indonesia. The agency was created under the name of LAMP-PTKes (Lembaga Akreditasi
Mandiri Pendidikan Tinggi Kesehatan). Therefore, the first outcome indicator, related to
component 1, was fully achieved.
54. With regard to the third KPI - The percentage of health professional schools (medical dentistry,
nursing, and midwifery) that have gone through the accreditation process and have publicized the
results. At the time of project closing 173 schools completed the accreditation process using
specifically designed instruments and accreditation process, against a KPI target of 124 schools.
Therefore, the third outcome indicator, related to component 1, had the target exceeded by 39.5%.
Although total number of schools/study programs accredited exceed the target, there was some
variation in the achievement across different professions. Medical and dentistry schools were
slightly below the target (21 medical schools and 10 dentistry schools were accredited, against the
target of 29 and 10 schools, respectively) and nursing and midwifery exceeded the target
significantly (81 nursing schools and 62 midwifery schools accredited, against the target of 52 and
33, respectively).14 For 2015, LAM-PTKes expects to accredit 788 schools.
55. The new accreditation system has the following characteristics: i) the accreditation process
includes both desk review of the schools self-assessment and site visits. LAM-PTKes adopted the
principle of continuous quality improvement, in which LAM-PTKes identifies weaknesses in
compliance to the standards of education, provide feedback and suggestion to improve and comes
back to reassess after an agreed period; ii) the selection of assessors for LAM-PTKes starts with
candidates submitting their application followed by several tests. Once selected, assessors attend
training provided by experts from BAN-PT and LAM-PTKes. The assignment of assessors to
schools is conducted by the Accreditation Directorate of LAM-PTKes and follows the core
principle of avoiding conflict of interest; iii) it follows the standard that requires that students be
involved in the curriculum management process (consistent with the European standards of
excellence for student engagement; and iv) it is centralized, LAM-PTKes conducts accreditation
in the entire country (for the seven health programs included in HPEQ).
ii) Developing national competency standards and testing procedures for certification and
licensing of health professionals.
14 At the time of the ICR mission (February 2015) additional four medical schools were undergoing accreditation.
15
56. With regard to the second KPI - The establishment of an independent National Agency for
Competency Examination of Health Professionals (NACEHealthPro), The National Agency for
Competency Examination was legalized as independent agency on December 2013 through the
Ministry of Justice and Human Rights Decree (No. AHU-291). The agency was created under the
name of LPUK-NAKES (Lembaga Pengembangan Uji Kompetensi Tenaga Kesehatan). Under
the new legal framework the authority to implement the competence test is given to the National
Committee for Competence Examination (for medicine and dentistry) and to the Indonesia Health
Workforce Council (MTKI) (for the remaining five professions). LPUK-NAKES has full authority
to develop and ensure the quality of testing, which includes: provide training for examination item
development, item bank management, examination management and standard setting, and running
examination trials. Therefore, the second outcome indicator, related to component 2, was fully
achieved.
57. With regard to the fourth KPI - The percentage of graduates of health professional schools
(medical, dentistry, nursing, and midwifery) passing national competency testing at the first
attempt, the results are mixed. The KPI was not achieved for medical students (actual 76%
against the target of 84%) and for nursing (actual 58% for bachelors nurses and 47.81% for
diploma nurses - DIII, against the target of 65% for both bachelors and diploma nurses - DIII).15
The target was achieved for dentistry (target of 83% and actual value equal to 88%) and for
midwives (target of 65% and actual equal to 64.65%).
iii) Building institutional capacity to employ results-based grants for encouraging the use of
accreditation and certification standards in the development of medical school quality
58. With regard to the fifth KPI - The mean test score of graduates from the FAP recipient schools
who have taken the National Competence Test, results are not fully conclusive given the absence
of a concrete target for this indicator. However, the mean competency examination scores of FAP
recipients showed an increase of around 10% from baseline (60.88 in 2010) to project closing
(67.07 by the end of 2014). Additionally, when comparing FAP recipients with non-FAP recipients
the mean score of FAP recipient graduates was higher than of those of non-FAP recipients during
the entire period of project implementation. Moreover, this difference increased from the baseline
(3.96 in 2010) to project closing (5.76 in 2014).
3.3 Efficiency
59. The project efficiency is considered substantial. The reasons for this rate are: i) HPEQ
achieved most of the PDO targets without making use of the total project funds with no prejudice
to the project implementation; ii) the policy and regulatory reforms introduced, the creation and
functioning of the new agencies and the methodology for developing the various standards of
competences, accreditation instruments and competence test examination, were all triggered by
HPEQ.
15 DIII is a three year nursing diploma.
16
60. Disbursement rates varied across components, but overall disbursement at the time of
project closing was 86.26%. This disbursement level did not affect the achievement of the PDO
targets. The final disbursement level was influenced by the rupiah depreciation and efficiency
gains in the procurement of goods (FAP recipients benefited from discounts in the procurement of
computers for CBT and mannequins for OSCE and, therefore, spending was lower than the initially
planned).16 As previously noted, the majority of project funds was allocated to component 3 (FAP),
even though this component only directly affected one outcome indicator. However, in terms of
efficiency, this was justified because: i) medical schools had already an incipient system of
accreditation prior to the project approval, so it was more cost-effective to focus on them; ii) the
investments made by medical schools on equipment, computer stations, and mannequins for
OSCE, benefited other professions (by sharing equipment).
61. Although limited, the existing evidence has demonstrated that a better educated health
workforce improves quality of care and also saves lives. What is completely absent is the empirical
assessment of the economic costs and benefits of accreditation and certification processes. A full
economic analysis (such as cost-benefit analysis) of the HPEQ impacts is difficult and unlikely to
realistically capture full project impacts. Firstly because the main expected impact of the project
is the improvement in quality of health services through improved quality of health professionals.
But although there is a clear causality between improved quality of care and competences of health
professionals, the exact impact may not be possible to be measured and may vary across cadres,
levels of care, providers’ characteristics and broader systems of HRH management and incentives.
Secondly, the impacts of HPEQ will take time to be observed in service provision, when the newly
graduated health professionals start practicing and implementing competences learned during
training and tested. Finally, significant part of the HPEQ benefits will come to individual health
professionals who will likely have more opportunities within the local labor market (and regional,
international labor markets) as now they graduate from accredited schools and have their
knowledge assessed through standardized examination. The same applies to schools, in the sense
that in the future accreditation status will influence students’ choices of schools to enroll.17
62. The cost-benefit analysis presented in the annex 3 takes a rather conservative approach in terms
of impacts and timeline for HPEQ results. It links the (improved) quality of training to improved
quality of care over a practitioner’s career, and hence population health outcomes (in terms of
infant mortality rate and maternal mortality rate). It then monetizes the gains and compares them
to project costs. The discounted total benefits of the project, estimated in productive life years
gained, is estimated at US$838.29 million which is significantly higher than the total value of the
project costs (US$77.82 million). This results in a benefit-cost ratio equal to US$10.77, which
means that for each US$1 invested through the project there is an expected return of US$10.77
(only considering the period of five years after project closing). Although significantly high, this
result are based on conservative assumptions adopted and likely underestimates the total project
benefits.
16 Rupiah depreciation from US$ 1 = IDR 10,300 at project preparation to US$ 1 = IDR 12,275 at project closing. 17 GoI already selects civil servants applying a rule that exclude graduates from low performers’ schools.
17
3.4 Justification of Overall Outcome Rating
Rating: Satisfactory.
63. As discussed above, the project design was considered highly relevant due to its fine alignment
with government priorities and Bank’s strategy (CPS) and it addressed important health sector
challenge (quality of health professionals). The design also benefited from analytical work and
recommendations from the QER that helped to identify risks, opportunities and mitigation
measures. PDOs were defined with enough precision that allowed to measure their implementation
progress and project achievements. The activities funded under the four components helped to
achieve HPEQ PDOs satisfactorily, most KPIs were achieved or surpassed. Therefore, the overall
project rating is satisfactory.
Project
Relevance
Achievement of PDO
(efficacy)
Efficiency Overall Rating
High Substantial Substantial Satisfactory
3.5 Overarching Themes, Other Outcomes and Impacts
(a) Poverty Impacts, Gender Aspects, and Social Development.
64. Although HPEQ did not have any specific focus on gender, it is expected that its
achievements will benefit women considerably. The reason is that a large share of the health
workforce in Indonesia is female. In 2010, for example, percentage of contract female doctors in
the country was around 60% while the contract female dentists was 81%.18 This share for nursing
and midwifery is likely to be higher as these are female dominated professions worldwide.
Therefore, HPEQ will likely to impact on female employability in Indonesia.
65. The affirmative FAP included two universities (UNCEN and UNDANA) from poor regions
of the country (East Nusa Tenggara, and Papua). HPEQ also provided, under component 3, an
affirmation program by providing bachelor degree scholarship to poor medical student candidates;
152 scholarships were granted under this program.
(b) Institutional Change/Strengthening
66. HPEQ’s major contribution is the establishment of the legal and regulatory frameworks for
school accreditation and the competence-based examination for health professionals. Annex 9 lists
all regulations that were put in place to guarantee the functioning of the two independent agencies
and the implementation of the accreditation and competence-based examination. The close
participation of key stakeholders and the sense of ownership built during project implementation
helped to strengthen relationships among universities, professional associations, schools
associations, and different ministries.
18 Anderson et al. (2014),
18
67. HPEQ also improved capacity at DGHE/MoRTHE to develop accreditation systems for other
professions. The minister already signalized that the system developed under LAM-PTKes and
LPUK-NAKES will be a model for other fields. Another important contribution is the
strengthening of MoRTHE and MoH collaboration, which was less harmonious before HPEQ.
HPEQ also contributed to create a long-term cooperation among universities matched in the FAP.
During the ICR field visits it was noted that some universities had planned activities beyond project
timeframe and to be funded through own resources.
68. The ICT system implemented under HPEQ supported implementing agency in the project
management and M&E throughout implementation. This system will serve as the information
management system for the two new agencies (LAM-PTKes and LPUK-NAKES). The CPCU also
implemented a web-based knowledge portal to disseminate HPEQ products (documents,
publications, etc.), policy reforms and other materials related to health professionals’ education.
This is expected to be a channel for collaboration and debate among health professionals.19
(c) Other Unintended Outcomes and Impacts (positive or negative)
69. During the ICR visit the following unintended outcomes were noted:
Students reported that some schools focused more on providing short term training for students
to take the NCBE instead of implementing curriculum changes based on the test
requirements/contents;
Related to the above, some schools seem to have been selecting students that will take the
NCBE based on their try-out results. This may be influenced by the fact that NCBE results is
part of the accreditation process;
HPEQ also improved GoI capacity to develop and implement quality assurance systems for
higher education. The GoI has already signaled that the system developed under LAM-PTKes
and LPUK-NAKES would be a model for other fields;
HPEQ also supported the creation of HPEQ Student. This network includes students from the
seven professions from all universities in Indonesia. HPEQ Student aims to engage students in
health professional education policy formulation and to foster inter-professional collaboration
across health disciplines;
HPEQ helped to establish a network of health professionals associations, health professional
schools association, government entities, students and broader civil society, committed to long
term support for quality improvements in the education of health professional in Indonesia.
3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops
Not applicable.
19 www.kmshpeq.net.
19
4. Assessment of Risk to Development Outcome
Rating: Moderate.
70. There are moderate risks to the sustainability of outcomes, these are:
Financial: Besides the fact that the two agencies (LAM-PTKes and LPUK-NAKES) have
developed their business plans, there are issues regarding how they will be financed in the
medium and long run:
- This is particularly the case for the LAM-PTKes as the new accreditation arrangement
requires schools to bear the costs of accreditation. Under the previous model (BAN-
PT) accreditation was subsidized by the government, free of costs to schools. Under
the new model, the unit cost for accreditation is estimated at IDR87.500.000 for
bachelor level and IDR73.000.000 for vocational, master, doctoral and specialized
level.20 The GoI has guaranteed resources for the accreditation of 400 schools in 2015
(out of 788 expected to be accredited during this year). Professional associations and
association of education institutions had also made financial commitments to LAM-
PTKes. But these funds are not stable and there is a need to secure government support
for the initial years of the agency operation (beyond 2015);
- For LPUK-NAKES the funding sources are: contributions from members (professional
associations), fees on services provided to members (CBT try out, training of assessors,
etc.), and government subsidies (no specified amount). The funding mechanism for
LPUK-NAKES is more straightforward as most of the schools recovers the cost of the
CBT from the students, either directly (through specific fees) or indirectly (including
the costs in the tuition fees).
Technical: Both agencies face challenges to recruit full and part-time professionals with
qualification in health education. Particularly for LAM-PTKes given the need to recruit
facilitators, assessors and validators;
Institutional: There is a need to ensure the alignment of competences and curriculum to
accreditation and examination systems. Schools seem to have incentives to adopt opportunistic
behavior by training students specifically for the CBT and OSCE as opposed to strengthening
their curriculum (adopting CBC). Additionally, it is not clear whether the government
recognition of LAM-PTKes means that the agency is the only entity in charge of the
accreditation system or whether there is scope for other institutions to play a role in this
‘market’;
Political: Change in government leadership may affect political commitment and support to
the agencies and, consequently, the financial support. Additionally, agencies will need to
maintain the engagement of the different stakeholders beyond HPEQ closing. During project
implementation, stakeholders’ engagement was facilitated by the project capacity to mobilize
funds for travel, venue rental and other logistics necessary to carry out meetings and
20 Equal to US$6,664.34 and US$5,551.37 respectively (1 US$ = 13,129.58 IDR on May 7, 2015).
20
consultations. Additionally, the MoNE is under restructuring (to become MoRTHE) and this
may affect the housing of initiatives developed under the project within the ministry;
Finally, there is a need to ensure alignment between health services delivery strategies and
health workers competences. Ensuring alignment with government initiatives (e.g., PHC
competences) will increase the relevance of the new accreditation and examination systems to
the extent that they can help to support government health sector strategies (e.g., in expanding
PHC services).
5. Assessment of Bank and Borrower Performance
5.1 Bank Performance
(a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory
71. The project design built on priorities identified from earlier sector work and objectives
were in line with national priorities. More specifically, the Bank accompanied the GOI’s interest
in improving the quality of education as depicted in the medium term development plan for 2010-
2014. The project preparation benefited from Bank’s expertise from both health and education
sectors, and was rigorously supported by analytical work (particularly the stakeholder analysis),
which helped to map and integrate key stakeholders (governmental and non-governmental
organizations) based on their interest and capacity to influence and oversee project activities.
72. The early reviews and analytical work helped to improve project design. For instance, the
stakeholder analysis identified risks, opportunities and mitigation measures. This included
identifying key stakeholders’ views and interests regarding the accreditation and certification
procedures. The project benefited from the stakeholder analysis as, unquestionably, the project
achievements were a direct result of the ownership and commitment from key stakeholders. The
preparation period also benefited from a QER. The QER panel endorsed the project objectives and
design, and provided recommendations (see paragraph 30). Some of the issues raised during the
QER were observed during project implementation, as for example: i) the delays in the legislative
process to approve regulations. This affected the achievement of the KPIs within the proposed
period of time; and ii) the balancing between the size of the grants with the institutional capacity
and incentive to use the grants was anticipated during the QER. During the ICR visit the team
noted that the incentives as well as the capacity to use FAP resources varied significantly across
FAP recipients. Overall, these shortcomings did not affect the achievement of the projects’
objectives, at worst case they just delayed the time of achievement. Taking into account the
complexity of the project as well as what was achieved, these delays did not compromise the
quality of the project design.
(b) Quality of Supervision Rating: Satisfactory
21
73. HPEQ had two task team leaders (TTL) during the entire project. TTLs were based in
Jakarta during the entire project implementation allowing continuity of support to the government
and close monitoring of project activities. For example, the Bank team actively sought advice from
fiduciary and safeguards specialists to support implementation. The team also proactively
identified consultants, international and local experts, to assess the implementation of the
component 3 partnership scheme and the implementation of components 1 and 2. These initiatives
provided timely technical inputs to the development of policies and operational instruments for
accreditation and CBT.
74. There were regular implementation support missions, which allowed the team to provide timely
inputs to the government and up-to-date information to Bank management. The findings of each
visit, nine in total (two per year), were conveyed in a structured manner in the ISRs, including
specific sections with detailed reports of the progress towards achieving the PDOs, project status,
implementation shortcomings and courses of action.
75. The Bank team timely intensified efforts to address challenges as they began to surface. For
example: the delays in channeling government resources to the DGHE; the lengthy process to put
in place the legal basis for the functioning of the two agencies (for accreditation and CBT); the
complex management of the collaboration across different government units and non-government
organizations. Efforts to address these challenges were done through extensive reviews of the
implementation of project activities (field visits), frequent meetings with key stakeholders,
proactively facilitating restructuring and reallocation of project funds across components to
address project’s needs. Additionally, after the implementation progress was reduced to
moderately satisfactory the Bank team started to have interim implementation support missions
between regular (implementation support) missions.
(c) Justification of Rating for Overall Bank Performance Rating: Satisfactory
76. The ICR team rates the overall Bank performance as satisfactory given the satisfactory rating
for both quality at entry and supervision.
5.2 Borrower Performance
(a) Government Performance Rating: Satisfactory
77. The Government demonstrated strong commitment and full ownership throughout the project.
This is reflected in the engagement of different ministries in the process of setting the legal basis
for the accreditation and competence test systems and respective agencies (see Annex 9). For
example, the MoEC and MoH joint regulation determined that the national competency
examination for health professions would refer to the methodology designed by LPUK-NAKES.
For the accreditation system, the MoEC declared that LAM-PTKes is in charge of conducting
accreditation of health professionals’ schools in the country and allocated resources to cover
accreditation costs of 400 schools in 2015.
22
78. Despite some delays in setting up legal framework (mainly due to MoEC internal processes
and lengthy consultations with stakeholders), these were defined and implemented within the
project implementation period. Additionally, to minimize the impacts of the delays the GoI
implemented two task forces to exercise the functions of the two agencies until the legal
frameworks were in place. This allowed to carry out the supporting activities necessary to
implement the new accreditation and competence test systems.
79. Main challenges faced by MoEC during the implementation period were: i) changing
ministry structure (from MoNE to MoEC and finally to MoRTHE)21; ii) delays in the transfer of
resources from MoF to MoEC due to delays in the government budget approval; and iii) lengthy
process to finalize and build consensus around the legal framework for the functioning of the two
agencies (LPUK-NAKES and LAM-PTKes).
80. Government effectively overcame the above mentioned challenges and the ICR team rates the
GoI performance as satisfactory. The main reasons were: the mitigation measures taken to
guarantee the progress of project implementation given the unexpected delays, and the support and
strong commitment from different ministries (MoEC, MoH, Ministry of National Development
Planning (BAPPENAS), and MoF) to the reform and to effectively managing complex and often
conflicting interests among stakeholders. Additionally, the establishment of the two bodies
resulted in transferring of power from the government to a non-government entity, which is not
usual for a country like Indonesia.
(b) Implementing Agency or Agencies Performance Rating: Satisfactory
81. The executing agency for HPEQ was the MoEC while the implementing unit was the DGHE.
Although not an implementing agency, the MoH participated in all policy discussions and played
an important role in setting up of the legal framework for accreditation and examination systems.
The CPCU was established within the office of the Directorate of Learning and Student Affairs
(LASA). The project director and project manager were the Director General of Higher Education
and the Director of Learning and Student Affairs, respectively. A Project Steering Committee was
established with representatives from BAPPENAS, MoEC, MoH and MoF and was very
functional during project implementation. At university level (component 3), Project
Implementation Units (PIU) were established, prior to FAP contract being signed, to support
implementation and administration of the project and rectors had the ultimate responsibility for
project implementation.
82. At university level, the capacity to execute component 3 activities was an issue. The capacity
to implement the FAP varied widely among schools. In particular, the performance of the
partnerships between the schools A and B schemes varied widely due to the limited administrative
and technical capacity to implement the proposed activities. Furthermore, the lengthy process to
review TORs (by the CPCU) and the incomplete implementation manual influenced negatively
the ability of the FAP recipients to implement the planned activities on time.
21 MoNE before 2011, then MoEC in the period of 2011 – 2014, then MoRTHE from 2014.
23
83. The CPCU and the DGHE’s have performed remarkably well in pushing for the regulations to
establish the two bodies. Despite the significant bureaucratic hurdles within MoEC and often
conflicting views of key stakeholders. The CPCU and the DGHE played an important role in
advocating, lobbying, organizing numerous meetings/workshops, and going through numerous
redrafting of the regulations to satisfy all stakeholders and yet still maintain the quality and
commitment to the HPEQ goals.
(c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory
84. The ICR team rates the overall borrower performance as satisfactory. This is because rating
for borrower performance and implementing unit performance were both satisfactory for the
reasons discussed above.
6. Lessons Learned
85. HPEQ was an unique project given its importance, cross sectoral nature, complexity and
impacts. The lessons learned from HPEQ are useful from both operational perspective (design and
implementation of Bank’s project) as well as technical (in terms of providing a platform for higher
education quality assurance systems for other professions in Indonesia and for other countries).22
The key lessons learned are:
Project design was successful in engaging multiple stakeholders and creating a sense of
ownership of the project. It created a momentum to strengthen stakeholders’ commitment
to improve quality of health professionals education;
Strong collaboration between government and non-government entities was essential to
conduct the reform process. Commitment and active involvement of the professional
associations and the associations of health professional schools were determinant to
achieve HPEQ PDOs. These associations were the founders of LAM-PTKes and LPUK-
NAKES and they were also the main contributors to the preparation of: i) the academic
papers for the accreditation and national examination systems; ii) the standards of
education; iii) the standards of competencies; and iv) the accreditation instruments;
Incentives for schools were very different across regions, accreditation level and ownership
(public and private). The size of the FAP must be better balanced in future similar
operations to create more appropriate incentives for participation and to improve
disbursement;
The process to set accreditation and competence-base examination are lengthy and depend
on the existing legal and institutional framework. Additionally, the political economy of
transferring the authority (accreditation) from the government to non-government entities
22 Accordingly to WHO more than half the countries of the world appear to lack a credible, transparent and comprehensive
accreditation system (WHO, 2013. Transforming and Scaling Up Health Professional Education and Training. Policy Brief on
Accreditation of Institutions for Health Professional Education. World Health Organization, Geneva).
24
is challenging and requires a very long and difficult change process. In future similar
operations, consideration should be given on the selection of KPIs to take into account
lengthy political and legal processes;
For the FAP the comparison of mean test between FAP recipients and non-FAP recipients
are not useful to monitor implementation and achievement as there are too many
confounding factors influencing the results;
7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners
(a) Borrower/implementing agencies
See Annex 7.
(b) Cofinanciers Not applicable.
(c) Other partners and stakeholders Not applicable.
25
Annex 1. Project Costs and Financing
(a) Project Cost by Component (in US$ Million equivalent)
Component
Appraisal
estimate
(US$ millions)
Actual/Latest Estimate
(US$ millions)
Percentage of
Appraisal
Component 1- - Training and workshops,
incremental operating costs,
research expenditure, consultant services and
goods under Part 1 of the Project
7,184 7,023 97.8%
Component 2 - Training and workshops,
incremental operating costs, research
expenditure, consultant services and goods
under Part 2 of the Project
12,899 14,609 113.3%
Component 3 - FAPs for sub-projects under
Part 3 of the Project 61,400 44,681 72.8%
Component 4 -Training and workshops,
incremental operating costs, consultant services
and goods under Part 4 of the Project 5,239 5,478 104.6%
Total 86,722 71,791 82.78%
(b) Financing
Source of funds
Appraisal
estimate
(US$ millions)
Actual/Latest Estimate
(US$ millions)
Percentage of
Appraisal
International Bank for Reconstruction and
development 77,822 67,126 86.26%
Government of Indonesia & Higher Education
Institutions 8,900 4,665 52.41%
Total 86,722 71,791 82.78%
(c) Reallocation by Category of Expenditure (in million US$)
Category of expenditure Appraisal
2009
Reallocation
2012
Reallocation
2014
Actual
(disbursed)
* 2014
Component 1: Training and workshops, incremental operating costs, research expenditure, consultant services and goods under Part 1 of the Project
6,584 8,335 9,127 7,629
Component 2 -Training and workshops, incremental operating costs, research expenditure, consultant services and goods under Part 2 of the Project
12,099 14,047 14,047 12,470
Component 3-FAPs for sub-projects under Part 3 of the Project 55,000 51,301 49,784 46,695
Component 4-Training and workshops, incremental operating costs, consultant services and goods under Part 4 of the Project
4,139 4,139 4,864 4,679
Total 77,822 77,822 77,822 71,473
* According to the government ICR.
26
Annex 2. Outputs by Component
Indicator #
Indicator's name
according to the
PAD
Target value Actual end line value Result
Component 1: Strengthening Policies and Procedures for School Accreditation
KPI 1
The establishment of
an independent
National
Accreditation
Agency (NAA).
Establishment of
the Charter of the
NAA
Through the MoEC Decree
No 291/P, October 17 2014,
the independent accreditation
agency was formally
recognized as an independent
entity authorized to conduct
the accreditation of health
higher education (under the
name of Lembaga
Akreditasi Mandiri
Pendidikan Tinggi
Kesehata – LAMP-TKES).
Achieved
KPI 3
The percentage of
health professional
schools (medical
dentistry, nursing,
and midwifery) that
have gone through
the accreditation
process and have
publicized the
results.
medicine (29, 42%),
dentistry (10, 42%),
Nursing (52, 10%),
Midwifery (33,
10%). Total 124
schools.a
medicine (21), dentistry (9),
nursing (82), midwifery (56).
Total 168 schools
Surpassed
Intermediate
Outcome 1
Completion status of
preparatory activities
for the establishment
of the NAA
NAA has
independent and
adequate budget to
conduct
accreditation and
has access to
adequate numbers
of suitably trained
assessors
Y Achieved
Intermediate
Outcome 3
Accreditation
instrument for the
four health
professional schools
are ready for use.
medicine
(international peer
review of
instrument),
dentistry (piloting
and dissemination),
nursing (piloting
and dissemination),
Midwifery (piloting
and dissemination)
Y Achieved
27
Component 2: Certification of Graduates Using a National Competency-based Examination
KPI 2
The establishment of
an independent
National Agency for
Competency
Examination of
Health Professionals
(NACEHealthPro).
Establishment of
the Charter of the
NACEHealthPro
The National Agency for
Competency Examination
(Lembaga Pengembangan
Uji Kompetensi Tenaga
Kesegatan - LPUK-NAKES)
was legalized as independent
agency on December 20 2013
through the Ministry of
Justice and Human Rights
Decree No. AHU-291
Achieved
KPI 4
The percentage of
graduates of health
professional schools
(medicine, dentistry,
nursing, and
midwifery) passing
national competency
testing at the first
attempt.
medicine (84%),
dentistry (90%),
nursing (65%),
midwifery (65%)
medicine (72.47%), dentistry
(92.31%), nursing (57.81%),
midwifery (64.65%)
Achieved for
dentistry and
midwifery.
Not achieved for medicine
and nursing
Intermediate
Outcome 2
Standard of
Competencies and
Standard of
Education for the
four health
professions are
available.
nursing (both
standards
completed and
legalized),
midwifery (standard
of education
completed and
legalized)
Y Achieved
Intermediate
Outcome 4
Number of trained
assessors
medicine (86),
dentistry (39),
nursing (105),
midwifery (105),
nutrition (40),
public health (40),
pharmacy (40).b
medicine (132), dentistry
(34), nursing (123),
midwifery (70), public health
(17), nutrition (16), pharmacy
(41)
Surpassed
for medicine,
nursing and
pharmacy.
Not achieved
for dentistry,
midwifery,
public health
and nutrition.
Intermediate
Outcome 5
Introduction of CBT
and OSCE for NCE
CBT for all four
professions. OSCE
for medicine and
dentistry, OSCE try
out for midwifery
and nursing. CBT
try out for the added
professions, OSCE
preparation only for
pharmacy.a
CBT for medicine, dentistry,
and nursing Paper based test
for midwifery and diploma of
nursing (DIII). CBT try out
for nutrition and pharmacy.
OSCE for medicine and
dentistry, and under
preparation for pharmacy.
Achieved
except for
nutrition
28
Intermediate
Outcome 6
Number of National
OSCE trainers
72/profession
(medicine and
dentistry only).
36/pharmacy.a
medicine (4950), dentistry
(84)
Surpassed
for medicine
and dentistry.
Not achieved for
pharmacy.
Intermediate
Outcome 7
Number of National
MCQ Item Writers
1,044 /initial
professions), and
added 36/profession
for nutrition, public
health and
pharmacy.b
medicine (254), dentistry
(650), nursing (828),
midwifery (675), pharmacy
(219), public health (141) and
nutrition (144).
Not achieved
for the four
initial
professions.
Surpassed for the three
added
professions
Intermediate
Outcome 8
Number of National
OSCE Item Writers
72/profession
(medicine and
dentistry),
36/pharmacy.a
medicine (221), dentistry
(650), pharmacy (39). Surpassed
Intermediate
Outcome 9
Number of MCQ
Item Writers and
reviewers
1,044 /initial
professions), and
added 36/profession
for nutrition, public
health and
pharmacy.b
medicine (216), dentistry
(650), nursing (828),
midwifery (675), pharmacy
(78), public health (84) and
nutrition (132).
Not achieved
for the four
initial
professions.
Surpassed
for the three
additional
professions.
Intermediate
Outcome 10
Number of OSCE
Item Writers and
Reviewers.
1,044/profession
(medicine and
dentistry only),
36/pharmacy.c
medicine (86), dentistry (63),
pharmacy (39).
Not achieved
for medicine
and dentistry.
Surpassed
for
pharmacy.
Component 3: Results-based Financial Assistance Package (FAP) for Medical Schools
KPI 5
The mean test score
of graduates from
the Financial
Assistance Package
(FAP) recipient
schools who have
taken the National
Competence Test.
67.07
Target not
specified
(The mean
test scores of
non-FAP
63.83 at
project
closing)
Intermediate
Outcome 11
Number or
percentage of
medical schools
receiving finance
support to strengthen
the program
43 43 Achieved
a Revised targets; b The targets for the three new professions were added during MTR;
29
Annex 3. Economic and Financial Analysis
1. HPEQ focused on strengthening quality assurance systems for health professional education
in Indonesia. This was done through: i) rationalizing and strengthening the accreditation of public
and private medical, dental, nursing, and midwifery schools; 2) implementing competency-based
certification; and 3) providing results-oriented resources to assist health education institutions meet
these challenges. The project activities focused on the establishment of an independent body for
schools accreditation, the LAMP-TKes, and an independent national agency for competency
examination, the LPUK-NAKES. Project funded the implementation of grants for medical schools to
strengthen the quality of medical education in Indonesia.
2. A functioning quality assurance system for health professional education is a way to minimize
the asymmetry of information inherent to health care markets. Consumers may not be able to
determine the quality of services provided by the health workforce, therefore accreditation and
competence examination are mechanisms to assure consumers that the inputs into the health production
function (in this case, labor) are of sufficient high quality.23 Accreditation, for example, is expected to
reduce variations in the clinical practice, eliminate inappropriate care, and control costs.24
3. The need to improve the competence and knowledge of the Indonesia’s health workforce was a
consensus at time of project preparation. The 2007 Indonesia Family Life Survey (IFLS) provided the
evidence based for that. The IFLS included vignettes to measure the diagnostic and treatment capacity
among doctors, nurses and midwives. Results showed a low percentage of correct responses to vignette
questions for antenatal care (45%), for child curative care (62%), and for adult curative care (57%).25
Additionally, evidence from Barber and colleagues (2007) demonstrated that low-quality of care in
Indonesia was to a large extent results of poor quality of training among health professionals.26
4. There is evidence demonstrating the links between schools accreditation, health professional
certification and health outcomes. Silber and colleagues (2002) found that certification of
anesthesiologists in the Unites States is significantly associated with a 13% reduction in mortality, after
controlling for a number of observable variables.27 A systematic review conducted by Alkhenizana &
Shaw (2011) has shown a consistent positive association between accreditation programs and clinical
outcomes.28 Another study, by Clark and colleagues (1998), found positive effects of providers’
education on child health outcomes related to asthma.29 In developing country context, Peabody and
colleagues (2008) found that national level accreditation influenced quality of care in the Philippines.18
Barber and Gertler (2009) estimated that a one standard deviation increase in quality (of providers)
reduces the prevalence of child stunting by six percentage points in Indonesia.30 The authors measured
quality of care by the practitioners’ answers to the vignettes questions.
23 Nicholson and Propper (2008). Nicholson S and Propper C (2012). Medical Workforce. Handbook of Health Economics, Vol.
2: 873-925. 24 Peabody et al. (2008). Should we have confidence if a physician is accredited? A Study of the Relative Impacts of Accreditation
and Insurance Payments on Quality of Care in the Philippines. Soc Sci Med. 2008 August ; 67(4): 505–510 25 Anderson et al. (2014). 26 Barber et al. (2007). Differences in Access to High-Quality Outpatient Care in Indonesia. Health Affairs, 26(3): w352-w366. 27 Silber et al. (2002). Anesthesiologist Board Certification and Patient Outcomes. Anesthesiology, 96:1044–52 28 Alkhenizana and Shaw (2011). Impact of Accreditation on the Quality of Healthcare Services: a Systematic Review of the
Literature. Ann Saudi Med. 2011 Jul-Aug; 31(4): 407–416. 29 Clark et al. (1998). Impact of Education for Physicians on Patient Outcomes. Pediatrics, 101: 831-836 30 Barber and Gertler (2009). Health workers, quality of care, and child health: simulating the relationships between increases in
health staffing and child length, 91(2):148-55
30
5. Although limited, the existing evidence has demonstrated that a better educated health
workforce improves quality of care and also saves lives. What is completely absent is the empirical
assessment of the economic costs and benefits of accreditation and certification processes. A full
economic analysis (such as cost-benefit analysis) of the HPEQ impacts is unlikely to capture the full
project impacts. Firstly, because the main expected impact of the project is the improvement in the
quality of health services through improved quality of health professionals. However, despite the fact
that there is a clear causality between improved quality of care and competences of health
professionals, the exact impact may not be possible to be measured and may vary across cadres, levels
of care, providers’ characteristics and broader systems of HRH management and incentives. Secondly,
the impacts of HPEQ on service delivery will take time to be observed. This will be felt when the
newly graduated health professionals start to practice and to implement the competences learned during
training and tested through CBT. Finally, significant part of the HPEQ benefits will benefit individual
health professionals who will likely have more opportunities within the local (as well as regional and
international) health care labor market as now they graduate from accredited schools and have their
knowledge assessed through the CBT. The same applies to schools, in the sense that in the future
accreditation status will influence students’ choices of schools.31
6. Having these limitations in perspective, the ICR economic analysis adopts a conservative
approach. The approach is conservative in three dimensions: i) it suggests modest gains in the quality
of care provided by a certified health practitioner trained by an accredited institution; and ii) it only
considers dimensions of health improvements that are relatively easily translated into monetary terms;
iii) the analysis covers an horizon of five years only (while project benefits are permanent). The
analysis links the (improved) quality of training to improved quality of care over a practitioner’s career,
and hence population health outcomes. It then monetizes the health outcomes gains and compares them
to project costs.
7. The impact of the project can be modeled as a series of interventions that affect the following health
indicators applicable to the beneficiaries of the project: infant mortality rate and maternal mortality
rate. The next steps is to measure the impact of HPEQ outcomes on these two indicators, value these
impacts in monetary units and compare them to HPEQ costs.
8. Valuing gains from reduced mortality is a long standing debate in the health economics
literature. Major issues are the ethical and equity debate around the task of how to value a life saved
or improved. The available literature provide some basis for the monetary estimation of these gains.
Alderman and Berhman (2004), for example, estimates the savings of US$1,250 for saving an infant’s
life through a measles campaign.32 An alternative approach is to measure the impacts, and the monetary
benefits, in terms of productive life years gained due to reduced mortality. This is done by calculating
the number of years gained as a result of project interventions and calculating the economic benefit of
these years. Given the nature of the interventions and the likelihood of having great impact among
children, hence large gains in terms of future productive life years, the economic evaluation adopts this
approach to measure the economic benefits of the project.
9. The analysis uses population health and demographic indicators (table A3.1) and apply the following
assumptions to estimate the (economic) benefits of reducing child and maternal mortality, as follows:
31 GoI already selects civil servants applying a rule that exclude graduates from low performers’ schools. 32 Alderman and Behrman (2004). Estimated Economic Benefits of Reducing Low Birth Weight in Low-Income Countries. World
Bank, Washington, DC.
31
- For children, the productive years are assumed to range from 13 to 71 (Indonesia’s life
expectancy at birth). This means a child’s productive life years will only start to count when a
child is 13 years old. It is also assumed the average year of a saved child is 2 years old;
- For pregnant women, is assumed that their average age is 20 years old and the maximum age
for being active in the labor force is 72.9 (which is the women life expectancy at birth in
Indonesia);
- The GNI per capita is used to value each productive life years gained. Indonesia GNI per
capita in 2013 was US$3,580;
- The project benefits in terms of productive life years gained are discounted with a 3%
discounting rate which is standard in economic evaluations.33
Table A3.1: Population health and demographic indicators
Population under five 16,380,000*
Number of pregnant women (national) 2,876,700**
GNI per capita (US$, 2013) 3,580.00
Life expectancy at birth (women) 72,9
Life expectancy at birth 71
Infant Mortality Ratio (per 1,000) 29
Maternal Mortality Ratio (per 100,000) 190
Population growth rate (2013) 1.2% SOURCES: World Bank, WDI. * Indonesia DHS, 2012 and Population Reference Bureau, 2013.
** Indonesia DHS, 2012.
10. Table A3.2 display the results of the analysis. The discounted total benefits of the project, estimated
in productive life years gained, is estimated in US$838.9 million which is significantly higher than the
total value of the project costs (US$77.82 million). The benefit-cost ratio is estimated in US$10.77,
this means that for each US$1 invested through the project there is an expected return of US$10.77,
only considering the period of five years after project closing. Although significantly high, this result
is based on rather conservative assumptions adopted and likely underestimates the total project
benefits. For example, health care costs (for health systems and households) saved due to reduced
morbidity and mortality are not taken into account, efficiency gains are not included, and the effect of
project interventions on health outcomes are considerably low (from 0.3% to 0.7% reduction in
mortality rates for infants and pregnant women) and, most important, the timeline adopted to measure
impacts is rather short. Finally, the analysis does not take into account the (individual and social)
impacts on the Indonesia labor market.
33 Drummond et al. (2005). Methods for the Economic Evaluation of Health Care Programmes. Paperback.
32
Table A3.2: Cost-Benefit Analysis Results
2015 2016 2017 2018 2019 Total
Child Health Benefit
Number of Children under five
16,380,000
16,576,560
16,775,479
16,976,784
17,180,506
Saved Children under five
1,425
1,923
2,432
2,954
3,488
12,222
Gained productive life-years per child under five
(Present Value) 20.26 19.66 19.08 18.52 17.98 19
Total Gained productive life-years (Present value)
28,869
37,808
46,419
54,711
62,693
230,501
Economics Gains relate to improved child health
(US$ million, Present value)
103.35
135.35
166.18
195.87
224.44
825.19
Maternal Health Benefit
Number of pregnant women 2,876,700
2,911,220
2,946,155
2,981,509
3,017,287
Saved women form maternal death 16.40 22.13 27.99 33.99 40.13 141
Gained productive life-years per saved women
(Present value) 27.9 27.1 26.3 25.6 24.8 26
Total Gained productive life-years (Present value) 458.3 600.2 737.0 868.7 995.5 3659.6
Economics Gains relate to improved maternal
health (US$ million, Present value)
1.64
2.15
2.64
3.11
3.56
13.10
Total Health Benefit
Total Gained productive life-years (Present value)
29,327.36
38,407.96
47,155.99
55,580.12
63,688.83
234,160
Economic gains related to improved child and
maternal health (US$, million, Present value)
104.99
137.50
168.82
198.98
228.01
838.29
Total Costs
Total Costs (Nominal, US$ million)
15.56
15.56
15.56
15.56
15.56
77.82
BENEFITS/COSTS RATIO 6.75 8.83 10.85 12.78 14.65 10.77
33
Annex 4. Bank Lending and Implementation Support/Supervision Processes
(a) Task Team members
Names
Title Unit Responsibility/Specialty
At Preparation
Puti Marzoeki
Claudia Rokx
Pandu Harimurti
Jed Friedman
Yogana Prasta
Susiana Iskandar
Ratna Kesuma
Jamil Salmi
Andreas Blom
Christopher Smith
Novira Asra
Imad Saleh
Paulus Bagus Tjahjanto
Andrew Daniel Sembel
Dayu Nirma Amurwanti
Susan Stout
Pierre Jean
Gordon Page
Rosalia Sciortino
Ratih Hardjono
Tetty Rachmawati
Khadrian Adrima
Estie Suryatna
Sr. Health Specialist
Lead Health Specialist
Health Specialist
Sr. Economist
Operations Advisor
Sr. Education Specialist
Operations Officer
Tertiary Education Coordinator
Sr. Economist
Consultant
Financial Management Specialist
Sr. Procurement Specialist
Procurement Specialist
Environmental Specialist
Operations Officer
Consultant
Consultant
Consultant
Consultant
Consultant
Consultant
Consultant
Team Assistant
GHNDR
GHNDR
GHNDR
DECPI
EACIF
GEDDR
GEDDR
GEDDR
GEDDR
GEDDR
GGODR
GGODR
GGODR
GENDR
GSURR
GHNDR
GHNDR
GHNDR
GHNDR
GHNDR
GHNDR
GHNDR
GHNDR
Task team leader
Health cluster leader
Medical education
Economic analysis
Operations advisor
Education governance
School grants
Peer reviewer
Peer reviewer
Peer reviewer
Financial management
Procurement
Procurement
Environment safeguard
Governance
Monitoring and evaluation
Accreditation
Examination
Stakeholder Analysis
Communication
Communication
Project costing
Team assistant
At implementation
Puti Marzoeki
Pandu Harimurti
Yogana Prasta
Siwage Dharma Negara
Novira Asra
Aswin Hidayat
Budi Permana
Angelia Budi Nurwihapsari
Dayu Nirma Amurwanti
Achmad Affandi Nasution
Isono Sadoko
Pierre Jean
Diane Brown
Gordon Page
Satrio S. Brodjonegoro
Shita Listyadewi
Sr. Health Specialist
Health Specialist
Operations Advisor
Sr. Education Specialist
Sr. Financial Management Specialist
Consultant
Procurement Specialist
Procurement Specialist
Operations Officer
Consultant
Consultant
Consultant
Consultant
Consultant
Consultant
Consultant
GHNDR
GHNDR
EACIF
GEDDR
GGODR
GGODR
GGODR
GGODR
EACIF
GSURR
GSURR
GHNDR
GHNDR
GHNDR
GHNDR
GHNDR
Task team leader
Task team leader
Operations advisor
Education governance
Financial management
Financial management
Procurement
Procurement
Governance
Governance
Social Safeguard
Accreditation (medicine)
Accreditation (nursing)
Examination
FAP
Implementation reporting
34
Christina Sukmawat Program Assistant GHNDR Program assistant
At Completion
Puti Marzoeki
Edson Correia Araujo
Diana Iuliana Pirjol
Victoriano Arias
Christina Sukmawati
Sr. Health Specialist
Sr. Economist
Consultant
Program Assistant
Program Assistant
GHNDR
GHNDR
GHNDR
GHNDR
GHNDR
Task team leader
ICR Primary Author
ICR team member
Program assistant
Program assistant
(b) Staff Time and Cost
Stage
Staff Time and Cost (Bank Budget Only)
No. of staff weeks US$ ‘000 (including travel and
consultant costs)
Lending
FY09 30.21 43.14
FY10 38.47 76.95
Total: 68.68 120.09
Supervision
FY11 28.04 69.15
FY12 24.99 60.42
FY13 23.5 56.65
FY14 28.30 76.58
FY15 20.63 62.65
Total: 125.46 325.45
35
Annex 5. Beneficiary Survey Results
Not applicable.
36
Annex 6. Stakeholder Workshop Report and Results
Not applicable.
37
Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR
1. HPEQ project was unique as overall project implementation was attached to and followed the
regulations of DGHE’s Working Unit (Satuan Kerja - SATKER). This model was intended to
facilitate early integration of the project to the programs into the SATKER for maintaining
sustainability of the programs after project completion. On the other hand, by integrating the project
into the SATKER, HPEQ was vulnerable to the dynamics and changes of government planning and
budgeting policies.
Challenges in Project Implementation
2. The achievement of PDO indicators/KPI targets in 2012 and 2013 was affected by the
implementation of new acts: the Higher Education act No.12/2012 and the Medical Education Act
No.20/2013. Enactment of these two acts was not predicted during project preparation and had
significant impact on project implementation, particularly on the timing and the extra effort required
to establish LAM-PTKES and LPUK-Nakes. As the result, the target of establishing the two bodies
could only be met towards the end of the project.
3. During project mid-term review, some KPI targets and intermediate outcome indicators were
revised to become more realistic. Measurement of the indicators were re-negotiated and agreed. The
changes were explained in the supplemental letter of revised performance indicators from the Bank
and was reflected in the revised Project Management Manual.
4. The implementing agency had to be agile in coping with the ambiguity and uncertainty of
government decisions and has adopted action through learning principles. In addressing the
challenge, the implementing agency has developed an effective monitoring and evaluation (M&E)
system by designing specific methods and instruments for each component and applying structured
data recording and reporting system for strengthening evidence based-internal quality assurance of
the project. For the FAP program, the M&E system was developed in collaboration with the Board
of Higher Education using best practices from previous school grant programs financed by the World
Bank, such as DUE, QUE, and I-MHERE. The formative approach through nurturing/mentoring as a
part of the M&E strategy was very useful for the School Project Implementing Units in managing the
FAP program. The implementing agency has also developed a risk mitigation framework to guide the
development of the M&E strategy for each project component.
5. The implementing agency and the School Project Implementing Unit has continued to experience
problems with the annual audits by the Finance and Development Supervisory Board – Ministry of
Finance (Badan Pengawasan Keuangan dan Pembangunan/BPKP). The problems were mainly related
to unstandardized evaluation by the auditors (especially regional auditors) in interpreting project
guidelines. The GoI recommends the World Bank to evaluate the BPKP audit mechanisms to ensure
they are standardized.
38
To what extent have the objectives been met / results been achieved
6. From GoI and stakeholders’ perspective, HPEQ project has hugely contributed to national
priorities, particularly by facilitating and accelerating the formulation of the Medical Education Act
No.20/2013 to guide the fundamental reform of medical education in Indonesia. This Act has
endorsed Ministry of Health and the Parliament to publish the Health Workforce Act No.36/2014 and
the Nursing Act No.38/2014.
7. LAM-PTKes and LPUK-Nakes were key actors in changing the health education paradigm by
integrating the education and health systems, facilitating stakeholders’ engagement, and inter-
professional collaboration. The values driven by LAM-PTKes and LPUK-Nakes were fundamental
for conflict management among the health professional society. Other disciplines can learn from the
lessons of the health professional society in establishing LAM-PTKes and LPUK-Nakes.
8. The implementing agency has taken advantage of the momentum and opportunity created by the
project for developing several innovative programs aimed for supporting the achievement of KPI
target and at the same time, adding values to HPEQ programs. The innovative programs were also
essential for developing the strategy for maintaining the sustainability of project outcomes. These
programs included: the teaching hospital program, the qualification framework for health disciplines,
the medical specialist study program, various policy formulation such as the joint decree on
competence examination and on public university hospital, intermediate regulations of Medical
Education Act No.20/2013, and the program for empowering students and young health professionals.
9. For ensuring effective publication of HPEQ outputs, the implementing agency has also created a
social marketing division with a task to conduct evidence-based studies to support the publication of
HPEQ program and outputs through official social media. This effort was also intended to stimulate
the government, the stakeholders and the society in general to notice health education as an important
issue.
10. The implementing agency with the approval of the government has an open access policy for the
use of HPEQ data/outputs for further scientific research as evidenced by the use of HPEQ data by
three masters and one doctorate program students.
What will be the follow-up to sustain results post project
11. Consistent commitment from the government, the health professional society, young generations
and the parliament is a key factor for maintaining the sustainability of HPEQ program.
Government Institutionalization Arrangements
39
12. The Directorate of Academic and Student Affairs has formulated the KPI for quality improvement
of higher education following some of HPEQ project’s KPI, particularly the number of accredited
study programs and the number of graduates passing the national competence examination.
Furthermore, the new structure of MORTHE will include a directorate for accreditation and
competence examination.
Institutionalization Arrangement for the Associations of Health Higher Education and
Professional Organizations
13. HPEQ project has provided much lesson in capacity building to strengthen both organizations.
The methodology for developing various standards has been adopted by the organizations involved
in HPEQ for other purposes. The methodology has also been adopted by other disciplines not only to
develop standards, but also to develop the accreditation instruments, and the blue print for competence
examination.
14. HPEQ project has also empowered the associations of health higher education and professional
organizations at the sub-national level. These organizations have been active sub-nationally to
disseminate updated policies and information regarding health education, especially those related to
accreditation and competence examination. The organizations have also become more independent
financially in providing technical support activities, and have improved capacity in formulating
financial planning and unit cost for future programs.
Recognition from External Stakeholders
15. The accreditation system developed by LAM-PTKes and competence examination system
developed by LPUK-Nakes have been endorsed by international experts, such as the Liaison
Committee on Medical Education (LCME) and the National Board of Medical Examiners (NBME).
In addition, during the project MTR World Bank international expert has stated that the national
examination system established for medicine was impressive and could be considered world class.
Such international recognition is crucial for maintaining the sustainability of LAM-PTKes and
LPUK-Nakes.
Empowering Young Health Professionals and Health Students
16. GoI and health professional societies acknowledged the important participation and contribution
of the young health professionals and health students as key stakeholders. Both groups have actively
and importantly contributed in shaping health professional education policy making, particularly the
implementation of inter-professional collaboration and inter-professional education, as important
values embraced by the LAM-PTKes and LPUK-Nakes. HPEQ Project supported student
participation through the development of (i) health student alliance (HPEQ Student), which has been
40
acknowledged as a model for the national program of student organization, and (ii) the Indonesian
Young Health Professional Society (IYHPS). At the end of the project, HPEQ Students has committed
to continue the program through the Indonesian Health Student Organizations’ Alliance (IHSOA),
while the IYHPS members have committed to develop a professional and independent organization
as a legal entity soon to be acknowledged by the Ministry of Justice and Human Rights. The
empowerment of IYHPS and HPEQ Students is a key strategy for maintaining HPEQ sustainability.
ICT System as an Asset for Sustainability
17. ICT as the backbone of project management has supported the implementing agency in conducting
paperless office management system and in taking advantage of tele-media to reduce meeting costs.
In its early implementation, HPEQ has also developed a blue print of the IT system as the information
management system platform for LAM-PTKes and LPUK-Nakes.
18. By the end of the project, the implementing agency has developed the Knowledge Management
System (KMS) regarded as an asset for the health professional community and government. KMS
HPEQ is a web-based knowledge portal containing HPEQ products, recent policies and related
references on health education. From project management perspective, KMS HPEQ is a form of
accountability through systematic documentation and publication. KMS will also accommodate
communication and opinion channeling among health professional community post HPEQ project. It
is a user friendly interface easily accessed through www.kmshpeq.net.
41
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders
Not applicable.
42
Annex 9. List of Supporting Documents
Component 1: Strengthening Policies and Procedures for School Accreditation
i. Higher Education Act No. UU No.12/2012
ii. Ministerial Decree No.49/2014 on National Standard for Higher Education (SN-Dikti)
iii. Ministerial Decree No.50/2014 on Quality Assurance System Of Higher Education (SPM-Dikti)
iv. Ministerial Decree No. No.87/2014 on Accreditation for Study Programs And Institutions
v. Ministerial Decree No.291/P/2014 on Ratification of LAM‐PTKes Establishment
vi. Ministerial Decree of Law and Human Rights No. AHU 30.AH.01.07/2014 on Ratification of
LAM-PTKes as a Legal Entity
Component 2: Certification of Graduates Using a National Competency-based Examination
i. Higher Education Act No. UU No.12/2012
ii. Joint Ministerial Decree MoEC and MoH on Competence Examination for DIII Midwifery, DIII
Nurse and Ners
iii. Ministerial Decree on National Committee for Competence Examination of DIII Midwifery, DIII
Nurse and Ners for year 2014
iv. Ministerial Decree No.36/2014 on Competence Examination for Medical and Dentistry Students
v. DGHE Decree No. 27/2014 on Committee for Competence Examination of Medical Students
vi. DGHE Decree No. 35/2014 on Committee for Competence Examination of Dentistry Students
43
MAP
Related Documents
