Sustainable use of products & tools in NTB province Survey results 2010 – 2011 Consolidation Programme Health / Policy Analysis & Formulation in the Health Sector (PAF) Dr. Lieve Goeman Dr. Rahmi Sofiarini Anwar Fachry Maddi Djara With contributions of Dr. Paul Rueckert, Dr. Harmein Harun and Karsten van der Oord
50
Embed
Sustainable use of products & tools in NTB province Survey ...Sustainable use of products & tools in NTB province Survey results 2010 – 2011 ... Sustainable use of products and tools
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Sustainable use of products & tools in
NTB province Survey results 2010 – 2011
Consolidation Programme Health / Policy Analysis & Formulation in the Health Sector (PAF)
Dr. Lieve GoemanDr. Rahmi SofiariniAnwar FachryMaddi Djara
With contributions of Dr. Paul Rueckert, Dr. Harmein Harun and Karsten van der Oord
Sustainable use of products and tools in NTB province
Sustainable use of products and tools in NTB provinceSurvey results 2010 – 2011
Consolidation Programme Health / Policy Analysis & Formulation in the Health Sector (PAF)
Dr. Lieve GoemanDr. Rahmi SofiariniAnwar FachryMaddi Djara
With contributions of Dr. Paul Rueckert, Dr. Harmein Harun and Karsten van der Oord
Download at: www.ighealth.org
The Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH was formed on 1 January 2011. It brings together the longstanding expertise of the Deutscher Entwicklungsdienst (DED) gGmbH (German development service), the Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH (German technical cooperation) and InWEnt – Capacity Building International, Germany.
For further information, go to www.giz.de
Sustainable use of products and tools in NTB province
Sustainable use of products and tools in NTB province
Contents
1. EXECUTIVE SUMMARY 5
2. INTRODUCTION 8
3. METHODOLOGY 9
3.1. CONCEPTUAL FRAMEWORK 9
Products 9
Health Facilities 9
Use of Products 10
3.2. SAMPLE 11
3.3. SURVEY TEAM 14
3.4. DATA SOURCES AND DATA COLLECTION INSTRUMENTS 14
3.5. DATA ENTRY AND MANAGEMENT 15
3.6. DATA ANALYSIS 15
3.7. FOLLOW-UP SURVEY (FUS) 16
4. RESULTS 17
Extent and quality of the use of the products (scores) 17
4.1. BREAKDOWN PER PRODUCT AND LEVEL OF HEALTH FACILITY 18
The products in detail 22
WISN 23
HRIS 24
HMIS 24
Referral System 25
IHPB 26
Integrated Monev 26
P/DHA 27
Desa Siaga 28
QI-Action 28
1
Sustainable use of products and tools in NTB province
4.2. BREAKDOWN PER DISTRICT & GEOGRAPHICAL DISTANCE 29
BETWEEN HEALTH CENTERS/VILLAGES & THE DHO
4.3. INTEGRATION OF THE PRODUCTS INTO PLANNING 32
AND BUDGET DOCUMENTS
4.4. INPUT FOR PLANNING, ACTION AND POLICY CHANGE 35
4.5. QUALITATIVE RESULTS 36
Success Factors 36
Failure factors 36
4.6. LIMITATIONS OF THE SURVEY 38
5. INTERPRETATION OF THE RESULTS 40
Too many Failure Factors 40
Fear for change 41
Complexity of sustainability 42
Looking for perfection 42
Minimal to no differences between BLS and FUS 43
6. LESSONS LEARNT 44
7. CONCLUSION 45
8. ABBREVIATIONS 46
2
Sustainable use of products and tools in NTB province
3
Sustainable use of products and tools in NTB province
4
Sustainable use of products and tools in NTB province
1. Executive Summary
BackgroundDuring 2006-2009, SISKES (Strengthening of District Health Systems) & HRD (Human Resources
Development) Projects supported the province Nusa Tenggara Barat (NTB) in strengthening district
health systems through improvement of health system management, health services management,
clinical services quality, human resource development and community empowerment. More than 200
products, tools, guidelines, information systems, manuals and films were developed to support the
achievement of the projects objectives.
In 2010 both projects were merged into 1 Consolidation Program Health: Policy Analysis and Formulation
(PAF) to consolidate and further strengthen the implementation and sustainable use of these products.
The program focused on policy efforts to create legal frameworks and the necessary laws and regulations
to institutionalize and mainstream the products. One of the indicators for measurement of the overall
objective that “Health policy guidelines and implementation regulations are better geared towards the
requirements of a decentralized health care system” is that “A representative number of health facilities
(at least 80%) in all 10 districts in NTB confirm that they implement or use the guidelines supported by
the project”.
MethodologyA baseline Survey (BLS) was conducted in May 2010 and a Follow up Survey (FUS) in April 2011 in
collaboration with the provincial health office (PHO) and a local research center in NTB. The nine most
important products in terms of investments and contribution to achieve the Maternal and Neonatal Health
strategy (AKINO) of NTB have been selected. Their use was assessed in 459 health facilities through
a cluster sampling method resulting in a representative sample for the province and this through 776
interviews with key respondents familiar with the projects and products. Their perceptions and answers
(subjective values) were cross checked with available proof and evidence (objective values). The “Use
of the products” was assessed at four “levels of use”: the availability and existence of the product in
the health facility; the implementation of the product and existence of all required conditions enabling
correct use; the incorporation of the use of the product into planning and budget documents; and policy
change or action obtained by the use of the product. SPSS v15 was used for the quantitative analysis.
5
Sustainable use of products and tools in NTB province
The qualitative analysis focused on factors that influenced the use of products, and on the changes and
actions that resulted from the use of the products.
ResultsBoth surveys were able to demonstrate the availability and presence of the nine tools in the health
facilities (BLS 52.9 % and FUS 55.6%). When the quality and the extent of use were brought into
account with a scoring system, the results dropped to 25.8 for BLS and 26.4 for FUS. No significant
differences were noticed between BLS and FUS. The products with the highest scores were Village Alert
system (Desa Siaga), Health Resource Information System (HRIS) and Integrated Health Budgeting
and Planning (IHPB). Use of the products was the highest at District Health Office (DHO) level. Central
Lombok and Sumbawa were the two districts with the highest scores.
The surveys identified multiple factors which influenced successful use. Even though the products
provided valuable data and information and staff perceived the benefits of using the products in their
daily work, the quality of using the products and the degree of incorporation into policy were limited. Main
obstacles, identified by key staff, were absence of legal frameworks to make the use of the products
obligatory, commitment of decision makers, frequent staff rotation without proper handing over and
insufficient budget allocation.
InterpretationSISKES and HRD projects allocated high amount of resources in terms of time, money, human resources
and expertise to the developments of products and created the perfect conditions for sustainable use of
the products. This resulted into very positive and impressive results of coverage and use at the end of
the projects. Despite all these efforts the target of the Consolidation Program PAF e.g. 80% of all health
facilities are using the products, has not been achieved. The presence of too many failure factors, fear
for change, the complexity of sustainability and the effect of development agencies with high amounts
of funding, staff and expertise during a short period of time were the underlying reasons to explain the
disappointing results of the surveys. The consolidation phase focused during the past 1.5 year on policy
advice to mainstream the products in the health system but it seemed too short to ensure impact on the
use of the products.
Lessons learnedAll influencing factors should be considered and addressed. All conditions and requirements for
successful use of products have to be fulfilled. Without perceived benefits a product will not be used.
A health system with weak institutional and management structures will not facilitate the full integration
6
Sustainable use of products and tools in NTB province
and use of a new product and cannot ensure its sustainability. Commitment from high level decision
makers is needed for successful use of products. A period of less than two years is too short to create
legal structures for the institutionalization of products. This has to be built in from the start of product
development and design together with ownership. Products developed during projects have many
dangers: they are often agency (donor) driven in the need for quick, tangible and visible results, they
may create duplication and add to the administrative burden and workload of staff. Newly designed
products have to be compatible with existing formats and systems. At the same time they have to be
easily changeable and adaptable to new requirements of decision makers or they will be abandoned.
7
Sustainable use of products and tools in NTB province
2. Introduction
During 2006-2009, SISKES & HRD Projects supported NTB province in strengthening district health
systems through improvement of health system management, health services management, clinical
services quality, human resource development and community empowerment. More than 200 products,
tools, guidelines, manuals and films were developed to support the activities, to bring in norms and
standards and to reach the objectives of these projects. In 2010 both projects were merged into 1
Consolidation Program Health: Policy Analysis and Formulation (PAF) to consolidate and further
strengthen the implementation and use of these products.
The consolidation phase had as objective that “Health policy guidelines and implementation regulations
are better geared towards the requirements of a decentralized health care system”. One of the indicators
for measurement of the progress and achievement of the objective was that “A representative number of
health facilities (at least 80%) in all 10 districts in NTB confirm that they implement or use the guidelines
supported by the project”.
Therefore a baseline Survey (BLS) was conducted in May 2010 and a Follow up Survey (FUS) in
April 2011. The nine most important products have been selected to investigate the sustainable
use of products after the end of the projects. The survey answered the following questions: Did the
consolidation phase contributed to sustainable use of the products by offering limited technical advice
to finalize the implementation and by focusing on policy advice to support institutionalization of these
products at provincial and national level (mainstreaming)? What were the factors that influenced the
use? Which of the selected products had the best results, which district and at which level of health
facility are products best used?
8
Sustainable use of products and tools in NTB province
3. Methodology
3.1. Conceptual Framework
To be able to answer the indicator and research questions a conceptual framework has been developed
that functioned as a “theoretical map” to ensure a systematic and comprehensive way in obtaining
objective unbiased results. This is presented in Table 1 below.
Products The nine most important products have been selected. Each of them supported the Maternal, Neonatal
and Child Health (MNCH) strategy of the province, namely “AKINO” or “No maternal death in the village”
and contributed to the improvement of the management of and quality of services in the health system.
• WorkloadIndicatorsStaffingNeed(WISN): a tool to calculate staffing needs based on workload
indicators
• HumanResourceInformationSystem(HRIS): a database based on MS Access which enables
HR Management and Planning
• HealthManagement InformationSystem (HMIS): A “one gate data flow” health management
information system
• Referral System:a technical guideline to establish the referral system, from referring patients
using referral and counter referral letters, to referral of specimen and transfer of knowledge
• Integrated Health Planning and Budgeting (IHPB): a guideline to develop and conduct the
process of integrated health planning and budgeting using existing data from HMIS
• IntegratedMonev: a guideline to conduct joint monitoring at district and province level, to be used
as input for the next planning cycle
• District and Provincial Health Account (P/DHA): public health expenditures at district and
provincial levels
• VillageAlertSystem(DesaSiaga): a comprehensive toolkit for raising community empowerment
and participation in reducing maternal and infant death in the village
• QIAction: an effort to improve quality of health services by focusing on identified problems at a
specific unit in the hospital
Health FacilitiesFive levels of health facilities, institutions and offices were identified: The Provincial Health Office
9
Sustainable use of products and tools in NTB province
(PHO), the provincial hospital, the District Health Office (DHO), the district hospital, the subdistrict
Health Centre (Puskesmas), the sub health Center (Pustu, Polindes) and the village. The application of
each of the nine products has been verified at the appropriate level. For example HRIS was only used
at hospital, PHO and DHO.
Use of ProductsThe “Use of the products” was composed out of four “levels of use” in a hierarchical way. For each
product, firstly the availability and existence of the product (level 1) in the health facility was verified
followed by the implementation of the product (level 2). Then the incorporation of the use of the product
into planning and budget documents (level 3) was checked and if the product had led to policy change
or program action (level 4). The degree or extent of achievement of a higher level of use could not
be higher than the previous level. The argument is as follows: A product could not have led to action
and change if it was not used completely and producing the necessary results required for change. A
product had to be present in the facility before it could be used.
Table 1. Overview Conceptual Framework
No Product User:level of health facility
Availability or pres-ence of product in the health facility (level 1)
Use and implementa-tion of the product(level 2)
Incorporation into Planning and budget documents(level 3)
Action or change induced by the product(level 4)
1 WISN PuskesmasHospital
Guideline and docu-ment with the WISN results are available.
Report and proposal on personnel issues eg. HR mobilization, is made, based on the WISN results
WISN is stated in the planning document of the health facility (RKA).
Documentation of change, decree, local laws.
2 HRIS Hospital DHOPHO
• HR Information is available.
• Hardware and soft-ware are present.
Databank is complete and functioning.Proposal on per-sonnel issues eg. Training and HR mobilization is made based on the data from HRIS. Virus scan and up-dates are done.
HRIS is stated in the planning doc of the health facility (RKA) and a budget is allocated for implementation and maintaining HRIS.
Documentation of change, decree, local laws.
3 HMIS PuskesmasDHOPHO
• Information data bank is established. Health facilities are part of a HMIS network.
• One gate system is established.
• Databank is complete and func-tioning.
• Monthly reports are produced.
HMIS is stated in the planning doc of the health facility (RKA) and a budget is allocated for implementation and maintaining HMIS
Documentation of change, decree, local laws.
4 Referral System Pustu, Polindes Puskesmas,Hospitals DHOPHO
• Guideline is avail-able.
• Referral and coun-ter referral letters at Hospital and Puskesmas can be shown.
Availability of records and reports on referral and counter referral of patients. DHO and PHO pro-mote the use.
Planning docu-ments refer to the implementation of the referral guideline, budget is allocated to use it and print the formats.
Documentation of change, decree, local laws
10
Sustainable use of products and tools in NTB province
No Product User:level of health facility
Availability or pres-ence of product in the health facility (level 1)
Use and implementa-tion of the product(level 2)
Incorporation into Planning and budget documents(level 3)
Action or change induced by the product(level 4)
5 IHPB Puskesmas DHOPHO
IHPB guideline is available. Planning document (RUK) at Puskesmas, and the Integrated work plans (Renja Terpadu) at district & province are available.
Meetings of IHBP process are taking place. The RUK of puskesmas is ac-comodated into the district Renja, the results of Renja is put into RKA of P/DHO.
Planning documents of P/DHO (RKA) men-tion the IHBP process as planning instru-ment and a budget is allocated for the implementation.
Documentation of change, decree, local laws.
6 Integrated Monev
DHOPHO
• Guideline is avail-able.
• Annual Monev report of P/DHO exists.
Monev is conducted yearly. Annual report is used for next plan-ning cycle. Achieve-ments of previous year are compared and analyzed. Results and recommendations are followed up.
Planning doc mention the M&E process and a budget is allocated.
Documentation of change, decree, local laws.
7 P/DHA DHOPHO
• Guidelines are available.
• Availability of yearly P/DHA data.
Data is complete and results are used for management (Plan-ning and Monev) and for advocacy.
Planning documents of P/DHO (RKA) men-tion the development of P/DHA and a budget is allocated for the implementa-tion.
Documentation of change, decree, local laws.
8 Desa Siaga Village PuskesmasDHO
The 5 alert networks are present at village level (notification system, community fund, Transportation system, Blood dona-tion system, Family planning Post).
Recordings and Minutes of Meetings (MoM) on the use of the alert networks (Especially Recording on maternal death at village/sub-village level).
Establishing and maintaining the func-tion of Desa Siaga is mentioned in plan-ning documents and budget is allocated for it.
Documentation of change, decree, local laws.
10 QI-A Hospital • QI team exists.• Application of QI
activities.
• Documentation of the results QI ac-tion. SK that justi-fies existence of QI team. MoM.
Establishing and maintaining of the function of a QI team is mentioned in plan-ning documents and budget is allocated for it.
Documentation of change, decree, local laws.
3.2. Sample
The survey sample was representative for the entire NTB Province. The unit of analysis was the “health
facility” which enabled to answer the indicator. All health facilities have been inventoried for the province
and per district.
See the table on the following page.
11
Sustainable use of products and tools in NTB province
Table 2. List of health facilities by type and number
No Provinsi/Ka-bupaten/kota
Types and number of health facilities by district Number by Village
Provincial Hospital
PHO District Hos-pital
DHO Puskesmas
1 Provinsi 1 1
2 Kota Mataram
0 1 8 50
3 Lombok Barat 1 1 15 88
4 Lombok Utara 0 1 5 33
5 Lombok Tengah 1 1 24 139
6 Lombok Timur 1 1 29 119
7 Sumbawa Barat
0 1 6 49
8 Sumbawa 1 1 21 164
9 Dompu 1 1 9 63
10 Bima 1 1 20 168
11 Kota Bima 0 1 5 38
Total 1 1 6 10 142 911
The “cluster sampling” technique was chosen for efficiency reasons. NTB was divided into four regions,
namely: Lombok-1, Lombok-2, Sumbawa and Bima. In each of the regions health facilities were selected
using an ad random sampling technique taking into account the stratification for urban and rural facilities.
The stratification for urban and rural health facilities, defined as “distance to the district health office” was
used to ensure fair representation in the sample as these subgroups had maybe significant differences
in results.
At provincial and district level, the public provincial hospital, all district hospitals, the PHO and all DHOs
were automatically included in the sample (100 %). There were 4 districts without district hospital:
Mataram City, North Lombok, West Sumbawa and Bima City.
At subdistrict and village level, a random sampling technique was used to sample Puskesmas and
villages after stratification for geographical distance from the district’s capital city. In the districts where
the number of Puskesmas was considered limited (e.g. North Lombok and West Sumbawa district only
had 5 Puskesmas) all Puskesmas were selected. At village level, three health facilities (Pustu/Polindes/
Poskesdes) were selected adjacent to the selected Puskesmas and based upon the distance to the
Puskesmas (near, middle, far). This resulted at sub-district level in the selection of 63 Puskesmas out
of the total of 142 Puskesmas (44 %) and at village level in a selection of 189 villages out of a total of
911 villages (21 %).
12
Sustainable use of products and tools in NTB province
The total sample for this survey consists of 459 health facilities (see Table 3).
Table 3. Sample of health facilities by type and number
No Provinsi/Kabupaten/
kota
Jenis dan jumlah fasilitas kesehatan
VillageProvincial Hospital
PHO District Hospital
Puskes-mas
Polindes PosyanduPoskesdes
1 Provinsi 1 1
2 Kota Mataram
1 5 15 15
3 Lombok Barat
1 6 18 18
4 Lombok Utara
1 5 15 15
5 Lombok Tengah
1 8 24 24
6 Lombok Timur
1 10 30 30
7 Sumbawa Barat
1 5 15 15
8 Sumbawa 1 7 21 21
9 Dompu 1 5 15 15
10 Bima 1 7 21 21
11 Kota Bima 1 5 15 15
Total 1 1 10 63 189 189
Grand Total n=459
In a second stage, the unit of analysis was the ”product” to enable the provision of specific information on
the use and implementation of each product and the reasons why it has been used or not. These results
led to the formulation of lessons learnt and recommendations. Not all health facilities applied all products.
Some products are only being used at provincial or at village level. Over the 459 health facilities together
the nine products have been assessed 776 times, as shown in Table 4 on the next page.
Confidence intervals were calculated for the main results. The range of the confidence interval was
defined by the sample statistic + margin of error and the uncertainty was denoted by the confidence
level (95%). The margin of error was the critical value times the standard error, which was the Standard
Deviation divided by the Square Root of the sample size.
13
Sustainable use of products and tools in NTB province
Table 4. Overview of the Number of surveys per Product and per type of Health facility
Health Facility
ProductsHospital PHO DHO Puskes-
masPolindes Village Total
WISN 7 63 70
HRIS 7 1 10 18
HMIS 1 10 63 74
Referral System 7 63 189 259
IHPB 1 10 63 74
MONEV 1 10 11
P/DHA 1 10 11
Desa Siaga 63 189 252
QI-A 7 7
Total 28 5 50 315 189 189 776
3.3. Survey Team
The Terms of reference (TOR) were developed by the GIZ team and served as base for the development
of a concept paper, a proposal of sampling, methodology and budget by a local research team linked
to an university in NTB (UNRAM). The same team was contracted for both surveys (BLS and FUS) and
consisted out of one principal researcher, four researchers for the district and provincial health facilities
and a team of 20 enumerators for the health centers and villages. Four groups were formed, one for
each region in NTB. Each group had one researcher acting as supervisor and enumerator for the district
and provincial hospital, DHO and PHO. Each group had as well one enumerator functioning as field
coordinator and responsible for data entry. A training was held before each survey to familiarize them
with and refresh their knowledge of the nine products.
3.4. Data Sources and Data Collection Instruments
The surveys were conducted at facility level. The data was obtained from interviews with key informants
in the facilities. Their perception and opinion served as primary and subjective data sources. Only key
informants were interviewed to ensure that as much information as possible about the products was
retrieved. No feedback from patients or other staff members was asked. Their awareness, knowledge
and perception of the products have not been assessed. Calls and revisits were made up to 3 times
in case key informants were absent at the time of the survey. The observations by the survey team in
14
Sustainable use of products and tools in NTB province
the health facility and from documents such as the annual provincial and district planning and budget
documents served as secondary and objective data sources and were considered as evidence to
confirm the interviews.
Questionnaires for each product for a specific facility were developed as the use of each product
varied according to the health facility level (implementation versus coordination and supervision). The
questionnaires consisted of closed and open questions, respectively for quantitative and qualitative
measurement. For the closed questions multiple options reflecting the degree or scale of achievement
were created. The open questions served to document concrete examples, failure and success factors
for the use of the nine products and for feedback to GIZ.
3.5. Data Entry and Management
A data entry software program was developed, using CS-Pro v4.03, allowing the field coordinators to
enter the data in the field during the survey after checking the completeness of the questionnaires. Data
cleaning and validation was done after completion of data entry. This data was archived with a code
book.
3.6. Data Analysis
For the quantitative data analysis SPSS v15 was used. As the survey used a single-faceted framework,
simple statistical methods such as univariate/bivariate analysis were applied and single and composite
index were computed.
The results were presented in the form of percentages (proportions) and scores. For each result a
subjective value (based on the answers of the respondents) and an objective value (based on observation
and evidence) was calculated. The results were disaggregated for each product, district, level of health
facility and for the BLS for urban and rural facilities. No weighting factors were used.
The results were presented in percentages to answer the indicator “A representative number of health
facilities (at least 80%) in all 10 districts in NTB confirm that they implement or use the guidelines
supported by the project”. These results were based on the existence of the product in the health
facility and reflected “yes or no” questions but they did not bring the extent and quality of the use of the
products into account.
To consider the extent and quality of the use of the products a scoring system has been developed.
15
Sustainable use of products and tools in NTB province
The score for a product has been based on the four levels of use and the scales/degree of answers for
each of the closed questions (3 to 4 gradations possible). A maximum score of 100 meant that a product
was fully implemented as the guidelines and standards of the projects require, that all conditions were
fulfilled, that all levels of use were fully accomplished (including integration into planning and budgeting
documents and leading to policy change). A score of 100 has been equal to the ideal perfect situation.
All scores together of all nine products defined the overall score.
For the qualitative analysis, factors that influenced the implementation of a products were collected
and grouped under “success factors” if they had led to successful use and under “failure factors” if
they prevented the implementation of the products. An overview of the different types of influencing
factors has been computed and concrete examples for each of the tools have been collected as it was
part of each interview. Concrete examples of change and influence on existing policies as given by the
respondents and observed by the survey team were documented. The FUS focused more in depth in
collecting these qualitative data in comparison with BLS.
3.7. Follow-up Survey (FUS)
The same survey team was hired to conduct FUS. Teams were mixed so that enumerators surveyed other
health facilities than during BLS. A refreshment training and planning session took place beforehand.
The survey sample contained the same facilities and if possible the same respondents. The data
collection instruments were adapted to measure the period after May 2010. More focus was put on the
qualitative part to collect concrete examples; on the “why?” of use or no use of products and the change
the products might had provoked. Technical advisors from the GIZ PAF team accompanied the survey
team during the first 3 days to serve as resource person and to ensure the survey team was obtaining
the required information. The data entry, management and analysis were done in the same way as BLS.
Results of FUS have been compared with the results of the BLS to measure differences which might
had occurred over time. The statistical tests used for this were t-Test for the comparison of means and
the Chi-square test for nominal variables (categorical data) for the comparison of proportions.
16
Sustainable use of products and tools in NTB province
4. Results
Availability of the products in the health facilities (percentages)
The indicator of the PAF project sets a target that “At least 80 % of health facilities in all 10 districts in
NTB confirm that they implement and use the guidelines supported by the project and this by the end
of the project”.
BLS
The findings of the BLS showed with a 95% confidence that 55.8 ± 3.6 % of all 459 health facilities
confirmed to implement and use the products (subjective value). Evidence that confirmed the “yes/no
questions” on the presence of the product in the health facility led to 52.9 ± 3.4 % of all health facilities
(objective value).
FUS
The findings of the FUS showed with a 95% confidence that 63.4 ± 3.7% or 291 out of all 459 health
facilities confirmed to implement and use the products (subjective value). Evidence that confirmed the
“yes/no questions” on the presence of the products in the health facility led to 255 health facilities or
55.6% ± 3.4% of all health facilities which verifiably used the products (objective value).
Comparison BLS-FUS
Based on the respondents perception (subjective values) there was a significant increase from 55.8 to
63.4 % (p=0.003) but this could not be confirmed with evidence gathered by the survey team. Based
on the objective values, there was no significant difference between the BLS and FUS for the use the
products by the health facilities (p=0.015).
Extent and quality of the use of the products (scores)
To reflect the degree and quality of the use of the products a scoring system has been developed based
on the four levels of use: the availability of the product in the health facility; the implementation and the
fulfilling of all required conditions enabling perfect use of the product; the integration of the products
into planning and budgeting documents; and the influence of the products on planning processes and
policies which might have led to change or action. The scores gave detailed information on each of the
products, the health facility level, the districts and the difference between them.
Note: A score of 100 meant that all products were perfectly implemented and used, planned and
17
Sustainable use of products and tools in NTB province
budgeted for, and that they produced information and results which led to change or action.
BLS
The overall score for all products together was 34.6 ± 2.6 subjectively (respondents perception) versus
25.8 ± 2.5 objectively (confirmed by evidence) and this with a 95% confidence.
FUS
The overall score for all products together was 34.1 ± 2.7 subjectively versus 26.4 ± 2.4 objectively and
this with a 95% confidence.
Comparison BLS-FUS
There are no significant differences between BLS and FUS for the subjective (t-Test=0,453 and p=0,684)
and objective (t-Test=0,510 and p=0,613) scores.
4.1. Breakdown per Product and Level of Health Facility
Health facilities have different functions, scopes of work, responsibilities and authority at different levels.
Since this is also the case in regards to the products, the health facilities should not be generalized or
treated homogeneously. Therefore a breakdown of the results has been given per level of health facility.
BLS
The results of the BLS in terms of availability and presence of the products in the health facilities are
presented in Table 5 on the following page.
PHO
The PHO used all five products namely HRIS, HMIS, IHPB, Integrated Monev and PHA (100 %).
DHO
All DHOs used HRIS and IHPB (100 %), 8 out of 10 DHOs implemented HMIS and DHA (80 %), and
none of the DHO implemented integrated Monev (0 %).
Hospitals
All Hospitals confirmed to use HRIS (100 %), 86 % implement QI-Action, 57 % use WISN, and none
used the Referral System.
18
Sustainable use of products and tools in NTB province
Health Center
All Puskesmas claimed to implement Desa Siaga (100 %) but it could only be confirmed in 84 %
(objective value) of the Puskesmas. A similar finding emerged for IHPB, where 73 % of the Puskesmas
respondents confirmed the use but the evidence was only in 44 % (objectively) of the Puskesmas
available. 65 % of the Puskesmas implemented HMIS versus 62 % objectively, 8 % used WISN, and
none of the Puskesmas implemented the Referral System.
Village
At village level, the result showed that none of Polindes/Pustu implemented the Referral System
Guideline and that up to 91% of all villages had evidence for the use of Desa Siaga.
Taking the quality and degree of use into account at each level of health facility, the use of the products,
expressed in scores, was the highest at DHO level with 35 objectively followed by PHO with a score of
32 objectively. The scores for implementation at hospital and village level were the lowest. Table 6 on
the following page shows all details.
Products PHO DHO Hospital Puskesmas Village %
Polindes/Pustu
Village Govern-ment
WISN S 57 8 13
O 57 8 13
HRIS S 100 100 100 100
O 100 100 100 100
HMIS S 100 80 65 68
O 100 80 62 65
Referral System
S 0 0 0 27
O 0 0 0 0
IHPB S 100 100 73 77
O 100 100 44 53
MONEV S 100 0 9
O 100 0 9
P/DHA S 100 80 82
O 100 80 82
Desa Siaga S 100 93 94
O 84 91 90
QI - A S 86 86
O 86 86
Table 5. Percentage of Health Facilities using the products (BLS)
19
Sustainable use of products and tools in NTB province
Table 6. Overview of the Scores for each Product per level of Health Facility (BLS)
No. Health Facility WISN HRIS HMIS Referral System
IHPB MONEV P/DHA DesaSiaga
QI-A TOTAL
SUBJECTIVE SCORES (Respondent’s perception)
1 PHO 56 30 67 33 17 40
2 Hospital 11 51 0 46 27
3 DHO 52 41 75 0 50 43
4 Puskesmas 3 34 0 38 68 29
5 Village 0 54 27
TOTAL 7 53 35 0 60 17 33 51 46 34,6
OBJECTIVE SCORES (evidence based)
1 PHO 50 15 44 33 17 32
2 Hospital 10 35 0 29 18
3 DHO 41 39 64 0 32 35
4 Puskesmas 1 27 0 26 44 20
5 Village 0 44 22
TOTAL 5 42 27 0 45 17 24 44 29 25.8
Products PHO DHO Hospital Puskesmas Village %
Polindes/Pustu
Village Govern-ment
WISN S 43 27 29
O 43 22 24
HRIS S 100 100 100 100
O 100 100 100 100
HMIS S 100 80 79 81
O 100 80 75 76
Referral System
S 14 8 13 12
O 0 8 3 0
IHPB S 100 100 68 65
O 100 70 52 55
MONEV S 100 70 64
O 100 60 91
P/DHA S 100 90 91
O 100 90 91
Desa Siaga S 97 99 99
O 57 97 87
QI - A S 71 71
O 71 71
Table 7. Overview of the Scores for each Product per level of Health Facility (BLS)
20
Sustainable use of products and tools in NTB province
FUS
The results of the FUS in terms of availability and presence of the products in the health facilities are
presented in Table 7.
PHO
All five products were used at PHO, namely HRIS, HMIS, IHPB, Integrated Monev and PHA (100 %).
DHO
All DHO used HRIS (100%), 90% used P/DHA, 80 % HMIS, 70% IHPB and 60 % applied Integrated
Monev.
Hospital
All hospitals used HRIS, 71% of the hospitals implemented Quality Improvement Actions and 43% used
WISN. None of the hospitals used the Referral System.
Health Center
The use of products ranged between 75% for HMIS (47 out of 63) and 8% for the use of the Referral
System.
Village
Three % of the surveyed polindes implemented the Referral System. 97% of all villages used Desa
Siaga.
Taking into account the quality and degree of the use of the products, table 8 on the following page gives
a detailed overview of the scores of each tool at each level of health facility. The score was the highest
for DHO with 36 objectively followed by the score for Health Center with 26 objectively.
21
Sustainable use of products and tools in NTB province
No. Health Facility WISN HRIS HMIS Referral System
IHPB MONEV P/DHA DesaSiaga
QI-A TOTAL
SUBJECTIVE SCORES (Respondent’s perception)
1 PHO 56 15 0 44 17 26
2 Hospital 5 66 5 45 30
3 DHO 53 44 50 34 46 45
4 Puskesmas 17 37 7 38 62 32
5 Village 4 50 27
TOTAL 11 58 32 5 29 39 31 56 45 34.1
OBJECTIVE SCORES (evidence based)
1 PHO 40 15 0 33 17 21
2 Hospital 5 37 2 36 20
3 DHO 46 41 45 6.6 42 36
4 Puskesmas 13 35 6 26 48 26
5 Village 1 44 23
TOTAL 9 41 30 3 24 20 29 46 36 26.4
Table 8. Overview of the Scores for each Product per level of Health Facility (FUS)
Comparison BLS-FUS
Table 9, on the following page, shows that PHO, DHO and villages did not have significant differences
in scores between the BLS and FUS. Their overall score for the use of all products remained stable
between both surveys. But, health centers scored better during the FUS: The overall score for all products
together increased here significantly from 20 to 26 objectively (p=0.007). However, for hospitals the
score decreased significantly during the FUS from 32 to 20 (p=0.006).
The products in detail
Figure 1 shows the scores for each of the products as observed during the BLS and FUS.
BLS
IHPB, Desa Siaga and HRIS were the products with the highest scores, their objective scores were
above 40. They were followed by QI Action, HMIS and P/DHA with objective scores between 20 and 40.
Monev and WISN had very low scores with objective scores below 20. Referral System had a score of
zero.
FUS
Desa Siaga and HRIS continued to be the products with the highest score, their objective scores were
above 40. They were followed by QI-A, HMIS, P/DHA, IHBP and Monev with objective scores between
20 and 40. The lowest scores were for WISN and Referral System and were below 10.
22
Sustainable use of products and tools in NTB province
0
10
20
30
40
50
60
Table 9: Scores for each Product for BLS and FUS (n=776)
BLS SBLS OFUS SFUS O
WISN HRIS HMIS Referral IHPB MONEV P/DHA Desa Siaga QI-A System
WISN
Questions to calculate the score of WISN included the application of the WISN process and tool in
hospitals and Puskesmas, the production of a report with the results, the use of the report to inform
decision makers, proof of change and action due to WISN. The findings of the BLS showed that the score
for implementation of WISN was 7 subjectively versus 5 objectively. This means that WISN was poorly
applied and in only a few facilities. Based on the feedback from the users, this poor score was mainly
attributable to the lack of knowledge in applying this product as stated by 20 out of 63 respondents in the
Puskesmas. Other constraints were the lack of socialization and training, no supporting budget and staff
rotation. As decisions on deployment of health staff at Puskesmas were taken at a higher level (DHO
en local goverment), the Puskesmas and hospitals did not see the need to apply WISN as the results
produced at their level wouldn’t be taken into consideration by the decision makers. These constraints
were given by up to 17 respondents. In addition, the lack of a binding policy or regulation to implement
WISN (no official endorsement or obligation) prevented staff of computing WISN. Therefore, staff simply
continued to use the existing model based on the use of ratio’s to calculate health human resources
needs and for planning purposes instead of applying WISN.
23
Sustainable use of products and tools in NTB province
The findings of the FUS showed that the score for implementation of WISN increased to 11 subjectively
versus 9 objectively. WISN was being implemented in 3 out of the 7 hospitals and in 14 out of the 63
Health Centers. In comparison with the BLS this meant a significant increase (p=0.018, see Table 9).
The same reasons were given to explain the low use of WISN.
HRIS
The use of HRIS covered questions on whether HRIS runned properly and data were available
(including software, computer and database), whether databases were maintained and updated
(including anti-virus scans) and the extent to which reports were produced and information was used
for staff development and decision makers. The score for the use of HRIS was 53 subjectively versus
42 objectively for the BLS. The scores were higher at PHO and DHO in comparison with the hospital.
The perceived benefits of HRIS were the easy access to and management of HR information. The main
constraints for the optimal use of this product were related to the incompleteness of individual data
(staff was not obliged to submit data) and the lack of institutionalization of HRIS. The existing system, a
manual excel based system, was still used in parallel and was the preferred system when reports had
to be submitted.
The findings of the FUS showed a score for implementation of HRIS of 58 subjectively versus 41
objectively. Even though HRIS was being implemented at PHO, in all DHO and in all hospitals, the
quality of use remained of the same level as the BLS (p=0.692, see Table 9). The existence of parallel
manual excel based system was noticed during the FUS as well. Problems in the software to produce
valid reports for the management and decision makers inhibited the replacement of the old system by
the new HRIS system. Even though the staff of the Provincial hospital clearly experienced the benefit
of the new system, they could not abandon the old system and confirmed the existence of above
mentioned problems.
HMIS
The perfect use of HMIS included the availability of hardware and software, a functioning HMIS
team (SIKDA team), a LAN/data network, a complete regular updated database, monthly reports, an
established one gate system etc. The score for the implementation of HMIS was 35 subjectively versus
27 objectively during the BLS. HMIS scored the best at DHO level followed by health center level. The
score for HMIS at PHO level was the lowest. The main identified constraints were the absence of an
established SIKDA team, problems with the LAN network, a limited number of computers (stated by
18 respondents), malfunctioning software, viruses, lack of incentives for data entry staff and lack of
stakeholders’ commitment. However, HMIS was widely acknowledged to bring benefits and efficiency
in the daily work: 41 respondents mentioned the improvement of the data quality and completeness,
24
Sustainable use of products and tools in NTB province
the increase of knowledge, the possibility of having quickly data and reports available. 17 respondents
stated that HMIS has led to more effective and efficient services and 10 respondents answered that
planning did become easier, more accurate and punctual.
The findings of the FUS showed that the score for implementation of HMIS was 32 subjectively versus
30 objectively. This was a small but significant increase in comparison with the findings of the BLS
(0.018, see Table 9). DHO and PHO respondents summarized the reasons for the low score as follows:
the absence of basic conditions e.g. a establishment of a specific HMIS unit in the DHO, lack of anti-
virus programs in the computers, electricity interruption, unfinished software (6 respondents) and the
mismatch between DHO and PHO software (5 respondents). The PHO respondent also mentioned
the additional workload to use the HMIS system. 110 respondents in health centers gave the following
explanations of why the use of HMIS was hampered: lack of specific HMIS operators (29 respondents),
frequent electricity cuts (18 respondents), lack of sufficient budget (18 respondents) and lack of sufficient
and functioning hardware (29 respondents). Positive feedback and comments collected in the health
centers were: HMIS was easy and very practical to use (33 respondents); HMIS produced information
in a very easy and efficient way (28 respondents); HMIS accelerated the delivery of health services (21
respondents); HMIS improved the performance and evaluation activities of programs (12 respondents).
Four health centers claimed they had no problem at all to use HMIS.
Referral System
This product was piloted in West Lombok during 2009. Commitment was obtained by PHO and DHO’s
to implement the guidelines and formats from 2010 onwards. BLS could not retrace the existence of the
product (availability of the new guideline, the formats etc). Only the old government referral system was
used in the health facilities, not the project developed system. Therefore a score of zero was given for
BLS. Reasons for this score were the late start of the activities under the former SISKES project (lack
of time) with lack of commitment and socialization of the product. The question could be raised whether
the new referral system was perceived as being significant different and better than the existing referral
system.
Activities during the Consolidation Program (PAF) to review the Referral guideline and dissemination
of the formats led to slightly better results during FUS. The score for implementation of referral System
was no longer zero but increased to 5 subjectively versus 3 objectively (p=0.006, see Table 9). This low
score was due to the limited number of health facilities using the new referral system: Only 1 out of the
7 hospitals, 5 out of 63 health centers and 6 Polindes out of 189. The quality and degree of the use was
still very poor as different health facility levels were not linked to each other and this condition needed to
be fulfilled to obtain a high score. Hospitals stated that using the new system was not obligatory yet and
25
Sustainable use of products and tools in NTB province
the lack of support was perceived as an obstacle for successful use. The referral/back referral letters
were found at the provincial hospital covered under dust in a corner of an administrative unit. Positive
feedback from the health centers on the new referral system was received from 12 respondents and
described as no problem in applying the new system, easy data recording, practical formats to be filled
in. 14 respondents gave explanations on why referral scored so low and named the refusal of patients
to be referred, not being used to ask for back referral letters, the administration burden of filling in
the referral letters, and the lack of printed formats. At villages level benefits linked to the new referral
systems were counted 47 times in terms of leading to better services (29 times), a tidier administration,
and a more comprehensive system. Obstacles for a functioning system were transportation problems
(10 respondents), the low awareness of the community to ask for/bring the letters and to be referred (15
respondents), the costs of referral and the administrative burden.
IHPB
Implementation of IHPB was verified in terms of availability of IHPB guidelines, their use in the planning
process in 2007-2009 (BLS) and in 2010-2011 (FUS), and the extent to which the Puskesmas and
DHO’s planning was accommodated into respectively the DHO and PHO’s planning. Implementation of
IHPB scored 60 subjectively versus 45 objectively during the BLS. IHBP had the highest scores at DHO
level followed by PHO level. The scores were the lowest at Puskesmas level. The benefits of IHPB in
terms of more transparency in planning, less overlap and more synergy, better coordination and easier
planning development were confirmed 87 times.
The main constraint for the use of IHBP at Puskesmas level was the conflict IHBP created being a
bottom-up process while in reality planning and budgeting processes still take place top-down. This was
stated by 20 respondents. Other reported constraints included lack of understanding and facilitation of
the process, and limited resources and budget to support the implementation.
The findings of FUS showed scores dropping significantly from 60 to 30 subjectively and from 45 to
24 objectively (p=0.001, see Table 9). This was the only product with a significant decrease in score in
comparison with the BLS. The main reason for this score was the obligatory use of P2KT or the MoH
version for Integrated Health Planning and budgeting which was different from the project developed
IHBP.
Integrated Monev
The assessment of the use of Monev at PHO and DHO included questions on the conduct of Integrated
Monev in 2008, the follow up of the recommendations in 2009 for BLS and in 2010 for FUS, and the
extent to which the results were used as input in planning processes. The implementation of Monev
scored 17 subjectively and objectively during BLS. While Monev for each program ran regularly, the
conduct of Integrated Monev was constrained by the non compliance to the schedule by the different
26
Sustainable use of products and tools in NTB province
programs. The score at DHO level was zero as it was not conducted at district level. The absence of
budget for supporting Integrated Monev was predominantly stated by DHO’s respondents to explain
why. The benefits of Integrated Monev were identified as bringing improving effective and efficient
program implementation, identifying DHO performance more comprehensively and offering a tool to
find solutions for expressed problems. But these benefits were not convincing enough for the DHOs to
conduct Integrated Monev. The findings of FUS showed that the implementation of Integrated Monev
scored 39 subjectively and 20 objectively but without significant difference with the BLS results (p=0.180,
see Table 9). Since BLS, Integrated Monev was conducted once and the perceived benefits, stated by 5
DHO, were that solutions and feedback were given right away during the Integrated Monev mission and
that the result based approach was very useful. The PHO repeated how difficult it was to coordinate a
joint mission which matched the agenda of each program.
P/DHA
The implementation of P/DHA included questions on the availability of P/DHA for 2007-2008 (BLS)
and 2009 (FUS), the completeness of data, and the use of the data as input for the IHPB process and
advocacy. The score for P/DHA was 33 subjectively versus 24 objectively during BLS. P/DHA were
elaborated for 2007 and 2008 but not for 2009. The scores at DHO were higher than at PHO level.
Reasons for the low scores were the unavailability and incompleteness of data, the mismatch between
available data and the templates as reformatting and adjustments, required before data entry, were not
done. Lastly, producing P/DHA was not compulsory so in absence of a request, facilitation and technical
backstopping by the project, the production of P/DHA reports did not continue. Budget constraints to
fund all the work meetings were mentioned as well at DHO level which pointed out the dependence of
development agency’s support once more. Perceived benefits of conducting P/DHA were the increased
understanding of fund sources and budget allocation, and the increased availability of information that
could be used for advocacy and evaluation.
FUS findings showed scores for P/DHA of 31 subjectively versus 29 objectively which were slightly
better than the scores of BLS (p=0.003, see Table 9). PHO and 9 out of 10 DHO applied P/DHA but
the quality of implementation was weak. PHO stated that there was a difference between the tool
promoted by the project and perceived as rather complicated, and the tool promoted by the University
of Indonesia and more widely used. Feedback of the DHO was the incompleteness of required data (7
respondents) and complexity of the tool (4 respondents). As positive feedback DHO stated that DHA
produced data which can be used for advocacy to make budget allocation fairer according to priorities
(11 respondents).
27
Sustainable use of products and tools in NTB province
Desa Siaga
The score for the implementation of Desa Siaga referred to the availability (establishment) of the
alert network systems in the village and to the extent these functions were maintained. Desa Siaga
implementation scored 61 subjectively and 44 objectively for BLS. The success of Desa Siaga was due
to the perceived benefits by the villagers: increased solidarity and awareness in the community, better
access to information and quicker service delivery when needed. The constraints identified for the
implementation were the lack of coordination and facilitation by DHO and the departure and drop-out
of village facilitators.
FUS revealed for Desa Siaga implementation scores of 56 subjectively and 46 objectively. The increase
of the objective scores was significant (p=0.028, see Table 9). Perceived benefits and reasons why
Desa Siaga was working well were formulated by 92 respondents in terms of better services for the
community, increased community awareness on health problems and improved health status of pregnant
women and children. Feedback from the villages why Desa Siaga was not working well included the
lack of supporting budget and support of health facilities (21 respondents), lack of coordination (8
respondents); lack of socialization of the concept and unawareness of the community members (18
respondents). 18 respondents stated that Desa Siaga was not running in their village.
QI-Action
The QI-Action covered questions on the occurrence of QI-Activities in hospital units, the existence of a
functioning QI team, proof of change, and the existence of a plan and budget to implement QI-Action
in 2010 for BLS and in 2011 for FUS. The findings of BLS showed a subjective score of 46 versus 29
objectively. This difference is due to the absence of real evidence. Budget constraints and staff rotation
were stated by the respondents in the hospital as main reason for the low score. The findings of the FUS
showed that QI-Action was implemented in 5 out of 7 hospitals and that the score for implementation
of QI-Action was 45 subjectively versus 36 objectively. There was no significant difference with the BLS
(p=0.289, see Table 9). The same reasons as during BLS were stated as reason for the low use. In
addition the lack of commitment of decision makers and the missing legal framework to make QI-Action
obligatory in the hospitals were mentioned by the hospital staff.
Comparison BLS-FUS
Table 9 shows the differences between the BLS and FUS. For the overall score there was no significant
difference between the results of the BLS and FUS. However, when breaking down the results per
product, significant differences were found: Four products scored better, namely WISN, Referral System,
P/DHA and Desa Siaga but with minimal improvements of scores. Only one product scored significantly
worse. The score of IHPB dropped to half the value in the period between the BLS and FUS. The reason
28
Sustainable use of products and tools in NTB province
given was the mandatory use of the P2KT guideline. No difference in status for three products, namely
HRIS, Integrated Monev and QI-Action.
Table 9. Comparison scores BLS-FUS for each product and level of health facility.
HealthFacility
WISN HRIS HMIS ReferralSystem
IHPB MONEV P/DHA DesaSiaga
QI-A Total t-Testp-value
Objective Scores (evidence based) – BLS versus FUS
PHO BLS 35 15 44 33 17 18 t=1.697 p=0.103
FUS 40 15 0 33 17 21
Hospital BLS 10 50 0 29 32 t=2.950p=0.006
FUS 45 37 2 36 20
DHO BLS 41 39 64 0 32 35 t=0.95p=0.349
FUS 46 41 45 7 42 36
HealthCenter
BLS 1 27 0 26 44 20 t=2.885 p=0.007
FUS 13 35 6 26 48 26
Village BLS 0 44 22 t=1.238p=0.232
FUS 1 44 23
TOTAL BLS 5 42 27 0 49 17 24 44 29 25.826.4
FUS 9 41 30 3 24 20 29 46 36
t-Testv-value
t=2.556p=0.018
t=0.404 p=0.692
t=2.556p=0.018
t=4.812p=0.006
t=4.483p=0.001
t=1.431p=0.180
t=5.477p=0.003
t=3.070p=0.028
t=1.162p=0.289
t=0.510p=0.613
4.2. Breakdown per District and Geographical Distance between health centers/villages and the DHO
Use of the products per district
NTB province has 10 districts and all districts were equally supported in the use of all nine products
except for the Referral System. This was only piloted in West-Lombok district during SISKES project
(See section 4.1).
BLS
Table 10 shows the scores, objectively and subjectively, for each of the districts and for each of the
products. The scores for the products at provincial level were not included in the calculation of this
breakdown. Four districts did not have a district hospital so QI-A could not be assessed there. North
Lombok was a new district since 2009 so P/DHA could not be assessed as no budget data was available
for 2008.
29
Sustainable use of products and tools in NTB province
No Districts WISN HRIS HMIS Referral System
IHPB Monev P/ DHA Desa Siaga
QI-Action Total
Subjective Scores
1 West Lombok 15 56 28 0 57 0 42 70 50 37
2 North Lombok 0 50 43 0 69 0 42 37
3 Central Lombok 17 53 54 0 66 0 58 62 50 40
4 East Lombok 10 56 43 0 63 0 58 59 50 38
5 West Sumbawa 0 67 41 0 64 0 58 74 38
6 Sumbawa 0 59 38 0 71 0 58 74 50 39
7 Dompu 3 56 67 0 28 0 58 46 25 31
8 Bima 0 50 42 0 36 0 58 72 50 34
9 Bima City 50 0 0 42 0 25 77 24
10 Mataram City 67 9 0 52 0 17 25 21
Total 4 56 37 0 55 0 48 61 28 32
Objective Scores
1 West Lombok 13 44 28 0 53 0 25 47 50 29
2 North Lombok 0 37 31 0 58 0 25 22
3 Central Lombok 8 53 51 0 62 0 50 57 13 33
4 East Lombok 9 40 40 0 49 0 50 57 13 29
5 West Sumbawa 0 48 36 0 64 0 50 58 32
6 Sumbawa 0 56 38 0 62 0 25 63 50 33
7 Dompu 0 51 54 0 28 0 50 17 25 25
8 Bima 0 35 35 0 25 0 50 36 25 23
9 Bima City 0 15 0 0 11 0 0 31 7
10 Mataram City 0 56 7 0 28 0 17 19 16
Total 3 43 32 0 44 0 32 44 18 25
The objective and subjective scores had the same ranking for the districts. The differences between the
objective and subjective scores were minimal except for North Lombok where the subjective score of
37 was much higher than the confirmed objective score of 22. The best scoring districts were Central
Lombok and Sumbawa with an overall objective score of 33, followed by West Sumbawa, West Lombok
and East Lombok. The districts with the lowest score were Mataram City and Bima city with respectively
16 and 7 objectively. All products had low scores in these two districts except for HRIS in Mataram
City and Desa Siaga in Bima City. The survey could not give reasons for this difference between “city
districts” and “larger districts” as further Focus Group Discussions and in depth interviews would be
needed and this was beyond the scope of this survey.
Table 10. Breakdown of the scores for each product and district (BLS).
30
Sustainable use of products and tools in NTB province
The products with the highest scores at district level were IHBP, HRIS and Desa Siaga. IHBP and Desa
Siaga were best applied in West Sumbawa and HRIS is Mataram City (the district that in general scored
low for all products together). Referral System and Integrated Monev were zero in all districts. WISN was
only applied in 4 districts subjectively and confirmed in 3 districts but the extent of use was very limited
(maximum objective score of 13 in West Lombok).
FUS
Table 11 shows the scores, objectively and subjectively, for each of the districts and for each of the
products for FUS.
No Districts WISN HRIS HMIS Referral System
IHPB Monev P/ DHA Desa Siaga
QI-Action Total
Subjective Scores
1 West Lombok 25 61 43 54 58 63 38 63 50 50
2 North Lombok 0 50 35 0 51 63 38 56 37
3 Central Lombok 27 72 54 0 54 47 56 76 50 49
4 East Lombok 22 54 45 0 42 0 56 75 50 38
5 West Sumbawa 0 66 40 0 56 47 56 78 43
6 Sumbawa 0 76 43 0 53 47 28 70 50 41
7 Dompu 0 70 61 0 40 0 28 36 35 30
8 Bima 0 48 39 0 37 47 56 37 35 33
9 Bima City 0 20 18 0 25 47 0 42 22
10 Mataram City 33 76 27 0 25 0 28 26 24
Total 11 59 40 5 44 36 39 56 27 37
Objective Scores
1 West Lombok 23 42 37 27 37 33 33 53 50 37
2 North Lombok 0 35 30 0 32 33 33 48 23
3 Central Lombok 21 50 46 0 34 25 50 64 50 38
4 East Lombok 18 37 38 0 27 0 50 64 50 32
5 West Sumbawa 0 46 34 0 35 25 50 66 28
6 Sumbawa 0 53 37 0 34 25 25 61 50 32
7 Dompu 0 49 52 0 25 0 25 29 25 23
8 Bima 0 33 34 0 24 25 50 31 25 25
9 Bima City 0 14 15 0 16 25 0 35 12
10 Mataram City 27 53 24 0 16 0 25 22 18
Total 9 41 35 3 28 19 34 47 25 27
Table 11. Breakdown of the scores for each product and district (FUS).
31
Sustainable use of products and tools in NTB province
The objective and subjective scores had more or less the same ranking for all districts. The subjective
scores were in average 5 to 10 points higher than the objective scores. The biggest difference between
both scores was again for North Lombok with a subjective score of 50 versus the confirmed objective
score of 37.
The best scoring districts were Central Lombok and West Lombok with an objective score of 38 and
37, followed by East Lombok and Sumbawa with a score of 32. The districts with the lowest score were
Mataram City and Bima city with respectively 18 and 12 objectively due to the scores of zero for some
of the products. Reasons why “cities” scored lower than “districts” were not given by the Survey team
as exploring this was beyond the scope of the survey.
The products with the highest scores at district level were HRIS and Desa Siaga. IHBP was no longer
part of the best three products at district level. HRIS was best applied in Sumbawa, Mataram City (the
district that in general scored low for all products together) and Central Lombok. Desa Siaga was best
applied in West Sumbawa, East and Central Lombok. Referral System and Integrated Monev had no
longer zero scores for all districts. Referral System had been applied now in West-Lombok. Integrated
Monev was now applied in seven out of ten districts. WISN continued to be used in only four districts
and the maximum score for implementation was 27.
Stratification for geographical distance
The assumption was that the further away health facilities and villages were located from the DHO,
the lower the scores would be, as it was assumed that DHO through its regulating, coordinating and
supervisory role could influence the implementation of products. The results of the BLS however showed
no statistically significant difference between health centers and villages close by or far away from
DHO. The only exceptions were the use of HMIS data for the IHPB process and the accommodation of
Puskesmas planning into the district planning. Here the survey showed that the scores decreased with
increasing geographical distance from DHO to the health centers (p< 0.050). Therefore this breakdown
had not been repeated during the FUS.
4.3. Integration of the products into planning and budget documents
The integration of the use of the products in planning and budget documents was the third level of use
of the products that was assessed during the survey. Integration of the use of products in planning
documents was considered as a required condition for sustainability. But, as integration into planning
documents did not automatically mean that the activities would be executed and that there would have
been budget allocated for implementation, the survey only assessed budget documents. The results
32
Sustainable use of products and tools in NTB province
referred to confirmation from the respondents whether budget was allocated for the implementation and
further use of the products and the amount of the budget allocation. A possible bias could have occurred
here as in principle not all products needed budget each year to guarantee their sustainability.
For example for WISN, once all staff members were trained (this included costs for travel, manuals,
training room, resource person etc), doing the exercise of calculating the staffing needs and producing
a report did not need resources except time allocation of staff. But “no budget, so no activity” was stated
as common practice.
In general the surveys pointed out that there was limited budget allocation for the products as such and
that proof of budget allocation was difficult to retrieve. Table 12 shows the percentage of health facilities
during the BLS that had a budget allocated for each of the products. Table 13 shows the same results
for the FUS. Mostly it was integrated into a more general budget line, for example, budget for HRIS
in the provincial hospital was part of the overall information department’s budget. The amounts were
perceived by the respondents as being insufficient for full implementation of the product.
Table 12. Percentages of Health Facilities having Budget in 2010 to support the implementation of Products (BLS)