UNICEF Lebanon Terms of reference: Accelerated Immunization Activities evaluation PROJECT/ASSIGNMENT TITLE: Evaluation of the Accelerated Immunization Activities within the Health and Nutrition programme at UNICEF country programme in Lebanon (2017-2019) BRIEF SUMMARY The evaluation of the Accelerated Immunization Activities (AIA) covering the period from November 2017 till June 2019 is aimed to understand primarily the impact of such intervention and secondarily how the approach and strategy shifted from EPI System Strengthening (in phase I) to mitigating the spread of measles cases (in Phases II and Phase III). The AIA started in November 2017, aiming to strengthen the existing Expanded Programme of Immunization (EPI) of the Ministry of Public Health (MoPH), and to prevent the transmission of Polio from Syria in 2017 and other vaccine preventable outbreaks in Lebanon (such as measles) through working on both the supply and quality of service at the health center level and to increase demand through finding children defaulting from immunization and enhancing the immunization seeking behavior at community level. AIA was first piloted in Bekaa and Mount Lebanon in September 2017 for almost one-month period before expanding to more areas. However, with scarcity, unpredictability and the increase in earmarked funding, UNICEF now would like to evaluate the Accelerated Immunizations Activities to measure their effectiveness, efficiency, relevance, sustainability and impact of the approach both on the measles outbreak response, as well as their contribution to Immunization System Strengthening and to provide specific recommendations on the way forward to strengthen the MoPH immunization programme. The evaluation is planned to start in December 2019 for 3 months period. The evaluation team will be supported by and reporting to the M&E Specialist in coordination with the Health and Nutrition section. BACKGROUND During the development of the UNICEF Lebanon Country Programme Document (2017-2020), all stakeholders (MoPH, UN agencies and implementing partners) agreed that immunization should be a main pillar under the Health and Nutrition programme for the coming 4 years. As such, UNICEF in its CPD, added Outcome (RAM) indicators to measure the progress towards children’s immunization, against Penta 1 and Penta 3 and measles. Throughout various country programmes, UNICEF has been supporting the MoPH for both Routine Immunizations and Campaigns through procurement of routine vaccines and vaccination commodities, in the quality cold chain at the EPI points for quality storage of vaccines, and campaigns implementation. In addition, UNICEF in past years, supported MoPH and UN agencies (namely WHO) in conducting several immunization campaigns. Lebanon constantly reported high national immunization coverage (JRF reports). However, the EPI cluster survey (WHO, 2016), identified low immunization cadasters in Lebanon. In addition, a circulating vaccine derived Polio outbreak in Syria was declared in 2017, with high risk of contamination to Lebanon. All of this indicated that Lebanon’s existing EPI system needs further strengthening. The Accelerated Immunization Activities project was initiated to originally target the low immunization coverage cadasters (identified in the EPI cluster survey and in consultation with central and regional MoPH focal points). The AIA was modified a bit later in 2018 to respond to the measles outbreak and mitigate the spread of measles cases and try to control the outbreak. In 2019 (Phase III) and due to continued increase in measles cases, especially in Great Bekaa and Great North, AIA was prioritized in these governorates to control the spread of cases. Overall, the AIA implementation was prioritized in cadasters, with the following consideration: - Equity: based on areas with low immunization coverage cadasters, to reach vulnerable children defaulting from immunization - Quality: under the leadership of the MoPH, to provide quality vaccines, with priority to Measles and Polio containing vaccines, but with a referral to the EPI system, to ensure access to quality vaccines and quality vaccine service provision and full immunization of the child. - Vulnerability: most of the areas identified as low immunization coverage, were also part of the vulnerable localities identified in the UN vulnerability mapping - Engagement with different stakeholders: to implement AIA, the Ministry of Public Health at central and regional levels, with UNICEF support, involved implementing partners (national and international NGOs) and local authority (such as municipalities, gatekeepers) and with the personnel at the identified primary health care facilities (PHCC or a dispensary). The AIA intervention, is aligned with meeting the indicators set in the CPD and are aligned to the SDG 3.
12
Embed
PROJECT/ASSIGNMENT TITLE - Daleel Madani · 2020. 1. 22. · PROJECT/ASSIGNMENT TITLE: Evaluation of the Accelerated Immunization Activities within the Health and Nutrition programme
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
UNICEF Lebanon Terms of reference: Accelerated Immunization Activities evaluation
PROJECT/ASSIGNMENT TITLE: Evaluation of the Accelerated Immunization Activities within the Health and
Nutrition programme at UNICEF country programme in Lebanon (2017-2019)
BRIEF SUMMARY
The evaluation of the Accelerated Immunization Activities (AIA) covering the period from November 2017 till June 2019 is aimed to
understand primarily the impact of such intervention and secondarily how the approach and strategy shifted from EPI System
Strengthening (in phase I) to mitigating the spread of measles cases (in Phases II and Phase III).
The AIA started in November 2017, aiming to strengthen the existing Expanded Programme of Immunization (EPI) of the Ministry
of Public Health (MoPH), and to prevent the transmission of Polio from Syria in 2017 and other vaccine preventable outbreaks in
Lebanon (such as measles) through working on both the supply and quality of service at the health center level and to increase
demand through finding children defaulting from immunization and enhancing the immunization seeking behavior at community
level. AIA was first piloted in Bekaa and Mount Lebanon in September 2017 for almost one-month period before expanding to more
areas. However, with scarcity, unpredictability and the increase in earmarked funding, UNICEF now would like to evaluate the
Accelerated Immunizations Activities to measure their effectiveness, efficiency, relevance, sustainability and impact of the approach
both on the measles outbreak response, as well as their contribution to Immunization System Strengthening and to provide specific
recommendations on the way forward to strengthen the MoPH immunization programme.
The evaluation is planned to start in December 2019 for 3 months period. The evaluation team will be supported by and reporting to
the M&E Specialist in coordination with the Health and Nutrition section.
BACKGROUND
During the development of the UNICEF Lebanon Country Programme Document (2017-2020), all stakeholders (MoPH, UN
agencies and implementing partners) agreed that immunization should be a main pillar under the Health and Nutrition programme for
the coming 4 years. As such, UNICEF in its CPD, added Outcome (RAM) indicators to measure the progress towards children’s
immunization, against Penta 1 and Penta 3 and measles.
Throughout various country programmes, UNICEF has been supporting the MoPH for both Routine Immunizations and Campaigns
through procurement of routine vaccines and vaccination commodities, in the quality cold chain at the EPI points for quality storage of
vaccines, and campaigns implementation. In addition, UNICEF in past years, supported MoPH and UN agencies (namely WHO) in
conducting several immunization campaigns. Lebanon constantly reported high national immunization coverage (JRF reports).
However, the EPI cluster survey (WHO, 2016), identified low immunization cadasters in Lebanon. In addition, a circulating vaccine
derived Polio outbreak in Syria was declared in 2017, with high risk of contamination to Lebanon. All of this indicated that Lebanon’s
existing EPI system needs further strengthening.
The Accelerated Immunization Activities project was initiated to originally target the low immunization coverage cadasters (identified
in the EPI cluster survey and in consultation with central and regional MoPH focal points). The AIA was modified a bit later in 2018
to respond to the measles outbreak and mitigate the spread of measles cases and try to control the outbreak. In 2019 (Phase III) and due
to continued increase in measles cases, especially in Great Bekaa and Great North, AIA was prioritized in these governorates to control
the spread of cases.
Overall, the AIA implementation was prioritized in cadasters, with the following consideration:
- Equity: based on areas with low immunization coverage cadasters, to reach vulnerable children defaulting from immunization
- Quality: under the leadership of the MoPH, to provide quality vaccines, with priority to Measles and Polio containing
vaccines, but with a referral to the EPI system, to ensure access to quality vaccines and quality vaccine service provision and
full immunization of the child.
- Vulnerability: most of the areas identified as low immunization coverage, were also part of the vulnerable localities identified
in the UN vulnerability mapping
- Engagement with different stakeholders: to implement AIA, the Ministry of Public Health at central and regional levels, with
UNICEF support, involved implementing partners (national and international NGOs) and local authority (such as
municipalities, gatekeepers) and with the personnel at the identified primary health care facilities (PHCC or a dispensary).
The AIA intervention, is aligned with meeting the indicators set in the CPD and are aligned to the SDG 3.
AIA Conceptual Framework – Theory of Change:
The AIA is divided into 3 phases:
-Phase I: November 2017 till December 20171
-Phase II: June 2018 till December 2018
-Phase III: January 2019 till current
When AIA Phase I started, in 2017, its main aim was to: strengthen the existing MoPH EPI system and prevent Polio transmission
from Syria and vaccine preventable outbreak in Lebanon. As such, the AIA Phase I approach focused on:
- Targeting low immunization cadasters identified in the EPI cluster survey conducted in 2016 (WHO) and cadasters there were
identified as vulnerable based on joint discussions between UNICEF and MoPH central and regional level.
- Trained community outreach workers, data entry clerks, vaccinators from identified PHC facilities, qadaa level MoPH staff,
on Interpersonal communication skills, AIA approach, data organization and flow of work
- Implementing partner, jointly with MoPH PHC coordinators and public health officers, scheduled meetings with local
authorities (mainly municipalities) and briefed them about the AIA project and asked them to support in accessing households
in their localities.
- Implementing partner mapped each cadaster
- Outreach teams visited all houses in the cadasters and screened the vaccination cards of all children under age of 15 years old;
identify defaulting children and provide them with a referral voucher with unique barcode.
- Referred children go to the identified PHCs near them and they are provided with needed vaccine and their barcode and their
information is registered
- Children who were identified as defaulters but did not go to the identified PHCs for their vaccination as advised had their data
shared with the implementing partner to follow-up on their immunization
- Implementing partner followed-up on children through phone calls and visits
1 Some implementing partners continued till February 2018 (based on extensions of their PDs)
Also, during Phase I, WHO with MoPH mapped all Informal Tented Settlements (ITSs) into 2 categories A&B, where by all ITSs
under category A (essential): children under 15 years of age, were provided on-spot immunization for defaulting children (measles,
MMR, PCV 13, OPV and IPV) and referred defaulting children to identified PHCs for follow-up on remaining vaccines. And children
in category B were screened and referred to nearest PHC (treated as a Household visit).
AIA phase II – specifically after measles outbreak; MoPH with support of UNICEF adapted the original AIA approach, to include
‘campaign mode’ option in areas with high measles attack rate and that do not have a nearby PHC facility. As such the AIA phase II
approach focused on:
- First, cadasters with measles cases; second, cadasters with low immunization coverage that were targeted in AIA phase I, but
were not completed; finally, cadasters with low immunization coverage that were not approached in Phase I.
- During planning phase jointly with MoPH and implementing partner(s), each cadaster was assigned as potentially needing a
Mobile Vaccination Unit or only outreach workers screening and referring children to nearest PHC and then following up on
them.
- Data is collected electronically using KOBO with direct link to MOPH at central level.
AIA Phase III: UNICEF continued to focus on areas with measles cases; as such the intervention in first 6 months of 2019 was like
that adopted in phase II. In addition:
- UNICEF with MoPH mobilized the health sector, especially in Great North and Great Bekaa, where some sector partners
adopted AIA approach and supported MoPH in targeting children in areas with measles cases.
- UNICEF started discussions with UNICEF partners from different sections, who already conduct outreach, to also support
without additional cost in reaching out to children under 15 years of age and referring defaulters to nearest identified PHC.
- All data is collected through MERA at outreach and center level and data is triangulated at MOPH central level to provide
feedback to health centers and partners on children defaulting from immunization who did not attend the service following
the referral.
Objectives, Purpose & Expected results:
Purpose:
UNICEF is commissioning a summative evaluation on the accelerated immunization activity program to generate substantial
knowledge and learning on the results of AIA project in order to understand how the project (i) reached its objective of strengthening
MoPH national EPI system; (ii) increased immunization coverage in Lebanon, and (iii) had an effect on the measles outbreak
(controlling the spread of cases, or decreasing the number of measles cases, or had no effect on the outbreak). The evaluation is also
intended to propose recommendations for such a project, to make it sustainable, cost effective and included within MoPH EPI
system.
UNICEF will be the primary user of the evaluation report. Ministries, donors and implementing partners will be the secondary users.
Objectives:
The evaluation will explore how the AIA project has contributed towards improving access and coverage of routine immunization and
its effect on the measles outbreak.
More specifically, the objectives of the evaluation are to:
1. Assess the effectiveness of different AIA approaches (door to door mobilization, temporary vaccination sites, mobile
vaccination units…) on enhancing routine immunization, including uptake by most vulnerable girls and boys and attitudes,
knowledge and behaviors.
2. Assess how AIA II and III had any effect on controlling the spread of measles cases in Lebanon; and if AIA I had any
‘protective factor’ on some cadasters and prevented or at least delayed the onset of measles outbreak
3. Assess the effect of AIA on strengthening the MoPH expanded programme of immunization (EPI) system; including the
effect of AIA on the enhancement of EPI data/registry system through KOBO/MERA application.
4. Provide recommendations on how to move forward with an immunization strengthening approach, which is relevant, efficient,
cost-effective and sustainable, especially with less and less funding available in the health sector. These recommendations
will be utilized bythe MoPH, UNICEF and the main health stake-holders to feed into future strategies for the Health and
Nutrition programme and upcoming UNICEF Lebanon Country Programme Document and could influence the health chapter
of the LCRP.
a. Scope of work:
The scope of the evaluation will focus on the UNICEF Lebanon AIA project, from 2017 till June 2019 in all governorates of Lebanon
including Akkar, North, Bekaa, Baalbeck-Hermel, Beirut, Mount Lebanon, South and Nabatieh.
The evaluation will specifically look at the following pillars:
1. Sustainable strengthening MoPH national EPI system;
2. Increasing immunization coverage in Lebanon;
3. Controlling the measles outbreak
The evaluation will take into consideration, the CPD as the highest level of outcome, whereby the AIA project falls as one of the tools
to reach the CPD Child Survival outcome. In addition, the rolling work plans of Health & Nutrition section of 2017, 2018 and 2019
will be considered when measuring the outcome of AIA project.
b. Time covered by the evaluation
The period 2017-2019 will be considered as the time frame for the evaluation which includes the 3 phases (AIA I+II+III)
c. Timing of the evaluation
The evaluation will be used to inform next years rolling work plan and shape the needed Immunization strategy. It will also guide in
the planning of the next country programme document (2022-2026). The start of the evaluation will be December 2019 – for a period
of 3 months.
Evaluation questions:
The following questions for this evaluation are initial suggestions formulated based on the OECD DAC criteria, and will be further
developed by the research institute/consultancy firm, that will contribute in developing the evaluation methodology and instruments
(including questionnaires). Overall, the evaluation aims to answer the following questions focusing to understand the relevance,
effectiveness, efficiency, coverage, and sustainability of the project:
1. How relevant was the AIA project to MoPH EPI national priorities and the needs of the most vulnerable boys and girls in Lebanon?
a. To what extent did the AIA project suit the priorities of the national MoPH strategy, MoPH EPI programme, UN LCRP,
Health sector plan and UNICEF CPD, and annual and rolling workplans?
b. To what extent are the objectives of the AIA programme still valid and will be valid - in regards to epidemiology (outbreaks)
and immunization coverage of children in Lebanon - for the coming country programme document?
c. To what extent are the activities and outputs of the programme consistent with the overall goal and achievement of its initial
objectives considering the shift from phase I to II and III? (Was the shift in the approach successful and useful or UNICEF
just had to remain in the initial AIA approach?)
2. To what extent was the project effective in meeting the intended goal?
a. To what extent was the project effective in strengthening the functionality of the MOPH EPI system?
b. How did provision of immunization data to health centers enable them to know the immunization coverage within their
catchment area and informed them on the children defaulting from immunization within their area?
c. What real difference has the project made to targeted PHC staff and centers in terms of immunization knowledge, motivation
and ownership?
d. To what extent did UNICEF collaborate with the right (key) partners to ensure achievement of the results? Were there any
challenges in achieving the intended goal?
e. To what extent did the AIA project result in changes in the knowledge and behavior of caregivers of children (boys and girls)
under 15 years of age towards immunizing their children and following up on their routine vaccination? And increasing their
trust in PHCs?
f. To what extent did the AIA project protect from vaccine preventable outbreaks? And to what extent did it control the spread
of measles outbreak?
3. To what extent did the AIA project increase routine immunization coverage at national level, or at least at targeted cadasters?
a. To what extent was the project effective in reaching most vulnerable children (boys and girls) in Lebanon including children
defaulting from routine vaccines? To what extent was the project generating evidence to inform equity EPI priorities? To what
extent was it gender equitable?
b. What reasons accounted for not reaching an identified defaulter child (boys and girls) with measles and polio vaccination?
What lessons learnt can be drawn and applied for strategy revision/strengthening in the near future?
c. To what extent was the project tailored to meet the needs of Lebanese and non-Lebanese children(boys and girls) in rural and
urban setting?
d. To what extent was the project tailored to target children (boys and girls) with disabilities?
4. To what extent was the project cost efficient (reflecting any differences between urban and rural areas; between one implementing
partner and another)
5. To what extent is the AIA project sustainable?
a. To what extent did the benefits of AIA project continue when interventions (due to limiting funding) stopped? Or intervention
was moved from one area to another? (what does sustainability for a vaccination programme look like? Will MoH continue
the programme? Are vaccines available without UNICEF project? Will the staff continue to use the knowledge?)
-If yes: What were the major factors that influenced the achievement of sustainability of the project?
-If no: what were the major factors that influenced the non-achievement of the sustainability of the project?
b. To what extent have the AIA project contributed to or embed sustained local ownership and involvement of local
communities in increasing demand and utilization of routine immunization services considering gender disaggregation of
caregivers?
c. Did the AIA project contributed to sustainable strengthening of the MoPH EPI system (in capacity building, in community
outreach, in EPI registry system)?
d. To what extent did MoPH, PHCs and local authority have sufficient capacity to take on tasks without UNICEF support
(especially that with decrease in funds, UNICEF will not be able to provide HR support to MoPH and PHCs)?
Evaluation stakeholders:
A preliminary mapping of relevant stakeholders of the AIA programme identified the below list of stakeholders. In addition, these
stakeholders are divided into four sub-categories depending on their relative level of “Interest” and relative “Influence”. These
stakeholders are at the core of the design of the programme with whom the team will engage.
Lo
w
I
NF
LU
EN
CE
H
igh
Municipalities Ministry of Public Health;
World Health Organization
Municipalities
Primary Health Care Center Directors
Primary Health Care Center Directors
Caregivers of children under 15
Primary Health Care Center Directors
Implementing partners
Caregivers of children under 15
Low INTEREST High
Evaluation Methods:
The methodology for this evaluation method will be a mix of quantitative (analysis of existing secondary data) and qualitative
tools (KII, FGDs). . The overall methodology should be participatory and should ensure that the various stakeholder including
the most vulnerable communities, municipalities, mayors, partners and others will be reached and their voices elicited.
It is expected that the evaluation will use the following methods:
- Desk Review of key documents: analysis of data from partners and MoPH reports, national surveys and internal compilation
of data and documentation of AIA.
- Key informant interviews with:
• Staff from MoPH PHC department (head of PHC department, senior PHC coordinator, EPI focal point), MoPH qada
offices (qada physicians, PHC coordinators, supporting staff), MoPH IT department (IT project manager and IT AIA
focal points), PHCs (nurses, data entry clerks, directors).
• Project manager from main UNICEF implementing partners and non-implementing partners: WHO, IOCC,
Makhzoumi, Amel, Islamic health association, Al Resaleh Scouts, Al Midan, LRC, Medecin sans frontiers, Nudge. Staff
from UNICEF Health & Nutrition, UNICEF Planning, Monitoring and Evaluation and Communication for