Framework Contract Beneficiaries 2013 Lot 1 European Union Delegation to Zambia FINAL EVALUATION OF PEP AND OPERATIONALISATION OF PEP II Contract N° 2015/371442/2 FINAL EVALUATION REPORT & STRATEGIC RECOMMENDATIONS FOR PEP II JULY 2016 This project is funded by the European Union Project implemented by The contents of this publication is the sole responsibility of AESA Consortium and can in no way be taken to reflect the views AESA Consortium [email protected]
64
Embed
FINAL EVALUATION OF PEP AND OPERATIONALISATION OF PEP II · FINAL EVALUATION OF PEP AND OPERATIONALISATION OF PEP II Contract N° 2015/371442/2 FINAL EVALUATION REPORT & STRATEGIC
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
0
Framework Contract Beneficiaries 2013 Lot 1
European Union Delegation to Zambia
FINAL EVALUATION OF PEP AND OPERATIONALISATION OF PEP II
Contract N° 2015/371442/2
FINAL EVALUATION REPORT
& STRATEGIC RECOMMENDATIONS FOR PEP II
JULY 2016
This project is funded by the European Union Project implemented by
9.4. Learning and Innovation Platforms for Research and Extension ..................................................... 43
9.6. Focal Districts for PEP II.................................................................................................................... 45
9.7 Knowledge Management in Ministries ............................................................................................. 46
Page | 3
Annex 1: Detailed Evaluation Method, Limitations and Tools .................................................................... 48
Annex 2: List of Persons/Organizations Consulted ..................................................................................... 58
Annex 3: Literature & Documentation Reviewed ....................................................................................... 63
Page | 4
Summary
1. Programme Context
As a programme, PEP was a response to the widely held view by national stakeholders
around 2009 that the agricultural sector was struggling to achieve its undoubted potential
to make a vital contribution to economic growth, rural development, and poverty reduction
in the country. At the centre of those concerns and challenges was the then MOAC – the
ministry expected to lead sector initiatives and development programmes. It had
capacities that were known to be weak and it needed substantial revitalization of its
policies, structures, personnel, technologies, and extension services systems.
The EU, with the strong support of Cooperating Partners (SIDA, JICA, IFAD & Finnish
Cooperation), engaged the ministry and it was agreed that a comprehensive assessment of
the Ministry’s capacities be conducted. Following from that exercise, the Performance
Enhancement Programme (PEP) was conceived and designed. It was not envisioned as yet
another project, but as a programme with the ambition, and indeed, courage to take on the
wide and deep system-wide capacity challenges facing the Ministry.
PEP eventually started in 2012 with four very overloaded key result areas across the
ministry covering change management; sector policy, planning and financial management;
human resources and ICT; and monitoring and evaluation. The goal was to tackle all the
major areas of capacity weakness together, with little prioritization and over a relatively
short time frame. This ultimately proved to be the most fundamental weakness of PEP –
trying to do too many things in too short a time – a commendable array of activities were
started but many were never properly or fully completed, while some unplanned activities
were added for implementation at the behest of MAL.
2. Strengths and achievements
PEP initiatives in management and staff training, providing computers, developing data
bases, strengthening planning, monitoring and evaluation skills, and updating the curricula
of training institutes were all well appreciated by ministry staff. This was partly because of
the extent of the previous protracted neglect of those areas, coupled with the hope and
impetus that those initiatives brought to the ministry staff which benefited. Given the
absence of modern support facilities and systems for middle-tier professional and support
staff in the ministry, the programme support to improve ICT workplace technology,
training, and the development data bases and information systems made most MAL staff
familiar and proficient with computer and internet use, knowledge of data bases, and
reporting in their day-to-day work.
Page | 5
3. Weaknesses and challenges
A central expectation of stakeholders and a core purpose of PEP was to facilitate and
manage change actively across and within MAL. To begin to fulfill this role satisfactorily,
the programme needed to have and communicate widely a comprehensive change
management strategy spearheaded by champions of change from the senior levels of the
ministry. The programme did not have such a strategy, including gaining and sustaining a
strong and evidence-based commitment from senior management. The Change
Management Team (CMT) operated in the prevailing administrative mode of MAL, as a
Head Office based coordination unit, rather than leading a dynamic system-wide change
process across all levels of the ministry. The programme was not communicated
adequately or cascaded effectively to the provincial or district levels as the feedback and
records on information, training and evaluations at those levels clearly reveal. Both those
shortcomings will need to be addressed in the next phase.
The programme should have conducted a benchmarking exercise, such as a ministry-wide
staff survey, at the outset. For a performance improvement programme centered on staff
proficiencies, such a comprehensive exercise should have been mandatory so as to enable
meaningful assessments of progress in skills acquisition by staff over the course of the
programme. The generally disenabling conditions in the ministries mitigated against
optimal performance improvement (e.g. limited and late budget allocations and lack of
opportunities to put skills gained from training into practice in their work places). Periodic
appraisals on the constraining effects of those latter factors will be necessary for inclusion
in the next phase. There were also instances of wastage of resources, despite the prevailing
budget shortfalls in MAL. PEP funded an exercise to devise improved budget resource
allocation criteria for MAL which the ministry subsequently decided not to accept or utilize,
casting some aspersions on programme decision-making processes and the quality of
engagement and ownership of PEP by MAL.
4. Conclusion
Overall, the PEP experience was a learning journey for the EU and MAL that revealed many
instructive lessons for the design, planning, facilitation and implementation of such
programmes in future. A central one is that a programme should be implemented in a well-
defined programme mode – not as a collection of projects. Another is that the result areas
should be limited and not overloaded – the so called “doability” mantra. While PEP brought
benefits across all levels (HO, provinces and districts) of MAL, the exclusion of large
numbers of staff (half of the frontline field extension complement) from the allocations of
computer tablets led to disappointment, and perceptions of favouritism and exclusion, by
half of a very important cadre of MAL staff. Change management programmes and their
strategies need to challenge existing norms and modes and bring about demonstrable
Page | 6
change in the way public service ministries perform. To do that, a change strategy and
process was needed together with a way of knowing or gauging the point of departure and
then how well it is progressing and achieving. PEP II needs to do just that and avert the
pitfalls of the first phase.
In a nutshell, PEP certainly sought to achieve noble and worthy deeds. There is no doubt
that PEP set out to do the right thing and had some very worthy accomplishments. It
facilitated increased awareness, knowledge and insights among many staff on management
and performance issues in MAL. Its major drawback was that it attempted to do some of
those “right things” in the wrong way. Examples included embarking on a staff
performance improvement programme without conducting a comprehensive baseline
study on existing proficiencies; seeking to improve MAL budget allocation criteria without
first securing purposeful and definite commitments by the relevant GoZ ministries;
providing wide-ranging support to improving ICT capacity across MAL while being unable
to secure the commitment of senior management to put in place a dedicated unit to
optimize the management and benefits of the investment.
Page | 7
1. Introduction
1.1 Background and Context
The Government of Zambia’s development priorities are framed by the vision of “a
prosperous middle-income nation by 2030” that “provides opportunities for improving the
well-being of all”. The revised Sixth National Development Plan (R-SNDP) 2013-2016 aims
“to accelerate growth further and to make it more relevant to improving the livelihood of
the Zambian people, especially in rural areas”.
Given that about 80% of the population depends on agriculture as their main source of
livelihood, the agricultural sector has the greatest potential to significantly impact poverty
reduction and to achieve increased economic diversification. This policy shift to refocus the
agricultural sector’s development strategies and programmes towards drivers of pro-poor
agricultural growth is at the core of the draft National Agriculture Policy (NAP) and the
National Agriculture Investment Plan 2014-2018 under the Comprehensive African
Agriculture Development Programme.
The NAP and the Agricultural Sector Chapter of the R-SNDP acknowledge that to achieve
this goal the Government must focus on providing public goods that are indispensable to
creating broad-based and inclusive agricultural growth. These include agricultural
research, effective agricultural extension services, control of pests and diseases of national
importance, and the development of basic rural infrastructure. Vision 2030 and the NAP
also recognise as key the "gender responsive sustainable development" of the agriculture
sector with an emphasis on women’s empowerment.
Beyond the sector policy framework, there is still a gap between stated policy intentions
and their realisation. Although budgetary allocations to the then Ministry of Agriculture
and Livestock have increased during the last seven years, they remain below the
recommended CAADP 2 minimum of 10% of the annual national budget and expenditure
within that remains overwhelmingly driven by the Government’s maize purchases through
the Food Reserve Agency (FRA) and input subsidies under the Farmer Input Support
Programme (FISP). The maize input subsidy programmes consumed 52% of the 2014
agriculture budget. Emphasis on the two programs has diverted resources from critical
programs with potential to contribute to the desired growth in the agricultural sector such
as irrigation development, extension services delivery and research while limiting the
scope for optimal private sector participation in national agricultural development.
Page | 8
1.2 EU Support to Strengthen the Agricultural Sector and MAL Performance
The underperformance of the agricultural sector is also linked to institutional weaknesses
and organizational constraints that prevent the Ministry of Agriculture and Livestock
(MAL) from delivering on its core functions such as extension and research, directly
affecting farmers' productivity and undermining the potential role of the private sector.
The EU provides consistent support to improve the capacity of the Ministry of Agriculture
and to the Ministry of Fisheries and Livestock to deliver quality services to smallholder
farmers towards increased productivity and enhanced diversity of Zambia's agricultural
sector.
Against this background, the EU in 2009, supported the then Ministry of Agriculture and
Cooperatives (MACO) to carry out a Participatory Review of the Ministry's Performance
that further fed into an in-depth assessment of MACO's capacity development needs and
challenges. The outcome of this exercise served as the basis for the design of the "Support
to Agriculture Sector Performance Enhancement Programme" (known as PEP)
implemented through four subsequent programme estimates under the 10th European
Development Fund (EDF). The project, worth €8.9M, aimed at ensuring that "the
agricultural sector contributes to growth and poverty reduction in Zambia" (Overall
Objective) by "strengthening the capacity of the Ministry of Agriculture and Livestock
(MAL) in an effective and sustainable way so that their operations are improved" (Project
Purpose). The project focused on four results areas namely 1) Change management and
service delivery, 2) Sector policy, planning and financial management, 3) Human resources
management and Information and Communication Technology, and 4) Monitoring and
Evaluation. Following a four-month closure phase PEP was completed in April 2016.
1.3. Objectives of the Evaluation
The assignment had two objectives and involved two separate, although interlinked
phases, consisting of 1) the final evaluation of PEP; and 2) the development of
recommendations for PEP II. The aim was to conduct an independent assessment of the
performance and impact of the "Support to the Agricultural Sector Performance
Enhancement Programme" (PEP), paying particular attention to the project's achievements
against its expected results and purpose. This involved an in-depth assessment of the
strategic objectives and implementation mechanisms of PEP as well as of the extent to
which these have been effective in delivering the desired results.
Page | 9
1.4. Approach and Methodology
Guided by the ToRs, and the Inception Report approved for the evaluation, the focus of the
methodology for this evaluation was on the completion of two separate, but closely linked,
assignments:
1) The final evaluation of PEP; and, 2) The development of recommendations for PEP II.
The central focus was on the assessment of PEP achievements under the national
prevailing economic and institutional realities; key indicators and modalities to achieve
result-oriented approaches; and core institutional capabilities and commitments required
to underpin improved performance by key sector actors; and the quality of outcomes,
results and lessons from the focal interventions.
This evaluation sought to deepen understanding of the cause and effect linkages between
activities and results, including the Theory of Change (ToC) as a key learning platform for
the European Union and its partners on agricultural sector performance enhancement and
capacity development programmes to understand not only (1) what works and what does
not, and (2) critically why, and under what circumstances. While the ToC per se was not
formally highlighted and elaborated in programme planning documents and reports, the
evaluation team (as per the Inception Report) used a ToC lens in assessing the programme
processes to achieve overall outcomes and impact.
In addition to the normative evaluation methodology for projects, centered on the purpose,
indicators and outputs of the logical framework, the methodology focused on key learning
outcome areas and operational attributes reflecting staff performance improvement in
MOA & MFL This focused on knowledge, skills and insights acquired by management and
staff on individual and collective performance during the course of the programme.
Examples of such Performance Enhancement Insights, as perceived and related directly by
Ministry staff, are described in the boxes included later in this report.
This cross cutting and multi-intervention evaluation was conducted in accordance with the
guidelines for the OECD DAC Evaluation Criteria on which the evaluation questions were
based and agreed with the Reference Group.
Those evaluation questions were formulated in the ToR to provide qualitative insights from
the overall programme experiences to inform future programme design, implementation
modalities, management, operations, and funding. The answers to those questions focus on
the overall programme performance outcomes in strengthening capacity in MAL, including
overarching lessons arising for future phases. Under the guidance of the Reference Group,
the team focused its work on evaluating the four key output areas as delineated in the
Page | 10
ToRs. Based on guidance & advice from the Reference Group, an itinerary of interviews,
meetings and discussions (see Annexes) were held with (1) key personnel involved in PEP
oversight, management, and implementation including relevant GRZ ministries (e.g.
Decentralization Directorate in the Cabinet Office), and (2) local institutions or
organizations (public sector/NGOs/Farmer Organizations) involved in PEP capacity
development activities, and (3) cooperating partners including IFAD & JICA.
Using a progress and performance matrix grid, an appraisal of all PEP key result areas
(targets, indicators & outputs ) as per the PEP Log Frame, was initially conducted, including
status of performance reviews/or evaluations, analyses of outcomes and results against
targets, and findings determined on PEP implementation. The four key result areas for PEP
were assessed comprehensively under the above matrix grid.
This was combined with more areas (also using a systematic matrix approach) to examine
the log frame, targets, indicators, achievements, and outcomes. The key lessons arising, and
recommendations covering institutional capacity development processes, including the
appropriateness of change management and organization development (OD) approaches
implemented by PEP were also assessed. The follow-through actions on the Mid-Term
Evaluation Report, (September 2014) were assessed in the context of the subsequent
implementation and completion of the programme. The overall programme was assessed
for consistency with the EU country strategy and complementarity with other donor
programmes. Two EU specific criteria (EU added value in design and implementation &
coherence with the EU country strategy, member states and other donors) were also
assessed.
The Consolidated FCRs (Findings, Conclusions and Recommendations) Matrix across all
outputs and activities was completed in the synthesis phase of the evaluation. The
evaluation questions were then addressed and answered. The priorities for
operationalizing PEP II were also identified, based on the lessons from the evaluation of
PEP I and both were presented on May 17th for validation to a stakeholder’s workshop in
close consultation with the Reference Group.
Page | 11
2. Evaluation Questions Answered
2.1 Relevance
To what extent was the project design appropriate for the then MAL's institutional capacity development needs? The parent MAL had remedial capacity needs in critical functional areas, as documented in needs assessment reports of 2009 and 2013. As such, the PEP programme design was appropriate to the overall capacity needs of the ministry. This is because the PEP Programme was designed to address the identified areas were capacity was lacking, such as, in policy analysis and decision-making processes; services provision; budgeting, planning and financial management; human resources management; and in knowledge management and Information and Communication Technology (ICT). As an illustrative example, while many MAL personnel were very highly qualified technically, leadership and management capabilities were lacking. This strongly justified training and capacity development on leadership, management and administration. A number of strategic documents viz: Vision 2030, the Revised Sixth National Development Plan (R-SNDP) 2013 – 201, the National Agricultural Policy (NAP), and the NAIP 2014 – 2018 also emphasize the importance of providing know-how and support to the development of the key ministry (MAL) and the wider agricultural sector. Have these needs changed since the mid-point of the project? No significant changes occurred within the programme context during implementation. Available development data, national indicators and web-based insights by development partners such as the African Development Bank and the World Bank show that there have been inconsequential variations in socio-economic indicators from project inception in 2012 till project completion in 2016. Despite efforts by the TA Team, Change Management Teams and implementation teams within MAL to continuously re-strategize, the experiences in the implementation of various activities did not seem to have evident influence/impact on prioritization and re-planning of activities on the programme. In other words, the programme continued throughout its implementation period to try to do too many activities, despite evidence that that it was struggling to achieve its targets. What amendments in the concept, theory or implementation modality should be made for a future phase of the project? The programme lacked a definite overall change management strategy, apart from the implementation of activities under programme estimates across the key result areas. PEP started in project mode with a number of activities but with no overall programme orientation and/or strategy. There is, therefore, need to understand Programme-Based Approaches (PBAs) to capacity development e.g. the need for a mindset shift from a project to programme mode with appropriate modalities. A future phase of the programme should also be more focused on fewer result areas; have a more robust communication strategy
Page | 12
between and within MOA & MoFL and with sector stakeholders; and have programme estimates with more coherent and integrated activities that facilitate more process-based approaches to implementation involving monitoring, learning, and evaluation (MLE). Were the project institutional arrangements and mechanisms relevant and have they contributed to increased ownership by the then MAL? The programme’s institutional arrangements and mechanisms were also ostensibly relevant. The institutionalization of the PEP programme, with the programme being housed and embedded in MAL rather than as a stand-alone and/or semi-autonomous endeavor, together with PEP programme interventions. put the programme in the best position to directly influence management and administration processes within MAL. However, these were largely functioning in administrative mode, with no evidence of commitment and active engagement with the processes of capacity development and ownership. As such, the programme did not contribute to increased ownership in any substantive way. The programme struggled with ownership issues, with evidence of limited commitment, participation and quality of decision-making by the programme steering committees and change management teams. A major factor diluting the quality of implementation throughout the programme cycle was that most of the implementation responsibilities and tasks, pertaining to those key senior roles, were delegated to lower ranking personnel in MAL departments whose authority levels, influence and proficiencies were limited. In some cases, this led to critical delays in the finalization of key processes e.g. the MAL Strategic Plan that remained unapproved up until the completion of PEP in April 2016. Another major mitigating challenge was that the programme tried to do too many activities concurrently and was trying to accomplish too much. The programme then quickly evolved into a set of often disconnected projects rather than a coherent programme. As discussed later, this reduced programme effectiveness, efficiency and overall impact. What changes should be considered for future interventions? For the future, similar interventions should be programme rather than project-based, hinged on an understanding of Programme-Based Approaches (PBAs) to capacity development. A future phase of the programme should also be more focused on much fewer activities across less result areas. There is also need for a delineated and phased approach to programme implementation (with orientation/inception, implementation phases, and an exit phase) and a very clear and comprehensive change management strategy (including a robust communication strategy given the limited vertical and horizontal communication between and within MoA and MoFL departments). A more focused approach, with fewer and less loaded result/output areas, will improve on programme relevance. Programmes such as PEP should also avoid making substantial ad hoc contributions to recurrent budget expenditure (e.g. E-voucher programme). Similar performance enhancement programme initiatives in Ghana and Rwanda have improved performance and service delivery, notably since these initiatives were integrated with policies and strategies for decentralization; lessons should be learned from those country
Page | 13
experiences in view of the impending decentralization in Zambia. Within the country, the MoA and MoFL can draw lessons in service delivery from the Ministry of Health.
2.2 Effectiveness
To what extent have the Ministry's capacity and performance been enhanced? The PEP programme enhanced the capacities of individual management, staff and units within MoA and MFL. This was largely through leadership training, induction and in-service training, M&E training, the development and establishment of an HR database, and the distribution of ICT equipment. However, the prevailing conditions in the ministries (e.g. staffing levels & field mobility, conditions of service, inadequate operational budgets), mitigated against optimal performance improvement. Key setbacks included the limited budget allocations and the frequent lack of opportunities to put skills gained from training into practice in staff work places. Has flexibility contributed to the achievement of the purpose or rather contributed to hindering it? The E-voucher system introduced innovation to farmer registration and FISP. On the other hand, emoluments payments enabled the ministries to fill positions and free up accommodation for new recruits. However, programme flexibility made the PEP programme a utility fund for shortfalls in government budgets which is neither desirable nor sustainable. In other words, programme flexibility has hindered rather than contributed to the achievement of the programme purpose. As an example, resources used for e-voucher pilots and emoluments payments (typical government recurrent expenditure) could have been used to complete planned and bona fide PEP commitments e.g. ICT equipment such as tablets made available to all district field staff, M&E training, and curriculum development. While recognizing the foregoing, it is also appreciated and acknowledged that the provision of some of this support was sometimes unavoidable given that this is a programme that was implemented within a set context i.e. the MoA and MoFL work in an environment where flexibility was imperative to ensure that critical deficit areas were addressed in the wider overall interest of progress by both MAL, PEP and the EU Country Strategy. The PEP programme had also previously supported the pilot electronic farmer registration process, which made support to the E-voucher system a logical follow-up programme to assure optimal benefits by MAL and recipient farmers from the initial investment. Were the project's design and approaches conducive to increased effectiveness? While the programme design facilitated increased effectiveness, the approaches adopted mitigated against effectiveness in some respects. Examples include the implementation of too many activities concurrently; the lack of communication on MTE recommendations; and the fact that most of the training conducted was not thoroughly evaluated for skills acquisition and application by trainees. Examples of cases where the project design improved effectiveness include the establishment of the Human Resource Information System (HRIS). This has gone a long way in providing timely and reliable information on and about the HR situation in the entire ministry. Similarly, the reinstatement of In-Service
Page | 14
and Induction Training Programmes has brought a sense of purpose to the Human Resource Development function in the ministries. However, there is no evidence yet of mainstream capacity and performance enhancement throughout the ministries. To what extent have the policies and guidelines developed on HIV/AIDS and gender been implemented and/or mainstreamed (as recommended by the MTR)? One of the programme’s milestones was the development of HIV/AIDS and gender policies and guidelines. Training conducted on the HIV/AIDS and gender policies and guidelines raised awareness at head office and provincial level and in the training institutes. However, more needs to be done at all levels of staffing. Furthermore, based on information in both the MTE and Final PEP Reports, there is no evidence yet of substantive implementation and/or mainstreaming of the policies and guidelines across the ministries.
2.3 Efficiency
Have the project implementation arrangements been efficient? Efficiency assessments by the evaluation team found that programme implementation arrangements have been unsatisfactory. This is because most of the planned activities were implemented late into the programme (leadership and management training, M&E training, procurement of ICT and other support equipment). In addition, some planned programme activities were not completed within the timeframe of the programme (e.g. M&E training and ICT training). Facilities (e.g. the ministry’s HR and M&E data bases, and website) were developed but were either not used at all, or under-utilized. As elaborated elsewhere, the programme tried to do too much with the consequent partial completion of some activities, while also concurrently implementing a lot of unplanned activities. Such approaches often propel ostensible efficiency in spending of programme estimate funds but raise searching questions about the planning and prudence of such spending in achieving desired core results and overall impacts. Budget figures (NAO/PEP Final Report) indicate a range of 81 – 100% expenditure under the four key result areas, which appears relatively efficient, but needs to be considered in conjunction with the overall findings on effectiveness and impact later. How should they be replicated or amended under PEP II? The programme estimates need to be better informed and planned in terms of budget allocations, activities, sequencing and monitoring. The introduction of incentives and awards should be a priority as a strategy for improving facility utilization and overall efficiency e.g. incentives for colleagues to make inputs for the MoA and MFL website. To what extent partnerships with other cooperating partners have contributed to the project performance? The collaboration with CPs enhanced programme performance (e.g. the Extension Strategy with JICA, the M&E Manual with IFAD, and the IFMIS with the Finnish Cooperation).
Page | 15
To what extent have lessons and recommendations from the MTE been implemented and/or followed up? The recommendations of the Mid-Term Evaluation (MTE) on the log frame were followed and contributed to its adjustment and updating (December 2014). However, the revised log frame was not officially adopted through an official rider at EU level and may not have brought about the proposed improvements set out in the MTE. In addition, there was no evidence that the proposals of the MTE for a more comprehensive change management process by the programme were engaged with. The MTE recommendations were also not shared, or communicated to, the provincial and district levels.
2.4 Impact
What impact (intended, unintended, positive or negative, direct or indirect) has the project had on the MAL's capacity and performance? Some individual beneficiaries of change management efforts (e.g. leadership and management training) exhibited a change of attitude and mindset shift towards performance issues (e.g. DACOs becoming involved in mentoring and coaching of their staff). According to ZARI reports, distributed ICT equipment also improved the timeliness, quality and consistency in reporting. As an example, the distributed ICT equipment facilitated the collection and utilization of real time data using the CABI Plant Wise which created plant disease clinics in farming communities run by camp extension officers. The payments of retirement packages indirectly contributed to employment creation. Worth noting is the fact that the direct impact of the PEP programme is largely restricted to individuals given its focus as a performance enhancement programme. impact assessments were also restricted to the activity/output levels. Impact at outcome level was difficult to assess due to the limited timeframe of the PEP programme. Performance enhancement is also a long-term process so the PEP programme should not be expected to have achieved much impact at the outcome level. On the other hand, some opportunities for learning and development that were presented to the ministry were not utilized (e.g. failure by a technical working group to develop proposals for a study tour to Brazil). Which of these effects have been observed or experienced by Zambia's small-scale male and female farmers? This evaluation also notes that the programme was primarily not designed to have impact at farmer level given that there are no direct targets or indicators at farmer level. The timeframe was also too limited for the programme to achieve any trickle down impact at farmer level. However, the programme was designed to enhance performance of ministry staff so that they would improve service delivery to farmers and improve their output in the long run. As already alluded, it was not possible to see major changes in the actual performance of staff during the tenure of the programme as attitudes and work culture change take time to happen. On the other hand, impact assessments based on the “with” and “without” project scenarios, show that access to ICT extension aids and service delivery improved much better as opposed to a scenario without the project. The PEP programme also made a difference overall within the ministry and with regards to service provision by
Page | 16
both the ministry departments and their sister institutions. These improvements would not have occurred without PEP programme support. What could have been done differently for the project to achieve wider impact? To achieve wider programme impact there was need for consolidation and strengthening of induction training. This is because the current approach is focused strongly on personnel and administrative procedures but remains weak on technical/subject matter content, There is also need for consolidation of staff skills beyond introductory training/ information giving (e.g. mentoring and coaching). This being a performance programme, there should also have been much greater attention paid to performance assessment across the ministry As an illustrational example, APAS was still as irrelevant at the end of the programme as it was at the beginning and no effective measures were taken to change that. Again, these are issues the follow-up phase (PEP II) could draw lessons from and improve upon.
2.5 Sustainability
To what extent has the support provided generated effects that can economically and institutionally be sustainable? The leadership and management training transformed the capacities of individual staff which can be cascaded down to the peers and subordinate staff (multiplier effect). Other examples of sustainability included an improvement in the capacity of HR staff and PPD staff (admittedly more at HQ than in the provinces and districts), an improvement in ICT skills at many levels, and improved efficiency as a result of the introduction of the HRIS. Beyond those accomplishments, the programme became involved in a lot of activities that should rightly have been undertaken and funded from the ministries’ recurrent budgets, and therefore objectively unsustainable (e.g. emoluments, induction and in-service training, and ICT equipment). In addition, unless there was a follow-up phase, as is the case, the chances for sustainability of a number of those initiatives were low. It is, therefore, paramount that the follow-up phase of the PEP programme should look at ways of ensuring the sustainability of key programme initiatives. Examples include the need for building upon the lessons from PEP I (a change management strategy, refocus attention from addressing the hardware issues to the software issues in agricultural extension, and use of Innovation Platforms to enhance learning and strengthen required capacities for innovation, and operationalising performance appraisal and incentive systems. ). Those, and other proposals, are elaborated under the later section on Strategic Recommendations for the Operationalization of PEP II. Sustainability of most of the PEP programme initiatives could also be assured if GOZ-MOF reinstates adequate recurrent budgets for both MoA & MoFL.
Page | 17
3. Overall Assessment of PEP
3.1. Programme Context
As a programme, PEP was a response to the widely held view by national stakeholders
around 2009 that the agricultural sector was struggling to achieve its undoubted potential
to make a vital contribution to economic growth, rural development, and poverty reduction
in the country. At the centre of those concerns and challenges was the then MOAC – the
ministry expected to lead sector initiatives and development programmes. It had
capacities that were known to be weak and it needed substantial revitalization of its
policies, structures, personnel, technologies, and extension services systems.
The EU, with the strong support of Cooperating Partners (SIDA, JICA, IFAD & Finnish
Cooperation), engaged the ministry and it was agreed that a comprehensive assessment of
the Ministry’s capacities be conducted. Following from that exercise, the Performance
Enhancement Programme (PEP) was conceived and designed. It was not envisioned as yet
another project, but as a programme with the ambition, and indeed, courage to take on the
wide and deep system-wide capacity challenges facing the Ministry.
PEP eventually started in 2012 with four very overloaded key result areas across the
ministry covering change management; sector policy, planning and financial management;
human resources and ICT; and monitoring and evaluation. The goal was to tackle all the
major areas of capacity weakness together, with little prioritization and over a relatively
short time frame. This ultimately proved to be the most fundamental weakness of PEP –
trying to do too many things in too short a time – a commendable array of activities were
started but many were never properly or fully completed, while some unplanned activities
were added for implementation at the behest of MAL.
3.2. Strengths and achievements
PEP initiatives in management and staff training, providing computers, developing data
bases, strengthening planning, monitoring and evaluation skills, and updating the curricula
of training institutes were all well appreciated by ministry staff. This was partly because of
the extent of the previous protracted neglect of those areas, coupled with the hope and
impetus that those initiatives brought to the ministry staff which benefited. Given the
absence of modern support facilities and systems for middle-tier professional and support
staff in the ministry, the programme support to improve ICT workplace technology,
training, and the development data bases and information systems made most MAL staff
familiar and proficient with computer and internet use, knowledge of data bases, and
reporting in their day-to-day work.
Page | 18
3.3. Weaknesses and challenges
A central expectation of stakeholders and a core purpose of PEP was to facilitate and
manage change actively across and within MAL. To begin to fulfill this role satisfactorily,
the programme needed to have and communicate widely a comprehensive change
management strategy spearheaded by champions of change from the senior levels of the
ministry. The programme did not have such a strategy, including gaining and sustaining a
strong and evidence-based commitment from senior management. The Change
Management Team (CMT) operated in the prevailing administrative mode of MAL, as a
Head Office based coordination unit, rather than leading a dynamic system-wide change
process across all levels of the ministry. The programme was not communicated
adequately or cascaded effectively to the provincial or district levels as the feedback and
records on information, training and evaluations at those levels clearly reveal. Both those
shortcomings will need to be addressed in the next phase.
The programme should have conducted a benchmarking exercise, such as a ministry-wide
staff survey, at the outset. For a performance improvement programme centered on staff
proficiencies, such a comprehensive exercise should have been mandatory so as to enable
meaningful assessments of progress in skills acquisition by staff over the course of the
programme. The generally disenabling conditions in the ministries mitigated against
optimal performance improvement (e.g. limited and late budget allocations and lack of
opportunities to put skills gained from training into practice in their work places). Periodic
appraisals on the constraining effects of those latter factors will be necessary for inclusion
in the next phase. There were also instances of wastage of resources, despite the prevailing
budget shortfalls in MAL. PEP funded an exercise to devise improved budget resource
allocation criteria for MAL which the ministry subsequently decided not to accept or utilize,
casting some aspersions on programme decision-making processes and the quality of
engagement and ownership of PEP by MAL.
3.4. Overall insights
Overall, the PEP experience was a learning journey for the EU and MAL that revealed many
instructive lessons for the design, planning, facilitation and implementation of such
programmes in future. A central one is that a programme should be implemented in a well-
defined programme mode – not as a collection of projects. Another is that the result areas
should be limited and not overloaded – the so called “doability” mantra. While PEP brought
benefits across all levels (HO, provinces and districts) of MAL, the exclusion of large
numbers of staff (half of the frontline field extension complement) from the allocations of
computer tablets led to disappointment, and perceptions of favouritism and exclusion, by
Page | 19
half of a very important cadre of MAL staff. Change management programmes and their
strategies need to challenge existing norms and modes and bring about demonstrable
change in the way public service ministries perform. To do that, a change strategy and
process was needed together with a way of knowing or gauging the point of departure and
then how well it is progressing and achieving. PEP II needs to do just that and avert the
pitfalls of the first phase.
In a nutshell, PEP certainly sought to achieve noble and worthy deeds. There is no doubt
that PEP set out to do the right thing and had some very worthy accomplishments. It
facilitated increased awareness, knowledge and insights among many staff on management
and performance issues in MAL. Its major drawback was that it attempted to do some of
those “right things” in the wrong way. Examples included embarking on a staff
performance improvement programme without conducting a comprehensive baseline
study on existing proficiencies; seeking to improve MAL budget allocation criteria without
first securing purposeful and definite commitments by the relevant GoZ ministries;
providing wide-ranging support to improving ICT capacity across MAL while being unable
to secure the commitment of senior management to put in place a dedicated unit to
optimize the management and benefits of the investment.
Page | 20
4. Programme Plans, Targets and Performance Outcomes
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
OVERALL OBJECTIVE
The agricultural sector contributes to growth and poverty reduction in Zambia
Target Increased national growth and reduced rural poverty. Indicators(s) Budget allocation to the sector according to CAADP objective to allow 6% growth and contribute to MDG 1 (halving poverty) Reduction in subsidies (FRA/FISP) to the sector to allow improvement of productive investment along CAADP lines.
Reduction in agricultural production, inter alia, is adversely weighing on growth prospects (TORs). Poverty levels among rural dwellers that make up the majority of the country's population have remained a significant problem with poverty levels at more than 77% for more than a decade and the gap between urban and rural poverty continues to widen (TORs).
Real GDP growth has been declining. This was 7.3% in 2012, 6.7% in 2013 and 5.7% in 2014. www.afdb.org 60% of the population lives below the poverty line, while 42% are considered to be in extreme poverty. www.worldbank.org Poverty levels have remained the same since 2010. Maize subsidy programmes consumed 52% of the 2014 agriculture budget.
PROGRAMME PURPOSE
The capacity of MAL is strengthened in an effective and sustainable way and their operations are improved
Target Strengthened MAL and improved service delivery Indicator(s): Number of new policies, legislation & plans approved / promulgated. Level of satisfaction of MAL staff with working environment. Percentage of filled positions against the establishment.
2 out of a target of 3 new policies crafted. Agricultural Transformation Plan and National Agricultural Policy. Revised Strategic Plans await approval. Baseline not completed. PEP final statistics indicated employment levels of 8,379 against a total staff establishment of 12,784 (66%).
PEP facilitated the development of a Strategic Plan by the ministry. PEP also supported the development of the National Agricultural Extension Strategy. Both are yet to be approved. No evidence of a baseline having been conducted. MOA & MFL statistics on staffing levels are inconsistent depending on the source. The number of extension officers in
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
Quality of planning, budgeting and reporting.
MoA now amongst best 50%, but still work to be done to ensure equitable distribution. Resource allocation work under way.
position increased from 48% to about 90% in the past 5 years (MOA KII). There are 1,400 camp extension officers in 1,810 camps giving a staffing level of 77% (PEP Ref Group). In ZARI the staff establishment is 60 – 70%. Significant changes and improvements still required in all three (3) areas.
OUTPUT 1
Change Management and Service Delivery improved
1. Change Management and Service
Delivery improved.
Target Adoption of a high performance culture focused on world class service delivery and customer satisfaction. Indicator(s): MAL structure reviewed and operationalised. Level of satisfaction of MAL staff with management performance. % of staff with reviewed job descriptions.
MAL structural review initiated. The programme put in place a Programme Steering Committee (PSC), Change Management Teams (CMTs), and Provincial Change Management Teams (PCMTs) to initiate and manage the change process. Survey not conducted. Old job descriptions, but there has been no updating since the last job evaluation exercise in 2010
Organizational review still at planning stage, awaiting approval of Strategic Plan (PEP Final Report). The PSC and CMT provided useful guidance to the programme, notably at the planning stages. The concept of “Champions of Change” was not integrated into the PEP programme. Baseline not completed (PEP Final Report). Restructuring has been delayed, awaiting approval of Strategic Plans (PEP Final Report).
Page | 22
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
OUTPUT 2
Sector Policy, Planning and Financial Management improved
2. Sector Policy, Planning and
Financial Management improved
Target MAL provides a clear policy framework for the sector, based on empirical evidence, disseminates this widely and uses it as the basis for planning its own operations and budget expenditures. Indicator(s): Number of policies/legislation reviewed or developed. Number of evidence-based policy advisory notes prepared annually. Number of policy dialogue meetings held annually at national, provincial and district level. Number of SAG meetings per year.
Three (3) policy documents have been completed out of the planned five (5). (PEP Final Report). Two (2) evidence-based policy advisory notes have been prepared. Three (3) policy dialogue meetings held annually. One (1) SAG meeting per year.
The two policies that have not been completed for various reasons are: The Strategic Plan and National Agricultural Policy. The former has been recently split into two strategic plans, one of each new ministry. The two (2) evidence-based policy advisory notes are the Effects of FISP reforms and Effects of Fish Ban on farmers’ livelihoods. Policy dialogue meetings at HQ level on E-voucher, Agricultural Transformation Plan, and M&E. At Provincial level annual review meetings held in three provinces with support from PEP. One AgSAG meeting supported each year, on specific themes including approval of the drafts of the Strategic Plan, National Agricultural Investment Plan and stakeholder contributions to the sector (GAFST framework).
Page | 23
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
MAL budget expenditure as % of budget released. % of departmental and district/provincial plans that are structured in line with the framework of the strategic plan.
Not specified. But should be well over target with increased FISP allocations alone (PEP Final Report). Whilst not approved, SPs remains among several policy documents used for MAL planning and budgeting.
MoF remain committed to PFM reforms. GRZ-wide M&E system under development, District and provincial plans being brought within the framework of the strategic plan, but also the NAP and NAIP. Progress has been made in harmonising these documents, but they still retain different structure and emphases.
OUTPUT 3
Human Resources Management and ICT improved
3. Human Resources Management
and ICT improved.
Target To begin to see a Human Resource Management function demonstrating a clear understanding of the Ministry’s business and become a strategic business partner, challenging and supporting the line departments and regions, and taking a lead on cultural and change management issues that inspire MAL employees to contribute to the continuous improvement in the service given to customers. Indicator(s): Payroll cleaned up for retired staff.
Completed in PE3 (2015). (PEP Final Report).
HRIS system modified, MAL staff trained, data entry complete for 95% of the staff. The activity would have been completed earlier if it were not for lack of commitment and sense of urgency on the part of Senior HRA Managers that were expected to lead and drive the process. (PEP Final Report).
Page | 24
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
Outstanding payments of retirement packages for MACO/MLF retirees made. HR Strategy developed. Number of training programmes re-introduced. % of staff trained in line with re-introduced programmes. HRIS operational. Number of departments regularly updating their website module.
PEP contributions complete and audited. Development of HR Strategy still at planning stage. More than three (3). These include the leadership, management and ICT end-user training, three (3) in-service training programmes, and induction training. 90% of targeted provinces, 10% of staff trained nationally. Training included Leadership, Management and Supervisory courses (PEP Final Report). Data entry took long. All departments providing information to NAIS for uploading onto website.
Audit was lengthy due to poor record keeping, but eventually competed. HR Strategy not yet developed A number of new courses developed and run with PEP support. MAL should include these in their recurrent budgets in the future, not to rely so much on external funding and overseas training. Induction of new staff still remains a challenge owing to insufficient funds allocated to this activity. Data entry completed by the end of October 2015. Testing and commissioning the system followed. More pressure should be applied by senior management to ensure on-going updating of website.
Page | 25
OUTPUT 4
Monitoring and Evaluation improved
4. Monitoring and Evaluation
improved.
Target Effective and efficient M&E system leading to evidence-based policy, planning and investment strategies. Indicator(s): % of quarterly M&E reports generated timely i.e. in line with M&E system M&E database system operational
Results not specified. M&E database under development and being piloted.
The template to be used for M&E reporting across government ministries is still being worked out. (PEP Final Report). M&E database developed but yet to be commissioned. .
CROSS-CUTTING ISSUES
Gender, HIV & AIDS improved
Gender
Target
Gender awareness is mainstreamed in all MAL activities. Indicator(s)
MAL Gender Guidelines and Checklists developed and applied to planning to ensure equal opportunity and access to both women’s and men’s needs and interests.
Gender mainstreaming guidelines were developed and distributed at all levels. A TOT for mainstreaming for gender was also conducted for senior management and provincial staff.
Gender mainstreaming, sensitization and training conducted.
HIV & AIDS
Target
HIV and AIDS is mainstreamed in all MAL activities.
Page | 26
Indicator(s)
MAL Wellness and HIV/AIDS Workplace Policy developed and staff sensitised at all levels.
The MAL Wellness and HIV/AIDS Workplace Policy was put in place and senior management were sensitized.
MAL Wellness and HIV/AIDS Workplace Policy developed and sensitization initiated.
Page | 27
5. Findings, Conclusions and Recommendations Output(s) Findings
Conclusions Recommendations
OVERALL OBJECTIVE
The agricultural sector contributes to growth and poverty reduction in Zambia
Real GDP growth has been declining. This was 7.3% in 2012, 6.7% in 2013 and 5.7% in 2014. www.afdb.org 60% of the population lives below the poverty line, while 42% are considered to be in extreme poverty. www.worldbank.org Poverty levels have remained the same since 2010. Maize subsidy programmes consumed 52% of the 2014 agriculture budget.
Overall the programme has had very little direct impact so far on the agricultural sector, priorities, maize subsidies, economic growth and poverty reduction in Zambia.
There is need for the programme to play a more influential role in policy formulation, implementation and review.
PROGRAMME PURPOSE
The capacity of MAL is strengthened in an effective and sustainable way and their operations are improved
PEP facilitated the development of a Strategic Plan by the ministry. PEP also supported the development of the National Agricultural Extension Strategy. Both are yet to be approved. No evidence of a baseline having been conducted. MOA & MFL statistics on staffing levels are inconsistent depending on the source. The number of extension officers in position increased from 48% to about 90% in the past 5 years (MOA KII). There are 1,400 camp extension officers in 1,810 camps giving a staffing level of 77% (PEP Ref Group). In ZARI the staff establishment is 60 – 70%.
Efforts to provide strategic directions to operations have been hindered by the structural decisions to have two ministries and protracted decision-making processes. With no baseline, monitoring and evaluation of programme progress is difficult and cumbersome. There doesn’t seem to be coherent, centralized and reliable information on personnel and staffing in the ministries.
PEP II should facilitate the development, approval/promulgation of strategic plans by both the MOA and the MFL. PEP should also go beyond policy formulation by promoting implementation in order to improve service delivery. Baseline studies in target districts should be given priority during the orientation phase of PEP II. The Human Resource Information System (HRIS) and supporting data bases need to be more professional and reliable.
Significant changes and improvements still required in planning, budgeting and reporting.
While the programme made valiant efforts to improve these three areas, the ministry’s low capacity base at the outset, coupled with the ministry’s current dysfunctional resource allocation and disbursement system which undermines bona fide attempts to strengthen professional planning, budgeting and reporting.
The two ministries need to review and reform the existing budget system and take advantage of Core Functions Analyses exercises to guide and assist them in modernizing their budgeting process, especially in view of the impending decentralization process.
OUTPUT 1
Change Management and Service Delivery improved
1. Change Management and Service
Delivery improved.
Organizational review still at planning stage, awaiting approval of Strategic Plan (PEP Final Report). The PSC and CMT provided useful guidance to the programme, notably at the planning stages. The concept of “Champions of Change” was not integrated into the PEP programme. Baseline not completed (PEP Final Report). Restructuring has been delayed, awaiting approval of Strategic Plans (PEP Final Report).
While these committees were effective at the national level, they were not sufficiently effective at the provincial and district levels. Without the baseline, it will be difficult to make assessments of the level of satisfaction of MAL staff and PEP performance in general. Restructuring remains key for effective change management, enhanced performance and service delivery.
Organizational review to be a priority for PEP II, The PEP focal persons at provincial and district levels need to be included in higher level change management teams. There is also need for active change management and facilitation teams at provincial and district levels. It is critical that benchmarking surveys be conducted. These can take the form of staff surveys or farmer perception surveys. This can be implemented during the orientation or start up phase of PEP II. There is need to hasten the development, finalization and approval of separate strategic plans for the MoA and MFL that will serve as the basis for restructuring.
Page | 29
Output(s) Findings
Conclusions Recommendations
OUTPUT 2
Sector Policy, Planning and Financial Management improved
2. Sector Policy, Planning and
Financial Management improved.
The two policies that have not been completed for various reasons are: The Strategic Plan and National Agricultural Policy. The former has been recently split into two strategic plans, one of each new ministry. The two (2) evidence-based policy advisory notes are the Effects of FISP reforms and Effects of Fish Ban on farmers’ livelihoods. Policy dialogue meetings at HQ level on E-voucher, Agricultural Transformation Plan, and M&E. At Provincial level annual review meetings held in three provinces with support from PEP. One AgSAG meeting supported each year, on specific themes including approval of the drafts of the Strategic Plan, National Agricultural Investment Plan and stakeholder contributions to the sector (GAFST framework). MoF remain committed to PFM reforms. GRZ-wide M&E system under development,
The government has been dragging its feet in approving the policy documents. This can be a reflection of lack of commitment, lack of political will or a failure to assume ownership of the PEP programme. Policy impact assessments are critical for improving policy relevance, impact and harmonization. Policy dialogue meetings initiated but more still needs to be done. These are yet to be fully decentralized. The Agricultural Sector Advisory Group served as the all stakeholder platform. The PEP programme, however, failed to work with the Agricultural Consultative Forum (ACF) for various reasons. Very little progress has been achieved in adopting the recommended budgetary allocation criteria/formulae.
There is need for better commitment, political will and for the government to assume ownership of the PEP programme to ensure greater support of the programme’s initiatives. Policy formulation, implementation and review should never be once-off processes, but rather provide platforms for the inclusive formulation of objective and consistent policies, constant reviews, and linked communication systems. To achieve greater impact policy dialogue should be facilitated and supported at all levels including the provincial and district level. Such platforms should also be initiated in all provinces and districts. The AgSAG should be further re-vitalised to provide a continuous dialogue between GRZ, CPs and stakeholders, rather than only meeting when a critical document has to be developed. The ministries continual problems with their budgetary processes and disbursement shortfalls need to be considered by the AgSAG group.
Page | 30
Output(s) Findings
Conclusions Recommendations
District and provincial plans being brought within the framework of the strategic plan, but also the NAP and NAIP. Progress has been made in harmonising these documents, but they still retain different structure and emphases.
Policy formulation exercises in themselves have limited value unless they are linked to and used for implementation.
The MoA and MFL strategic plans should be finalized as a matter of urgency.
OUTPUT 3
Human Resources Management and ICT improved
3. Human Resources Management
and ICT improved.
HRIS system modified, MAL staff trained, data entry complete for 95% of the staff. The activity would have been completed earlier if it were not for lack of commitment and sense of urgency on the part of Senior HRA Managers that were expected to lead and drive the process. (PEP Final Report). Audit was lengthy due to poor record keeping, but eventually competed. HR Strategy not yet developed More than three (3). These include the leadership, management and ICT end-user training, three (3) in-service training programmes, and induction training.
Payroll cleaned up. All outstanding payments of retirement packages cleared. The HR Strategy is long overdue. A number of new courses developed and run with PEP support.
There is need for a change management strategy that can facilitate a mindset change during the orientation phase of PEP II to ensure a common vision and assumption of programme ownership by all MoA and MFL staff. Record keeping to be improved. The HR Strategy is the overarching guiding documents on how to proceed and deal with critical HR issues. Its development should therefore be a priority. MoA and MFL should include these new courses in their recurrent budgets in the future, not to rely so much on external funding and overseas training. This will ensure sustainability of the PEP initiative.
Page | 31
Output(s) Findings
Conclusions Recommendations
Induction of new staff still remains a challenge owing to insufficient funds allocated to this activity. Data entry completed by the end of October 2015. Testing and commissioning the system followed. All departments providing information to NAIS for uploading onto website.
Induction of new stuff remains a peripheral priority for government. HRIS operational. Innovation is required to ensure that websites are consistently updated.
The induction of new stuff is key for enhancing performance and for improving service delivery. It should be a top government priority for both implementation and funding. HRIS to be consistently maintained and updated. There is need for incentives and awards to ensure that websites are consistently updated. More pressure could also be applied by senior management to ensure on-going updating of website.
OUTPUT 4
Monitoring and Evaluation improved
4. Monitoring and Evaluation
improved.
The template to be used for M&E reporting across government ministries is still being worked out. (PEP Final Report). M&E database developed but yet to be commissioned.
The government, with EU support, is developing a government-wide M&E system. Encouraging progress has so far been achieved in M&E.
M&E, which is one of the three areas to be addressed by PEP II, should incorporate lessons learnt and recommendations from the government-wide M&E system. Commissioning, adoption and utilization of the M&E database have to be hastened.
Page | 32
Output(s) Findings
Conclusions Recommendations
CROSS-CUTTING ISSUES
Gender, HIV & AIDS improved
Gender
Gender mainstreaming guidelines were developed and distributed at all levels. A TOT for mainstreaming for gender was also conducted for senior management and provincial staff.
Policies and guidelines put in place, but there is no dedicated full-time officer. The function is currently under PPD.
There is need for a full-time gender officer, preferably under the HRA departments.
HIV & Aids
The MAL Wellness and HIV/AIDS Workplace Policy was put in place and senior management were sensitized.
Policies and guidelines put in place, but there is no dedicated full-time officer.
The position of an HIV & AIDS coordinator should also be a full-time position i=under the HRA departments.
Page | 33
6. Programme Oversight and Management A Programme Steering Committee (PSC) provided oversight and policy direction and was
to meet twice a year while a national Change Management Team (CMT), comprising the
departmental directors, met quarterly to deal with programme implementation issues.
Management arrangements, in line with the Financing Agreement, had programme
supervision provided by the Director of the Policy & Planning Division (PPD), MAL who
was the Imprest Administrator while the Chief Accountant was the Imprest Accounting
Officer. In addition to those arrangements, a member of staff of the Policy and Planning
Department was appointed as a MAL/NAO Liaison Officer. The PSC struggled to have
meetings in accordance with the frequency stipulated in its ToRs; supervised PEP in an
uninspiring, administrative mode; and displayed little innovation in communicating PEP
across MAL e.g. no evidence of national seminars on change management processes or
interaction or lesson learning from other countries implementing agricultural sector and
ministry change programmes in SSA.
Provincial Change Management Teams (PCMTs) were nominally established at all
provincial head offices, with representation from the districts, but were not actively
engaged in programme implementation as the programme was centrally managed.
Provincial staff from MAL departments, however, were involved in coordinating PEP
supported activities (e.g. ICT equipment distribution, SAO & SEO training, HRIS data
collection).
The programme was supported by a team of two long-term technical assistants, comprising
the Team Leader and Institutional Development/Change Management Specialist and the
Human Resource Management Specialist. Programme administration was provided by the
TA team and a Programme Support Unit based in the offices of PMTC (Zambia) Limited,
comprising an accountant, an administrator and a driver.
The CMT should give more priority to strengthening planning processes at provincial and
district levels and facilitate more interaction, inputs and meetings with ministries middle
management staff. Given the stated commitment for PEP II, supported by MoA & MAL, to
shift capacity development efforts and resources more substantially to the provinces
during the next phase, it is recommended that there be Provincial and District Change
Facilitation and Implementation Teams in target provinces/districts, and that the provincial
teams have formal representation on the PSC.
The MoA and MOFL management, at both provincial and district levels, should ensure
integrated planning i.e. PEP activities as part of the normative ministry quarterly/monthly
planning processes with reviews conducted by M&E officers at district and provincial
levels. As decentralization becomes imminent, it is suggested that PSC meetings be held
Page | 34
rotationally in focal PEP provinces and involve periodic direct presentations from the
Provincial & District Teams. This will facilitate the much needed closer interaction and
stronger mutual understanding between oversight and implementing entities in the second
phase of the programme.
Page | 35
7. Financing and Budget Despite its delayed start, the programme succeeded in spending a high proportion of its allocated funds (see 2.3 on Efficiency).
The amount of the overall budget allocated and disbursed under PE 2 was very high, most probably arising from the need to
compensate for the delayed start-up. While HRM & ICT allocations were the highest elements in the overall budget, the
inability to make provision for the procurement of tablet computers for half of the field extension staff complement was a
major omission in terms of financial planning, equitable treatment of staff, prudent use of funds, and programme reputation.
For the future, there is need for a timeous inception phase and the addition of an exit phase with a separate PE.
and age structure, profile of farmer groups and associations and commodity foci, profiles of
NGOs active in district, MOA/MFL staff lists, agri-input companies and dealers, names and
enterprises of lead/high performing farmers, young farmers groups/clubs ..et al. Again,
depending on the performance outcomes achieved in the 10 districts, this can be extended
to other districts – and with lessons and insights -- probably can be completed more cost
effectively than for the first 10.
As recommended in the final evaluation of PEP 1, baseline surveys will be conducted in all
focal districts to benchmark the extension capacities and provide a basis for assessment of
progress during the second phase.
9.4. Learning and Innovation Platforms for Research and Extension
The need to “institutionalize joint research-extension planning and strengthened
dissemination” (Action Fiche 2.2) is fully acknowledged by ZARI, MoA & MFL and how best
to do this is the key challenge for the ministries, their agriculture and livestock
departments, research institutes, farmers associations, and private sector stakeholders.
It is recommended to facilitate and support the setting up of innovation platforms ( Box
XX) comprising personnel from research, extension, farmers and their associations, ago-
dealers, NGOs, and the private sector. Zambia has some experience of such platforms at
district levels to promote the adoption of conservation agricultural practices (2010) and
this, coupled with considerable SSA experience over the past 15 years, should be harnessed
to inform the setting up of innovation platforms under PEP II.
44
There are 10 ZARI research stations located around the country and three should be
selected during the Orientation Phase ( in close consultation with ZARI, MOA & MFL, ACF,
NFU) to establish learning and innovation platforms, each with circa 10-12 members
initially, including PACOs and DACOs from the hinterland province and districts. Together,
preferably under a rotating chairperson (extension, research, farmer association leader),
and meeting twice a year, the members facilitate multi-stakeholder input and feedback on
local agricultural development priorities, farmer demand, and the technical services and
training responses needed from research and extension for the focal hinterland farming
systems and enterprises. Based on identified needs, joint and integrated training
programmes covering technical subjects, farmer organization & market development, and
value chains (3-5 days involving ZARI, DOA/Livestock, SCCI, NGOs) should be planned and
organized for District staff, CEOs and farmer associations/groups.
Innovation platforms should arrange for topic/subject sessions and invite agricultural
practitioners and experts periodically to provide inputs and advice on chosen subjects. The
learning and innovation experiences with the three initial innovation platforms should be
carefully evaluated after two years and comprehensively documented as case studies and,
depending on the outcomes, further platforms considered for initiation and support in the
subsequent phases of the programme.
9.5. Incentives to Recognize and Reward Distinctive Achievements in Performance
Long-run experiences with workforces globally (in both public and non-public
organizations) provide ample evidence of how incentives can spur and boost the
performance of individuals and teams across all cultures and nations. Awards have been
successfully used in some countries in the SADC region over the past 20 years as one way
to recognize and spur improved performance in public extension service systems.
While not included per se in the PEP II Action Fiche, it is recommended that incentives and
awards should be included, probably under the following areas (Action Fiche: 1.4, 2.1, 2.2,
& 2.3). The incentive/award system would be first discussed with public service ministries
in the context of revitalizing the public service performance appraisal system. If given the
go-ahead, it would be introduced under PEP II for the first 2 years after which there would
be eventual co-sponsorship (50:50) by the ministries and PEP over the last two years, with
the ministries then mainstreaming the principles and practices of incentives across their
key performance areas,
• Under 1.4, MOA/MFL individuals that achieve outstanding merit as “Champions of
Change” in leading the change process or highly motivated teams would receive, for
example, the PEP Change Management Award (annual?). PEP peers, a HR representative,
45
and the TA TL would develop the award criteria for approval by the PSC . The prize
could be a sponsored place on an appropriate training course/programme linked to
advancing their professional or personal development.
• Under Result 2, an annual award for the Best Block or Camp Extension Officer of the
Year is commended for each ministry. The criteria would be drawn up by a group
comprising peer CEOs, Chief Extension Officer, HR Officer, and TA Extension Advisor.
The award could be sponsored participation in a relevant study tour related to field
extension systems or an appropriate training course/programme linked to advancing
their professional or personal development. There is plenty of scope for awards for
other job categories and for other areas e.g. Best District Extension Performance –
possibly a biennial award.
Other possible areas for awards could be Best District Profile Award, Best District Team of
the Year -- especially for the new districts targeted under PEP II. Under Action Fiches 1.3,
there could be an annual award for Best Input to the Ministry Website. Other worthy areas
and awards can be chosen or nominations/ideas sought from staff by the Change
Management Teams.
9.6. Focal Districts for PEP II
Based on extensive discussions during the evaluation with senior management and staff in
MOA/MFL, it is proposed to focus primarily on the new districts of which 30 have been
created over 10 years or so and are spread across most provinces. It is proposed to focus
PEP II on circa 30 districts in total, 20 of which would be the new districts. An overall
guiding criterion, based on discussions with ministry management, for all PEP II focal
districts would be that there is no other CP or NGO active in them currently.
The principal reason for focusing decentralized development on 20 “new districts” is that
they are recognized to have “foundation capacity development needs” in developing basic
skills and supports for agricultural extension planning and services provision. A GRZ
programme like PEP needs to begin to correct what have been described by PPD as the “the
imbalances in development” where remote and peripheral regions/districts have been
marginalized and experienced relative neglect over the past 15-20 years.
The key guiding criteria for targeting these districts is that:
• Poverty levels are high among communities
• The nutrition status of households is generally poor with relatively high levels of child malnutrition and stunting.
46
• They were created relatively recently
They have relatively high numbers of newly recruited staff (especially MFL) all of
whom require foundation training in providing extension services for resource-poor
communities.
So there is both a need and an opportunity to develop core capacity based on modern
extension service delivery systems, including management and staff training in
participatory extension approaches that focus strongly on assuring food and nutrition
security at household levels. Those districts present an opportunity to provide integrated
extension training in communities and implement the operational guidelines for food and
nutrition officers in collaboration with CEOs, thus mainstreaming nutrition into the
extension systems of both MOA and MFL.
The other 10 districts would be selected based on the following criteria:
• Management and staff have had “threshold” capacity development to provide
extension services based on their experiences over 5- 10 years with donor-funded
projects (e.g. SIDA, JICA, IFAD, World Bank) or with NGOs (e.g. Concern Worldwide,
World Vision, Self-Help Africa).
• The districts have considerable potential to achieve impacts in the commercialization
of smallholder agriculture and in contributing to local economic and sector growth.
• There is scope to develop public-private partnerships for commodity enterprises and
value chain development.
• The districts have relatively good infrastructure including feeder roads and proximity
and access to markets.
Case studies on learning and innovation from all 30 districts would be continuously and
comprehensively documented; used in training programmes and for out scaling to other
districts; and be on ministry websites.
9.7 Knowledge Management in Ministries
PEP I had major achievements in improving farmer registration, contributing to the
enhancement of the M&E system, renovating and relaunching the website, and supporting
the provision of introductory training in ICT in MAL. PEP also advised and advocated for a
fully-fledged ICT Unit in the ministry, but this was not acted upon by the end of the project.
Further strengthening of the ministries information systems across an array of areas is
proposed (Action Fiche: 1.3). To operationalize this commitment effectively, it is proposed
47
that the ministries adopt a more coherent and integrated approach to their “itsy-bitsy” and
disjointed collection of activities in data and information systems management and
operations. PEP II should have a strong knowledge management component to
professionalize and bring cohesion to the ministries information systems from data
collection/capture right through to storage, access and dissemination.
Advantage should be taken of the CFA to assess the current situation thoroughly and, based
on previous PEP I consultancy exercises and lessons from experiences from other
comparable ministries in COMESA countries, develop and capacitate a knowledge
management system for both ministries, including appropriate structures, systems and
staffing for modern information management across all departments and units.
48
Annex 1: Detailed Evaluation Method, Limitations and Tools
APPROACH AND METHODOLOGY
Preamble
The focus of the methodology for this evaluation was on the completion of two separate, but
closely linked, assignments:
1) The final evaluation of PEP; and,
2) The development of recommendations for PEP II.
Phases & Tasks Involved
The central focus was on the assessment of PEP achievements under the national prevailing
economic and institutional realities; key indicators and modalities to achieve result-oriented
approaches; and core institutional capabilities and commitments required to underpin
improved performance by key sector actors; and the quality of outcomes, results and lessons
from the focal interventions.
This evaluation undertook to further enhance and deepen understanding of the cause and
effect linkages between activities and results as a key learning platform for the European Union
and its partners on agricultural sector performance enhancement and capacity development
programmes to understand not only (1) what works and what does not, and (2) critically why,
and under what circumstances – the latter providing qualitative insights to inform future
programme designs, implementation modalities, management, operations, and funding..
This cross cutting and multi-intervention evaluation was implemented in accordance with the
guidelines for the OECD DAC Evaluation Criteria, including those on capacity building and
development. Under the guidance of the Reference Group, the team focused its work on the
four key output areas as delineated in the ToRs. The methodology to complete those phases
included the following sequential steps and processes:
INCEPTION PHASE
1. Initial Meetings with EU Delegation staff and key MOA/MFL personnel involved with PEP,
including implementing partners as advised by the Reference Group/EUD.
2. Inventory and review (desk studies) of all relevant project documents including
preparatory processes; contracts; proposals; log frames; annual plans of operations; work
49
packages; baseline surveys, needs assessments; training programmes, reports, and interim
evaluations.
3. Briefing meetings before the start of the field phase at the EU Delegation/ Reference Group
with staff and key stakeholders presenting the findings of the desk phase, as well as the work
plan -- envisioned approach, methodology and itinerary/organization -- for the field phase. The
team proposed, discussed and agreed with the Reference Group the key evaluation questions
(as per the 5 OECD DAC Criteria) to be addressed during the evaluation.
FIELD PHASE
4. Field Phase: Based on guidance & advice from the Reference Group, an itinerary of
interviews, meetings and discussions was held with (1) key personnel involved in PEP
oversight, management, and implementation including relevant GRZ ministries (e. g. PSMD),
(2) institutional implementers and participants in PEP and (3) local institutions or organizations
(public sector/NGOs/Farmer Organisations) involved in PEP capacity development activities,
and (4) cooperating partners including IFAD, JICA, and Finnish Cooperation and (5) multilateral
agencies involved in agricultural sector programmes such as the World Bank, & Africa
Development Bank.
5. Analyses of Programme Performance: Using a progress and performance matrix grid, an
appraisal of all PEP key result areas (output & outcome) as per the PEP Log Frame, was initially
conducted, including status of performance reviews/or evaluations, analyses of outcomes and
results against targets, findings, conclusions and recommendations on PEP implementation
effectiveness. The four key result areas for PEP was assessed under the above matrix grid,
coupled with the key evaluation questions.
This was combined, as appropriate, with more in-depth performance assessment(s) in key
process areas (also using a systematic matrix approach) to examine the log frame, plans of
operations, targets, indicators, achievements, and outcomes. The key lessons arising, and
recommendations covering institutional capacity development processes, including the
appropriateness of change management and organization development (OD) approaches
implemented by PEP was also assessed.
The overall programme was assessed for consistency with the EU country strategy and
complementarity with other donor programmes. This responded to the key evaluation
questions derived from the five OECD-DAC criteria and the two EU specific criteria (EU added
value in design and implementation & coherence with the EU country strategy, member states
and other donors). Following the above comprehensive analysis, the hypothesis for PEP as a
performance enhancement programme was also systematically examined and verified.
50
6. The follow-through actions on the Mid-Term Evaluation Report, (September 2014) was
assessed in the context of the subsequent implementation and completion of the programme.
A perspective was elaborated on how effectively and efficiently the PEP four output areas were
implemented, including overall resource utilization, using the OECD DAC evaluation criteria.
SYNTHESIS PHASE
7. Across all key outcome areas, a Consolidated FCRs (Findings, Conclusions and
Recommendations) Matrix across all projects was completed by the team and the Reference
Group was fully debriefed via a slide presentation on the key findings, conclusions, and
emerging recommendations coming from the evaluation. The priorities for operationalizing PEP
II was also identified from the FCRs and presented for validation to the stakeholders workshop
at 15 below.
8. Drafting of final report including narrative based on FCRs including the emerging priorities
for PEP II.
9. Assessment of the major lessons learnt for ongoing/future design and implementation of
performance enhancement programmes. This included the priorities for design of such
programmes; types of framework(s) to guide programmes; modalities for planning and
operations, including M&E; and approaches and processes for capacity development. The
lesson arising from PEP was fed into into strategic recommendations for PEP II.
OPERATIONALISATION OF PEP II (FIELD PHASE)
10. Briefing meeting with Reference Group
11. Meetings with key stakeholders
12. Field Visits (3 days)
13. Drafting by team of provisional recommendations for PEP II
14. Debriefing of the Reference Group
15. FCRs was presented to a half-day Stakeholder Validation Workshop, broadly representative
of key principals and partners of PEP to discuss, obtain feedback on, and finally validate the
FCRs, including key lessons and recommendations for PEP II. The team was advised by the
Reference Group on stakeholder representation at the workshop.
REPORT FINALISATION
51
16. Following the completion of steps 1 – 15 above, including feedback from the Stakeholder
Validation Workshop and approval of recommendations for PEP II; and a final debriefing
meeting with the Reference Group, the draft final report of the evaluation was prepared by
the team within two weeks of the stakeholder validation workshop.
4. Guiding Evaluation Criteria
As per DAC OECD guidelines for programme and project evaluations, the following five criteria
guided the overall process:
Relevance – appraise congruence between the perception of what is needed as
proposed by the programme planners, and the reality of what was needed based on the
demands and economic circumstances of beneficiaries, government priorities and the
prevailing donor strategy. The intended outputs and outcomes of PEP was also be
assessed in terms of cross-cutting issues on gender, HIV/AIDS and environmental
sustainability.
Effectiveness – measure the extent of progress towards outputs or outcomes that has
been achieved. This includes three steps;
Measuring change in the observed output or outcome (i.e. progress made
towards changes to be observed for outputs or outcomes).
Attributing observed changes in output or outcomes to progress made in the
implementation of programme activities.
Judging the value of the change for the component beneficiaries – principally
staff in MoA & MFL.
Efficiency – measures how economically resources are being converted to results to
date. The focus is on exploring the extent to which resources are being used to produce
the intended outputs and if/how the resources available could have been used more
efficiently to achieve the intended results.
Impact – assesses the progress in changes (positive or negative, intended or unintended,
direct or indirect) brought about by the programme in counterpart capacity
development and changes in people’s well-being. Specifically, impact assessments
dwelled on what and how much change occurred at implementation level that is
attributable to the programme?
52
Sustainability – assessed the capacity to which present relevant social, economic,
institutional and other factors/players present are able to maintain, manage and ensure
project results endure into the future.
5. Theory of Change
The Theory of Change (ToC) was used to analyze the intervention logic of the PEP programme
and its translation into the logical framework, by exploring the linkages between the PEP
activities and the programme results. This is based on the understanding that an approach such
as the ToC – or one based on similar principles and logic – was used during the programme
design in identifying programme-level outputs, outcomes and impacts and linking these to the
activities under each of the four result areas under PEP: Change Management and Service
Delivery; Sector Policy, Planning and Financial Management; Human Resources Management
and ICT; Monitoring and Evaluation. The ToC is also used as a framework for analyzing the
linkages between PEP programme activities, the logical framework (log-frame) and PEP’s
Monitoring and Evaluation (M&E) system, all of which are built around a common set of
outputs, outcomes and impacts. This approach ensures a logical and systematic analysis of
cause-effect relationships in assessing and evaluating the programme.
The logic embedded in the ToC as outlined above is represented graphically in Figure 1.
Figure 1: A Generalized Theory of Change for PEP.
53
Sources: Patrizi, H and Patton, M.Q. (2010) “New Directions for Evaluation”, Evaluating Strategy, 128. Clark, H. (2004) Deciding the Scope of a Theory of Change, New York: ActKnowledge.
DIFFICULTIES ENCOUNTERED AND METHODOLOGICAL LIMITATIONS
The adoption of multiple evaluation approaches and tools was designed to improve on
analytical rigor, validation, accuracy and the reliability of collected information. As such,
reviews of literature from and related to the PEP programme, key informant interviews, focus
group discussions with secondary beneficiaries and the validation workshop were used to
ensure content validity, encompassing guidance on theoretical, conceptual and empirical
insights. Detailed discussions of the evaluation tools during the Inception Phase and their
subsequent revisions were also designed to improve on the reliability of collected information.
However, where challenges were encountered this was due to the adoption or use of the case
study in selecting provinces and the districts to visit during the Field Phase of the final
evaluation. For various reasons, it was unfeasible to conduct several case studies to allow for
greater generalization. Thus the generalization of findings might be an issue for debate. The
second phase of the assignment, i.e. operationalization of PEP II, was also conducted within a
“time squeeze”. This made it difficult for the consultants to undertake another field visit. The
consultancy team, however, used the opportunities presented by the initial key informant
interviews and field visit, to make search for insights for PEP II.
____________________________________
54
DATA COLLECTION TOOLS
PEP Final Evaluation - Evaluation Matrix based on Mission ToRs & DAC-OECD Criteria
Key evaluation areas Focal questions Data collection methods and sources of information
a) Relevance Core Evaluation Questions To what extent was the project design appropriate for the then
MAL's institutional CD needs? Have these needs changed since the mid-point of the project? What amendments in the concept, theory or implementation
modality should be made for a future phase of the project? Were the project institutional arrangements and mechanisms
relevant and have they contributed to increased ownership by the then MAL?
What changes should be considered for future interventions?
Literature Review (Programme Proposal, Project Estimates, etc.)
KIIs. FGDs.
b) Effectiveness Core Evaluation Questions To what extent have the Ministry's capacity and performance
been enhanced? Has flexibility contributed to the achievement of the purpose
or rather contributed to hindering it? Were the project's design and approaches conducive to
increased effectiveness? To what extent have the policies and guidelines developed on
HIV/AIDS and gender been implemented and/or mainstreamed (as recommended by the MTR)?
Literature Review (log frame, baseline & MTR reports)
Generating new partnerships or coalitions 6. New Implementation Know-How
Evidence in formulated plans, strategies and operations at unit/dept. levels
Evidence in implemented plans, strategies and operations at unit/dept. levels Sources: EU Toolkit for Capacity Development, 2010. UNDP: Capacity Development Guides, 2004- 2009. World Bank Institute: Capacity Development Results Framework, 2009.
58
Annex 2: List of Persons/Organizations Consulted
Name Position Organization Telephone/Cell Number
Email
EUD and PEP Reference Group
Matteo Sirtori Head of Section (Agriculture) EU Delegation + 260 211 250 711 [email protected]
Marion Michaud Advisor (Agriculture & Rural Development)
Annex 3: Literature & Documentation Reviewed To date the following documents have been fully or partially reviewed1:- 1. Vision 2030.
2. Revised Sixth National Development Plan (R-SNDP) 2013-2016.
3. National Agriculture Policy (NAP) 2004 – 2015.
4. Comprehensive African Agriculture Development Programme - Zambia Compact.
5. National Agriculture Investment Plan (NAIP) 2014-2018.
6. Draft Strategic Plan of the Ministry of Agriculture and Livestock, 2013-2016.
7. PEP Proposal, Financing Agreement (FED/2010/022-057), Start Up Programme Estimate and Programme Estimates no. 1, 2 and 3.
8. PEP Logical Framework – 18 July 2013 + December 2014.
9. Project deliverables including thematic studies, training materials, policy briefs, etc.
10. PEP Training Reports.
11. Results-Oriented Monitoring (ROM) Reports and Background Conclusions Sheets, 2012 and 2013
12. Participatory Review of the Ministry of Agriculture and Co-operatives Performance [...]", including an "Assessment of Current MACO Capacity, Capacity Development Needs and Challenges" and a "Strategy and Roadmap for a Performance Enhancement Programme in MACO/GRZ", Cardno Agrisystems Ltd, July 2009.
13. PEP Mid-term Evaluation Report – September 2014.
14. PEP Final Report – February 2016.
15. Activity and lessons Learnt Report.
16. PEP Quarterly Progress Reports.
17. PEP Meeting Minutes.
18. Participatory Extension Approaches (PEA) Assessment & Harmonization Study – July 2015.
19. National Agricultural Extension and Advisory Services Strategy (NAEASS) 2016.
20. HIV & AIDS and Wellness Policy for Agriculture Sector Document – February 2014.