-
Document of
The World Bank FOR OFFICIAL USE ONLY
Report No: ICR00004512
IMPLEMENTATION COMPLETION AND RESULTS REPORT
(LOAN 8446-MX)
ON A
LOAN
IN THE AMOUNT OF US$350 MILLION
TO THE
UNITED MEXICAN STATES
FOR A
SCHOOL BASED MANAGEMENT PROJECT
June 24, 2019
Education Global Practice Latin America And Caribbean Region
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
-
CURRENCY EQUIVALENTS
(Exchange Rate Effective January 29, 2019)
Currency Unit = Mexican Peso (MXN)
MXN 18.99 = US$1.00
FISCAL YEAR
July 1 - June 30
Regional Vice President: Axel van Trotsenburg
Country Director: Pablo Saavedra
Senior Global Practice Director: Jaime Saavedra Chanduvi
Practice Manager: Emanuela Di Gropello
Task Team Leader(s): Monica Yanez Pagans
ICR Main Contributor: Abril Alicia Ibarra Castaneda
-
ABBREVIATIONS AND ACRONYMS
AEL Local Education Authority (Autoridad Educativa Local) AGE
School-based Management Support (Apoyo a la Gestión Escolar) CEAI
Colegio de Estudios Avanzados de Iberoamérica CONAPASE Consejo
Nacional de Participación Social en la Educación CONOCER Consejo
Nacional de Normalización y Certificación de Competencias Laborales
CPS Country Partnership Strategy DGAG Directorate of School
Management (Dirección General Adjunta de Gestión) DGDGE General
Directorate of School Management and Education (Dirección General
de
Desarrollo de la Gestión Educativa) DGE General Directorate of
Evaluation (Dirección General de la Evaluación) DGEI General
Directorate of Indigeous Education (Dirección General de
Educación
Indígena) ENLACE National Evaluation of Academic Achievement in
Schools (Evaluación Nacional del
Logro Académico de Centros Escolares) FM Financial Management FY
Fiscal Year GDP Gross Domestic Product GNI Gross National Income
GoM Government of Mexico IBRD International Bank for Reconstruction
and Development ICR Implementation Completion and Results Report IE
Impact Evaluation ISR Implementation Status and Results Report IERR
Internal Economic Rate of Return INEE National Institute for the
Evaluation of Education (Instituto Nacional para la
Evaluación de la Educación) INEA National Institure for Adults
Education (Instituto Nacional para la Educación de los
Adultos) IPP Indigenous People Plan ITEA Tlaxcalteca Institute
for Adults Education (Instituto Tlaxcalteca para la Educación
de los Adultos) M&E Monitoring and Evaluation NAFIN Nacional
Financiera, S.N.C., I.B.D. NDP National Development Plan (Plan
Nacional de Desarrollo) OECD Organization for Economic Co-operation
and Development OIC Internal Control Body (Órgano Interno de
Control) PAD Project Appraisal Document PDO Project Development
Objective PEC Quality Schools Program (Programa Escuelas de
Calidad) PEC-FIDE Programa Escuelas de Calidad- Fortalecimiento e
Inversión Directa a las Escuelas PED Dignified School Environment
Program (Programa Escuelas Dignas) PEEARE Excelence School Program
to Tackle Educational Gaps (Programa Escuelas de
Excelencia para Abatir el Rezago Educativo)
-
PETC Full Time Schools Program (Programa Escuelas de Tiempo
Completo) PISA Program for International Student Assessments PLANEA
National Plan for Learning Assessment (Plan Nacional para la
Evaluación de los
Aprendizajes) PRE Education Reform Program (Programa de la
Reforma Educativa) PSE Education Sector Program (Programa Sectorial
de Educación) RCTs Randomized Control Trials SBM School Based
Management s.d. Standard Deviation SEB Subsecretariat of Basic
Education (Subsecretaría de Educación Básica) SEMS Subsecretariat
of Upper Secondary Education (Subsecretaría de Educación Media
Superior) SEP Secretariat of Public Education (Secretaría de
Educación Pública) SHCP Ministry of Finance and Public Credit
(Secretaría de Hacienda y Crédito Público) SIGED Education
Management Information System (Sistema de Información y Gestión
Educativa) SISAT Early Warning System (Sistema de Alerta
Temprana) SPD Professional Teacher Service (Servicio Profesional
Docente) WMS World Management Survey
-
TABLE OF CONTENTS
DATA SHEET
..........................................................................................................................
1
I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES
....................................................... 5 A.
CONTEXT AT APPRAISAL
.........................................................................................................5
B. SIGNIFICANT CHANGES DURING IMPLEMENTATION
................................................................9
II. OUTCOME
....................................................................................................................
10 A. RELEVANCE OF PDOs
............................................................................................................
10
B. ACHIEVEMENT OF PDOs (EFFICACY)
......................................................................................
11
C. EFFICIENCY
...........................................................................................................................
19
D. JUSTIFICATION OF OVERALL OUTCOME RATING
....................................................................
19
E. OTHER OUTCOMES AND IMPACTS
.........................................................................................
19
III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME
................................ 20 A. KEY FACTORS DURING
PREPARATION
...................................................................................
20
B. KEY FACTORS DURING IMPLEMENTATION
.............................................................................
21
IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT
OUTCOME .. 23 A. QUALITY OF MONITORING AND EVALUATION (M&E)
............................................................ 23
B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE
..................................................... 25
C. BANK PERFORMANCE
...........................................................................................................
26
D. RISK TO DEVELOPMENT OUTCOME
.......................................................................................
27
V. LESSONS AND RECOMMENDATIONS
.............................................................................
28
ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS
........................................................... 29
ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION
......................... 47
ANNEX 3. PROJECT COST BY COMPONENT
...........................................................................
49
ANNEX 4. EFFICIENCY ANALYSIS
...........................................................................................
50
ANNEX 5. BORROWER COMMENTS
......................................................................................
55
ANNEX 6. SUPPORTING DOCUMENTS
..................................................................................
57
ANNEX 7. THEORY OF CHANGE
............................................................................................
59
ANNEX 8. CHANGES TO KEY INDICATORS
.............................................................................
61
-
The World Bank Mexico School Based Management Project
(P147185)
Page 1 of 62
DATA SHEET
BASIC INFORMATION Product Information Project ID Project
Name
P147185 Mexico School Based Management Project
Country Financing Instrument
Mexico Investment Project Financing
Original EA Category Revised EA Category
Not Required (C) Not Required (C)
Organizations
Borrower Implementing Agency
United Mexican States Secretaria de Educacion Publica
Project Development Objective (PDO)
Original PDO The objective of the Project is to improve schools'
managerial capacity and parental participation to reduce dropout,
repetition and failure rates among PEC Schools and PETC Schools.
Revised PDO The objective of the Project is to improve schools'
managerial capacity and parental participation to reduce dropout,
repetitionand failure rates among Participating Schools.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 2 of 62
FINANCING
Original Amount (US$) Revised Amount (US$) Actual Disbursed
(US$) World Bank Financing IBRD-84460 350,000,000 350,000,000
322,064,712
Total 350,000,000 350,000,000 322,064,712
Non-World Bank Financing 0 0 0
Borrower/Recipient 469,950,000 469,950,000 469,950,000
Total 469,950,000 469,950,000 469,950,000
Total Project Cost 819,950,000 819,950,000 792,014,712
KEY DATES
Approval Effectiveness MTR Review Original Closing Actual
Closing 24-Oct-2014 09-Jul-2015 10-Oct-2017 31-Dec-2018
31-Dec-2018
RESTRUCTURING AND/OR ADDITIONAL FINANCING
Date(s) Amount Disbursed (US$M) Key Revisions 26-Jul-2017 122.56
Change in Project Development Objectives
Change in Results Framework Change in Components and Cost
Reallocation between Disbursement Categories Change in Legal
Covenants Change in Financial Management Other Change(s)
15-Aug-2018 270.48 Reallocation between Disbursement Categories
29-Nov-2018 303.77 Reallocation between Disbursement Categories
KEY RATINGS
Outcome Bank Performance M&E Quality
Satisfactory Satisfactory Substantial
-
The World Bank Mexico School Based Management Project
(P147185)
Page 3 of 62
RATINGS OF PROJECT PERFORMANCE IN ISRs
No. Date ISR Archived DO Rating IP Rating Actual
Disbursements (US$M)
01 12-Dec-2014 Satisfactory Satisfactory 0
02 22-Jun-2015 Satisfactory Moderately Satisfactory 0
03 15-Dec-2015 Satisfactory Moderately Satisfactory 70.00
04 21-Jun-2016 Satisfactory Satisfactory 122.56
05 23-Dec-2016 Satisfactory Satisfactory 122.56
06 30-Jun-2017 Moderately Satisfactory Moderately Satisfactory
122.56
07 08-Dec-2017 Satisfactory Satisfactory 225.36
08 26-Jun-2018 Moderately Satisfactory Satisfactory 270.48
09 17-Dec-2018 Satisfactory Satisfactory 303.77
SECTORS AND THEMES
Sectors Major Sector/Sector (%)
Education 100
Early Childhood Education 33 Public Administration - Education 1
Primary Education 33 Secondary Education 33
Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Social
Development and Protection 0
Social Inclusion 49 Participation and Civic Engagement 49
-
The World Bank Mexico School Based Management Project
(P147185)
Page 4 of 62
Human Development and Gender 0 Education 52
Access to Education 26
Education Financing 26
ADM STAFF
Role At Approval At ICR
Regional Vice President: Jorge Familiar Calderon Axel van
Trotsenburg
Country Director: Gerardo M. Corrochano Pablo Saavedra
Senior Global Practice Director: Claudia Maria Costin Jaime
Saavedra Chanduvi
Practice Manager: Reema Nayar Emanuela Di Gropello
Task Team Leader(s): Rafael E. De Hoyos Navarro Monica Yanez
Pagans
ICR Contributing Author: Abril Alicia Ibarra Castaneda
-
The World Bank Mexico School Based Management Project
(P147185)
Page 5 of 62
I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES
A. CONTEXT AT APPRAISAL
Context 1. At the time of Project Appraisal, Mexico's long-term
economic growth was somewhat below expectations, limiting a
potential rise in average living standards and more rapid progress
on poverty reduction. In 2013, its Gross National Income (GNI) per
capita was approximately 39 percent of the level observed in high
income countries in the Organization for Economic Co-operation and
Development (OECD),1 the same proportion observed two decades
earlier, signaling a lack of progress in economic convergence. Over
the past three decades, annual Gross Domestic Product (GDP) growth
averaged 2.4 percent and only 0.8 percent per capita. Growth
decomposition exercises pointed to insufficient average
productivity growth as the main cause of growth performance. To
raise productivity and unleash growth, the Government launched in
2013 an ambitious structural reform agenda and enacted legislative
changes in the areas of labor market regulation, education,
telecommunication and competition policy, financial sector
regulation, energy, and fiscal policy. There was a clear potential
for the reforms to enhance potential output growth, which at
Project Appraisal was estimated in the range of 2.5-3.0 percent. 2.
Low learning outcomes comprised the biggest challenge faced by
Mexico’s education system. Student learning, as measured by the
Program for International Student Assessment 2012 (PISA) placed
Mexico in the last position among OECD countries and in 53rd of the
65 participating countries.2 In terms of education coverage,
although Mexico was making significant progress in basic education,
there were still students left out of school. At appraisal, net
coverage was 87 percent for preschool (5 years-old), 96 percent for
primary (6 to 12 years-old), and 87 percent for secondary (12 to 14
years-old).
3. The Education Reform was one of the 2013-18 administration’s
priorities and aimed to accelerate growth and development by
improving the quality of education while reducing access and
achievement gaps between rich and poor. The 2013 Education Reform
mainly consisted of: i) establishing a professional system for
hiring, evaluating, training, and promoting teachers; ii) providing
full autonomy to the National Institute for the Evaluation of
Education (Instituto Nacional para la Evaluación de la Educación,
INEE) to oversee all evaluation functions throughout the education
system; (iii) establishing a federal census of education data and a
national Education Management Information System; and (iv)
fostering school autonomy and school-based management (SBM).
4. The Project supported the school autonomy component of the
2013 Education Reform. More autonomy at the school level sought to
improve school infrastructure, provide relevant materials, and
solve basic school operational problems, while promoting social
participation so that students, teachers, and parents, under the
school director’s leadership, could overcome their challenges.
Changes to the General Education Law, derived from the Reform,
stated that SBM programs should use evaluation results as feedback
for continuous improvement and to develop annual school improvement
plans (shared with the education authority and the school
community), specifying their priorities and linking them to 1
Mexico is an upper middle-income country and a member of the OECD
and the G20. 2 Mexico had slightly improved its Mathematics results
in PISA over time (from 387 in 2000 to 413 in 2012), but reading
results stagnated (from 422 in 2000 to 424 in 2012).
-
The World Bank Mexico School Based Management Project
(P147185)
Page 6 of 62
verifiable targets. In addition, three of the seven strategies
of the Programa Sectorial de Educación (PSE) 2013-18 focused on
schools as the units of change and improvement. These included: (i)
creating the conditions for schools to be at the center of the
education system by allowing them to receive the support needed to
achieve their objectives; (ii) increasing school management
capacity to improve learning outcomes; and (iii) strengthening the
relationship between the school and the broader community to
promote social participation, transparency, and accountability. The
school autonomy policy was mainly implemented through four
programs: (i) the Schools of Excellence to End Educational Delay
Program (Programa Escuelas de Excelencia para Abatir el Rezago
Educativo, PEEARE); (ii) the Dignified Schools Environment Program
(Programa Escuelas Dignas, PED); (iii) the Quality Schools Program
(Programa Escuelas de Calidad, PEC); and (iv) the Full Time Schools
Program (Programa Escuelas de Tiempo Completo, PETC). Under this
framework, the World Bank loan was designed to support the school
autonomy policy through PEC and PETC programs. 5. The financial and
technical support to schools partially provided through the
Bank-financed Project was the single most important means for
schools to exert their autonomy, granted by the 2013 Education
Reform. By supporting the school autonomy policy, the Project aimed
to improve schools’ managerial capacity and parental participation
to reduce dropout and improve the quality of education services in
public basic education schools in Mexico. In the long run, the
Project aimed to contribute to the Bank’s twin goals of eliminating
extreme poverty and boosting shared prosperity by improving
students’ future income through their staying in school longer and
learning more. The Project was also an important component of
services put forward in the World Bank Group’s Country Partnership
Strategy (CPS) for FY14-FY19 (Report No. 80800-MX), discussed by
the Executive Directors on December 12, 2013, which is fully
aligned with the goals of Mexico's National Development Plan (NDP)
for 2013-18.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 7 of 62
Theory of Change (Results Chain)
Project Development Objectives (PDOs)
6. The objective of the Project was to improve schools’
managerial capacity and parental participation to reduce dropout,
repetition, and failure rates among Participating Schools.3
Key Expected Outcomes and Outcome Indicators 7. The PDO was to
be measured through the following indicators:
a. Proportion of basic education schools in programs (PEC and
PETC) to strengthen SBM with a director that has a “sufficient”
level of managerial capacity.
b. Proportion of basic education schools in PEC with parent
associations participating in the design, monitoring, and
adjustment of the school improvement plan.
c. Dropout rate among basic education schools in programs to
strengthen SBM (PEC and PETC). d. Gross failure rate among basic
education schools in programs to strengthen SBM (PEC and PETC). e.
Repetition rate among secondary school students in schools in
programs to strengthen SBM (PEC
and PETC).
3 Participating Schools were clearly defined in the Project
Restructuring, the Loan Agreement, and all other relevant Project
documents as schools participating in the Programa de la Reforma
Educativa (PRE) and in Programa de Escuelas de Tiempo Completo
(PETC).
-
The World Bank Mexico School Based Management Project
(P147185)
Page 8 of 62
Components 8. The Project included three components: 9.
Component 1: Increasing School Autonomy and Parent Participation
(Original allocation: US$342 million IBRD funding; Actual
disbursed: US$321 million IBRD funding4) Provision of direct
financial support to PEC Schools to implement School Improvement
Plans (Ruta de Mejora) agreed with parent associations. The school
improvement plan is a strategic plan whereby the school’s technical
council (comprised by the school director and teachers) identifies
the school’s main challenges and goals and the means to achieve
them. The direct support amounts are defined by a resource
allocation formula using two criteria: exogenous conditions (i.e.
enrollment, level of marginalization) and performance in quality
indicators (i.e. student learning, graduation rate). Priority is
given to schools serving marginalized and vulnerable populations
and those with low achievement levels in the national standardized
test and high dropout rates. The resources are used to support
activities related to the school improvement plans, as opposed to
recurrent costs of the school’s everyday functioning. 10. Component
2: Improving Schools’ Managerial Capacity (Original allocation:
US$3.5 million IBRD funding; Actual disbursed: US$0 IBRD funding5).
Provision of technical assistance to PEC Schools and PETC Schools
to strengthen school-based management through:
a. Support for the development of a school dashboard for school
supervisors and directors including
performance indicators and best practices, among others. b.
Support for: (a) the development and implementation of a
capacity-building strategy for school
directors and supervisors on: (i) the use of the school
dashboard and its role in improving school management practices;
(ii) the use of a classroom observation tool; (b) the improvement
of schools’ managerial practices through the development and
implementation of capacity-building activities for parents; and (c)
the development of the information system needed to keep the school
dashboards running, relevant, and up to date, including the
preparation of guidelines describing an implementation and
maintenance protocol for dashboards.
11. Component 3: Research and Innovation (Original allocation:
US$3.6 million IBRD funding; Actual disbursed: US$0 IBRD
funding).
a. Provision of support for the development of a new instrument
to measure Participating Schools’ managerial capacity, through the
piloting, redesign (if necessary), implementation, and comparison
of alternative questionnaires to measure managerial capacity.
b. Provision of support to States to improve adherence to PEC’s
Operational Rules through the carrying out of workshops and
knowledge exchange activities.
c. Provision of support for carrying out an assessment in at
least two self-selected States which will estimate and quantify the
effect of the intervention supported by the Project.
4 The disbursed portion of the loan allocated to Component 1
represents 39.9% of the original projection of the Component cost
(US$804 million). The rest of the costs were financed by the GoM. 5
The GoM financed all costs for Component 2 (US$9.7 million) and
Component 3 (US$5.4 million).
-
The World Bank Mexico School Based Management Project
(P147185)
Page 9 of 62
B. SIGNIFICANT CHANGES DURING IMPLEMENTATION
Revised PDOs and Outcome Targets 12. The PDO was formally
revised in the July 26, 2017 restructuring to replace the
terminology “PEC and PETC schools” with “Participating Schools.”
This change had no substantive impact on the Project; therefore,
there were no major implications on the original theory of change
or the originally expected outcomes and targets. The revised PDO
reads: “the objective of the Project is to improve schools’
managerial capacity and parental participation to reduce dropout,
repetition and failure rates among Participating Schools.” The
change was needed since the PEC program, one of the programs
supported by the Project and stated in the PDO, was consolidated
into a meta-program called the Education Reform Program (Programa
de la Reforma Educativa, PRE). For consistency, reference to PEC
and PETC was replaced throughout the Project by "Participating
Schools." Participating Schools were clearly defined in the Loan
Agreement and other relevant Project documents as schools that are
part of PRE and/or PETC.
Revised PDO Indicators 13. None of the PDO indicators suffered
reductions in targets. While some indicators were replaced, their
replacements were simply more accurate measures of the same
outcomes. The PDO indicators were adjusted to reflect data
availability and the change in the Project beneficiaries from "PEC
and PETC Schools" to "Participating Schools” (mainly baseline
conditions and progress among PRE beneficiaries). Of the 5 original
PDO indicators, three were revised and two were replaced in the
July 2017 restructuring. For a full explanation of the changes by
indicator, please see Annex 8. The following are the revised key
indicators:
i. Proportion of a representative sample of Participating
Schools with a director that scores 3 or above in the World
Management Survey (WMS). Added to replace original PDO Indicator
1.
ii. Proportion of a sample of Participating Schools that
increase parental participation (score more than 1 in the
School-based Management Support (Apoyo a la Gestión Escolar, AGE)).
Added to replace original PDO Indicator 2.
iii. Dropout rate among (primary and secondary) Participating
Schools. iv. Primary and Secondary school gross failure rate among
Participating Schools. v. Secondary school repetition rate among
Participating schools.
Revised Components 14. The three Project components were
slightly revised in the July 2017 restructuring as follows:
i. Component 1. Increasing School Autonomy and Parent
Participation was adjusted to support the provision of Direct
Support to foster school autonomy and implement school improvement
plans in “Participating Schools” (PRE and PETC) instead of PEC
schools only. At the time of restructuring, the Government of
Mexico (GoM) requested that direct support to PETC schools also be
included under Component 1. Although this implied more complex
financial management of Project funds (funding two government
programs instead of only one), it also helped PETC schools benefit
from Component 1 and helped disburse Project funds more
promptly.
ii. Component 2. Improving Schools' Managerial Capacity was
adjusted to support the provision of technical assistance to
“Participating Schools” (instead of PEC and PETC schools) to
strengthen SBM. The reference to the school dashboards was also
removed because the Federal Government chose to have each state
select the interventions to improve managerial skills of school
directors and communities based on their specific context.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 10 of 62
iii. Component 3. Research and Innovation was adjusted to
provide support to States to improve adherence to Operational Rules
of "Participating Schools" (PRE and PETC) instead of PEC's
Operational Rules.
Other Changes 15. The Project's Financial Management (FM),
including flow of funds, was modified to reflect PRE arrangements,
which differ from those under PEC. In addition, FM included the
flow of funds to support SBM in PETC schools. The restructuring
also included a reallocation between disbursement categories to
provide more financing for Direct Support to Participating Schools
(Component 1). Finally, the Loan Agreement was amended to reflect
the above changes and make eligible expenses for the PETC
retroactive as of January 2016.
Rationale for Changes and Their Implication on the Original
Theory of Change
16. The July 2017 restructuring was needed to respond to the
consolidation of national programs to strengthen SBM in Mexico.
However, the description of the PDO, components and key activities
mostly remained unchanged. The overall objective of the Project
remained the same, since the Project continued supporting school
autonomy and managerial practices of schools, but now under: (i)
PRE's School Autonomy component by providing direct support to
schools to implement school improvement plans; and (ii) PRE’s
Managerial Practices component by developing and implementing
capacity-building activities for school supervisors and directors.
The PRE components supported by the Project follow the same
approach as the former PEC program. Therefore, the theory of change
remained unchanged.
II. OUTCOME A. RELEVANCE OF PDOs
17. The PDO is aligned to the current World Bank Group’s CPS for
FY14-FY19. It aligns with the CPS Pillar 2 “Increasing Social
Prosperity” as updated through the Performance and Learning Review
(Report No. 104752-MX) which has the expected objective of
“improved access and quality in target education programs”. The CPS
and the PDO are also consistent with Mexico’s National Development
Plan (NDP) 2013-18, which has five main components: Peace,
Inclusion, Quality of Education, Prosperity, and Global
Responsibility. All the NDP components emphasize improving the
quality of education and reducing access and achievement gaps
between rich and poor to increase productivity and long-term
growth. In addition, the Project fully supported the school
autonomy aspects of the 2013 Education Reform. Finally, the Bank’s
engagement with and support to Mexico, an upper-middle income
country, is highly relevant since around half of its population is
still poor. In addition, the evidence of best practices in key
education policies in Mexico potentially has a valuable
demonstration effect for other lower-income countries.
Assessment of Relevance of PDOs and Rating Rating: High
18. The relevance of the PDO is rated High since the Project
provided clear evidence of the alignment of the PDO to the expected
outcomes of the current CPS and Mexico’s most current NDP
(2013-18), as described above.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 11 of 62
B. ACHIEVEMENT OF PDOs (EFFICACY)
19. Even though the Project underwent a level 1 Restructuring in
July 2017, this Implementation Completion and Results Report (ICR)
does not conduct a split evaluation of the efficacy of the outcome
indicators before and after the restructuring since the scope of
activities and expected outcomes of the Project remained the same.
The change in PDO merely reflects support to the same activities
under different denominations in a recently consolidated government
program. From the outset, the Project was designed to support the
school autonomy policy as specific government programs evolve or
shift. Given that PEC was consolidated into the PRE program, the
Project was restructured to shift resources from PEC to PRE. In
addition, the revised indicators reflected more detailed and
accurate measures of the Project’s achievement without changing the
Project’s scope or reducing its targets. Therefore, the achievement
of each of the objectives is measured through the five PDO
indicators revised at the July 2017 restructuring. 20. The
Project’s expected outcomes were to improve managerial capacity and
parental participation among Participating Schools to reduce
dropout and improve quality of education, as measured by failure
and repetition rates among Participating Schools. The ICR goes
beyond the measurement of the PDO outcomes as stated in the Project
Appraisal Document (PAD) by adding learning results as an expected
outcome of this Project. Improved student learning (as measured by
test scores) provides a broader view of the achievements of the
Project in terms of its contribution to improving the quality of
education. Although students’ learning results were not explicitly
mentioned in the PDO, it was clear in both the PAD and the July
2017 restructuring paper, that the Project aimed to improve the
quality of education, including learning results. For instance, the
PAD mentions that “the two programs (initially supported by the
Project) share the goal of contributing to students’ learning
within a framework of more school autonomy, the use of evaluations
to improve pedagogical strategies, better managerial capacities at
the school level, and active social participation.” As explained in
the objectives section (see paragraph 53), learning results were
not included from the outset because: (i) the Bank was overly
cautious of adding learning results as a key outcome indicator due,
among others, to the short span of the Project to impact learning
and to technical reasons (given the transition from the ENLACE
national standardized test to the National Plan for Learning
Assessment (Plan Nacional para la Evaluación de los Aprendizajes,
PLANEA)) and the uncertainty on timing and availability of PLANEA
results. Instead, failure and repetition rates were included as
proxies for learning outcomes to measure the quality of education.
In addition, it has become increasingly feasible and important to
measure and include student learning outcomes as key outcome
indicators to ensure children attain real learning in the classroom
and are thus able to enter the job market as healthy, skilled, and
productive adults (World Bank, Human Capital Project, 2018).
Assessment of Achievement of Each Objective/Outcome Improve
managerial capacity in Participating Schools. 21. This objective
was measured through the key indicator “proportion of a
representative sample of Participating Schools with a director that
scores 3 or above in the World Management Survey (WMS).”6
6 The WMS is a well-known instrument developed by Bloom and
others at the London School of Economics that has been widely used
by the World Bank in several countries to measure managerial
capacity in different areas, including education. The WMS has a
scale from 1 to 5 to measure managerial capacity (1 being the
lowest and 5 being the highest score). In the case of Mexico and
other countries with low managerial capacity, there are two more
intermediate scores, 1.5 and 2.5, to allow for a greater dispersion
of the results. For more information, please visit:
www.developingmanagement.org
http://www.developingmanagement.org/
-
The World Bank Mexico School Based Management Project
(P147185)
Page 12 of 62
The share of Participating Schools with directors scoring 3 or
more in the WMS increased from 21 percent in school year 2015-16 to
27 percent in 2017-18, exceeding the 25 percent target set at
restructuring. School directors scoring 3 or more can use existing
resources in an effective way to provide better education services.
School directors’ managerial capacity was measured in a sample of
1,665 (PEC/PRE) schools. 22. Additional evidence from the results
from 7 randomized control trials (RCTs) conducted in 7 of the 32
states in Mexico show that after the three years of implementation
of the interventions supported by this Project, the school
management index improved by 0.086 standard deviations (s.d.) per
year7 in Participating Schools where Project activities were
closely monitored and implemented. 23. The Project (under Component
1) contributed to improving the managerial capacity of
Participating Schools by providing direct support (based on
national criteria) to implement the school improvement plans agreed
with parents. The provision of direct financial support to schools:
(i) worked as an incentive for school directors and local actors
(including teachers and parents) to establish a common dialogue on
how to address school needs to improve student learning (through
the school improvement plans); (ii) allowed schools to exert their
autonomy; and (iii) helped increase local transparency and
accountability, since the school directors needed to report on the
use of funds received to both the school community and education
authorities. 24. The development and implementation of the
capacity-building strategy for school directors and supervisors
(Component 2), helped improve school directors’ managerial capacity
(De Hoyos, Silveyra and Yanez 2017). All the supervisors and over
96 percent of school directors in Participating Schools received
capacity-building activities in SBM. The capacity-building strategy
mainly consisted of two sets of interventions. The first set
designed and provided cascade training in school management to
school supervisors who, in turn, trained school directors. This
training strategy was conducted by the Dirección General de
Desarrollo de la Gestión Educativa (DGDGE) in all 32 States.
Examples of the cascade training in management and tools to improve
learning provided include the following:
a. Comprehensive school supervision course focused on how to
help schools improve their student
learning outcomes. The course, titled “Una Supervisión Efectiva
para la mejora del aprendizaje de nuestros alumnos,” benefited
18,447 school supervisors. This included all school supervisors in
basic education in the country and those that were on a clear track
to become school supervisors.
b. Training on the use of classroom observation tools to improve
pedagogical practices and collection of performance indicators to
assess basic reading, writing, and math skills. Overall, 7,694
supervisors were trained on a classroom observation tool and around
8,817 supervisors undertook the basic reading, writing, and math
exploration tools training. Cascade training from supervisors to
school directors benefited over 70,060 primary and secondary school
directors (of 72,611).
c. Eight workshops for trainer of trainers to a core group of
1,000 school supervisors, including key topics on best practices in
SBM.
d. Each school was also offered a paid time slot (one school day
every month) to prepare, implement, and monitor its school
improvement plan. This gave school directors and teachers a space
to analyze school-level data and use it to improve students
learning.
7 The school management index represents schools’ managerial
capacity as measured by the 5 WMS aspects: leadership, operations,
monitoring, setting of objectives, and personnel.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 13 of 62
25. The second set of interventions related to capacity-building
under Component 2 consisted of training for school directors
tailored to their specific management needs in the 7 States
participating in the impact evaluation. This training aimed to help
school directors improve their managerial capacity through a shared
leadership approach, collaborative planning, and teamwork to
improve learning. Examples of the training provided by the States
include the following:
a. Tlaxcala offered the Diplomado en Liderazgo para la Autonomía
de Gestión Escolar to improve school leadership and certify school
directors on the competency of Liderazgo para la Autonomía de
Gestión en Centros de Trabajo Educativos, validated by the Consejo
Nacional de Normalización y Certificación de Competencias Laborales
(CONOCER). Training was conducted by the Colegio de Estudios
Avanzados de Iberoamérica (CEAI) and included hands-on and
theoretical approaches under a blended modality (4 face-to-face
sessions/24 hours, 13 online sessions/136 hours, and 90 practice
hours). The training focused on sharing experiences on how to solve
specific school needs among a diverse group of facilitators
including teachers, school leaders, and education experts.
b. Puebla offered a Diploma program on school leadership
imparted by WorldFund. Capacity-building activities under the
Diploma program included 308 face-to-face training hours, 16 online
hours, and personal coaching to school directors. Contents included
the importance of building a school culture, understanding the role
of teachers, parents, and students, and coaching to improve
learning outcomes (including social and emotional skills).
26. The Secretariat of Public Education (Secretaría de Educación
Pública, SEP) also developed a school dashboard information system
called the Sistema de Alerta Temprana (SisAT). School directors and
supervisors received training on the use of this dashboard. SisAT
helped school supervisors and directors systematically collect
relevant and up to date information on basic learning skills and
performance indicators to monitor school improvement plans (i.e.
students’ basic reading, writing and math skills). This was
combined with the collection of frequent data on student attendance
and student grades. Teaching practices (through the classroom
observation tool), and the provision of feedback to teachers to
improve their pedagogical practices was also a common practice at
the school level, although this information was not always uploaded
to the SisAT. The SisAT helped school directors access data to
inform the monitoring of school improvement plans; these in turn
helped them strengthen their managerial capacity. The SisAT is now
working in 27 of 32 states in Mexico. 27. Finally, the research and
innovation activities under Component 3 helped measure, monitor and
thus improve schools’ managerial capacity (and assess the impact of
the Project). Before the Project, there was no formal or robust
measurement of school managerial capacity. The Project provided
support to select and adjust an instrument to measure school
managerial capacities in the Mexican context following
international best practices. The adapted WMS provided, for the
first time, a reliable measure of school managerial capacities in
the country and their main dimensions (operations management,
monitoring, target setting, people management, school leadership,
and pedagogical practices). This measure helped fuel a discussion
for schools to understand their specific challenges and some of the
practices that effectively-managed schools follow.8
8 As previous research shows (i.e. De Hoyos, García, Patrinos,
2017), the sole provision of information to schools may have a
positive effect on learning.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 14 of 62
Improve parental participation 28. This objective was measured
through the key indicator “proportion of a sample of Participating
Schools that increase parental participation.” Parental
participation increased from 83.9 percent in the 2015-16 school
year to 94.66 in 2017-18, exceeding the 90 percent target set at
restructuring. Parental participation was measured with the School
Management Autonomy questionnaire (Cuestionario de Autonomía de
Gestión Escolar, AGE). The AGE is a national survey created to
measure parental engagement and other aspects of SBM in 7 states.
The baseline for this indicator was collected in a representative
sample of 1,091 Participating Schools using the AGE questionnaire.
29. The Project (Component 1) contributed to increased parental
involvement by providing direct support to Participating Schools to
implement school improvement plans agreed with parents. Parents in
schools that received support participated in the development of
the school improvement plan and were involved in the monitoring
process to ensure transparent use of the funds. 30. The Project
(Component 2) also helped design capacity-building activities on
SBM for parents to increase their participation. Although not all
parents in Participating Schools were trained, 78.21 percent
received some kind of SBM training. For instance, the DGDGE, in
coordination with the Consejo Nacional de Participación Social en
la Educación (CONAPASE), designed and delivered a capacity-building
strategy for parents to increase their impact on students’ learning
and development. The program, named Familias Educadoras, offered
information and guidance for parents to effectively support their
children at school. Familias Educadoras focused on: helping
increase self-esteem and healthy emotional family relationships;
guidance to set clear rules at home; specific techniques to support
children with reading, writing and math at home; and parents’ role
in school transparency and accountability. The results of an
electronic beneficiary satisfaction survey of Familias Educadoras
show that, overall, parents find this strategy useful and engaging.
In terms of coverage, Familias Educadoras had only reached 3,000
schools by Project closing, but is expected to reach 18,386 basic
education schools by 2018-19. 31. In addition, the 7 states
participating in the impact evaluation designed tailored
capacity-building activities to increase parental participation on
school matters. For instance: 32. In Puebla, parents in
Participating Schools received a 6-hour workshop (with two monthly
sessions consisting of three hours each) called Familia Presente,
delivered by school directors and SEP personnel. The workshop’s
objective was to improve parental knowledge and practices that are
effective to help strengthen student learning. Parents improved
their communication skills and built confident relationships with
their children, were informed of the importance of learning
together with their children and understood the importance of
having high expectations for their children’s performance. In
addition, parents received information on the importance of
participating in school activities and asking for support from
teachers and other school actors when needed.
33. In Tlaxcala, intensive training was provided to a select
group of 98 parents that were responsible for training and bringing
in other parents to their children’s schools. The workshop,
delivered by CEAI, lasted 20 hours (4 sessions of 5 hours each) and
aimed to increase joint teacher-parent responsibility to improve
students’ learning outcomes, as well as their involvement in the
monitoring of the school improvement plans. Tlaxcala also developed
a network called Un Domingo por Nuestros Hijos e Hijas,
-
The World Bank Mexico School Based Management Project
(P147185)
Page 15 of 62
where parents met regularly over the weekends to brainstorm on
how to improve their children’s learning and school autonomy from a
social participation perspective. In addition, in 2015, the
Instituto Tlaxcalteca para la Educación de los Adultos (ITEA) and
Instituto Nacional de Educación de Adultos (INEA) helped design
communication campaigns to help parents in Participating Schools
meet their own learning needs, such as acquiring basic literacy and
numeracy skills. 34. Finally, as mentioned in the previous PDO
indicator, school supervisors and directors were also trained to
increase parental involvement as part of the capacity-building
strategy. Reduce Dropout Rates 35. This objective was measured
through the key indicator “dropout rate among (primary and
secondary) Participating Schools.” In primary, the dropout rate
fell from 3.18 percent in 2014-15 school year to 2.56 in 2016-17
school year, exceeding the 2.91 target for 2017-18. In secondary
schools, the dropout rate decreased considerably from 5.31 percent
in school year 2014-15 to 2.12 percent in school year 2017-18,
greatly exceeding the 4.80 percent target for the 2017-18 school
year. 36. School grant schemes, such as the one supported under the
Project, have shown positive impacts on lowering dropout rates
(World Bank, 2014; Gertler, Patrinos and Rubio-Codina, 2012;
Skoufias and Shapiro, 2006; Murnane, Willet and Cardenas, 2006).9
This Project, in particular, contributed to reducing dropout rates
by increasing school autonomy, parental participation, and schools’
managerial capacity. The Project helped school directors, teachers,
students, and parents identify and address their local priorities.
The school dashboard information system (SisAT) also allowed
directors and even teachers to detect students at risk of dropping
out and to focus on different strategies to keep them in the
system. Schools may have also absorbed some direct costs of
schooling, reducing the financial parental contributions and, thus,
decreasing dropout rates among children whose parents were unable
to cover such direct costs. Parents (with higher opportunity costs
of sending their children to school) may have also decided to keep
their children in school due to their increased involvement in the
school’s issues. Parents also perceived that the relevance and
quality of education increased with the activities supported under
the Project.10 For instance, parents perceived that there was more
autonomy to make decisions at the school level and that school
actors were better prepared to identify local needs and improve
student learning outcomes. Reduce Gross Failure Rates 37. This
objective was measured through the key indicator “primary and
secondary school gross failure rate among Participating Schools.”
For primary schools, the gross failure rate was calculated as the
proportion of total students that failed the school year. For
secondary schools, the gross failure rate was calculated as the
proportion of total students that failed at least one subject in
the school year.
9 The evidence cited comes from impact evaluations that isolated
the impact of SBM interventions on the studied outcomes. However,
since dropout rates are due to many other factors, it is hard to
associate dropout to SBM interventions without an impact
evaluation. 10 Evidence in Mexico suggests that school grants may
contribute to improving learning results (Santibañez, Abreu and
O'Donoghue, 2014; Cabrera and Campusano, 2016).
-
The World Bank Mexico School Based Management Project
(P147185)
Page 16 of 62
Participating Primary schools’ gross failure rate slightly
increased from 1.10 percent to 1.44 percent in the school year
2016-17, falling behind the 1.00 percent target. The increase in
the gross failure rate for primary Participating Schools, however,
should be taken cautiously. The changes observed are economically
small and may be reflecting statistical noise, making them
equivalent to no change11. In addition, gross failure rates are
driven by many other factors outside of the Project’s scope; for
instance, changes in the national failure policy for primary school
children may have negatively affected the gross failure rate in
Participating Schools. Impact evaluations taking into account SBM
interventions have found positive impacts on failure and repetition
rates in Mexico (i.e. Skoufias and Shapiro, 2006). 38. The
Participating Secondary schools’ gross failure rate slightly
increased from 7.79 percent in 2013-14 to 7.84 percent in 2017-18,
falling behind the 3.26 percent target. The large difference
between the current rate and the expected outcome is due to a
mistake in the calculation of the baseline and target values during
the Project restructuring (please see paragraph 63 and 68). The
correct figures, however, show that secondary schools’ gross
failure rate remained practically unchanged, with an increase of
0.06 percent throughout the span of the Project.
39. Although the Project’s activities may have a small positive
impact on the overall gross failure rates, the activities under the
Project are expected to mitigate in the short to medium term
further increments in the gross failure rate among primary and
secondary Participating Schools, making it likely to achieve the
set targets in the future. As explained previously, the direct
support to schools and capacity-building activities financed under
the Project helped schools improve their managerial capacity and
parental participation, which should increase schools’
responsiveness to local challenges, such as increments in gross
failure rates (see theory of change). For instance, participating
schools are now able to use data and design school improvement
plans to support students that are more likely to fail the school
year. Similarly, capacity-building interventions (such as training
of school directors and supervisors in the classroom observation
tools to provide feedback to teachers and monitor their school
improvement plans), coupled with the collection and use of
performance indicators through the SisAT, help identify low
performers and find strategies to improve learning of students at a
greater risk of failing the school year, which should contribute to
reduced gross failure rates. Reduce Repetition Rates 40. This
objective was measured through the key indicator “Secondary school
repetition rate among Participating Schools.” Secondary school
repetition rate among Participating Schools slightly increased from
0.63 percent (2014-15 school year) to 0.97 percent (2017-18),
falling behind the 0.60 target. This result should be interpreted
with caution given that the baseline values were already small and
may fluctuate rapidly due to non-Project factors. The changes
observed are economically small and may be reflecting statistical
noise and are, therefore, equivalent to no change. In addition,
repetition rates are driven by many other factors outside of the
Project interventions. 41. Although the Project’s activities may
have a small impact on helping decrease the overall repetition
rates for participating secondary schools, Project activities are
expected to contribute to achieving this
11 The last ISR for this Project suggested that the increase in
primary gross failure rates could have been affected by the
substantial reduction in dropout rates. However, there is not
enough evidence to substantiate this argument. In addition, the
increase is less than 0.4 percent and therefore the explanation in
the main text seems more reasonable.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 17 of 62
objective in the medium term in the same way as they are
expected to help reduce gross failure rates (see paragraph 39). 42.
The indicators used to measure dropout, gross failure and
repetition rates do not allow the establishment of a causal
relationship between Project activities and the results obtained,
mainly because these learning outcomes are affected by factors
outside of the Project’s scope. Additional activities in both PRE
and PETC may have also contributed to the results in dropout, gross
failure and repetition rates. For instance, PRE had a large
infrastructure component that helped schools with greater
infrastructure needs to build classrooms, which could have
contributed to helping parents think their children were better-off
at school. Better infrastructure may have also contributed to
improved learning. In the case of PETC, an impact evaluation
(Silveyra, Yáñez & Bedoya, 2018) showed that the program had a
positive impact on reducing dropouts and improving student learning
outcomes. The impact evaluation cannot assess the mechanisms
through which the different components of PETC, including school
autonomy as described above, reduced dropouts and improved
students’ learning outcomes. In addition to the extended school day
and extra-curricular activities (i.e. music, English, etc.), PETC
schools sometimes also include a meal program. The meal program,
for example, could have worked as a magnet to retain students, and
improve learning outcomes, especially among the poorest. Learning
Results 43. This objective was measured through the key indicator
“proportion of a representative sample of Participating Schools
that improve their student’s learning,” measured by the National
Standardized Test (Plan Nacional para las Evaluaciones de los
Aprendizajes, PLANEA). Based on the overall results of impact
evaluations conducted in 7 States, learning results were better in
(T1) schools where Project interventions were closely monitored and
implemented directly by the federal government (DGDGE) in
coordination with the 7 States, as opposed to the (T2) schools that
received similar interventions as part of the federal government’s
national strategy to strengthen school autonomy (these schools may
or may not have received the interventions, since the
implementation was not overseen). In particular, students’ scores
in language skills in T1 schools increased considerably and
students’ learning improved on average (See Table 1). According to
PISA equivalence scales, improving mathematics scores by 0.41 s.d.
is equivalent to one additional year of education. This means that
students in beneficiary schools accumulated 1.5 months of extra
schooling per year.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 18 of 62
Table 1. Learning Results
44. The impact evaluations show that managerial capacity has a
causal, positive but small impact on learning in Participating
Schools in Mexico. Although the effects may not be
transformational, they are cost-effective (see efficiency section)
and given the impact evaluation results, are worth the investment.
Furthermore, Project interventions, and in particular increased
school managerial capacity, are expected to have lasting and larger
effects in the future. As a meta-analysis12 of SBM interventions
shows, results in students learning are seen after 5 to 8 years of
implementation (Gertler, Partinos and Rubio-Codina 2007).
Therefore, the Project’s long-term effects on the targeted
populations are expected to increase with time. 45. The results of
the impact evaluations show with certainty that the Project
contributed to improved learning results in the short term (3
years) by increasing schools’ managerial capacity (parental
involvement may have also contributed to student learning but was
not part of the impact evaluation). Overall, school autonomy, as
measured by the activities supported by this Project, directly
impacted school management (and parental participation) which, in
turn, improved student learning outcomes. Justification of Overall
Efficacy Rating Rating: Substantial 46. The overall efficacy rating
of this Project is considered Substantial since the operation has
achieved improved managerial capacity and parental participation,
while decreasing dropout rates and improving learning in
Participating Schools. Gross failure and repetition rates did not
change, but it is difficult to assess if this was due to the
effects of Project activities because there are many other factors
that affect these variables. This ICR considers that the Project
almost fully achieved its objectives since the results of the
impact evaluation (De Hoyos et al forthcoming) show that it
contributed to improved student learning results, which are a more
robust measure of the quality of education than failure and
repetition rates.
12 Examination of data from a number of independent studies of
the same subject, in order to determine overall trends.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 19 of 62
C. EFFICIENCY
Assessment of Efficiency and Rating
47. The Project’s benefits are Substantial. The economic
analysis of the first two components of the Project shows positive
results in both the cost-benefit analysis and the Internal Economic
Rate of Return (IERR). After three years of implementation of
Project interventions, the school management index improved by
0.086 s.d. per year, and student learning assessments in Math and
Spanish improved, on average, by 0.054 s.d. per year. According to
PISA equivalence scales, improving mathematics scores by 0.41 s.d.
is equivalent to one additional year of education. This means that
students in Participating Schools accumulated 0.132 more years of
education per year of exposure to the program (0.054/0.41 = 0.132),
which corresponds to 0.395 years over the duration of the program
(0.132 × 3 = 0.395) or 1.5 months of extra schooling per year
(0.132*12). Therefore, students who benefited from the Project will
earn MX$784 more per year than those that did not through the
effect of the Project on learning outcomes (50,771 × 0.117 × 0.132
= 784). This corresponds to an increase of 1.5 percent in per
capita annual wages. (Please see Annex 4 for a complete
explanation). 48. On average, the annual cost per student was of
MX$291.2. Students who benefited from the Project earn MX$784 per
year more than their peers. With an interest rate of 10 percent,
the net present value of the income benefits of the average Project
beneficiary is 6.7 times higher than its cost. Under these
assumptions, the Project’s IRR is 21.5 percent. (Please see Annex 4
for a complete explanation and the methodology used).
49. DGDGE used existing systems and resources within SEP
ensuring smooth project implementation, but disbursements faced a
one-year delay given the consolidation of the SBM programs in
Mexico, the fluctuations in the exchange rate, and the 2017
earthquake (please see section on Key Factors During
Implementation). At closing, the Project had disbursed 92 percent
of the funds; however, an extension to disburse the remaining funds
until June 30, 2019 was granted. By the extended deadline, it is
expected that about 99.4% of the Loan will be disbursed, leaving
less than one percent undisbursed. D. JUSTIFICATION OF OVERALL
OUTCOME RATING Rating: Satisfactory
50. The overall outcome rating of the Project is deemed
Satisfactory. The relevance rating for the Project objectives is
High, the efficacy rating is Substantial, and the efficiency rating
is Substantial. There were only minor shortcomings in the
operation’s achievement of its objectives. The combination of these
ratings yields an overall Satisfactory rating. E. OTHER OUTCOMES
AND IMPACTS Gender 51. Mexico has achieved gender parity in basic
education. Female beneficiaries at the end of the Project accounted
for 50.04 percent, while the set target was of 48.7 percent. Half
of the student beneficiaries in Participating Schools were female,
thus achieving gender parity.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 20 of 62
Institutional Strengthening 52. Since 2000, Mexico has provided
direct support to schools to improve school autonomy and parental
participation and the World Bank has supported SBM efforts since
2006. As the third World Bank Project supporting SBM in Mexico,
this Project contributed to consolidating and strengthening a
culture of school autonomy and SBM that are now embedded in the
system. In particular, this Project helped reinforce the 2013
Education Reform and supported the institutionalization of the
school autonomy policy. The policy has endured despite changes in
the programs through which the provision of direct support to
schools is implemented. The new administration has also continued
to support the federal programs to improve school autonomy despite
reducing its budget.
Poverty Reduction and Shared Prosperity
53. The Project contributed to the Bank’s twin goals of
eliminating extreme poverty and boosting shared prosperity. The
Project helped strengthen schools’ managerial capacity and parental
participation which, in turn, contributed to reduced dropout rates
in Participating Schools. The Project also improved learning
outcomes in Participating Schools. All Participating Schools are
public schools, many of which are situated in poor areas (on
average, 78.33 percent and 51.28 percent of PRE and PETC schools
are in highly and very highly marginalized areas). Because of the
Project, students in these schools will stay in school longer and
will learn more. As more Project beneficiaries enter the labor
market (and the additional years of schooling and learning
achievement of the workforce increases), the economic benefits of
the Project are also expected to increase (see Annex 4). More human
capital will translate to higher personal or household incomes,
less poverty and, if the increases in human capital take place
among disadvantaged households (which is the case in this Project),
it will translate to lower income inequality in the long run. Other
Unintended Outcomes and Impacts 54. The school autonomy and SBM
policies in basic education have permeated into the upper secondary
education system. The Secretaría de Educación Media Superior (SEMS)
launched in recent years the Fondo para fortalecer la autonomía de
gestión en planteles de Educación Media Superior to provide direct
financial and technical support to schools following the school
autonomy approach that has been undertaken for several years in
basic education.
III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY
FACTORS DURING PREPARATION
55. Objectives. The objectives laid out in the PDO were clear,
measurable, and based on the available evidence on the impact of
SBM initiatives. Based on the literature, the PDO aimed to improve
schools’ managerial capacity and parental participation to decrease
dropout, failure and repetition rates (World Bank, 2014; Gertler,
Patrinos and Rubio-Codina, 2012; Skoufias and Shapiro, 2006;
Murnane, Willet and Cardenas, 2006). Learning results were not
included in the PDO from the outset because the Bank was cautious
of adding learning results as key outcome indicators due to, among
others, a short Project lifespan that would not allow enough time
to impact learning. In addition, there was no baseline for test
scores for Participating Schools. While the Project was being
designed, the National Evaluation of Academic Achievement in
Schools (Evaluación Nacional del Logro Académico de Centros
Escolares, ENLACE) was under revision and there was no certainty on
how learning results were going to be
-
The World Bank Mexico School Based Management Project
(P147185)
Page 21 of 62
measured in the future,13 so gross failure and repetition rates
were chosen as proxies to measure learning. However, dropout,
failure, and repetition rates (as well as learning results) are
education outcomes that may be affected by factors outside of the
Project activities. The only way of measuring the real impact of
the Project on such outcomes is through a rigorous impact
evaluation, which was also conducted under the Project. 56. Design.
The Project design was simple and sought a smooth implementation of
the Project. The operational logic behind Component 1 was
straightforward: the funds should reach and be administered
directly by schools. Component 2 was in tune with the available
international evidence and designed to ensure that schools were
able to use the resources from Component 1 to improve education
service delivery. Component 3 was crucial to estimate and quantify
the effect of the Project interventions by including the design and
implementation of a rigorous impact evaluation. 57. A drawback of
the Project’s design was to allocate funds to Components 2 and 3.
The two previous Projects’ experiences warned about the challenges
associated with the procurement of technical assistance services,
mainly related to the client’s preference to follow local
procedures (see procurement section for a complete explanation).
All the activities under Components 2 and 3 were undertaken using
Government budget instead of the Loan as originally planned in the
PAD. It would have been desirable to include Components 2 and 3 as
financed with counterpart funding only from the outset. 58. Risk
and Mitigation Measures. Few risks that materialized may have
needed more attention at appraisal. Nevertheless, none of the
materialized risks mentioned below jeopardized the achievement of
the PDO: (i) The consolidation of PEC into PRE was not considered a
major risk at Project design, but it materialized in 2016 requiring
a Level 1 Restructuring (the risk of PEC being discontinued was
rated as moderate); (ii) Macroeconomic risks were rated as
moderate, but oil prices went down in 2015 and 2016, severely
affecting the education budget for SBM programs and thus the number
of Project beneficiaries. In addition, the exchange rate increased
considerably from the beginning of the Project to closing. A
clearer mitigation strategy for these risks would have been
desirable. B. KEY FACTORS DURING IMPLEMENTATION Coordination and
engagement 59. The three Government agencies involved in the
implementation of the Project: Secretaría de Hacienda y Crédito
Público (SHCP), National Financial Development Bank (Nacional
Financiera, S.N.C., I.B.D, NAFIN), and SEP-DGDGE worked as a team
to ensure that the Project was implemented as planned to achieve
its development objectives. Disbursements slowed in 2016 due to the
consolidation of programs to strengthen SBM in Mexico and the need
to restructure the Project. This restructuring took almost a year
to complete due to uncertainty from the Government on how and when
the Project should finance the new program and if the Project
should also finance direct support to PETC schools (given the
increases in the exchange rate). Another challenge during Project
implementation was that there were
13 In the June 2017 restructuring, the Bank team did not add
learning outcomes as part of the PDO even though the new national
standardized test (Plan Nacional para las Evaluaciones de los
Aprendizajes, PLANEA) was already established. As previously
stated, the Bank was cautious in using learning results as measured
by test scores as a key outcome indicator. Measurement of learning
outcomes, however, was maintained in the Project’s impact
evaluation.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 22 of 62
cases where the direct support to schools arrived late in the
school year, preventing schools from implementing their school
improvement plan as planned. SEP addressed this by speeding up the
process for transferring funds to schools, for example by keeping a
trust fund to manage the direct support to schools’ funds and
transferring resources directly from the federal government
(bypassing the States). 60. Finally, the States’ uneven capacity to
implement the national school director and supervisor training
strategy was mitigated by SEP by having the DGDE carry out the
coordination and provision of training for all school supervisors
in the country. Nevertheless, the capacity-building for school
directors was achieved at different levels and paces by states and
districts given their own implementation capacity and limitations.
For instance, in Tabasco and Durango, there is frequent rotation of
personnel, affecting the implementation of Project activities.
Therefore, it is important to provide each state with a suite of
services to implement school autonomy and accountability
initiatives, according to their particular needs and shortcomings.
Factors subject to World Bank control Adequacy of Supervision 61.
The Bank conducted regular supervision missions at least twice a
year that included meetings with SHCP, NAFIN, and SEP, as well as
field visits to Participating Schools, all of which helped identify
implementation challenges in a timely manner. For instance, through
the July 2017 Level 1 Restructuring, the Project was adjusted to
respond to the consolidation of the SBM government programs in
Mexico. 62. The Bank also sustained close collaboration with SEP in
the design and implementation of the impact evaluation. For
instance, the Bank provided: (i) experts’ time to ensure the robust
design of the impact evaluation and the accuracy of the data
collection and analysis; (ii) help to select an adequate instrument
to measure schools’ managerial capacity; (iii) financial support
for the piloting of the instrument, training of enumerators, and
data analysis; (iv) continuous technical assistance tailored to the
federal government and the 7 states participating in the impact
evaluation, including the design and coordination of numerous
conferences and workshops to share best practices and international
experiences to inform the selection and implementation of adequate
interventions to improve schools’ managerial capacity; and (iv)
guidance to select/develop VALORA, an instrument to measure the
degree to which the capacity-building strategy was implemented in
the 7 states. Overall, the Bank accompanied the Government
throughout the impact evaluation process, monitored its timeline
closely, and helped ensure that any technical and administrative
challenges were overcome. Furthermore, in coordination with SEP,
the Bank created a webpage as a repository for the evaluation,
including information on the: (i) design and preparation; (ii)
lines of intervention; (iii) measurement instruments; (iv)
implementation; (iv) results; and (v) good practices. The website
is available under the domain: http://escuelaalcentro.com/. 63. The
Bank also provided guidance and support related to M&E. For
instance, after a mistake in the baseline data and target for
secondary schools’ gross failure rates was reported by SEP, the
Bank hired a consultant to review all data related to the Project
indicators. SEP collaborated fully with the consultant, providing
all relevant databases and information. After a detailed review, no
further errors were detected.
http://escuelaalcentro.com/
-
The World Bank Mexico School Based Management Project
(P147185)
Page 23 of 62
Factors outside the control of the Government or the World
Bank
Macroeconomic Environment 64. As mentioned in the risks section,
the fall in the world price of oil in 2015 and 2016 produced fiscal
austerity in Mexico that resulted in, among others, the reduction
of the education budget from 2016 onwards. The fiscal austerity
affected the budget of SBM programs, including PRE and PETC and,
although at the beginning of the Project a significant increase in
students participating in SBM programs in Mexico was expected, the
number of beneficiaries actually decreased. The exchange rate also
increased considerably from the beginning to the end of the
Project. At Project preparation, the Peso to Dollar exchange rate
was around 13:1, while it was almost 20:1 by Project closing. In
the July 2017 restructuring, the Government of Mexico decided to
add PETC schools under Component 1 to ensure that all Project funds
could be disbursed despite the changes in the exchange rate.
Natural disasters 65. The earthquake that affected Mexico in
September 2017 affected approximately 15,000 schools (5,000 had
major damages), requiring a significant amount of resources to be
redirected towards the reconstruction of the affected schools. The
reduction in the amount of available resources from the Government
significantly affected the projected number of beneficiaries of
both the PRE and the PETC. To redirect existing limited resources,
however, the Government adopted strict targeting criteria towards
more vulnerable schools, including indigenous, rural, and those
located in highly marginalized and marginalized communities. This
focus had a significant impact on improving equity in allocating
resources to schools. Although the targeted number of beneficiaries
was not met, better-targeted schools received more resources. 66.
The September 2017 earthquake also damaged several SEP buildings,
including the data analysis buildings. This natural disaster
further slowed M&E reporting since SEP needed to recover lost
data and reallocate personnel responsible for analyzing and
reporting indicator data.
IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT
OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E)
M&E Design 67. The operation’s theory of change was clear
and based on available evidence on similar SBM initiatives (see
Annex 7). Each of the key Project indicators were matched to
monitor the achievement of each of the Project objectives. The
M&E design mainly relied on official data from the Encuesta 911
(the main survey used by the Government to update official
education indicators). The M&E design also contributed to
incorporating a concrete measure of schools’ managerial capacity
(one of the PDO objectives) into SEP’s M&E agenda. The Project
provided support to choose an instrument through the piloting,
redesign, implementation, and comparison of alternative
questionnaires to measure schools’ managerial capacity. The M&E
plan helped SEP systematically measure school directors’ managerial
capacity for the first time in Mexico. 68. The results framework
had some flaws, most of which were corrected in the 2017
restructuring (see Annex 8). Nevertheless, the following remained
through the Project: (i) the target for the dropout
-
The World Bank Mexico School Based Management Project
(P147185)
Page 24 of 62
rate for Participating Schools was underestimated; (ii) the
baseline and targets for gross failure and repetition rates were
already small from the outset and did not represent significant
changes (for instance, the difference between the baseline and the
target for participating secondary schools’ repetition rates was
only of a 0.03 percent reduction); (iii) the participating
secondary schools’ gross failure rate baseline and target were
inaccurate in the 2017 restructuring;14 (iv) the number of project
beneficiaries was not updated; and (v) learning results were not
added as PDO indicators. M&E Implementation 69. The methodology
to calculate each of the Project indicators was clearly described
in the PAD and in the July 2017 restructuring paper. The
methodology for most indicators followed conventional and local
measurements of education indicators such as dropout, failure, and
repetition rates. In the case of school managerial capacity, the
methodology to collect and analyze data to calculate this indicator
followed the WMS guidelines, which has been used in more than 15
different countries. The methodology to measure parental
participation was adjusted to the AGE national survey guidelines in
the July 2017 restructuring to make data collection for this
indicator less expensive and more reliable. There were delays and a
few inaccuracies in the reporting of information by SEP to the
Bank, as well as high rotation of SEP personnel responsible for
analyzing and updating data for Project indicators. Despite this,
SEP was diligent in systematizing the information collected (i.e.
keeping all relevant files together). M&E Utilization 70. For
the first time, the SEP generated information related to the
schools’ managerial capacity by using the WMS. The WMS results have
been used to inform the school autonomy policy design at the
federal, state, and school levels. Each of the 7 States that
undertook the WMS received a diagnostic with specific feedback
pointing to management areas for improvement, which helped them
adjust and tailor capacity-building strategies for schools
principals and supervisors. A similar diagnostic has been received
by each of the schools participating in the WMS so that each school
director can identify and address its managerial shortcomings. In
addition, SEP has used the WMS results to shape the national school
autonomy policy. Similarly, the information on the AGE
questionnaire has been useful to adjust the strategy to increase
parental participation and engagement at federal, state, and local
levels. Justification of Overall Rating of Quality of M&E
Rating: Substantial 71. The rating for the overall quality of
M&E is considered Substantial given the moderate shortcomings
in the M&E system’s design and implementation. For instance,
there were delays in the collection and analysis of the data to
update PDO indicators. Nevertheless, the M&E system was
generally sufficient to assess the achievement of the objectives
and test the links in the results chain.
14 The baseline at restructuring was calculated by SEP at 3.49
percent for PRE and PETC schools for the 2014-15 school year but in
March 2018, SEP reported that the correct baseline figure for the
2014-15 school year was 7.79 percent instead. Since the PAD
baseline was reported at 9.64 percent for PEC and PETC schools, it
seems feasible that the 3.49 percent reported at restructuring was
incorrect.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 25 of 62
B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE
72. This was a category C Project with a low Social Risk Rating.
The Project triggered OP/BP4.10 Indigenous Peoples. Therefore, an
Indigenous Peoples Plan (IPP) was approved and published on the
Bank and SEP webpages in August 2014. The IPP was updated at the
mid-term review (2016), in consultation with the Dirección General
de Eduacación Indígena (DGEI), teachers and indigenous leaders, and
published in early 2017. The overall implementation of the IPP was
satisfactory, especially among school communities that speak
Indigenous languages (in Mexico, there are 68 indigenous languages
and 364 variants). 73. The main results of the IPP were:
i. Increased awareness of SEP staff at the central level, as
well as supervisors, directors and teachers, that Indigenous
Peoples may be bilingual and require differentiated strategies to
ensure their participation in SEP Programs.
ii. Promotion of the coordination of DGDGE with other
departments within SEP around the topic of Indigenous schools and
the pertinence of processes and instruments used for M&E and
Supervision (DGDGE coordinated with the DGEI, the General
Directorate of Curriculum (DGDC), and others).
iii. Improved communication of guidelines, norms, plans, and
other important messages by addressing communications in the
language of the Indigenous communities to reach parents and
students in their native language – i.e. (a) in the SisAT Program,
the instructions given to students for reading and writing
exercises were translated into two main Indigenous languages
(Tutunacú and Náhuatl) thus enabling students to understand
instructions well before performing the exercise in Spanish; (b)
instruments to monitor the Indigenous PRE and PETC schools were
improved. Monitoring of Indigenous schools increased from 8 percent
in 2016 to 16 percent in 2017; (c) translation of administrative
information (banners/posters) to local languages to improve
governance and fiduciary transparency is underway.
iv. Yearly published PRE guidelines now include a clause to
alert education authorities (Autoridades Educativas Locales, AEL)
that information on Programs, processes, and plans must be
communicated to the community in the local language.
v. Indigenous schools were incorporated into the pilot program
of a school-based strategy. 104 of 997 schools in this program were
Indigenous.
74. Financial Management (FM) and audit. Project Aide Memoires
indicated a consistent and satisfactory FM performance, except for
the FM supervisions that took place from November 2016 through
October 2017, where performance was rated Moderately Satisfactory
(MS). FM performance was downgraded during this period mainly due
to a lack of disbursements while the Project was restructured to
disburse funds to schools in the newly created PRE program. 75. The
audit performance was uniformly strong. Annual, independent audits
of Project accounts were delivered on time and produced in general
clean, unqualified opinions for all years of implementation, except
for the 2017 audit report where the opinion was qualified due to
observations identified by the auditors (Órgano Interno de Control,
OIC) from SEP, which were promptly addressed by the Project. The
final Implementation Status and Results Report (ISR) rated FM
performance as Satisfactory.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 26 of 62
76. FM supervision was conducted in accordance with arrangements
stipulated in the legal agreement. The FM risk rating remained
Substantial throughout Project implementation due to the Project’s
size, scope and complexity. The Bank was able to dedicate greater
time and attention to FM oversight as implementation progressed,
which translated into visible results. 77. Procurement. As per the
Project Procurement visits to participating states, school-level
procurement was conducted as agreed in the Operational Manual. SEP
could not use the funds allocated to Categories 2 and 3 mainly due
to budgetary and regulatory reasons related to the internal
budgetary calendar and budgetary constraints that were aggravated
by the state of emergency caused by the September 2017 earthquakes.
78. Project design in Mexico should consider the challenges
associated with the procurement of technical assistance services
with Bank financing. SEP, as other federal entities, is required to
follow local budgetary procedures that are difficult to reconcile
with national procurement procedures, making procurement of
capacity-building activities burdensome for SEP. SEP could thus not
finance training or impact evaluation expenses with Project funds
as they are not considered “investments” and by Law are not subject
to be financed with foreign debt.
C. BANK PERFORMANCE Quality at Entry 79. The Bank provided
technical assistance to the Government to prepare a Project with
strategic relevance to strengthen schools’ managerial capacity and
parental participation, in line with the school autonomy component
of the 2013 Education Reform. The Bank’s contribution for quality
at entry included sharing international best practices and
available evidence on SBM initiatives to inform Project
preparation. For instance, the Bank conducted research on available
instruments to measure schools’ managerial capacity to help the
Government select an adequate instrument for the Mexican context
and measure schools’ managerial capacity for the first time. The
Bank also provided information on relevant interventions, such as
the use of classroom observation tools and students’ performance
data to improve teaching practices. Project design considered
poverty and social development aspects by targeting public
marginalized schools and preparing a plan to address indigenous
populations’ specific needs. In terms of FM, the Project used the
supported programs’ flow of funds. The safeguards and fiduciary
teams were continuously involved in relevant aspects of the Project
design. On the other hand, the Bank missed the opportunity to
assign counterpart funding instead of loan funds to
capacity-building and research activities under the Project, even
though lessons learned from the two previous Projects warned of the
challenge of using loan funds for these activities. While the
Project’s M&E framework had moderate shortcomings, including
the underestimation of school dropouts and overestimation of gross
failure and repetition rates, it was sufficient to measure the
Project impacts. Quality of Supervision 80. The Bank sustained a
continuous dialogue with the client, which facilitated the close
monitoring of Project implementation. For instance, the Bank began
a Level 1 Restructuring in 2016 as soon as the PEC program was
consolidated into the PRE program, and disbursements were put on
hold until the restructuring was completed. If the Bank had not
started the restructuring as soon as the changes in the SBM
programs occurred, disbursements would have lagged even
further.
-
The World Bank Mexico School Based Management Project
(P147185)
Page 27 of 62
81. The Bank also helped overcome technical, administrative, and
procurement challenges to undertake the impact evaluation of the
Project. Under the impact evaluation, the Bank helped ensure that
the interventions to improve schools’ managerial capacity were
undertaken as designed at Project appraisal. This engagement was
crucial to the successful completion of the impact evaluation,
which was a major achievement of this Project since the two
previous projects were unable to carry out or finalize the planned
impact evaluations.
82. The safeguards and fiduciary teams continuously supervised
and developed follow-up reports on the implementation of the IPP
and followed up on detailed aspects of the direct support to
schools’ disbursements, audits, etc. This was coupled with building
staff capacity in the Directorate of School Management (Dirección
General de Desarrollo de la Gestión Educativa) on both fiduciary
and technical aspects. On the other hand, there were moderate
shortcomings. For instance, the above-mentioned restructuring
missed the opportunity to update the number of beneficiaries and
further adjust the Project’s indicators (i.e. removing gross
failure and repetition rates and adding learning r