Policy and Program Evaluation Logical Framework What is Program Evaluation Case 1: Indonesia PQEP Case 2: Honduras Educatodos Case 3: School Quality Program.

Policy and Program Policy and Program EvaluationEvaluation

Logical FrameworkLogical FrameworkWhat is Program EvaluationWhat is Program EvaluationCase 1: Indonesia PQEPCase 1: Indonesia PQEPCase 2: Honduras EducatodosCase 2: Honduras EducatodosCase 3: School Quality Case 3: School Quality ProgramProgramSumming UpSumming Up

Logical FrameworkLogical Framework

The Logical Framework Approach is a The Logical Framework Approach is a project design methodology. Also know project design methodology. Also know by the initials of the method in German by the initials of the method in German ZOPP (Ziel Orientierte Projek Planung, ZOPP (Ziel Orientierte Projek Planung, or Objectives Oriented Project Planning or Objectives Oriented Project Planning or OOPP). This methodology, the or OOPP). This methodology, the Logical Framework Approach, calls for Logical Framework Approach, calls for a structured sequence of activities that a structured sequence of activities that leads to the production of a Logical leads to the production of a Logical Framework (a Document). Framework (a Document).

The main steps in this process The main steps in this process include:include:

Analysis of the Present SituationAnalysis of the Present Situation1.1. Analysis of Stakeholder Groups and Analysis of Stakeholder Groups and

Institutions Affected by the ProblemInstitutions Affected by the Problem2.2. Problems AnalysisProblems Analysis3.3. Objectives AnalysisObjectives Analysis4.4. Alternatives AnalysisAlternatives Analysis5.5. Activities PlanningActivities Planning6.6. Logical FrameworkLogical Framework

The Logical Framework was developed and The Logical Framework was developed and popularized in the design of USAID projects popularized in the design of USAID projects in the end of the 1960s and has been in the end of the 1960s and has been adopted widely by development adopted widely by development organizations.organizations.

Key themes:Key themes: Objectives OrientedObjectives Oriented Target Group OrientedTarget Group Oriented ParticipatoryParticipatory

Universal Primary Education =Universal Primary Education = Poverty ReducedPoverty Reduced

Quality of Education Increased Quality of Education Increased National Productivity IncreasedNational Productivity Increased

Confict Resolution Program Confict Resolution Program

Implemented Implemented Social Violence Social Violence ReducedReduced

Distinguishing between Immediate Distinguishing between Immediate and Development Objectives:and Development Objectives:

Both outside project control. Project Both outside project control. Project hypothesis.hypothesis.

Cause-Effect. Cause-Effect.

Increased productivity of the labor Increased productivity of the labor forceforce

Increased literacy of the populationIncreased literacy of the population

Increased healthy practices in Increased healthy practices in populationpopulation

Higher educational attainmentHigher educational attainment

Increased value added in information Increased value added in information technology industriestechnology industries

Greater scientific knowledge of college Greater scientific knowledge of college graduatesgraduates

Greater peace and securityGreater peace and security More sophisticated international relationsMore sophisticated international relations More knowledge and awareness of global More knowledge and awareness of global

issuesissues

Projects transform inputs into outputs. Projects transform inputs into outputs. Activities are the processes that create the Activities are the processes that create the outputs using inputs. These three are outputs using inputs. These three are within the project control.within the project control.

Outputs are results than can be Outputs are results than can be guaranteed by the project as a guaranteed by the project as a consequence of activitiesconsequence of activities

Activities are actions necessary to Activities are actions necessary to transform given inputs into planned transform given inputs into planned outputs within a specified period of timeoutputs within a specified period of time

Inputs are the resources of a project Inputs are the resources of a project necessary to produce the intended outputsnecessary to produce the intended outputs

OutputsOutputs 100,000 graduates of literacy campaign100,000 graduates of literacy campaign

ActivitiesActivities 10,000 literacy workshops conducted in 10,000 literacy workshops conducted in

20012001

InputsInputs 100,000 literacy booklets100,000 literacy booklets 2,000 literacy promoters2,000 literacy promoters 200 literacy coordinators200 literacy coordinators 500 all purpose vehicles500 all purpose vehicles 2,000 sites for training2,000 sites for training

External Factors. Necessary and External Factors. Necessary and Sufficient Conditions.Sufficient Conditions.

External factors are necessary for External factors are necessary for project success but outside the project success but outside the control of project management.control of project management.

Outside and inside the project. Outside and inside the project. Project Environment.Project Environment.

Examples:Examples: Participants engage in training Participants engage in training

opportunitiesopportunities Trained participants exhibit skills Trained participants exhibit skills

gained in traininggained in training Complementary resources (e.g. Complementary resources (e.g.

textbooks) are available.textbooks) are available. Jobs are availableJobs are available Parents send their children to new Parents send their children to new

schools.schools.

Critical factors need to be integrated as project Critical factors need to be integrated as project outputs, when possible. Alternatively they need to be outputs, when possible. Alternatively they need to be monitored carefully.monitored carefully.

Indicators and Means of Verification.Indicators and Means of Verification. Should specify:Should specify: Target groupTarget group QuantityQuantity QualityQuality TimeTime LocationLocation

Examples:Examples: Increased literacy, defined as ability Increased literacy, defined as ability

to read and understand a short text.to read and understand a short text. Ability to design a project using a Ability to design a project using a

logical framework approachlogical framework approach

The Logical Framework Matrix is a useful tool The Logical Framework Matrix is a useful tool to systematize the hypotheses implicit in the to systematize the hypotheses implicit in the project. It can be used by individuals or project. It can be used by individuals or project teams, and often teams design their project teams, and often teams design their projects first and then retrofit them into the projects first and then retrofit them into the Logical Framework Matrix. This approach, Logical Framework Matrix. This approach, obviously, misses the most important benefits obviously, misses the most important benefits of the approach, the possibility to organize of the approach, the possibility to organize thinking and to facilitate communication and thinking and to facilitate communication and dialogue among a range of stakeholders dialogue among a range of stakeholders about problems and options for intervention.about problems and options for intervention.

In its original and most appropriate form, In its original and most appropriate form, however, the Logical Framework Matrix however, the Logical Framework Matrix should be the end product of a Logical should be the end product of a Logical Framework Analysis (process). The Matrix Framework Analysis (process). The Matrix is always understood as a living document, is always understood as a living document, a compact expressing temporary a compact expressing temporary agreements and understandings among agreements and understandings among those with a stake in a project. It can (and those with a stake in a project. It can (and should) be amended from time to time, as should) be amended from time to time, as implementation proceeds and more is implementation proceeds and more is learned about the effects of the project learned about the effects of the project and about the problems it is trying to and about the problems it is trying to affect.affect.

SITUATION ANALYSISSITUATION ANALYSIS

1.1. Situation Analysis. Situation Analysis.

2.2. Stakeholder Analysis (also called Stakeholder Analysis (also called Participation Analysis)Participation Analysis)

3.3. Problem Analysis.Problem Analysis.

4.4. Objectives Analysis.Objectives Analysis.

5.5. Analysis of alternatives.Analysis of alternatives. PROJECT DESIGNPROJECT DESIGN

6.6. Specific Planning of Activities.Specific Planning of Activities.

7.7. Preparation of a Detailed Logical Preparation of a Detailed Logical Framework.Framework.

Situation Analysis. Situation Analysis. This is an initial report to kick off the This is an initial report to kick off the

conversation. It describes the main issues conversation. It describes the main issues defining the problem. It could be a needs defining the problem. It could be a needs assessment, a proposed project and assessment, a proposed project and feasibility, or an education sector-review. feasibility, or an education sector-review.

Develop comprehensive picture of the Develop comprehensive picture of the interest groups, individuals and institutions interest groups, individuals and institutions involved. TARGET groups whose views will involved. TARGET groups whose views will be given priority in analyzing problems.be given priority in analyzing problems.

2. Stakeholder Analysis (also called 2. Stakeholder Analysis (also called Participation Analysis)Participation Analysis)

This consists of political mapping of This consists of political mapping of the groups affected by the problem the groups affected by the problem under consideration, or with the ability under consideration, or with the ability to influence the problem. This political to influence the problem. This political mapping answers the question: which mapping answers the question: which interests are affected by this problem? interests are affected by this problem? This process should clarify the groups This process should clarify the groups and their specific interests. and their specific interests.

3. 3. Problem Analysis.Problem Analysis.In this step of the Logical Framework In this step of the Logical Framework

Process key stakeholders negotiate an Process key stakeholders negotiate an agreement on the problem, it’s agreement on the problem, it’s underlying causes and principal effects. underlying causes and principal effects. One way to proceed is to get participants One way to proceed is to get participants in the workshop identify their own ideas in the workshop identify their own ideas of problems and then connect those ideas of problems and then connect those ideas in a problem tree (a sequence of cause-in a problem tree (a sequence of cause-effect relationships). Then a group effect relationships). Then a group discussion seeks to integrate individual discussion seeks to integrate individual ideas into a group problem tree. ideas into a group problem tree.

Identify major problems and main causal Identify major problems and main causal relationships between them.relationships between them.

4. O4. Objectives Analysis.bjectives Analysis.

In this step statements about In this step statements about problems are translated into positive problems are translated into positive statements about objectives. An statements about objectives. An ‘objective tree’ is a sequence of cause-‘objective tree’ is a sequence of cause-effect relationships necessary to bring effect relationships necessary to bring about desired objectives. Multiple about desired objectives. Multiple objective trees can be generated.objective trees can be generated.

5. 5. Analysis of alternatives.Analysis of alternatives.At this stage participants assess the At this stage participants assess the

alternative courses of action proposed by alternative courses of action proposed by each objective tree in terms of each objective tree in terms of effectiveness, costs, feasibility, impact on effectiveness, costs, feasibility, impact on priority groups, risks or other criteria. priority groups, risks or other criteria.

Criteria can be:Criteria can be: TechnicalTechnical FinancialFinancial EconomicEconomic InstitutionalInstitutional Social/distributionalSocial/distributional EnvironmentalEnvironmental

6. 6. Specific Planning of Activities.Specific Planning of Activities.

Select Development and Immediate Select Development and Immediate Objectives.Objectives.

Once a particular course of action has Once a particular course of action has

been agreed upon, participants in the been agreed upon, participants in the workshop identify measurement workshop identify measurement indicators and means of verification indicators and means of verification for each objective and identify the for each objective and identify the necessary activities and inputs necessary activities and inputs necessary to achieve the objectivesnecessary to achieve the objectives

7. Preparation of a Detailed Logical 7. Preparation of a Detailed Logical Framework.Framework.

LFA is a useful communication tool to be LFA is a useful communication tool to be used not just during project design, but also used not just during project design, but also during implementation and evaluation.during implementation and evaluation.

Define Project Elemenents (Inputs, Define Project Elemenents (Inputs, Activities and Outputs)Activities and Outputs)

Identify External FactorsIdentify External Factors IndicatorsIndicatorsArbitrary at which level to pitch the project in Arbitrary at which level to pitch the project in

a problem ladder or tree. Definition of focal a problem ladder or tree. Definition of focal problem is an arbitrary decision, reflecting problem is an arbitrary decision, reflecting value preferences. value preferences.

A LFA can be used at various stages of the A LFA can be used at various stages of the project cycle:project cycle:

Project IdentificationProject Identification Feasibility StudyFeasibility Study Project DesignProject Design Detailed Planning (Planning for Detailed Planning (Planning for

Implementation)Implementation) Monitoring (Implementation)Monitoring (Implementation) Project Review (Mid-term project Project Review (Mid-term project

evaluation)evaluation) EvaluationEvaluation

What is evaluation?What is evaluation? Why is it done?Why is it done?

Evaluation of school improvement Evaluation of school improvement through an educational effectiveness through an educational effectiveness model: The case of Indonesia’s PEQIP model: The case of Indonesia’s PEQIP ProjectProject

Van Der Werf, G., B. Creemers, R. de Jong and E. Van Der Werf, G., B. Creemers, R. de Jong and E. Klaver.Klaver.

Why was the evaluation done?Why was the evaluation done? What was the program about?What was the program about? What are the conclusions?What are the conclusions?

1992 Government initiated the Primary 1992 Government initiated the Primary Education Quality Improvement Project (PEQIP). Education Quality Improvement Project (PEQIP). Study conducted between 1992 and 1997Study conducted between 1992 and 1997

School effectiveness study.School effectiveness study. An integrated school effectiveness model was An integrated school effectiveness model was

developed which took into account PEQIP inputs developed which took into account PEQIP inputs and intended outputs, school and classroom and intended outputs, school and classroom level indicators of effective schools in Western level indicators of effective schools in Western countries, indicators of effective schools in countries, indicators of effective schools in developing countries, and the local context of developing countries, and the local context of schools, teachers and pupils in Indonesia.schools, teachers and pupils in Indonesia.

Research questionsResearch questions

How large are the differences in student How large are the differences in student achievement between schools?achievement between schools?

Are between school differences related Are between school differences related to school participation in PEQIP?to school participation in PEQIP?

Which school and classroom factors, Which school and classroom factors, indicating PEQIP implementation, are indicating PEQIP implementation, are related to between-school differences in related to between-school differences in student achievement?student achievement?

PEQIPPEQIP Launched 1990Launched 1990 440 schools440 schools Books and materialsBooks and materials School fundsSchool funds New instructional approachesNew instructional approaches Professional development for teachers and administratorsProfessional development for teachers and administrators School clustersSchool clusters

Four components:Four components: Teacher developmentTeacher development Educational management –and community participation—Educational management –and community participation— Books and learning materialsBooks and learning materials Evaluation and monitoringEvaluation and monitoring

Study in two provinces. 27 PEQIP schools from all three Study in two provinces. 27 PEQIP schools from all three participating subdistricts. In each subdistrict three clusters of participating subdistricts. In each subdistrict three clusters of schools participated. Each cluster had a center school (Inti schools participated. Each cluster had a center school (Inti school) and six to nine satellite schools (Imbas schools). Plus school) and six to nine satellite schools (Imbas schools). Plus 14 control schools in each province.14 control schools in each province.

Bahasa Indonesia and Mathematics pre and post tests.Bahasa Indonesia and Mathematics pre and post tests. Science post-testScience post-test School year 1996-1997School year 1996-1997 PEQIP participation did indeed have effects on student PEQIP participation did indeed have effects on student

achievement, after taking prior achievement into account, but achievement, after taking prior achievement into account, but these effects are small, not always significant, and not always these effects are small, not always significant, and not always present for every subject in every category of school. Large present for every subject in every category of school. Large unexplained differences remain between schools after unexplained differences remain between schools after controlling for PEQIP participation and context.controlling for PEQIP participation and context.

Implementation of components is not higher in PEQIP schools. Implementation of components is not higher in PEQIP schools. Pequip is not implemented better in PEQIP schools than in Pequip is not implemented better in PEQIP schools than in control schools, which explains why the participation in PEQIP control schools, which explains why the participation in PEQIP in general was not very effective. Copy table 5 page 346in general was not very effective. Copy table 5 page 346

Multilevel model to examine the extent to which the variables Multilevel model to examine the extent to which the variables in PEQIP explain achievement. The variables indicating the in PEQIP explain achievement. The variables indicating the implementation of PEQIP account for almost half of the school-implementation of PEQIP account for almost half of the school-level variance.level variance.

Pretests administered in fifth year of project.Pretests administered in fifth year of project.

EducatodosEducatodos

Quality and Efficiency in a Quality and Efficiency in a Complementary Middle School Complementary Middle School Program: The Educatodos Program: The Educatodos Experience in HondurasExperience in Honduras

Marshall, J., M. Mejia and C. AguilarMarshall, J., M. Mejia and C. Aguilar

Focus on Middle School. Alternative schools.Focus on Middle School. Alternative schools. Outcome attritionOutcome attrition Origins radio courses in primary grades for adultsOrigins radio courses in primary grades for adults Now audiotaped lessons in community centersNow audiotaped lessons in community centers Evidence of high rates of attritionEvidence of high rates of attrition We analyze attrition in 55 learning centers in We analyze attrition in 55 learning centers in

2003-52003-5 Predictors of staying in the program are use of Predictors of staying in the program are use of

community projects, having a remunerated community projects, having a remunerated facilitator and type of learning center.facilitator and type of learning center.

Based on official curriculum, learning materials Based on official curriculum, learning materials and facilitator.and facilitator.

Purpose examine relationship between features Purpose examine relationship between features of learning centers and attrition.of learning centers and attrition.

Contracted to investigate root causes of Contracted to investigate root causes of attrition.attrition.

Examined 55 centers in 8 clusters.Examined 55 centers in 8 clusters. Centers located in:Centers located in: Primary schoolsPrimary schools BusinessesBusinesses Private homesPrivate homes Vocational training centersVocational training centers ChurchesChurches Only 32% of students remained in the Only 32% of students remained in the

program 2 years later.program 2 years later. 20% of the centers closed.20% of the centers closed.

Frequency of community participation projects Frequency of community participation projects related to low attritionrelated to low attrition

Attrition is also lower where the facilitator receives Attrition is also lower where the facilitator receives more moneymore money

Also social status of the centerAlso social status of the center Facilitators’ pay related to effort.Facilitators’ pay related to effort. Why did learning centers close?Why did learning centers close? No facilitatorsNo facilitators No materialsNo materials Why did they dropout? Disproportionately Why did they dropout? Disproportionately

interested in higher income. Perhaps dropouts had interested in higher income. Perhaps dropouts had higher expectations.higher expectations.

Ability to pay is not key… do not focus exclusively Ability to pay is not key… do not focus exclusively on making schooling more affordable to poor on making schooling more affordable to poor people.people.

Studying School Studying School Autonomy in Autonomy in

MexicoMexico

¿Who Benefits from School ¿Who Benefits from School Based Management?Based Management?

Fernando Reimers and Sergio Fernando Reimers and Sergio CardenasCardenas

Background for this studyBackground for this study

Evaluation of 4 large scale policies in Evaluation of 4 large scale policies in Mexico in 2006Mexico in 2006

The study of SQP effects in grade The study of SQP effects in grade repetition, dropout and promotion repetition, dropout and promotion (Murnane, Willett and Cardenas)(Murnane, Willett and Cardenas)

The study of Who Benefits, The study of Who Benefits, differences in learning outcomes and differences in learning outcomes and school organization.school organization.

The Quality Schools ProgramThe Quality Schools Program

School Based ManagementSchool Based Management School grantsSchool grants Participatory PlanningParticipatory Planning AccountabilityAccountability TeamworkTeamwork

What influenced the choices we What influenced the choices we made about the studymade about the study

Interests:Interests: Better understand who benefits, how and Better understand who benefits, how and the organizational dynamics of this school improvement the organizational dynamics of this school improvement project.project.

Resources:Resources: Existing evaluations in Mexico and research in other Existing evaluations in Mexico and research in other

countries.countries. Cross sectional data on student achievement in SQP Cross sectional data on student achievement in SQP

and non SQP schools and responses of principals to a and non SQP schools and responses of principals to a surveysurvey

Social network built over the years and reinforced with Social network built over the years and reinforced with the national evaluation.the national evaluation.

Opportunities:Opportunities: Invitation to participate at a State Congress of SQP Invitation to participate at a State Congress of SQP

staff in Baja California.staff in Baja California. Institutional Relationship with University in Baja Institutional Relationship with University in Baja

CaliforniaCalifornia

MethodsMethods

Literature ReviewLiterature Review Secondary Analysis of an existing Secondary Analysis of an existing

DatasetDataset Survey to 300 teachers and Survey to 300 teachers and

principals attending the Conference principals attending the Conference in Baja Californiain Baja California

Findings.Findings.Who benefits? Percentage of schools in Who benefits? Percentage of schools in

QSP…QSP…

29%

9%

12%

0%

5%

10%

15%

20%

25%

30%

35%

Urban Rural Indigenous

QSP Schools

Percentage of students who receive Percentage of students who receive scholarshipsscholarships

33%

19%

43%

79%

39%

19%

45%

79%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Total Urban Rural Indigenous

QSP

Non QSP

0.21

0.16

0.29

0.20

0.25

0.220.20

0.33

0.05

0.10

0%

5%

10%

15%

20%

25%

30%

35%

Very Low Low Medium High Very High

QSP I QSP II QSP III QSP IV QSP V

Schools by Poverty LevelSchools by Poverty Level

Student achievementStudent achievement15

.25

16.1

9

20.8

3

22.4

8

22.6

8

19.2

2

16.6

3

17.7

2

20.5

7

21.7

8

23.1

2

23.1

5

1012141618202224

No QSP

Indigenous

Schools

QSP No QSP

Rural Public

Schools

QSP No QSP

Urban Public

Schools

QSP

Language

Math

Institutional CapacityInstitutional Capacity

80%

91%

79%97%

98%

99%

0% 20% 40% 60% 80% 100%

Indigenous

Rural

Urban

Has the school a Technical School Council?

QSP

Non-QSP


82%

94%

81%95%

96%

98%

0% 20% 40% 60% 80% 100%

Indigenous

Rural

Urban

Has the school a Technical Teaching Comission?

QSP

Non-QSP


96%

100%

92%98%

78%

99%

0% 20% 40% 60% 80% 100%

Indigenous

Rural

Urban

Has the school a school project?

QSP

Non-QSP


74%

84%

74%

81%

58%

78%

0% 20% 40% 60% 80% 100%

Indigenous

Rural

Urban

Does the school project address the main academic priorities?

QSP

Non-QSP


60%

84%

64%83%

56%

81%

0% 20% 40% 60% 80% 100%

Indigenous

Rural

Urban

Has the school project fostered team work?

QSP

Non-QSP


58%

76%

60%

78%

45%

80%

0% 20% 40% 60% 80%

Indigenous

Rural

Urban

Has the school project been shared with supervisors, parents and students?

QSP

Non-QSP


42%

51%

47%

63%

39%

62%

0% 20% 40% 60% 80%

Indigenous

Rural

Urban

Did the implementation of the program begin in a timely fashion?

QSP

Non-QSP


51%

56%

49%

57%

37%

57%

0% 10% 20% 30% 40% 50% 60%

Indigenous

Rural

Urban

Is implementation of the School Project proceeding on track?

QSP

Non-QSP


50%

67%

57%

67%

46%

67%

0% 20% 40% 60% 80%

Indigenous

Rural

Urban

Is the School Project achieving its goals?

QSP

Non-QSP


92%

94%

91%

91%

90%

91%

88% 89% 90% 91% 92% 93% 94%

Indigenous

Rural

Urban

Has the school implemented programs for at-risk children ?

QSP

Non-QSP

Expenditure in Professional Expenditure in Professional Development ActivitiesDevelopment Activities

4.64%5.45% 5.83%

4.42%5.41%

4.46%5.26%

3.03%

13.53%

1.60%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

Very low Low Medium High Very high

Poverty Level

Schools with one year in the program

Urban

Rural


5.34%4.18% 4.76%4.06%

7.18%7.33%

4.47%4.72%

16.47%

2.44%

0.00%2.00%4.00%6.00%8.00%

10.00%12.00%14.00%16.00%18.00%


Poverty Level

Schools with two years in the program

Urban

Rural


10.62%

1.06%

6.89%

3.98%5.00%4.56%

7.31%

4.71%

13.95%

4.28%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%


Poverty Level

Schools with three years in the program

Urban

Rural


9.91%9.24%7.37%

5.59%6.77%

9.22%10.17%

5.03%

16.90%

3.87%

0.00%2.00%4.00%6.00%8.00%

10.00%12.00%14.00%16.00%18.00%


Poverty Level

Schools with four years in the program

Urban

Rural

Principals and teachersPrincipals and teachers’’ perception perceptionReasons why socially marginalized Reasons why socially marginalized

schools did not participate in the QSP schools did not participate in the QSP programprogram

They lack stability of school staff to They lack stability of school staff to participate in the programmeparticipate in the programme

56%56%

Poor school organization prevents Poor school organization prevents them from preparing a proposalthem from preparing a proposal

48%48%

They lack a shared vision for the They lack a shared vision for the schoolschool

47%47%

They are unable to provide They are unable to provide counterpart fundingcounterpart funding

46%46%

Principals and teachersPrincipals and teachers’’ perception perception

Reasons why socially marginalized Reasons why socially marginalized schools did not participate in the QSP schools did not participate in the QSP

programprogram

They do not know the programThey do not know the program 39%39%

They are not interestedThey are not interested 24%24%

School principals are not trainedSchool principals are not trained 24%24%

They do not prepare good They do not prepare good proposalsproposals

12%12%


Reasons why Reasons why QSP schools had not had QSP schools had not had greater impact in student learninggreater impact in student learning

Schools are not adequately organized to focus Schools are not adequately organized to focus on student learningon student learning

59%59%

Teachers are not prepared to teach effectivelyTeachers are not prepared to teach effectively 31%31%Educational materials are not adequateEducational materials are not adequate 9%9%It is not the purpose of the programIt is not the purpose of the program 5%5%


The selection mechanisms of the programme make it easier for those schools with greater institutional capacity to participate,

particularly urban schools...

16%

38%8%

22%

15%Completely agree

Agree

Do not know

Disagree to some extent

Completely disagree


The program works better in schools where principals are more effective leaders

67%

28%

1% 4% 1%

Completely agree

Agree

Do not know


Completely disagree


The program works better in those schools where there is shared vision among teachers

76%

20%0%1%1%

Completely agree

Agree

Do not know


Completely disagree


The program has not had its intended effects because there is a tension between its participatory and local management focus and the way in which other programmes reinforce an

approach that discourage local participation .

17%

43%9%

22%8% Completely agree

Agree

Do not know


Completely disagree


Program management undermines local management in the most vulnerable schools

9%

32%

18%

21%

8%Completely agree

Agree

Do not know


Completely disagree


The program increases conflicts among school staff.

5%14%

2%

26%

52%

Completely agree

Agree

Do not know


Completely disagree


The program has created conflict among program managers and managers of other programs

5% 14%

2%

26%

52%

Completely agree

Agree

Do not know


Completely disagree


The main problem is that teachers need more professional development to teach effectively.

37%

33%

1%

18%

11%Completely agree

Agree

Do not know


Completely disagree

GarawayGarawayFive key differences in evaluation in developing Five key differences in evaluation in developing

countriescountries

Education systems, structures and contextEducation systems, structures and context these factors matter because the these factors matter because the impact program’s delivery. Outsider evaluators may have difficulty understanding how impact program’s delivery. Outsider evaluators may have difficulty understanding how context impacts implementation dynamics.context impacts implementation dynamics.

Socio-Political and Cultural IssuesSocio-Political and Cultural Issues degree of verticality in the society. Cultural degree of verticality in the society. Cultural values shape how individual voice and dissent is valued. Consensus building may be values shape how individual voice and dissent is valued. Consensus building may be preferred over dialogue.preferred over dialogue.

Players and their rolesPlayers and their roles Greater power for the evaluator.Impact on donors. Greater power for the evaluator.Impact on donors. Client often a donorClient often a donor Evaluators often foreignEvaluators often foreign Authority structuresAuthority structures Evaluation ImpactEvaluation Impact During the evaluation processDuring the evaluation process Post evaluation, use of findingsPost evaluation, use of findings Language of the report, level at which it is writtenLanguage of the report, level at which it is written Ethical ConsiderationsEthical Considerations Should local custom be observed? How and when to give voice to women, children and Should local custom be observed? How and when to give voice to women, children and

vulnerable groups.vulnerable groups. Is the evaluator accountable to the local populationIs the evaluator accountable to the local population Whose concerns take precendence? Donor or program participants?Whose concerns take precendence? Donor or program participants?

Policy and Program Evaluation Logical Framework What is Program Evaluation Case 1: Indonesia PQEP Case 2: Honduras Educatodos Case 3: School Quality Program.

Documents

project outputs

project necessary

project success

project hypothesis

project environment

outside project control

project design methodology

control of project management