Dr. G. Johnson, Program Evaluation and the Logic Model Research Methods for Public Administrators Dr. Gail Johnson.

Dr. G. Johnson, www.researchdemystified.org

1

Program Evaluation and the Logic Model

Research Methods for Public Administrators

Dr. Gail Johnson


2

What to Evaluate?

Projects: a single intervention in one location or a single project implemented in several locations.

Programs: an intervention comprised of various activities or projects which are intended to contribute to a common goal.

Organizations: multiple intervention programs delivered by an organization.


3

When to Evaluate?

Before program starts: To improve design

During the implementation: To improve implementation Identify barriers to be removed Lessons learned about implementation


4

When to Evaluate

Mid-term evaluation Relevance, effectiveness, efficiency Lessons learned: management tool

Impact evaluation Either at the end of the project or a few years after the

program has been operating: assessing a mature program

Can also look at: effectiveness, efficiency, early signs of impact and sustainability

Lessons learned: future projects


5

Why Is Evaluation Useful?

Feedback Accountability Learning Improvement Results Testing underlying assumptions or theory Funding decisions


6

Fear of Evaluation

If evaluation is so useful, why do some people fear evaluation?


7

Evaluation QuestionsCompliance/ Accountability Questions

Did the promised activities actually take place as they were planned?

“How” Questions What was the sequence or processes that led to successful (or not) outcomes

Impact Questions

Did the program achieve the desired results?


8

Types of Evaluations

Auditing: accounting for money Is the money being spent according to plan? Efficiency and effectiveness.

Monitoring: measuring implementation and results Is the intervention producing the intended

results? Process: measuring operations and service delivery

Are there problems in service delivery?


9

Types of Program Evaluations

Feasibility evaluations Before the program begins Intended to improve program design

Evaluability assessments Assesses potential usefulness of the evaluation Used to test out different strategies for

conducting an evaluationWhat is doable given the situation?


10

Evaluability Assessment

Helps to define the actual objectives, implementation and management of a program. The actual objectives may differ from those

initially planned.

Determines the coherence of the program: are goals, activities, program infrastructure linked?


11


Key steps in the process: Interview key program staff to actual program

mission, goals, objectives and activities. Site visits to observe and get a sense of what is

going on. May include interviews with key stakeholders. Observe program delivery.


12


Reach agreement as to: Whether to conduct the evaluation. Scope and objectives of the evaluation.

The decision could be to not conduct the evaluation.


13

Evaluability Assessment: Challenges Key components of the program may not be

well defined: Lack of agreement on program objectives. Lack clear, measurable indicators of

performance and/or impact. Target group may not be clearly defined. The delivery system is poorly articulated.


14


Formative evaluations During implementation Feedback about operations and processes Used to make mid-course corrections


15

Definition: Performance Monitoring

Performance monitoring: the continuous process of collecting and

analyzing data to compare how well a project, program or policy is being implemented against expected results.

Traditional: focus on inputs, activities and outputs.


16

Types of Evaluation:Monitoring

On-going review: On-time On-budget On-target

Linked with on-going management Measured against established baselines Indicators of progress toward targets


17


Summative Evaluations At the end of the program or after the program has been

running long enough to achieve its goals (with “mature” programs)

Identify lessons learned Other issues: unintended outcomes, program

sustainability, program efficiency, costs and benefits Sometimes called impact evaluations and ex-post

evaluations


18

Program Evaluation

Summative Evaluation Question:

Do Public Programs Work? Implied cause-effect relationship

Did the program cause a desired outcome? Performance-based:

Focus on outcomes, results, impacts, goal achievement.


19

Differences

Formative Evaluations Project Monitoring Early Years of Implementation Key Question:

Are we doing things right?– Have we hired the right people with the right skills?

– Have we marketed the program effectively?

– Have we met our strategic objectives?

– Have we spent our money according to our plan?


20

Differences

Summative Evaluations Measuring Results or Impacts A longer time before results or impacts are visible Key Question:

Are we doing the right thing? This gets back to the theory or underlying assumptions of the

program:– We can do an excellent job at training people but if the problem

is not about the larger structural economic issues, a training program, no matter how well implemented, may show little result.


21

Working with Models

Visualize a program in context Systems approach, within an environment

Identify the relationships between various components

Identify cause and effect Identify key assumptions


22

Models: Cause and Effect:

Did the program cause something to happen?

Education Employment


23

Reduced Poverty

Improved Quality of Life

Increased Income

Job

Training

Unemployed

Hierarchy of Objectives


24

Logic Models

The focus is on results or impacts rather than inputs and activities We are not training people just for the sake of training

people We believe if we train the chronically unemployed,

then there quality of life will be improved and poverty will decrease.

Our goal is to reduce poverty Also called Program Outcome Model or

Measuring for Results


25

Logic Model

Inputs

Activities

Outputs

Outcomes

Impact


26

Elements of the Logic Model

Inputs: what resources are usedUniversity inputs: budget, number of

faculty, number of staff, number of buildings, number of classrooms

Activities: what the program doesUniversity activities: teaching, research,

and service


27


Outputs: the services or products produced University outputs: number of students that

graduate, number of articles and books published by faculty

Outcomes: what happened: immediate results Graduates are sought after, get good jobs,

active alumni who donate big bucks Faculty well-known, obtain big grants, enhance

rating of university


28


Impacts: the “so what.” Larger, long term results, usually tied to program goals. A more informed and engaged citizenry,

preserves democratic institutions, future leaders. Faculty research contributes to knowledge.


29

Logic Model

Logical Connections: Inputs to do activities Activities lead to outputs Outputs lead to one or more outcomes Outcomes lead to impacts


30

Logic Model: Training Program

Inputs Activities Outputs

Resources

•money

•staff

•Supplies

•mentors

•Training Programs

•Dress for success coaching

•Interview coaching

•Resume assistance

Products

•Number of graduates per training session

•% graduate rate

Outcomes

Benefits changes

•Increased skills

•% Obtain jobs

•% Obtain high paying, quality jobs

•Increased self-esteem

Impacts

Goals

•Increased income

•Self-sufficiency

•Family stability

•Reduction in poverty


31

Takeaway Lesson

Program evaluation takes many forms but all follow the same research planning process.

Evaluation of programs have shifted from reporting inputs and activities to attempting to measure results: the difference the program actually made.


32

Takeaway Lesson

Measuring results is harder than it appears. Program goals and objectives may be fuzzy. Sites may vary in how they have implemented

the program. It takes money to collect and analyze data. Results may not be observable for many years. The operating environment may make it hard to

see results.


33

Takeaway Lesson

Measuring results is harder than it appears. But there is much that can be learned from

engaging in this process and doing the best job possible.

Remember: do not quickly conclude a program does not work just because you cannot measure the result. The research tools available may not be up to the job.


34

Creative Commons

This powerpoint is meant to be used and shared with attribution

Please provide feedback If you make changes, please share freely

and send me a copy of changes: [email protected]

Visit www.creativecommons.org for more information

Dr. G. Johnson, Program Evaluation and the Logic Model Research Methods for Public Administrators Dr. Gail Johnson.

Documents

program delivery

implementation slide

program infrastructure

mature program

gail johnson

actual program mission

evaluation useful

fear of evaluation