Community-Based Research Workshop Series CBR 304 A Participatory Approach to Programme Evaluation.

Community-Based Research Workshop Series

CBR 304 A Participatory Approach to

Programme Evaluation

2

Things we will cover

• Why program evaluation? • Review lots of jargon.• Creating program logic models.• Using PLMs as a tool for evaluation design.

3

Things we will not cover

• Specific evaluation designs• Qualitative/quantitative methods

4

Introduction

• Your name• Organization you are associated with• What is the first word that comes to mind when you hear

evaluation?

5

Introduction continued

• What is evaluation?• Why should we evaluate?• Who should do evaluations?• How often should evaluation be done?• Who uses evaluation findings?

6

What is Evaluation?

• “Program evaluation is a collection of methods, skills and sensitivities necessary to determine whether a human service is needed and likely to be used, whether the service is sufficiently intensive to meet the unmet needs identified, whether the service is offered as planned, and whether the service actually does help people… at a reasonable cost without unacceptable side effects” – Posavac & Carey, 1997, p.2

7

Why Evaluate?

• To assess the needs of the community• To devote resources to unmet needs• To verify that a program is providing intended services • To determine which services provide the best results• To assess what processes are effective in delivering and managing

programs• To provide information needed to maintain and improve quality of

program

8

Who Evaluates?

• Program staff• Independent / external consultants• Academics• Program users• Funders

9

How often?

In a regular program cycle:• Too early – not enough to evaluate• Too late – become crisis management

10

Who Uses Evaluation Findings?

• Program planners• Program staff• Program management• Program funders• Policy makers• Legislators• Service users

11

Traditional vs CBR Evaluation

Traditional CBR

Outside expert Team of stakeholders

Expert defines problems and solutions

Stakeholders collectively decide focus of evaluation

Report may or may not be used for change

Early buy-in from stakeholders increases likelihood of uptake

Capacities leave with expert Capacity is built internally

12

Participatory Continuum

Participatory nature depends on where questions originate from and where decision making power lies:

• Researcher /Funder • Researcher consults community • Community based

13

Small Group

• What freaks you out about evaluation?• What makes it challenging?• What makes it exciting?

14

D2D CASE STUDY

• D2D is a street youth serving agency that wants to respond to the needs of street involved youth in thecommunity. D2D offers an all night drop in with meals, access to health care workers, computer classes, and a mentorship program. They run from 7 pm to 7 am, 5 days a week. They have 10 staff and serve 500 kids a weeks. Their funding comes from the Ministry of Children and Family, drug prevention money, and a ‘youth resiliency and empowerment’ grant from the Z Foundation. They want to evaluate their services.

15

D2D CASE STUDY

1) Who do you think should be on your evaluation team? (Why?)

2) What are some immediate questions you have for the D2D?

3) Where would you start?

Where to start?

A good programme plan!

17

Planning & Evaluation Cycle

Plan Programme

Establish Need

Act on findings

Assess Results

ImplementProgramme

18

Establishing Need

• Walking tours• Interviews with formal and informal leaders• Community forums• Voting with your feet• Visioning process• Photovoice• Literature review• Client data

The first step in evaluation:

Articulating what you are doing and why…

(in other words clarify – your goals, objectives and activities)

What you are doing? (practice)Why you think it should work? (theory)

What will change as a result of your efforts? (evaluation)

20

Essential Components of Programme Plans

• Goals: broad visioning statements– e.g. “To promote the birth of healthy babies”

• Objectives: Specific things you would like to see changed– e.g. “To reduce substance use among pregnant women”

• Activities: What you will do to make your goals and objectives happen– e.g. “Provide substance use treatment program for pregnant women”

21

D2D

How would you help the D2D articulate its • Goals?• Objectives?• Activities?

22

Program Logic Model

A flow chart which depicts the logical relationships between program activities and the changes expected to occur as a results of these activities.

- United Way PEOD

23

Program Logic Models – Elements

INPUTS:

Resources dedicated to program

e.g. money, staff, volunteers

facilities, supplies

ACTIVITIES:

What the programdoes with

inputs

e.g. sheltering,

feeding, training, education

OUTPUTS:

Direct productsof programactivities

e.g. # of youth accessing

centre,hours of contact,

meals served

OUTCOMES

Benefits forParticipants:1) Immediate2) Short term3) Long term

CHANGES!

24

Teen Sexual Health Information Program

INPUTS:2 staff

$130,000/yearTraining space

Web ServerPhone Lines

ACTIVITIES:

Trains 100 peersexual health

counsellors

Provide face to facepeer counselling

Host peer web site

Host peer phone line

OUTPUTS:

Meet with 100 youthper week face to face

Field 100 calls/night

Field 1,000 onlinequestions/month

OUTCOMES

Goal: To empower teens to make healthy sexual decisions.

25

Outcomes

• Immediate: youth get advice they need, youth learn new things (knowledge)

• Short term: greater self-esteem, increased condom use

• Long term: Fewer STIs, fewer pregnancies, youth empowered to make ‘healthy’ sexual decisions

26

Create a logic model for D2D!

INPUTS:

ACTIVITIES:

OUTPUTS:

OUTCOMES

1) Immediate2) Short term3) Long term

CHANGES!

What to evaluate?

So many options…

28

Aspects of a program that can be evaluated

• Effort – resources available and used

• Execution – adequacy of delivery

• Efficacy – benefits to clients

• Effectiveness – attainment of outcome

• Efficiency – achievement/costs

29

3 Types of Evaluation

• Formative or Process Evaluation• Outcome or Impact Evaluation• Economic Evaluation

30

Relationship between Types of Evaluation: Quit Program

PROCESS IMPACT

Short term

OUTCOME

Long term

ECONOMIC

Practice/

programme

Effect Benefits Cheaper

e.g.

What happened? Did people like it? Why?

e.g.

Did people stop smoking?

e.g.

Lower rates of smoking disease?

e.g.

Is prevention cheaper than treatment?

31

Process? Outcome?

• Number of people attending the sessions• Level of satisfaction with sessions• Behaviour change (short and long term)• Number of clients that come back to a session• Fewer illnesses resulting

32

Making decisions

• If we only focus on process – we will never know about outcome

• If we only focus on outcome – we will never know why a programme works or doesn’t

A good evaluation should have elements of both that inform each other!

33

Deciding on D2D

• Will you focus on process? Why?• Will you focus on outcome? Why?• Which elements of process or outcome are you

interested in zeroing in on?

34

From model to indicators

INPUTS:

ACTIVITIES:

OUTPUTS:

OUTCOMES

1) Immediate2) Short term3) Long term

CHANGES!

35

From model to indicators

Indicator Definition: Indicators are ways of phrasing your evaluation strategies…

• Indicators should be directly related to your expected outcomes

• Indicators should be measurable• Indicators should have a time element• You can have both ‘process’ and ‘outcome’ indicators!

36

From model to indicators

PROGRAMME OUTCOME INDICATOR

Homework programme Students perform at grade level

% participants who earn passing marks in next report card

Prenatal care for substance abusing women

Reduction in alcohol consumption

% participants who report no alcohol consumption in 3rd trimester

37

TYA Indicators

• Try and create some indicators for D2D (remember they should come directly from your outputs and outcomes)

• How will you collect them?• What resources will you need to put in place?

38

Wrap-up

• Outstanding Questions

39

Workshop Evaluation

Your feedback is extremely important!

Please complete the workshop evaluation….

Thank you!