Measuring and Communicating the Impact of your Coalition ... · Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute

Measuring and Communicating

the Impact of your

Coalition and Programs

Brian Bumbarger

Coalition Leadership Institute

June 2010

If you don’t know where you’re going,

any road will get you there!

The Cheshire Cat, Alice in Wonderland

It is not enough to be busy. So are the ants. The question is: What are we busy about?

Henry David Thoreau

Don’t mistake movement for progress.

They might feel the same, but just shaking your butt won’t get you to

Disney World

Brian Bumbarger

• Create community-level outcomes.

• Investigate potential programs, policies and practices.

• Select programs, policies and practices.

• Create program-level outcomes.

• Plan to evaluate.

• Create a written Community Action Plan.

Evaluate (v.): to determine the significance,

worth, or condition of usually by careful

appraisal and study

Assess (v.) : to determine the importance,

size, or value of

Pre-test Post-testPreventionProgram

We can compare the difference between pre-test and post-testBut…

• What if something else caused the change?

• If the post-test isn’t better, does that mean the program didn’t work?

• What would have happened in the absence of the program?

• Are these kids representative of all kids?

Pre-test Post-testPreventionProgram

Post-testPre-testPreventionProgram

Randomization

Pre-test PreventionProgram Post-test

Random

Se l ect i on

Fo l l owup

Data

• Create community-level outcomes.

• Investigate potential programs, policies and practices.

• Select programs, policies and practices.

• Create program-level outcomes.

• Plan to evaluate.

• Create a written Community Action Plan.

• Gives clear direction toward achieving community vision

• Provides built-in evaluation measures and accountability

• Required by many grant makers

(10 to 15 years)

• Increase in protective factors

• Decrease in risk factors

• Decrease in problem behaviors

• Changes in participant knowledge, attitudes, skills or behavior

• Program implementation fidelity

(3 to 10 years) (6 months to 2 years)(1 to 4 years)

The Community Board will learn how to develop outcome-focused prevention plans.

Participants will be able to:

1. Define outcome-focused planning.

2. Write desired outcomes:

– Behavior outcomes

– Risk-factor outcomes

– Protective-factor outcomes

3. Prepare for community evaluation.

• Measuring changes

• Measuring progress

• Knowing what actions will take place

1. Change in what

2. As measured by

3. The baseline or starting point for comparison

4. By how much, by when

Community Plan Implementation Training 1-<#>

1. Priority risk-factor outcome to change

2. Indicator used to measure the outcome

3. Baseline data

4. How much change, by when

Community Plan Implementation Training 1-<#>

• Identify targets for prevention-planning efforts.

• Evaluate the impact of prevention efforts over time.

• Engage Key Leaders and community members in supporting prevention.

• Know the data to be collected.

• Know who is responsible for data collection.

• Know how outcomes will be reported.

• Determine what resources will be needed for future assessments.

Community Plan Implementation Training 1-<#>

Participants will learn how to develop participant and implementation outcomes in preparation for implementing and evaluatingtheir selected programs,policies and practices.

Participants will be able to:

1. Develop participant outcomes.

2. Develop implementation outcomes.

3. Identify the elements of implementation.

• Participant outcomes measure the changes a program produces.

• Implementation outcomes measure the process by which a program produces desired changes.

• Knowledge

• Attitudes

• Skills

• Behavior.

• Who the program will be delivered by

• When the program will be delivered, including how often and how long

• Where the program will be delivered

• How the program will be delivered

• Number of people to be affected by the program

• Who your target audience will be.

Community Plan Implementation Training 1-<#>

Participants will be able to:

1. Identify the uses of evaluation.

2. Explain basic evaluation concepts.

3. Assess the need for an expert evaluator.

Evaluation helps you:

• monitor implementation fidelity

• monitor progress toward outcomes

• identify problems with program design or selection

• demonstrate achievements

• determine cost-effectiveness.

• What do you want to evaluate?

• What is the purpose of your evaluation?

• Community Board members

• Key Leaders

• Program implementers and site administrators

• Media

• Local interest groups

• Other community members

• Sources of funding

• Monitor implementation fidelity.

• Identify and correct implementation problems before they result in program failure.

• Records

• Activity logs

• Observation

• Questionnaires

• Measure the program’s impact on participant knowledge, skills, attitudes or behavior.

• Monitor progress toward the community’s vision.

• Observation

• Interviews

• Questionnaires

• Records

An expert evaluator can help:

• select appropriate evaluation design

• develop evaluation measures

• analyze statistical information

• interpret results.

• The program requires significant technical assistance

• Evaluation tools will need to be developed

• Data collection will be expensive and complex

• Complex data analysis will be required

• The Board lacks the expertise to complete the evaluation.

• Local universities

• Research and consulting firms

• The nature of the evaluation

• Who will be doing the evaluation

• Resources needed for the evaluation.

Community Plan Implementation Training 3-42

To help ensure that programs, policies and practices in the Community Action Plan are implemented with fidelity.

Community Plan Implementation Training 3-43

Participants will be able to:

1. Explain the elements of successful, high-fidelity implementation.

2. Develop implementation plans.

3. Identify methods of monitoring implementation for fidelity.

Community Plan Implementation Training 3-44

High-fidelity implementation means that:

• the target audience is reached

• all components of the program are delivered according to the original design

• the program is delivered at the intended dosage (duration, frequency).

Community Plan Implementation Training 3-45

Studies have found that:

• the effectiveness of any program is significantly increased when the program is implemented with fidelity

• some programs are only effective when implemented with a high degree of fidelity.

(Blueprints, 2001)

Community Plan Implementation Training 3-46

• Support of site administrators

• Qualified implementers who support the program

• Planning meetings and trainings

• A detailed implementation plan(who, what, where, when and how)

• Ongoing monitoring and technical assistance

Community Plan Implementation Training 3-47

Select implementers who:

• have the credentials prescribed by the program

• support the program approach

• are able to reach/communicate with target audience.

Community Plan Implementation Training 3-48

• Involve implementers in the planning process.

• Provide implementers with clear information about the program.

• Train implementers to deliver the program.

• Consider a written participation agreement.

Community Plan Implementation Training 3-49

• To educate stakeholders and build support for the program

• To negotiate implementation logistics

• To develop detailed implementation plans

• To obtain signed participation agreements

Community Plan Implementation Training 3-50

Implementers should leave trainings with:

• a clear understanding of the program goals, rationale, desired outcomes, components and methods

• confidence that they have the skills and expertise to deliver the program effectively.

Community Plan Implementation Training 3-51

Include:

• target audience and expected number of participants

• plans for recruiting participants

• who the implementers are

• schedule for implementing all components.

Community Plan Implementation Training 3-52

Community Plan Implementation Training 1-<#>Community Plan Implementation Training 3-53

• To ensure that the program is implemented with fidelity to the original design.

• To identify and correct implementation problems.

• To fulfill requirements of funders.

• To provide “lessons learned” for future implementation efforts.

• To identify and celebrate early successes.

Community Plan Implementation Training 3-54

• The target audience is not participating in great enough numbers.

• The program components are not being delivered at the intended dosage.

• Program delivery methods are not being used.

Community Plan Implementation Training 3-55

Monitor:

• planning meetings before implementation

• training of implementers

• program delivery.

Community Plan Implementation Training 3-56

• Who attended the trainings?

• What was covered at the trainings?

• Did implementers leave trainings with the skills, confidence and motivation to deliver the program effectively?

Community Plan Implementation Training 3-57

• How were participants recruited?

• How many members of the target audience participated?

• Was the program delivered at the correct dosage?

• Did implementers use methods prescribed in the program design?

• Did implementers communicate effectively with the target audience?

Community Plan Implementation Training 3-58

Community Plan Implementation Training 1-<#>Community Plan Implementation Training 4-5

1. Engage stakeholders.

2. Define what will be evaluated.

3. Select an evaluation design.

4. Decide how you will collect and measure data.

5. Gather, analyze and interpret data.

6. Report your findings and use the results.

Community Plan Implementation Training 4-7

• Use the instrument designed for the program.

• If no instrument is available, seek help from an expert evaluator.

Community Plan Implementation Training 4-15

Consider:

• commitment to the project and to the evaluation

• objectivity

• ability to collect complete and accurate data.

Community Plan Implementation Training 4-18

Determine:

• who will oversee the evaluation

• a central storage place for data from each program

• evaluation deadlines

• decision-making processes.

Community Plan Implementation Training 4-19

• Choose an appropriate format.

• Write for your most important audience.

• Discuss results in terms of the overall program goals.

• Be honest.

Community Plan Implementation Training 4-20

Identify individuals with:

• skills in designing evaluation instruments

• skills in designing and conducting evaluation studies

• skills in data analysis and presentation

• the ability to objectively assess outcome results and recommend possible solutions

• the ability to credibly communicate results to key stakeholders.

Community Plan Implementation Training 4-24

• Celebrate success.

• Write a report detailing the results.

• Fulfill accountability requirements.

• Identify causes of unmet expectations.

• Revise and update the Community Action Plan.

Community Plan Implementation Training 4-25

• Weak connection between the program and outcomes

• Unrealistic outcomes

• Failure to implement a program with fidelity

• Participant-related reasons

• External factors

• Measurement problems

• Decreases in sample size (attrition)

Community Plan Implementation Training 4-26

• Celebrate success.

• Identify changing priorities.

• Select new tested, effective prevention strategiesto address changing priorities.

• Assist resource allocation decisions.

• Evaluate policies and practices.

Community Plan Implementation Training 4-30

If you want to build a ship, don't drum up the

men to gather wood, divide the work, and

give orders. Instead, teach them to yearn for

the vast and endless sea.

-- Antoine de Saint-Exupéry

Thank You!Brian K. Bumbarger

Evidence-based Prevention and Intervention Support Center

Prevention Research Center, Penn State University

206 Towers Bldg.

University Park, PA 16802

(814) 865-2618

www.prevention.psu.edu

Programs

(6 mos. - 2 years)

What kind of

future do you

envision for

your

community’s

children ?

What changes in

ATOD use,

delinquency,

violence, teen

pregnancy,

dropout will be

necessary to

achieve the

vision ?

What changes in

protective

factors are

needed to

reduce problem

behaviors ?

What changes in

risk factors are

needed to

reduce problem

behaviors ?

How will your

programs &

services change

risk &

protective

factors ?

Outcome-Based Planning Logic Model

Reduction in

problem

behavior

(3 - 10 years)

Increase in

Protective

Factors

(1-4 years)

Decrease in

priority risk

factors

(1-4 years)

Vision

(10 – 15 years)

Vision

Statement

Problem

Behaviors

Protective

Factor

Prevalence

Risk Factor

Prevalence

Program

Outcomes