Driving Quality: Making the Difference through Program Evaluation Processes Ben Kearney, PhD Guidestone.

Driving Quality: Making the Difference through Program

Evaluation Processes

Ben Kearney, PhD

Guidestone

“I think you should be more explicit here in Step Two.”

… thought Alice and she went on, “Would you tell me, please, which way I ought to go from here?”

“That depends a good deal on where you want to get to,” said the Cat.

“I don’t much care where —” said Alice.“Then it doesn’t matter where you go,”

said the Cat.“— so long as I get somewhere,” Alice

added as an explanation.“Oh, you’re sure to do that,” said the

Cat, “if you only walk long enough.”(Carroll, 1865, p. 51)

• What gets measured gets done

• If you don’t measure results, you can’t tell success from failure

• If you can’t see success, you can’t reward it

• If you can’t reward success, you’re probably rewarding failure

• If you can’t see success, you can’t learn from it

• If you can’t recognize failure, you can’t correct it.

• If you can demonstrate results, you can win public support.

Osborne and Gaebler, 1992

Accountability

Improvement Process

• Data

• Information

• Knowledge

• Wisdom

Inherent Questions

• Does the program work as designed—correlated process measures with outcome measures?

• How does the program compare with other programs and against organizational/contractual/external benchmarks?

• If effectiveness indicators are met, what efficiency changes can be made to maintain quality, but increase client saturation?

A priori assumptions

– Therapy works– Common factors help facilitate change

• Empathy, increasing awareness, therapeutic relationship

– Specific interventions permits change to happen even faster

– We must be responsible stewards– Make an IMPACT

Components of Program Evaluation

• Theory of Change• Logic Models

– Process Measures (including Fidelity Measures)

– Outcome Measures • Short term—during treatment • Mid term--At the end of treatment • Long term—at some point following treatment—

depends on logic model and change theory

• Status reports

Theory of Change vs. a Logic Model

• a theory of change defines how and why you expect the desired outcomes to occur and usually applies to several programs or the whole organization– theories of change clarify why you are doing what you

are doing • a logic model visually presents your

understanding of the relationships among your program’s resources, planned activities and anticipated results and usually applies to a single program. – Logic models clarify what you are doing

Theory of Change vs. a Logic Model

Theory of Change• Links outcomes and activities to explain how and why

the expected change will occur• Usually starts with a goal before deciding on

programmatic components• Requires justification for program components; specifies

the hypothesis about why something will cause something else

• Requires identifying indicators to measure outcomes• Best used to design and evaluate a complex initiative

Theory of Change vs. a Logic Model

Logic Model• Graphically illustrates program components, identifies,

inputs, activities and outcomes• Usually starts with a program and illustrates its

components• Requires identification of program components, but

doesn’t show why activities are expected to produce outcomes

• Sometimes includes indicators to measure outcomes• Best used to demonstrate you have carefully identified

the inputs, outputs and outcomes of your work

Evaluation

• A logic model is the first step in evaluation. Through evaluation, we test and verify the reality of the program theory – how we believe the program will work.

• A logic model helps us focus on appropriate process and outcome measures.

An Expanded Logic Model

Logic Model Components

• Situation

• Inputs

• Outputs—the measurement of:– Activities– Personnel

• Outcomes

The situation…

• …is the foundation for logic model development.

• The problem or issue that the program is to address sits within a setting or situation--a complex of sociopolitical, environmental, and economic conditions.

Situational Questions

• What is the problem/issue? • Why is this a problem? (What causes the problem?) • For whom (individual, household, group, community,

society in general) does this problem exist? • Who has a stake in the problem? (Who cares whether

it is resolved or not?) • What do we know about the problem/issue/people that

are involved? What research, experience do we have? What do existing research and experience say?

Inputs

• Inputs are the resources and contributions that you and others make to the effort, the resources invested that allow us to achieve the desired outputs. These include time, people (staff, volunteers), money, materials, equipment, partnerships, research base, and technology among other things.

• These inputs allow us to create outputs

Outputs

• Outputs are the activities, services, events, and products that reach people (individuals, groups, agencies) who participate or who are targeted.

• Outputs are "what we do" or "what we offer." They include workshops, services, conferences, community surveys, facilitation, in-home counseling, etc.

• These outputs are intended to lead to specific outcomes.

Outcomes

• Outcomes are the direct results or benefits for individuals, families, groups, communities, organizations, or systems. Examples include changes in knowledge, skill development, changes in behavior, capacities or decision-making, policy development. Outcomes can be short-term, medium-term, or longer-term achievements. Outcomes may be positive, negative, neutral, intended, or unintended.

Outcomes

• In the past, we've tended to focus on what is included in the outputs column - the "what we do and who we reach."

• We are anxious to tell our clients, funders and community partners what it is that we do, the services we provide, how we are unique, who we serve

• Now, the question is: "What difference does it make?" This is a question about OUTCOMES.

• The newest question is: “How does what you do produce the differences being made?” This question links OUTPUTS to OUTCOMES.

Impact

• the ultimate consequence or effects of the program--for example, increased economic security, reduced rates of teen smoking, improved air quality.

• the ultimate, long-term outcome or your programmatic goal.

• Impact refers to the ultimate, longer-term changes in social, economic, civic, or environmental conditions.

Assumptions

• Assumptions are the beliefs we have about the program and the people involved and the way we think the program will work. This is the "theory" we are talking about: the underlying beliefs in how it will work. These are validated with research and experience. Assumptions underlie and influence the program decisions we make.

More about Outcomes…

• Outcomes often fall along a continuum from shorter- to longer-term results. This continuum is called an "outcome line" (Mohr, 1995), the "outcome sequence chart" (Hatry, 1999), or "outcome hierarchy" (Funnell, 2000). This concept--a series of outcomes that are connected--is fundamental to a logic model.

More about Outcomes…

For our process:

• short-term—accomplished during the program

• medium-term—accomplished at the end of program

• long-term—accomplished at some point after the program has ended

Starting at the End

• When planning, start where you want to end. – Identify the long-term outcome(s). – What is your end goal? – What will be different? – How will the community, producers, local citizens, the

environment, be different as a result of the program?

• Once you have that long-term outcome (end result, goal) identified, then work backwards across the logic model

Outpatient Counseling Fidelity Checklist

• The worker has utilized Nurturing Parenting principles and philosophies to guide their interaction with the referred child and/or family.

• The therapist has established an optimal therapeutic alliance with the client and/or family by providing a secure base from which the client can explore the various unhappy and painful aspects of his/her life, past and present, through support, encouragement, sympathy, and guidance.

• The therapist has assisted the client in identifying important attachment relationships; in recognizing the role they play in their current functioning; to rework unhealthy attachment related behaviors/patterns; and to promote stronger relationships between the client and their children or loved ones in the present and for the future.

• The therapist has promoted resiliency and generational change in the client and/or family.

• The therapist has utilized Cognitive Behavioral Therapy (CBT) techniques to assist the client with monitoring and reduction of mental health symptoms.

Outpatient Counseling Fidelity Checklist

• The therapist has assessed for the client’s commitment to change; applied appropriate intervention technique; and promoted client belief in the notion that therapy will help.

• The therapist has facilitated the client’s awareness of the factors that maintain his or her difficulties.

• The therapist has encouraged and assisted the client to consider the ways in which he or she engages in relationships with significant figures in his or her current life.

• The therapist has helped the client recognize and change images of self and others that contribute to difficulties.

• The therapist has encouraged the client to engage in corrective emotional experiences.

• The therapist has emphasized ongoing reality testing in the client’s life.

• The therapist has worked to promote openness to empathy in the client.

Helpful evaluation tools

• Website data entry• Built-in specific reports• Specific tools for analysis• Statistical change• Clinical change• Analysis tools• Run Charts• Pareto charts• Fishbone-cause and effect• Correlations/regressions

Fishbone

Pareto Chart

Pareto Chart-example

Pareto Chart by Cottage - December 2010

7 7

4

2

1

0

33.3%

66.7%

85.7%

95.2%

100.0%

0

5

10

15

20

Open ResidentialBoys 8-17

(Crossman/BF)

Open Residential Girls8-17 (ML)

SHARP 8-12 Boys(Hilton)

SHARP 13-17 Boys(Burris)

STC Crisis/STEP

Cottage

Nu

mb

er o

f R

estr

ain

ts

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Run Chart--Example

Sharp 13-17 Boys 2003-2010.12 Restraint Numbers by Month

17.9UCL

7.7CL

-2.4LCL

-6.0

-1.0

4.0

9.0

14.0

19.0

24.0

29.0

34.0

39.0

44.0

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97

Month

Nu

mb

er o

f R

estr

ain

ts

Evaluation Potholes

• Be aware where the evaluator’s purpose is different than the clinicians

• Be aware when the evaluator’s purpose is the same as the clinicians

• Be clear about roles and boundaries, but be willing to be flexible when needed as well

ex: CCO, Evaluator

Evaluation Potholes

• Use well established instruments

• Computerization combines data gathering, editing, and entering into one step

• Computerization permits automatic analysis on predetermined factors

• Don’t be afraid to pilot—Pilot when you can. Force field analysis for pilot—driving and restraining forces.

Current work

Spirituality

• The openness to and the awareness of the transcendent, the sacred, that which is beyond us; and by this knowledge, recognizing the connectedness and responsibility we have to each other.

Current Work

• Moral emotional processes-build communication and collaboration, transmit culture– Quietness, openess, mindfulness, awareness,

regulation, empathy

• Transformational emotional processes– Hope, altruism, sympathy, truth (joy), narrative

(contentment)

References

• Beckworth, L., (2000). Prevention science and prevention programs. In C. H. Zeanah, Jr. (Ed.), Handbook of infant mental health (pp 439-456). New York: Gilford Press.

• Breyfogle III, F. W. (2003). Implementing Six Sigma: Smarter Solutions Using Statistical Methods (2nd ed.). Hoboken, NJ: John Wiley & Sons, Inc.

• Brusse, W. (2004). Statistics for Six Sigma Made Easy! New York, NY: McGraw-Hill.

• Carey, Ph.D., R. G., & Lloyd, Ph.D., R. C. (2001). Measuring Quality Improvement in Healthcare: A Guide to Statistical Process Control Applications. Milwaukee, WI: ASQ.

References

• Cavanagh, R. R., Neuman, R. P., & Pande, P. S. (2000). The Six Sigma Way: How GE, Motorola, and Other Top Companies are Honing Their Performance. New York, NY: McGraw-Hill.

• Cavanagh, R. R., Neuman, R. P., & Pande, P. S. (2002). The Six Sigma Way. New York, NY: McGraw-Hill.

• Hayes, R. A., & Stout, C. E. (Eds.) (2005). The Evidence-Based Practice: Methods, Models, and Tools for Mental Health Professionals. Hoboken, NJ: John Wiley & Sons, Inc.

References

• Langley, G. J., Moen, R. D., Nolan, K. M., Nolan, T. W., Norman, C. L., & Provost, L. P. (2009). The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd ed.). San Francisco, CA: Jossey-Bass.

• Parlakian, P. and Seibel, N.L. (2002). Building strong foundations: Practical guidance for promoting the social-emotional development of infants and toddlers. Washington: ZERO TO THREE Press

• Stiffman, A. R. (Ed.) (2009). The Field Research Survival Guide. New York, NY: Oxford University Press.