John R. Kasich, Governor Tracy J. Plouck, Director Kraig J. Knudsen, PhD, LISW * Slides have been adapted from UW Extension training on Logic Models.

DEVELOPING A LOGIC MODELEdited version April 20, 2015

John R. Kasich, GovernorTracy J. Plouck, Director

Kraig J. Knudsen, PhD, LISW

Promoting Wellness and Recovery

*Slides have been adapted from UW Extension training on Logic Models

Introduction and Overview

-Why evaluate your Program/Projects?

-What does evaluation do?

Why Evaluate?

• Determine program outcomes• Identify program strengths• Identify and improve weaknesses• Justify use of resources• Increased emphasis on accountability• Professional responsibility to show effectiveness of program

What Evaluation Does…

Looks at the results of your investment of time, expertise, and energy, and compares those results with what you said you wanted to achieve

A Logic Model is…

•A depiction of a program/project showing what the program will do and what it is to accomplish.

•A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes

•The core of program planning and evaluation

A LOGIC MODEL AT ITS MOST BASIC LEVEL

INPUTS OUTPUTS OUTCOMES

A Simplified Logic Model….?

A LOGIC MODEL CAN BE APPLIED TO:

• A Small Program• A Process (i.e. a team working

together)• A large, multi-component program• Or even to an organization or

business

“If you don’t know where you are going, how are you gonna’ know when you get there?”

Yogi Berra

Where are you going?

How will you get there?

What will show that you’ve arrived?

Many people say a logic model is a road map

Logic model may also be called…

• Theory of change• Program action• Model of change• Conceptual map• Outcome map• Program logic

Accountability Era

•What gets measured gets done•If you don’t measure results, you can’t tell success from failure•If you can’t see success, you can’t reward it•If you can’t reward success, you’re probably rewarding failure•If you can’t see success, you can’t learn from it•If you can’t recognize failure, you can’t correct it.•If you can demonstrate results, you can win public support. Reinventing Government, Osborne and Gaebler, 1992

What is the benefit of a Logic Model?

•Focus on and be accountable for what matters-OUTCOMES•Provides common language•Makes assumptions EXPLICIT•Supports continuous quality improvement•Promotes Communication•public support. Reinventing Government, Osborne and Gaebler, 1992

LOGIC MODEL COMPONENTS

Logic Model: Everyday Example

HEADACHE

Feel betterGet pills Take pills

Situation INPUTS OUTPUTS OUTCOMES

Logic Model: Everyday Example

HU

NG

AR

Y

Feel BetterGet food Eat Food

Situation INPUTS OUTPUTS OUTCOMES

Defining the Situation• What problematic condition exists that demands a programmatic response?

• Why does it exist? • For whom does it exist? • Who has a stake in the problem? • What can be changed?

• If incorrectly understood and diagnosed, everything that flows from it will be wrong.

• Factors affecting problems: protective factors; risk factors

• Review research, evidence, knowledge-base

• Traps: • Assuming we know cause: symptoms vs. root causes.• Framing a problem as a need where need is actually a program or service.

“Communities need leadership training” Precludes discussion of nature of the problem: what is the problem? Whose problem? Leads one to value provision of the service as the result – is the service provided or not?

INPUTS

WHAT WE INVEST– STAFF

– VOLUNTEERS– TIME

– MONEY– RESEARCH BASE

– MATERIALS– EQUIPMENT

– TECHNOLOGY– PARTNERS

OUTPUTS

WHAT WE DO– TRAIN, TEACH

– DELIVER SERVICES– DEVELOP PRODUCTS

– NETWORK WITH OTHERS

– BUILD PARTNERSHIPS– ASSESS

– FACILITATE– WORK WITH THE

MEDIA

WHO WE REACH– PARTICIPANTS

– CLIENTS– CUSTOMERS– AGENCIES

– DECISION MAKERS– POLICY MAKERS

OUTCOMES

SHORT-Learning– Changes in:

• Awareness• Knowledge• Attitudes• Skills• Opinion• Aspirations• Motivation• Behavioral Intent

MEDIUM-Action– Changes in:

• Behavior• Decision-making• Policies• Social action

LONG TERM-Conditions– Changes In:

• Conditions• Social (Well-Being)• Health• Economic• Civic• Environmental

Language Used

– Goal=Impact

– Impact=Long Term Outcome

– Objectives (participant focused)= Outcomes

– Activities=Outputs (also can be products as a result of activities)

Logic Model Basic Components

Resources

The inputs dedicated to or consumed by the program

Activities

The actions that the program takes to achieve desired outcomes

Outputs

The measurable products of a program’s activities

Outcomes

The benefits to clients, communities, systems, or organizations

Program Goal: overall aim or intended impact

How? Why? So what?

The Logic Model is a Series of “IF, THEN” statements that express a theory of Change…

Resources

Activities

Certain resources are needed to run your program

IF you have access to them, THEN you can accomplish your activities

IF you can accomplish these activities THEN you will have delivered the services you planned

IF you have delivered the services as planned THEN there will be benefits for clients, communities, systems or organizations

Outputs Outcomes

What Does A Logic Model Look Like

• Graphic display of boxes and arrows; vertical or horizontal

• Relationships, linkages• Any shape possible

• Circular, dynamic• Cultural adaptations; storyboards

• Level of detail• Simple• Complex

• Multiple models• Multi-level programs

• Multi-component programs

ASSUMPTIONS

•Assumptions underlie much of what we do. It is often these underlying assumptions that hinder success or produce less-than-expected results. One benefit of logic modeling is that it helps us make our assumptions explicit.

ASSUMPTIONS•The beliefs we have about the program, the participants, and how the program will work. Includes ideas about:

– the problem or existing situation– program operations– expected outcomes and benefits– the participants and how they learn, behave, their motivations– resources– staff – external environment: influences– the knowledge base– etc.

Everyday Example: Family Vacation

Family Members

Budget

Car

Camping Equipment

Drive to state park

Set up camp

Cook, play, talk, laugh, hike

Family members learn about each

other; family bonds; family has

a good time

INPUTS OUTPUTS OUTCOMES

A youth financial Literacy Program

Partners invest resources

A high school financial planning program – 7 unit curriculum - is developed and delivered in high schools

Teens gain knowledge and skills in money management

Teens establish sound financial habits

Teens make better decisions about the use of money

Staff

Money

Partners

Assess parent ed programs

Design- deliver evidence-based program of 8 sessions

Parents better understanding their own parenting style

Parents use effective parenting practicesResearch Facilitate

support groups

Parents gain skills in new ways to parent

Parents of 3-10

year olds

attend

Parents gain confidence in their abilities

Parent Education Program – Logic model

Improved child-parent relations

INPUTS OUTPUTS OUTCOMES

Parents identify appropriate actions to take

Reduced stress

SITUATION: During a county needs assessment, majority of parents reported that they were having difficulty parenting and felt stressed as a result

Trainer

Funds

Equipment

Research base

Training curriculum

Situation: Funder requires grantees to include a logic model in their funding request; grantees have limited understanding of logic models and are unable to fulfill the funding requirement

- Participants will increase knowledge of logic models

- Participants will increase ability to create a useful logic model of program

- Participants will Increase confidence in using logic models

Improved planning

Improved evaluation

INPUTS

•3 hour training

•Interactive activities

•Group work

•Practice

•Q and A

Create meaningful logic models

Use logic models in own work

OUTPUTS OUTCOMES

Grantees

Accountable here

Fulfill requirement of funder

Youth and community service

Staff

Partners

Youth improve skills in planning, decision making, problem solving

Youth learn about their community

Youth demonstrateleadership skills

Youth are connected with and feel valued by their community

Time

Youth gain confidence in doing community work

Youth engage in additional community activities

INPUTS OUTPUTS OUTCOMES

Youth identify project to work on

Plan project

Carry out the project

Evaluate how they did

Youth successfullycomplete projects

Grant

THEORY OF CHANGE

“ A theory of change is a description of how and why a set of activities – be they part of a highly focused program or a comprehensive initiative – are expected to lead to early, intermediate, and long-term outcomes over a specified period.”

(Anderson, 2000)

Logical chain of connections showing what the program is to accomplish

What we do

Who we reach

What results

INPUTS OUTPUTS OUTCOMES

Program investments

Activities Participation Short MediumLong-term

What we invest

Another Example…

ResourcesPrimary Care Clinic Coordinator

Community Health Director

Staff therapist

Staff pediatrician

Medical providers (nurses)

Money for supplies

ActivitiesTraining•Develop curriculum•Two one-hour didactic trainings to medical providers in behavioral health assessment•One-on-one training to medical providers on behavioral health

Outreach•Order supplies for well-child packets•Make up packets•Distribute to parents at end of each visit

Outputs

Training# of two-hour trainings held# of one-on-one trainings held# of medical providers trained

Outreach# of parents/children receiving packets

Outcomes

Medical providers demonstrate accurate behavioral health assessment, education and prevention activities

More children receive high-quality behavioral health assessment, education and prevention activities during well-child visits

Parents/children are more knowledgeable about behavioral health and caring for children’s behavioral health needs

Reduced incidence of adverse behavioral health events in children at the community health center

Program Goal: To improve the behavioral health of low-income children who receive primary care in a community health center

Complete Components of the LOGIC MODEL used in PROGRAM EVALUATION

EVALUATION: check and verify

What do you want to know? How will you know it?

EVALUATION: check and verifyEVALUATION: check and verify

Things Usually Not Included but Important

– Situation Statement– Priorities

– List of Assumptions– List of External Factors– Evaluation Methods

GOAL-Outcome Definition

– Goal answers: “What are the issues you would like the program to address”

– Outcomes answers: “What changes do you want to occur because of your program?”

Outputs vs. Outcomes

– Number of patients discharged from a hospital is an OUTPUT

– Percentage of discharged patients who are capable of living independently is an OUTCOME

Outputs vs. Outcomes

Program Outputs OutcomesCrime control Hrs of patrol

# responses to calls# crimes investigatedArrests made

Reduction in crimes committedReduction in deaths and injuries resulting from crime;Less property damaged or lost due to crime

Highway construction

Project designsHighway miles constructedHighway miles reconstructed

Capacity increasesImproved traffic flowReduced travel timesReduction in accidents and injuries

Picking the Right Outcome is Critical….

Ohio SBIRT Logic Model

Ultimate Goal

Alcohol and illicit or prescription drug use morbidity and mortality are decreased through an integration of SBIRT approaches into medical and behavioral health services.

Outcomes Ohio’s use of SBIRT is expanded in primary health care and other community settings.

Clinically appropriate services for people at risk for or diagnosed with a substance use disorder are supported.

Current technological strategies to imbed SBIRT as a clinical and business practice are enhanced and expanded.

The potential misuse of prescription drugs is reduced through the expanded use of the Ohio Automated RX Reporting system (OARRS) in conjunction with SBIRT.

System and policy changes to increase access to treatment in generalist and specialist settings are identified and implemented.

Outputs Performance Assessments Policy Steering Committee (PSC) recommendations Evaluation Plan

Performance Assessments SBIRT trained medical professionals System for

referral to specialty

treatment GPRA Data

Performance Assessments

Electronic SBIRT

screening modules Protected

SBIRT information in Health Information Exchanges (HIE)

Performance Assessments OARRS

requirement in SBIRT Guidelines Comprehensive

Rx medication review for co-occurring

patients

Sustainability plans Care

coordination Established

referral and linkage mechanisms for medical and

specialty treatment

Inputs Motivational Interviewing (MI) Electronic Health Records (E.H.R.) SBIRT training MI Training

ASAM guidelines for

Substance Use Disorders (SUD) Health

Navigator

HIPAA requirements

Care coordination

Technology workgroup

Support for inclusion of OARRS as part of SBIRT guidelines in primary care and other community settings

Policies and procedures for care coordination

Policies and procedures for referrals and linkages

Activities Hire SBIRT Director Create and convene PSC Create Performance Assessment E.H.R. training

SBIRT training Distribute

ASAM Guidelines GPRA training

Contract for IT development Coordinate with HIEs Review

available SBIRT technology best practices

Identify barriers

(including resistance and/or reluctance) to the use of OARRS Provide OARRS training as needed

Hire and train Health Navigators PSC identifies

the regulatory, policy, funding and health

disparity changes needed to increase access to treatment

Logic Model

Ultimate Outcome

Offenders released from state prisons and victims of human trafficking are restored to optimum behavioral, physical and occupational health, thereby improving family functioning and community safety.

Long- term Outcomes

1.1 Safer and more effective reintegration of offenders into Ohio communities.

1.2 Safer and more effective reintegration of human trafficking victims into Ohio communities.

1.3 State and local policies and practices that support and promote community reintegration of both target populations.

Intermediate Outcomes

2.1 A Reduced Recidivism. 2.1 B Reduced parole and probation revocations. 2.1 C Improved behavioral and physical health. 2.1 D Improved acquisition of employment, job training, housing and education.

2.2 A Reduced psychological effects of trauma. 2.2 B Reduced stigma associated with involvement in the sex industry, albeit forced. 2.2 C Improved behavioral and physical health. 2.2 D Improved acquisition of employment, job training, housing and education.

2.3 A Increased diversion of human trafficking victims from the criminal justice system. 2.3 B Increased use of alternative sanctions to avoid revocation. 2.3 C Improved inter-system collaboration. 2.3 D Improved methods for payment and/or reimbursement of OREP services.

Immediate Outcomes

3.1 A Increased referrals to treatment. 3.1 B Improved access to behavioral health treatment. 3.1 C Improved access to primary health services.

3.2 A Improved screening tools to better detect human trafficking victimization. 3.2 B Increased number of treatment staff trained to use the aforementioned screening tools. 3.2 C Increased referrals to treatment. 3.2 D Improved access to trauma-informed behavioral health services. 3.2 E Improved access to primary health services.

3.3 A Increased awareness of human trafficking victimization by policy makers and service providers. 3.3 B Increased partnerships between corrections, behavioral health and primary health providers. 3.3 C Increased targeted resources for both target populations. 3.3 D Improved data on the Incidence of human trafficking.

Outputs

4.1 A Pilot projects 4.1 B Outcome evaluation of offender reintegration portion of OREP project. 4.1 C GPRA data

4.2 A Pilot projects 4.2 B Outcome evaluation of human trafficking victim reintegration portion of OREP project. 4.2 C GPRA data

4.3 A Policies and procedures for use of alternative sanctions for offenders and victim diversion. 4.3 B MOUs between agencies and corrections.

Activities

5.1 A Hire additional staff. 5.1 B Institutional meetings with inmates 4 months prior to their release. 5.1 C Refer and connect offenders to treatment and ancillary services. 5.1 D Collect GPRA data.

5.2 A Hire additional staff. 5.2 B Screening inmates in jails to identify victims of human trafficking. 5.2 C Refer and connect Identified victims to treatment and ancillary services 5.2 D Collect GPRA data.

5.3 A Inform and educate the public. 5.3 B Develop alternative sanctions. 5.3 C Stakeholders seek input to develop policies and procedures.

LIMITATIONS

• Logic Model…• Represents intention, is not reality

• Focuses on expected outcomes

• Challenge of causal attributionMany factors influence process and outcomes

• Doesn’t address: Are we doing the right thing?

CHECK YOUR LOGIC MODEL

• Is it meaningful

• Does it Make Sense

• Is it Doable

• Can it be verified?

Logic Model in Evaluation

EVALUATION: check and verify

What do you want to know? How will you know it?

Logic Models Help with Evaluation

•Provides the program description that guides our evaluation process

• Helps us match evaluation to the program• Helps us know what and when to measure

• Are you interested in process and/or outcomes?• Helps us focus on key, important information

• Prioritize: where will we spend our limited evaluation resources?

• What do we really need to know??

LOGIC MODEL and COMMON TYPES OF EVALUTION

Needs/asset assessment: What are the characteristics, needs, priorities of target population?What are potential barriers/facilitators?What is most appropriate to do?

Process evaluation: How is program implemented? Are activities delivered as intended? Fidelity of implementation?Are participants being reached as intended? What are participant reactions?

Outcome evaluation: To what extent are desired changes occurring? Goals met?Who is benefiting/not benefiting? How? What seems to work? Not work?What are unintended outcomes?

Impact evaluation: To what extent can changes be attributed to the program? What are the net effects?What are final consequences? Is program worth resources it costs?

Possible Evaluation Questions using a Logic Model

To what extent is stress reduced? relations improved?

To what extent did behaviorschange? For whom? Why? What else happened?

To what extent did knowledge and skills increase? For whom? Why? What else happened?

Did all parents participate as intended? Who did/not not?Did they attend all sessions?...support groups?Level of satisfaction?

Were all sessions delivered? How well? Do support groups meet?

What amount of $ and time were invested?

Staff

Money

Partners

Assess parent ed programs

Design & deliver evidence-based program of 8 sessions

Parents increase knowledge of child dev

Parents better understand their own parenting style

Parents use effective parenting practices

Improved child-parent relations

Research

Facilitate support groups

Parents gain skills in effective parenting practices

Parents identify appropriate actions to take

Strong families

Parents of 3-10

year olds

attend

Reduced stress

QUESTIONS OR ASSISTANCE???

If you have any questions or need any assistance, feel free to contact me at [email protected]