Evaluation Research

EVALUATION RESEARCH

Evaluation Research How do we begin?

What are the different types of evaluation research?

How do these different types fit together?

What purpose do they serve?

Begin with a Logic Model

Step 1 – Start with

Program

Model

Step 2 –

Identify

Resources

Step 3 –

Identify

Inputs

Step 4 Identif

y Activiti

es

Step 5 –

Identify

Outputs

Step 6 –

Identify

Outcomes

Other Common Terms Used to Identify Steps

Step 1 – Identify Program Model

Step 2 – Identify Resourc

es

Step 3 – Identify Program Events

Step 4 – Identify Resultin

g Products

Step 5 – Identify ChangesSuccess

Program Model

Teachers Grant Money

Service LearningActivities

Service Learning Projects Complet

ed

Measure Leadership Skills

Example-Service Learning Program

ExampleProgram Model

# of TeacherAmount of Money for Building Materials

Number of Service

Learning Projects

Planned by Students

Number of Projects

Completed

Perception of

Leadership Skills

Developed

Program Model

Teachers Grant Money

Service LearningActivities

Service Learning Projects Complet

ed

Leadership Skills

Another Conceptualization of the Logic Model

Resources

• Step One – Identify and count resources used to instigate program

Activities

• Step Two – Identify and count activities that took place

Outputs• Step Three – Identify and count events or products

that resulted

Short-Term

Outcomes

• Step Four – Identify and count short-term changes that took place (i.e., positive changes or accomplishments immediately after participation)

Long Term

Outcomes

• Step Five – Identify and count long-term impacts that occurred (i.e., positive changes or accomplishments at specified future date)

Service Learning Example Using This Conceptualization

Resources

• Count number of staff and amount of money used to instigate program. Count number of networks established.

Activities

• Count number of planning meetings and number of service learning activities planned by students. Count number of hours students spent in activities.

Outputs• Count number of projects completed by students. This could be

subdivided into things such as number of picnic tables built for a park etc.

Short-Term

Outcomes

• Use survey to assess positive changes or accomplishments as perceived by service learning participants. Compare to control group.

Long Term

Outcomes

• At a specified future time, use survey to assess positive changes in the lives of students (i.e., number of hours spent in volunteer work etc.) Compare to control group

Another Example Using This Conceptualization

Resources

• Count number of staff and amount of money used to instigate 4-H program. Count number of networks established.

Activities

• Number of 4-H activities that occurred, and/or number of students who participated in planning activities

Outputs

• Number of students who participated in workshops and projects. Number of educational packets distributed.

Short-Term

Outcomes

• Assessment of their sense of accomplishment using survey. Assessment of increase in knowledge using survey.

Long Term

Outcomes

• Number of 4-H students who attend college after attending educational workshops. Compare to control group.

Types of Evaluation Research In order of Use

Outcome Evaluation

Process Evaluation

Formative Evaluation

Needs Assessment

Example of Each Type of Evaluation

Example - Surveys, intensive interviews and focus groups are used to determine (1) Long-term impact on USU students and (2)Long term -

impact on high school

Outcome Evaluation

Process EvaluationAn ongoing evaluation is used to monitor the program

Example -Grantee comes from time to time to “see how things

are going”.

Formative EvaluationInformation is gathered to formulate a plan to establish a

program.

Example-Planning meetings with high school administrators

etc.

Needs AssessmentA need for a program is identified

Example –need for mentoring program with public school

students

Needs Assessment Purpose

Evaluate the need for a program/intervention/ project

Basic Steps Use secondary data (i.e., census data, community

reports etc.) and/or surveys etc. to create a profile of the community that assess existing resources and current needs.

Determine if current resources are adequately meeting needs.

Identify one or more specific needs that are not currently being met.

Example - Needs Assessment

Purpose Determine the percent of local post secondary students

that are from an underrepresented population. Compare this to the percent of students in post secondary educational institutions. Based on this and programs already available determine the need to provide additional opportunities for this group.

Basic Steps Use secondary data to determine percent of population

that is Latino. Use USU data to compute the percent of USU that is Latino. Compare differences in percentages. Note any discrepancy.

Utilize local webpages and phone books, identify programs/organizations that focus on the Latino population and meeting their needs.

Formative Evaluation Purpose

Plan (form) a program that will met the needs of the community, keeping in mind the need to develop indicators that can be used to evaluate the success of the program.

Basic Steps Locate other similar programs to use as a model

Meet with “experts” in this area and get their advice

Organize a program and apply for a grant

Form an “ad hoc” evaluation committee to evaluate program

Formative Evaluation Purpose

Plan a mentoring program where USU students mentor high school students. Identify objectives (i.e., increase high school graduation rates, increase college retention rates of mentees).

Basic Steps Identify other mentoring programs

Meet with directors of programs, professors etc. who have been/are involved in mentoring programs.

Organize your mentoring program and apply for a grant to implement program.

Select members of your committee to engage in evaluation research

Purpose To monitor the implementation of your program on an

ongoing basis. It is a type of program monitoring.

Steps Specifically describe your program

Decide what products to count Consider objectives (verb) (target)

(date)

Decide on valid way to count products Identify process indicators

Quality Assurance Identify process indicators

Process Evaluation

Purpose Monitor the mentoring program and address two questions:

(1) Is the mentoring occurring as planned, (2) How are the mentees

responding? Steps

Describe Program The mentoring program connects high school mentees with

USU mentors through a monitored blog. Decide what products to count

The number of hours that mentors spent in contact with their mentee etc.

Decide on valid way to count products Have program administrator count hours spent on website

Quality Assurance High school counselors will be contacted periodically to

determine how mentees are responding

Example-Process Evaluation

Purpose To determine how well you meet the objectives of your

program.

Steps Identify short-term and long-term outcomes

Construct indicators to measure outcomes

Decide on data collection and sampling techniques

Collect data

Analyze data

Write Report

Outcome Evaluation

Purpose To determine how well you meet the objectives of your program.

Steps Short term outcomes are establish relationship between mentors

and mentees. Long term outcomes are higher graduation and retention rates for mentees.

Mentees will be asked questions about their experience, and then about their graduation/retention

Surveys will be administered to all mentees after completion of program and then again in three years.

Surveys are administered

Using SAS, data is entered and analyzed.

Report is written

Outcome Evaluation

Quality Assurance vs. Program Outcome Evaluation

Inform Administrators and

ClientsRely on “Expert”

Opinion

Inform Administrators

Rely on Statistics

Quality Assurance

Legislative Mandate

Client Satisfaction Is Focus

Outcome Evaluation

Not Mandated

Does Not Necessarily Focus on Client

Satisfaction

Dr. Carol Albrecht Extension Assessment Specialist Phone: 979-777-2421 [email protected]