The Formative & Summative Evaluation Process TS & EOC SAEOPP Priority 1 Training PY 2009.

The Formative & Summative The Formative & Summative Evaluation ProcessEvaluation Process

TS & EOC TS & EOC

SAEOPP Priority 1 Training

PY 2009

PresenterPresenter

Elizabeth C. Retamozo, Executive Director

NOSOTROS Education Center NOSOTROS Education Center www.nosotrosedu.org

Phone: 469-450-7137

Email:[email protected]

[email protected]

2

http://www.nosotrosedu.org/

mailto:[email protected]

Presentation MaterialsPresentation Materials

• PowerPoint Slide Handouts• Additional Documents & Materials located at:

• www.nosotrosedu.org– Go to Resources Page– Register Your Email Address– Click the Presentations & Links Tab– Click SAEOPP Priority 1 Training

3

http://www.nosotrosedu.org/

4

Session Summary Session Summary

This session will focus on providing Talent Search staff with an overview of:

• Program Evaluation• Types of Program Evaluations • Methods & Types of Data Collection • Quantitative & Qualitative Data • Formative & Summative Methods

5

EvaluationEvaluation

• Evaluation - Systematic method for collecting, analyzing, and using information to answer basic questions about your program.

• Helps to identify effective & ineffective services, practices, & approaches.

• Informs & assists program staff to make changes to improve program effectiveness.

6

Evaluation PlanEvaluation Plan

• Evaluation Plan - A written & comprehensive plan to determine the effectiveness of the program in meeting stated objectives & goals.

• Includes what you plan to do, how you plan to do it, who will do it, and when it will be done.

• Provides a mechanism for evaluating & implementing programmatic changes to improve program effectiveness

• Relates your project to GPRA & the TRIO Programs Performance Indicators

7

The GPRA ConnectionThe GPRA Connection

1) Government Performance and Results Act of 1993 (GPRA)

2) DOE Response to GPRA Requirements: Goals 1-63) TRIO Programs Performance Indicator: Goal 54) TRIO Programs’ Specific Program Measures:

Upward Bound, Talent Search, Educational Opportunity Centers Student Support Services & Ronald E. McNair

5) Prior Experience Criteria & Mandatory Objectives – APR6) Project Evaluation Plan7) Activities & Services Specific to Each Objective

8

Types of Evaluation PlansTypes of Evaluation Plans

Goals-Based Evaluation

Outcome-Based Evaluation

Process-Based Evaluation

9

Evaluation Plan “Buzz” WordsEvaluation Plan “Buzz” Words

Qualitative Values

Quantitative Values

Formative Methods

Summative Methods

10

Qualitative ValuesQualitative Values

• Measurement of “feelings”, impressions, & perception

Ways to describe effectiveness that are measurable, but may not aggregate in traditional, quantifiable methods

Information that is difficult to measure, count, or express in numerical terms

11

Collection of Qualitative Data Collection of Qualitative Data

• Interviews & Focus Groups Used to fully understand someone's impressions or

experiences, or learn more about their answers to questionnaires

• ObservationsUsed to gather accurate information about how a program actually operates, particularly about processes

• Questionnaires, Surveys, Checklists

Used to quickly and/or easily get info in a non-threatening way

12

Quantitative ValuesQuantitative Values

• Information that can be expressed in numerical terms, counted, or compared on a scale

Promotion to Next Grade Level Rates

Graduation From High School Rates

# of College Admissions Forms Completed

# of Financial Aid Applications Completed

College Enrollment Rates

13

Collection of Quantitative Data Collection of Quantitative Data

• Report Cards

• Transcripts

• Test Scores (NCLB, PSAT, SAT, ACT)

• National Student Clearinghouse Reports

14

Formative MethodsFormative Methods

• Continuous (daily, weekly, monthly, quarterly, semester) assessment of services/activities & progress towards achieving objectives

• Allows for changes in activities during the year so final objectives/outcomes will be met (changes in delivery of services)

15

Formative MethodsFormative Methods

• Weekly/monthly student service reports# of continuing student with 0 services # students with less than 2 services # of students with equal to or more than # services # students that have received a specific services

• Student evaluation of workshops & field trips

• Grading period report cards (6/9 weeks)

16

Progress Towards Achieving Goals & Objectives During the Year -

Formative Methods:

• Project staff document all services provided to students as they occur on Individual or Group Service Logs

• Project staff enter all data on service logs into the project database on a weekly basis

• Project database generates monthly/quarterly contact reports documenting which students have been served, the types of services provided, & the frequency of services provided.

17

Progress Towards Achieving Goals & Objectives During the Year -

Formative Methods:

• Project Director tracks progress towards the accomplishment of all objectives via reports (quantitative) generated by the project database on a weekly/ monthly basis

• Participants rate specific group activities and workshops to rate their level of satisfaction with the quality of services on evaluation forms (qualitative).

18

Summative MethodsSummative Methods

• End of year assessment that measures the results or outcomes of a project.

Concerned with a program's overall effectiveness

Quantitative data is analyzed for a given period

Qualitative evaluation supplements the quantitative data

19

Achieving Goals & Objectives at End of the Year - Summative Methods:

• Project database combines all monthly reports to produce a report documenting & confirming the achievement of project objectives at the end of the year (quantitative).

• Students complete end-of-year surveys (qualitative) to measure level of satisfaction with the overall quality of project services.

20

Achieving Objectives at the End of the Project Period - Summative Methods:

• Target school/agency/project personnel complete end of year evaluation forms (qualitative) to measure level of satisfaction with the overall quality of project services.

• Project Director reviews evaluation results to implement any corrective measures to eliminate potential problems by appropriately modifying & improving activities & services for the next program.

21

TRIO Evaluation PlansTRIO Evaluation Plans

• Goals Based Evaluation (Internal & Formative) Evaluates the extent to which projects are meeting

predetermined goals or objectives. INTERNAL GOALS

• Outcomes Based Evaluation (Internal & Summative) Looks at impacts/benefits/changes to your students as a

result of your project’s efforts during and/or after their participation in your project. PE OBJECTIVES

• Process Based Evaluation (External & Summative) Geared to fully understanding how a program works -- how

does it produce that results that it does.

22

Goals Based Evaluation Goals Based Evaluation

1) What project goals were established?

2) What is the status of the project’s progress towards achieving the goals?

3) Will the goals be achieved according to the timelines specified in the program implementation or operations plan?

4) Do personnel have adequate resources (supplies, facilities, training, etc)?

23

Goals Based Evaluation Goals Based Evaluation

5) How should priorities be changed to more focus on achieving the goals?

6) How should timelines be changed?

7) How should goals be established in the future?

24

Sample TS Internal Goals Sample TS Internal Goals

1) Identification, Recruitment, & Selection of Participants:

To Identify, recruit, & select new students to have

a total of 600 participants with 67% classified as

both low-income & potential first generation

college students within 90 days of the beginning

of the program year.

25


2) Development of Needs Assessment & Individual Service Plans:

100% of all participants will have their needs

assessed and an Annual Individual Service Plan

developed within 30 days of their initial selection

or renewal into the project each program year.

26


3) Delivery of Promotion Services To 9th - 11th Participants:

70% of all 9th - 11th grade participants will

receive a minimum of three (3) college, career,

and financial aid exploration services each

semester for a total of six (6) services by the end

of the program year.

27


4) Delivery of Individual College Admissions & Financial Aid Application Assistance to all College Ready Participants:

80% of all college ready participants will receive

a minimum of five (5) individual advising sessions

each semester for a total of ten (10) sessions by

the end of the program year.

28


5) Increased Awareness & Knowledge of College, Financial Aid, & Career Information:

80% of participants will demonstrate an increase

in knowledge of college, financial aid & career

information as demonstrated by a comparison

pre/post tests by the end of the program year.

29


6) Perception & Level of Satisfaction with Project Services:

85% of all participants and target school

personnel surveyed will rate the project as

"Excellent" on the Project Evaluation Form by the

end of each project year.

30

Outcome Based Evaluation Outcome Based Evaluation

TS Goals Based Evaluation Plan

31

Sample EOC Internal Goals Sample EOC Internal Goals

1) Identification, Recruitment, & Selection of Clients:

To Identify, recruit, & select new students to have

a total of 1000 clients with 67% classified as both

low-income & potential first-generation college

students within 180 days of the beginning of the

program year.

32


2) Development of Needs Assessment & Individual Service Plans:

100% of all clients will have their needs assessed

and an Individual Service Plan developed within

30 days of their initial selection into the project

each program year.

33


3) Delivery of GED/Continuing Education Services to Non-College Ready Clients:

100% of all Non-College Clients will receive a

minimum of three (3) GED/Continuing Education

services (individual advising, referrals,

information, etc.) by the end of the program year.

34


4) Delivery of Individual College Admissions & Financial Aid Application Assistance to all College Ready Clients:

100% of all college ready clients will receive a

minimum of three (3) individual advising sessions

by the end of the program year.

35


5) Perception & Level of Satisfaction with Project Services:

85% of all Clients and target agency personnel

surveyed will rate the project as "Excellent" on

the Project Evaluation Form by the end of each

project year.

36


EOC Goals Based Evaluation Plan

37


1) Identify Major Outcomes (Mandatory Objectives)

2) Specify Observable Measures & Indicators

3) Identify what Info is Needed to Show these Indicators

38


4) Decide How this Info Can be Realistically & Efficiently Gathered

5) Assign Staff Responsibility & State Timelines

6) Analyze & Report Findings

39


TS Outcome Based Evaluation Plan

EOC Outcome Based Evaluation Plan

40

External EvaluationExternal Evaluation

• Recommended in last grant proposal competitions

• Must be conducted if included in grant proposals

• Allowable cost for the project

• May be paid by the grantee agency/institution

• May be provided free of charge – in-kind/donation

41


• May be conducted by person working/employed by agency/institution BUT not for the project

• Must have experience conducting evaluations for educational opportunity programs & TRIO like programs

• Must state what type of evaluation will be conducted by the evaluator.

42


• Process-Based Evaluation

Process-based evaluations are geared to fully understanding how a program works -- how does it produce that results that it does.

• Useful if programs are long-standing and have changed over the years.

• Useful for accurately portraying to outside parties how a program truly operates.

43

Process-Based EvaluationProcess-Based Evaluation

• Examines the extent to which a program is operating as intended by assessing ongoing program operations & whether the targeted population is being served.

• Helps program staff identify needed interventions and/or change program components to improve service delivery.

44

Process Based Evaluation Process Based Evaluation

1) How does the project identify, recruit, & select students?

2) How does the project decide what specific services & activities to provide to its students?

3) How does the project monitor students’ academic progress?

45


4) How does the project deliver services to its students?

5) How does the project document services provided to student?

6) What is the project’s goals based evaluation plan?

7) What is the project’s outcome based evaluation plan?

8) How does the project plan for the next program year?

46


Process Based Evaluation Plan

47

Basic Guide to Program Evaluation

Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.

Adapted from the Field Guide to Nonprofit Program Design, Marketing and Evaluation.

Online Guide: A Basic Guide to Program Evaluation.pdf

Bibliography & Resources

48

Basic Guide to Goals-Based Evaluation for Nonprofit Organizations with Very Limited Resources

Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright 1997-2007.

Online Guide: Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.mht


49

US Department of Health & Human Services Administration for Children & Families - Office of Planning, Research & Development

The Program Manager's Guide to Evaluation

http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/pmguide_toc.html

This guide explains program evaluation -what it is, how to understand it, and how to do it. It answers your questions about evaluation and explains how to use evaluation to improve programs and benefit students, families, and staff.


50

Questions & Answers

The Formative & Summative Evaluation Process TS & EOC SAEOPP Priority 1 Training PY 2009.

Documents

program effectivenessrelates

assists program staff

written comprehensive

quantitative valuesinformation

gpra requirements

formative methodscontinuous

gpra doe response

talent search staff