AmeriCorps State and National Evaluation Requirements August 16, 2013 1.

AmeriCorps State and NationalEvaluation Requirements

August 16, 2013

1

Learning Objectives

• Increased understanding of broader federal policy context

• Increased understanding of AmeriCorps State and National evaluation requirements

2

Federal Policy ContextPresidential Administrations Office of Management and Budget

Guidance

President Clinton (1993 – 2001)

Government Performance and Results Act

President Bush (2001 – 2009)

Program Assessment Rating Tool

President Obama(2009 – 2017)

Memoranda•M-10 -01 Increased Emphasis on Program Evaluation•M-12-14 Use of Evidence and Evaluation in the 2014 Budget•M-13-17 Next Steps in the Evidence and Innovation Agenda

3

Code of Federal Regulations

The Code of Federal Regulations, or CFR, is the collection of all administrative rules and

regulations from the various departments and agencies of the federal government. It is

organized by titles, chapters, and parts that cover specific regulatory areas

4

Code of Federal Regulations

• http://www.gpoaccess.gov/ecfr– Select Agency List from left navigation bar – Scroll down to CNCS, select 45 CFR Chapters XII,

XXV– Scroll down to XXV, select 2500-2599– Select 2522– Select Subpart E – Evaluation Requirements

5

Code of Federal Regulations

Primary Focus of Today’s Webinar:Evaluating Programs: Requirements and

Procedures (§§.700 -.740)CNCS finalized these regulations July 8, 2005

6

Definition of Evaluation

…evaluation uses scientifically-based research methods to assess the effectiveness of programs by comparing the observed program outcomes with what would have happened in the absence of the program.

§2522.700

7

Definition of Performance Measurement

Performance measurement is the process of systematically and regularly collecting and monitoring data related to the direction of observed changes in communities, participants (members), or end beneficiaries receiving your program’s services. It is intended to provide an indication of your program’s operations and performance.

§2522.700

8

Evaluation RequirementsNational Direct & State Competitive ($500K+)

…you must arrange for an independent evaluation of your program, and you must submit the evaluation with any application to the Corporation for competitive funds…

§ 2522.710

9

Strengthening the Evidence BaseNational Direct & State Competitive ($500K+)

• Greater investments of public resources warrant more rigorous evidence and evaluation requirements

• Independent, impact evaluations designed to provide evidence of a causal relationship between program activities and participant outcomes are required

10

Evaluation RequirementsNational Direct & State Competitive ($500K+)

•Your evaluation must cover a minimum of one year but may cover longer periods.

§ 2522.720

11

Evaluation RequirementsNational Direct & State Competitive (less than $500K)

or Education Award Program Corporation

…you must conduct an internal evaluation of your program, and you must submit the evaluation with any application to the Corporation for competitive funds…

§ 2522.71012

Strengthening the Evidence BaseNational Direct & State Competitive (less than $500K) or

Education Award Program Corporation

• Encourage use of experimental or quasi-experimental designs but not required

• Select a study design most appropriate for developmental phase of AmeriCorps program (process, implementation, outcome or impact)

13

Evaluation RequirementsNational Direct & State Competitive (less than $500K) or

Education Award Program Corporation

•Your evaluation must cover a minimum of one year but may cover longer periods.

§ 2522.72014

Evaluation RequirementsState Commission

…you must establish and enforce evaluation requirements for your State formula

subgrantees, as you deem appropriate.

§ 2522.710

15

Evaluation RequirementsState Formula Grantees

If you are a State formula grantee, you must conduct an evaluation, as your State commission requires.

§ 2522.72016

Evaluation Requirements

• The Corporation may, in its discretion, supersede these requirements with an alternative evaluation approach, including one conducted by the Corporation at the national level.

• Grantees must cooperate fully with all Corporation evaluation activities.

§ 2522.710

17

Evaluation RequirementsWhat is due when?

• If you are an existing grantee re-competing for AmeriCorps funds for the first time, you must submit an evaluation plan… as part of your application for funding.

• If you again compete for AmeriCorps funding after a second three-year grant cycle, you must submit the completed evaluation with your application for funding.

§ 2522.730

18

CNCS “Cheat Sheet”19

If you are competing for…

You are…

Your first AmeriCorps grant Not required to submit an evaluation plan with the application or complete an evaluation during the initial grant period. You are encouraged to begin the evaluation planning process.

Your second AmeriCorps grant

Required to submit an evaluation plan with the application and complete the evaluation during the next three-year grant period.

Your third AmeriCorps grant Required to submit a report on the evaluation conducted during the previous grant period with the application, and submit an evaluation plan for the next grant period with the application.

An AmeriCorps grant beyond your third grant award

Required to submit an evaluation plan for the next grant period, along with a report that details findings from evaluations conducted in previous grant periods with the application. Evaluation plans should build on previous evaluation efforts.

Formula grantees applying for competitive funds

Required to submit an evaluation plan for the next grant period.

CNCS Priorities

• Strengthening the evidence base for national service using scientifically-based research methods

• Supporting grantee efforts to build the evidence base of their AmeriCorps programs

20

Strengthening the Evidence Base

“In order to maximize the impact of the public investment in national service, CNCS will fund programs that can demonstrate community impact and solve community problems using an evidence-based or evidence-informed approach.” (2013 ACSN NOFO, page 1)

“An evidence-based approach is one where the intervention is based on research or backed by statistically significant evaluation findings. “(2012 ACSN NOFO, page 19)

21

Identify a strong program design

Attain strong evidence of positive program

outcomes

Assess program’s outcomes

Ensure effective implementation

Obtain evidence of positive program

outcomes

Building Evidence of Effectiveness

Evidence Informed

Evidence Based

Supporting Grantees

• Grantees that have not completed the evaluation requirements in the proper timeframe are strongly encouraged to provide any evaluation findings available and provide an evaluation plan for the next grant period that would fulfill the requirements

• Evaluations that show null or negative findings will not automatically lead to an unacceptable assessment of your program. CNCS expects such findings to be used for program improvements.

Supporting Grantees: Summer 2013

• Disseminate Evaluation FAQs• Conduct Webinars: – Evaluation Capacity Building– CNCS Evaluation Requirements and FAQs

• Create central location for evaluation materials on website

Supporting Grantees: Fall 2013

• Grantee Symposium, Performance Measurement & Evaluation Track– Performance Measurement Pre-Conference

Workshop– Developing evaluation plans and reports– Evaluation 101

• Initiate individualized technical assistance– 1:1 coaching to help finalize evaluation plans, use

performance monitoring and evaluation for program improvement, and develop evaluation reports

Supporting Grantees: 2013 - 2014

• Individualized technical assistance• Evaluation Core Curriculum (webinars, online

courses, downloadable resources)– Developing evaluation plans and reports– Evaluation 101– Logic models– How to report evaluation findings– How to manage an external evaluation– Budgeting for evaluation

Questions and Answers

We welcome questions at this time.

Thank you.

27