NASCSP March 1, 2012. 2 Guidance About Evidence Success should be judged by results, and data is a powerful tool to determine results. We cant ignore.

Examining Evidence

Identifying elements of evidence.

How can we do better?

NASCSP March 1, 2012

2

Guidance About Evidence• “Success should be judged by results,

and data is a powerful tool to determine results. We can’t ignore facts. We can’t ignore data.”

President Barack Obama July 24, 2009

• “The test of a performance management system is whether it’s actually used….Federal managers and employees at all levels must use performance goals and measures to set priorities, monitor progress, and diagnose problems.”

Chief Performance Officer Jeffrey ZientsOctober 29, 2009

Two key points: ◦ Results are supported by data◦ Data from performance measurement systems

must be put to use.

In recent OCS guidance, several “models” and references to “evidence” have been identified.

Here is a quick review of these terms.

Use of “evidence”

3

Approaches to prevention or treatment that are validated by some form of documented scientific evidence.

This could be findings established through scientific research, such as controlled clinical studies, or other comparable and rigorous methods.

Identifies proven effectiveness, supported by objective and comprehensive research and evaluation.

Evidence-Based Practice

4

Approaches that use the best available research and practice knowledge to guide program design and implementation within context.

This informed practice allows for innovation and incorporates the lessons learned from the existing research literature.

Evidence-Informed Practice

5

An innovative and consistently applied policy, process, practice, or procedure that takes a comprehensive approach to developing and implementing activities using strategies that are related to the intended service recipients and community.

This practice model is culturally competent, data-driven, measurable, and replicable and incorporates a method for documenting programmatic results.

Exemplary Practice

6

A practice with at least preliminary evidence of effectiveness in small-scale interventions, or for which there is potential for generating data that will be useful for making decisions about taking the intervention to scale the results diverse populations and settings.

Promising Practice

7

A program, activity or strategy that has been shown to work effectively and produce successful outcomes

This is supported to some degree by subjective and objective data sources

Field Tested Practice

8

What do we know about our measurement tools?

Are they Accessible Reliable Valid (appropriate) Affordable Scalable (able to measure change)

Where’s the beef (aka data)?

9

Bank accounts Copy of Diploma or

Certificate Employment records Escrow accounts Financial reports Health or nutrition records Inspection results Lease agreements Legal documents

Loan monitoring reports Mortgage documents

Observation log Pre-post tests Progress reports Questionnaire Rent receipts Scales and Matrices Survey Testing results

Tools For Measurement

10

Examples of data sources

Agency database Case notes Centralized database Computer spreadsheets File cabinets Individual case records Manual tallies

Public database School records Specialized database Tax Assessor database Training center Work plan reports

IDENTIFY PROS AND CONS OF EACH SOURCE

11

If no one is assigned the task, no one will do it.

Consider various agency staff responsibilities

Include partner reports

Personnel

12

Home visits Office appointments Phone follow up Review of case notes/progress reports

(scales?)

Procedures

13

Daily Weekly Monthly Quarterly Biannually Annually Upon incident

Frequency

14

15

16

What ROMA Data IS

Data about community needs and resourcesThis can identify the scope of community issuesUsed to prioritize community needs for agency intervention

Data used during strategic planning Identify resources ($, facilities, staff, etc) the agency has/needsWhat results does the agency expect to achieve?

Data collected during implementation of servicesHow many individuals, families, communities are projected to be served? How

many are projected to achieve results?How many individuals, families, communities were actually served? Who were they?

Data collected to identify achievement of resultsHow many made movement toward their goals? How many achieved the end results previously identified?

17

What ROMA Data IS NOT (yet….)

Evaluation Data

Used in comparing CAA performance with a “control” group to determine the quality of the results

Done by an independent reviewer

Used to validate that results actually impact on identified community needs

18

Moving towards Evidence

Source: Child Trends

The collection of NPI data for a report to OCS is a comprehensive effort by the CSBG Network to identify the results of the efforts of local CAAs.

This is sometimes called the ROMA report, but we have just seen that ROMA data includes more than these indicators.

Who’s using ROMA data?

19

Measure and report performance Identify patterns and relationships Consider new actions based on analysis Create new outcome-focused goals Develop new resources

We use data to:

20

• When you have your data, remember that you have to turn them into information.

• They must be interpreted to become evidence

• Consider this piece of data and the questions that arise: 35 individuals got a job.

• Is that good?  What is the unemployment rate? Opportunity for

employment? Characteristics of those who got jobs. Etc.

So now what?

21

What are the top 10 services provided by your agency?

What are the top 10 outcomes achieved by your customers?

What is the connection between these two lists?

The top 10

22

Build on ROMA principles to identify ways to collect credible evidence in systematic ways (making the connections among mission, community need, agency strategies and well documented results)

Find ways to compare CAA performance with established standards and performance of other similar programs.

What’s next?

23

• We must recognize the value of evidence-based or evidence-informed interventions…. but our definition of what counts as credible evidence should be expanded to allow for continuing improvement and innovation.

• ‘‘Evidence-based’’ does not have to mean experimental-based.

• We draw on evidence from many kinds of research, including program evaluations, and practice.

From Lisbeth Schorr

Redefining Evidence

24

Lisbeth Schorr tells us to identify the clear, measurable results sought by a complex intervention as the essential first step toward both successful implementation and to a successful evaluation.

We have a start on this with the NPIs.

Begin with a results framework.

25

• It is essential to have some way of comparing results to establish that the observed change has a high probability of resulting from the practices.

• How you will compare results from your programs to those who have not been involved in your programs.

The community-specific nature of place-based interventions makes it very hard to find a comparison group that would allow for a clinical “control group.”

Expect to compare results

26

It is possible to compare outcomes among the populations those served by a specific initiative to

(a) similar populations in the geographic area before the intervention began for whom baseline data are available,

(b) current populations that did not receive similar services and supports, but for whom data are already available, i.e. does not have to be collected as part of the evaluation, and

(c) national, state or local norms.

Other kinds of comparisons

27

The elements of ROMA principles and practices can help agencies establish evaluation frameworks that include the evidence that will be used to support their results

Establish CSBG industry standards

Remember what baseball did that we haven’t yet done:

ROMA to the rescue

28

Clearly identified the indicator to be measured

Collect data consistently

Analyze the data against records of “wins”

Publish the data and the analysis, so the public recognizes success

Baseball’s example

29

Support knowledge collection, analyses, and evidence syntheses that yield a more complete body of evidence.

Develop network tools and capacities to gather knowledge at greater scale.

Expand the menu of available evaluative techniques that can be matched to different types of interventions.

Combine findings from research, theory, and practice, for informed decision-making

Promote use of a results framework to strengthen measurement for accountability and learning

Suggested Actions

30

• Real-time learning can be achieved by using a well developed results framework to track progress toward those results, and using the data for real-time learning.

Use day-to-day management of work

31

As Peter Drucker has pointed out, “The greatest danger in times of turbulence is not the turbulence; it is to act with yesterday’s logic.”

We do not want to rely on fads, hunches, anecdotes, or good intentions.

Nor are we reluctant to identify and end support for the efforts that are ineffective.

Today’s Logic

32

Barbara MooneyNational ROMA Training Project

243 E. High St.Waynesburg, PA724-852-2272

[email protected]

For more discussion:

33