Chapin Hall Center for Children Measuring Changes in Reclaiming Futures Communities: National Evaluation Results Annual Meeting of the Coalition for Juvenile.

Chapin Hall Center for Children

Measuring Changes in Reclaiming Futures Communities: National Evaluation Results

Annual Meeting of the Coalition for Juvenile JusticeWashington, DCJune 11, 2007

Jeffrey A. ButtsChapin Hall Center for Children

University of Chicago

Views expressed are those of the author and should not be attributed to the University of Chicago, Chapin Hall Center for Children, their trustees or funders..


Reclaiming Futures

An initiative of Robert Wood Johnson Foundation

Focused on substance abuse interventions in juvenile justice

Principal strategy was organizational change/system reform

Evaluation challenge: Tracking change in 10 completely different service systems with 10 different change strategies


Surveys of Reclaiming Futures Communities

National evaluation tracked change with bi-annual surveys

Surveys measured quality of systems as reported by 20 to 40 expert informants in each community

Six surveys conducted in:2003 – December 2004 – June and December 2005 – June and December 2006 – June

Response rate averaged 70 percent and ranged from 63% (December 2003) to 73% (December 2004)



58 questions about quality and effectiveness

All questions were brief statements

Respondents indicated whether they strongly disagreed, disagreed, were neutral, agreed, or strongly agreed

Answers coded so higher scores indicate positive opinions

Responses scored: -10 strongly negative -5 somewhat negative 0 neutral +5 somewhat positive +10 strongly positive



3 General Areas 13 Indices

ADMINISTRATION Access to ServicesData SharingSystems IntegrationResource Management

COLLABORATION Client InformationPartner InvolvementAgency Collaboration

QUALITY AOD AssessmentsTreatment EffectivenessTargeted TreatmentCultural IntegrationFamily InvolvementPro-social Activities







The overall effectiveness of treatment was measured with five statements:

In the past three months, …

1. The substance abuse treatment needs of youth in my community were adequately met.

2. The mental health needs of youth in my community were adequately met.

3. Graduated sanctions were used effectively to support treatment goals for youth.

4. Youth-serving agencies in my community generally did a good job serving youth.

5. Youth-serving agencies in my community were usually able to provide youth with the range of services they needed.








Survey Results for Each Index


Access to Services

ADMINISTRATION INDICES


Data Sharing



Systems Integration



Resource Management



Client Information

COLLABORATION INDICES


Partner Involvement



Agency Collaboration



AOD Assessments

QUALITY INDICES


Treatment Effectiveness

QUALITY INDICES


Targeted Treatment

QUALITY INDICES


Cultural Integration

QUALITY INDICES


Family Involvement

QUALITY INDICES


Pro-social Activities

QUALITY INDICES


1 AOD Assessment 28% Yes2 Treatment Effectiveness 26% Yes

3 Client Information 24% Yes4 Family Involvement 22% Yes

5 Systems Integration 19% Yes6 Pro-social Activities 18% Yes

7 Data Sharing 18% Yes8 Resource Management 16% Yes

9 Agency Collaboration 14% Yes10 Targeted Treatment 12% Yes

11 Cultural Integration 11% Yes12 Access to Services 9% Yes

13 Partner Involvement -2% No

Change in scores across all Reclaiming Futures communities between December 2003 and June 2006, ranked by percentage of possible improvement realized

Survey IndexRankPercent of Possible Improvement Realized Statistically

Significant?

Note: The total improvement possible in any index is limited by the score in Survey 1. Indices with high values in Survey 1 have less room to improve, given that +10 is the maximum possible score. “Possible improvement” realized was calculated by taking the total difference in survey scores (Survey 6 – Survey 1), and dividing by the maximum possible improvement, or 10 minus the first survey score. Change Indices are rank ordered from largest to smallest change, based on percentage of possible improvement realized. A t-test was used to determine each if change was significantly different from zero.


Significant increases in survey indices by RF community, 2003-2006

Note: Significant increases occurred when the difference in mean index scores on the first survey (December 2003) and last survey (June 2006) was statistically significant ( p < .05). An increase was considered to be linear when at least 4 of 5 possible score changes between the six surveys were in a positive direction.


Community Significant and Linear

Increases: 2003-2006 Community Significant and Linear

Increases: 2003-2006

Anchorage Access to Services AOD Assessment Treatment Effectiveness Cultural Integration

New Hampshire Access to Services Data Sharing AOD Assessment Treatment Effectiveness Targeted Treatment

Santa Cruz None Dayton Access to Services Data Sharing Resource Management Client Information AOD Assessment Pro-social Activities

Chicago Data Sharing Systems Integration Resource Management AOD Assessment Pro-social Activities

Portland Data Sharing Cultural Integration

SE Kentucky Data Sharing Systems Integration Client Information Agency Collaboration AOD Assessment Family Involvement

Rosebud Data Sharing Client Information

Marquette Data Sharing Pro-social Activities

Seattle Treatment Effectiveness Targeted Treatment

None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.


What About Bias?

Answers to survey questions could be measures of self-interest or wishful thinking

Regression analyses were performed to check on this possibility

Analyses tested the influence of several factors on the trajectory of changes in individual survey scores

Main variable of interest: Did survey scores change more for RF “Insiders” -- the 4 or 5 people in each community who were involved in planning the RF initiative and who were closer to RWJF?


Most Indices Were Not Affected by Bias



The values differ, but the trajectories are positive for both groups, indicating growing confidence in the effectiveness of treatment.




AOD Assessment


A Few Indices Were Affected by Bias


Cultural Integration


Partner Involvement















If we cross out the indices that were possibly affected by bias…


























































Conclusions• Respondent ratings of local juvenile justice systems improved significantly between 2003 and 2006.

• Increases were statistically significant in 12 of the 13 survey indices.

• Improvements were especially dramatic in the ratings for: - treatment effectiveness, - the use of client information in support of treatment, - the use of screening and assessment tools, and - overall systems integration.

• These findings suggests that the coordination and quality of substance abuse interventions for youthful offenders improved during the RWJF Reclaiming Futures initiative.


Jeffrey A. ButtsResearch FellowChapin Hall Center for ChildrenUniversity of Chicago1313 East Sixtieth StreetChicago, IL 60637(773) [email protected]

This presentation can be accessed at:www.jeffreybutts.net

For more information:

Chapin Hall Center for Children Measuring Changes in Reclaiming Futures Communities: National Evaluation Results Annual Meeting of the Coalition for Juvenile.

Documents

butts chapin hall center

positive slide

biannual surveys surveys

treatment goals

substance abuse treatment

quality of systems

different service systems

mental health needs