Using BOTH Randomized Trial and Meta-analytic … BOTH Randomized Trial and Meta-analytic Approaches to Prevent Delinquency and Reduce Recidivism: The Pennsylvania Case Study . Collaborative

Brian Bumbarger, M.Ed. & Stephanie Bradley, Ph.D. Bennett-Pierce Prevention Research Center

Penn State University

Using BOTH Randomized Trial and Meta-analytic Approaches to Prevent Delinquency and Reduce Recidivism: The Pennsylvania Case Study

Collaborative Policy Partners: Linda Rosenberg John Frain

James Anderson Keith Snyder Mike Pennington Clay Yeager

Investigators and Authors:

Brian Bumbarger Stephanie Bradley Brittany Rhoades Cooper Mark Feinberg Mark Greenberg

The EPISCenter and much of the research described here are supported by grants from the Pennsylvania Commission on Crime and Delinquency. Special thanks to the staff of the Office of Juvenile Justice and Delinquency Prevention (OJJDP)

2

§  Strong evidence-base §  Demonstrated effects

(confidence) §  Sponsored lists

§  E.g., Blueprints

§  Funding requirements

§  Not an EBP for every community need/context

§  Many programs/services

already in widespread use

§  Some of those may be effective

§  Local expertise/fit

A 2-pronged approach…

Promoting Evidence-based Programs

Developing Practice-based Evidence

History of Research-Based Prevention in Pennsylvania

§  1994: Key state leaders introduce Communities that Care (CTC) §  Data-driven strategic prevention planning framework

§  1994-2002: Initiation of CTC funding by PCCD

§  16 cycles of CTC model introduced in ~120 communities

§  1996: PCCD Co-funding of research to develop Blueprints EBP list

§  1998: Initiation of Evidence-based Program Initiative by PCCD §  12 cycles of EBPs funded over 15 years, resulting in over 200 EBP replications

§  2001: Targeted Technical Assistance by Prevention Research Center §  Focus on implementation quality and fidelity, outcome assessment, & sustainability

§  2008: Resource Center for Evidence-Based Prevention and Intervention Programs and

Practices created (EPISCenter)

§  Multi-agency Steering Committee Representing Justice, Welfare, Education, and Health §  More proactive engaged TA and system coordination

Intermediary  and    State-­‐level  Preven1on  Support  System  

Support  to    Community    Preven0on  Coali0ons  

Improve  Quality  of    Local  Innova0ve  Programs  and  

Prac0ces  

Support to Evidence-based

Programs

Multi-Agency Steering Committee (Justice, Welfare, Education, Health)

The  EPISCenter  is  a  project  of  the  Prevention  Research  Center,  College  of  Health  and  Human  Development,  Penn  State  University,    and  is  funded  by  the  Pennsylvania  Commission  on  Crime  and  Delinquency  and  the  Pennsylvania  Department  of  Public  Welfare  

 as  a  component  of  the  Resource  Center  for  Evidence-­‐Based  Prevention  and  Intervention  Programs  and  Practices.    

A  unique  partnership  between  policymakers,  researchers,  and  communities  to  bring  science  to  bear  on  issues  of  public  health  and  

public  safety    

EPISCenter’s  3  Key  Func0ons:  §  Build general

prevention capacity

§  Build program-specific capacity

§  Facilitate interaction/ communication between systems

Rhoades, Bumbarger & Moore (2012). The Role of a State-Level Prevention Support System in Promoting High-Quality Implementation and Sustainability of Evidence-based Programs. American Journal of Community Psychology.

Tran

sla'

onal  

Research EPISCenter  

(Preven0on  Support  System)

Build  general  preven'on  capacity  among  prac''oners  and  policy  makers  

Technical  Assistance

Con'nuous  Quality  

Improvem

ent

Providers,  Prac''oners,  &    Community  Coali'ons  (Preven0on  Delivery  System)

Preven'on  Researchers  and    Program  Purveyors  

(Preven0on  Synthesis  &  Transla0on  System) Policy  Makers  &  Funding  Agencies

Pennsylvania’s “Brand Name” EBP dissemination in 2014…

-­‐20

-­‐10

0

10

20

30

40

-­‐10.8

33.2

-­‐10.8

16.4

Delinquency Academic  PerformanceNegative  Peer  Influence School  Engagement

5 year Longitudinal Study of PA Youth % Change of CTC/EBP Youth Over

Comparison Group

419 age-grade cohorts over a 5-year period:

youth in CTC communities using

EBPs had significantly lower rates of

delinquency, greater resistance to negative

peer influence, stronger school engagement and

better academic achievement

Feinberg, M.E., Greenberg, M.T., Osgood, W.O., Sartorius, J., Bontempo, D.E. (2010). Can Community Coalitions Have a Population Level Impact on Adolescent Behavior Problems? CTC in Pennsylvania, Prevention Science.

Impact on Juvenile Court Placement Rates: Comparison of Placement Rates for Counties* With and Without an EBI

Bumbarger, B. K., Moore, J., & Rhoades, B. (2010). Impact of evidence-based interventions on delinquency placement rates. Presentation at 2011 Society for Prevention Research annual meeting. Washington, DC.

PA Return-on-Investment Analysis

Jones, D. J., Bumbarger, B. K., Greenberg, M. T., Greenwood, P., & Kyler, S. (2008). The economic return on PCCD’s investment in research-based programs: A cost-benefit assessment of delinquency prevention in Pennsylvania. The Prevention Research Center, The Pennsylvania State University.

Programs/services can be placed along a continuum of confidence based on their evidence or theory

How  confident  are  we  that  this  program  or  practice  is  a  good  use  of  resources    AND  improves  outcomes  for  children  and  families?  

Very  Confident  

ü   Evidence-­‐based  “This  program  has    been  rigorously  evaluated    and  shown  to  work”  

ü   Promising  Approaches  “We  really  think  this    will  work…    but  we  need    time  to  prove  it”  

•   Best  Practices  •  “We’ve  done  it    

and  we  like  it”  

EFFECTIVE  HARMFUL  

"    Iatrogenic  (Harmful)  “This  program  has  been  rigorously  evaluated  and  shown  to  be  ineffective  or  harmful”  

Promising  Ineffective   unknown  Very  

Confident  

*Bumbarger  &  Rhoades,  2012  

What is the Standardized Program Evaluation Protocol (SPEP)?

§  Partnership between probation/juvenile court & service providers

§  Quality improvement process informed by meta-analytic research

§  Specifically aimed at reducing recidivism

•  PA has adopted a data-driven, evidence-based approach in improving the juvenile justice system to reduce youth recidivism rates

•  PA average rate of recidivism from 2007-2009 was 22%, with a range of 6-33%

•  Five medium-sized urban counties self-selected to participate in pilot of SPEP quality improvement process

•  Pilot counties’ recidivism rates ranged from 12-24%

•  SPEP scores can range from 0-100; scores 50+ shown to be predictive of reduced recidivism at 6 and 12 months

Key Drivers of JJ Services Effectiveness

Service Type

Provider Delivery

SPEP Assessment

Probation Usage

Program Improvement Lifecycle SPEP

Assessment

Understanding SPEP (score)

Improvement Implications

Improvement Plan

Plan Implementation Service

Type

Provider Delivery

SPEP Assessment

Probation Usage

Program Improvement Plan

§  Addresses the areas identified during the SPEP process, as prioritized by stakeholders

§  Identifies the timeframe and method for improvements in accordance with the stakeholder capacities

§  Identifies the needed technical assistance and support which may be necessary to implement improvements.

§  Process for monitoring the progress and outcomes of the Program Improvement Plan.

Most services* score ≥ 50:

* important to consider selection bias of self-selected early adopters

•  23 fully scored services; avg. score was 58, range of 37-100

•  Most services scored well on staff training and supervision

•  Most services scored poorly on written protocols and response to drift

•  VERY poor fidelity to dosage and duration standards

•  Self-selection bias; “True” state of the field may be in worse shape than what is depicted here

•  Evidence of SPEP predictive validity is very gross measure (+/- 50); limits precision

•  Probably not quite ready for scale – better tools and training needed after this pilot phase

•  Nearly impossible to convince providers and JPOs NOT to focus on score…potential implications for competition and funding

•  Array of services insufficient to match risk assessment

•  Further muddies the waters concerning “evidence-based”

•  Lower recidivism?? (already quite low – stay tuned)

Limitations and Cautions…

•  Qualitative interviews lead JPOs better understanding what programs “really” offer (and for whom services are best suited)

•  JPOs are now more routinely sending YLS risk score to providers (better service matching and treatment plans)

•  Ongoing education of juvenile court system (Judges) re: relationship of dosage and duration to youth/service outcomes

•  Improved relations between probation and providers

Early Benefits from the Pilot Phase

•  Service providers are encouraged to more clearly articulate the theory of behavior change of their services

•  Providers are developing more specific procedures for coaching, monitoring implementation and addressing drift

•  Early discussions of moving to performance-based contracting?

•  Evidence that these two models of “evidence” can be complimentary rather than competing!

Early Benefits from the Pilot Phase (cont.)

Thank  You!            

Brian  Bumbarger  Preven'on  Research  Center,  Penn  State  University  

www.preven'on.psu.edu  www.episcenter.org