CPS Recidivism Associated with a Home Visiting Program: A Quasi Experimental Analysis Ed Byrnes, Ph.D. Eastern Washington University Michael Lawson, M.S.

CPS Recidivism Associated with a Home Visiting Program: A Quasi Experimental Analysis

Ed Byrnes, Ph.D.Eastern Washington University

Michael Lawson, M.S.University of California at Davis

Evaluation Questions

1. What is the CPS recidivism rate for Birth & Beyond families in comparison to similar families who were involved with CPS?

2. How does the time to onset of CPS recidivism differ between Birth & Beyond families and similar families who were involved with CPS?

Evaluation Design• Quasi Experimental• External Comparison Group• Fixed Follow Up Periods• Data Sources

– B & B Database– Sacramento County CPS Database

• Between Groups Comparability– Multiple Points of Comparison

• Outcomes– CPS Recidivism and Onset of Recidivism

• Client Predictors of Outcomes– Demographic Characteristics– Assessment Results

Birth and Beyond Group

• 384 B & B Families

• All Had A Prior CPS Referral

• All Had At Least One B & B Home Visit

• All Had A First Home Visit On Or Before July 11, 2006 For One Year Follow Up

CPS Comparison Group• 327 Families• NOT B & B Clients• All Had A First CPS Referral Between January

1, 1999 and July 11, 2006• Sample Trimming

– Children’s Ages– Most Severe CPS Allegation– Children’s Language and Race

• Stratified Random Sampling– Most Severe CPS Allegation– Children’s Gender, Language and Race

Between Groups Comparability• No Significant Differences in

– Children’s Race– Children’s Primary Language – Children’s Gender

• Very Unlikely to Influence Results• Children’s Ages Differed Significantly

– CPS Comparison Group Had More Neonates– CPS Comparison Group Was Then Older– Children’s Age and Group Membership Correlated

at .25, a Weak Relationship– Significance An Artifact of Sample Size of 711

Families• If Children’s Ages Have Any Influence on

Results It Is Quite Small

Between Groups Comparability

Child's Race x BB & CPS Groups

42%

28% 31%33%42%

25%

0%

25%

50%

75%

100%

White Black Latino

Race

Pc

t o

f G

rou

p

CPS Comparison BB Group

Between Groups ComparabilityChild's Language x BB & CPS Groups

90%

10%

97%

3%0%

25%

50%

75%

100%

English Spanish

Language

Pc

t o

f G

rou

p

CPS Comparison BB Group

Between Groups Comparability

Child's Gender x BB & CPS Groups

52% 48%45%55%

0%

25%

50%

75%

100%

Female Male

Gender

Pc

t o

f G

rou

p

CPS Comparison BB Group

Between Groups Comparability

Child's Age x BB & CPS Groups

0

1

2

3

4

0 25 50 75 100

Percentile

Ch

ild's

Ag

e

CPS Comparison BB Group

CPS Recidivism• Data From CPS Database

• All Referrals With A Definitive Finding– Substantiated, Inconclusive and Unfounded– Reflect Contact With The CPS System

• One Year Follow Up Period– From First Home Visit for B & B Group– From First CPS Referral for Comparison

Group– Contrast Voluntary Additional Services with

Usual CPS Processing

CPS Recidivism• Of the 384 B & B families, 140 (36%) had a new

CPS referral during the one year follow up period

• Of the 327 CPS Comparison Group families 159 (49%) had a new CPS referral during the one year follow up period

• This difference was statistically significant• B & B participation versus regular CPS services

was weakly correlated with CPS Recidivism (Phi = .12)

• Participating in B & B home visiting influenced CPS recidivism in the desired direction

CPS Recidivism1 Year CPS Recidivism

49%

36%

0%

25%

50%

75%

100%

CPS Comparison BB Group

Group

Pc

t w

/ Ne

w C

PS

Re

ferr

al

CPS Recidivism Onset

• Data From CPS Database

• Same Criteria As CPS Recidivism

• One Year Follow Up Period

• Survival Analysis

• Time to First CPS Referral– From First Home Visit for B & B Group– From First CPS Referral for Comparison

Group

CPS Recidivism Onset• Average time elapsed from the beginning of

services to their first CPS referral during the one year follow up period– For B & B families was 279 days (sd = 6.5)– For CPS Comparison Group families was 256 days

(sd = 7.45)• This difference was statistically significant• Relationship between group membership and

time to the onset of CPS recidivism was moderate-to-strong– Magnitudes of the test statistics

• Participating in B & B home visiting influenced the time to the onset of CPS recidivism in the desired direction

CPS Recidivism Onset

CPS Recidivism Onset

49%

36%

0%

25%

50%

75%

100%

30 60 90 120 150 180 210 240 270 300 330 360

Days From Start

Cu

mu

lati

ve

CP

S

Re

cid

ivis

m R

ate

CPS Comparison BB Group

CPS Recidivism Onset

• B & B and CPS Comparison Groups have similar patterns of cumulative CPS recidivism rates through the first 150 days

• After 150 days the cumulative CPS recidivism curves diverge

• The relationship between participating in B & B home visiting and a delayed onset of CPS recidivism is more beneficial as time progresses

Predicting CPS Outcomes• Additional Evaluation Questions

– What demographic, service and assessment characteristics of cases are strongly related to having a new CPS referral within 150 days of the first home visit?

– What demographic, service and assessment characteristics of cases are strongly related to having a new CPS referral within one year of the first home visit?

• Members of the CPS comparison group never engaged in B & B services they could not be included in these analyses

Predicting CPS Outcomes• Logistic Regression Primer

– Predict the occurrence of an event

• Risk and Protection– Can be quantified using Logistic Regression

• Odds Ratios– Value of 1 means no relationship between a

factor and an event– Value greater than 1 means a risk factor for

an event– Value less than 1 means a protective factor

against an event

Predicting CPS Outcomes• Outcome Variables

– New CPS Referral Within 150 Days– New CPS Referral Within 1 Year– Same Definitions As Recidivism and Onset

• Predictor Variables– Demographics– Services– Assessment

Predicting CPS Outcomes• Model Building Process• Two Stages

– Model by Variable Category for Each Outcome– Final Model for Each Outcome

• Criteria For Final Model Inclusion– Statistical Significance– Odds Ratio Greater Than 2

• Evaluating Models– Goodness of Fit, Improved Case Classification,

Outliers In The Solution v. Over-identification– Sample Size– Explained Variance: How Much Variability In The

Outcome Can Be Attributed To Variables In The Model – 20% or higher is acceptable

Predicting CPS Outcomes• Predicting 150 Day CPS Recidivism• No Service Variables Met Final Model Criteria• Predictors At Intake

– Being a first time parent– Being pregnant– Being under the age of 25– Being a member of the Black race– Being a member of the White race– Being a member of the Hispanic race– Meeting the criteria for clinical depression– Having a high total score on the APPI (indicating

more skillful parenting)– Having a low score on the MSSI (indicating a low

level of social support)

Predicting CPS Outcomes• Predicting 150 Day CPS Recidivism• 85 Cases With Complete Data• 36% Explained Variance• Met Other Model Quality Criteria• Risk Factors

– Pregnant at the Time of Intake– Being Black– Having Low Levels of Social Support

• Protective Factor– Being Under Age 25

Predicting CPS OutcomesRisk & Protection for 150 Day CPS Referral

13.2

0.1

3.22.0

-5.0

0.0

5.0

10.0

15.0

Pregnant atIntake

Under 25 Black* Low SocialSupport*

Factor

Od

ds

Ra

tio

RISK

PROTECTION

* = Not Statistically Significant

Predicting CPS Outcomes• Predicting 1 Year CPS Recidivism• No Service Variables Met Final Model Criteria• Predictors At Intake

– Being fluent in English– Being pregnant– Having a low total score on the APPI (indicating less

skillful parenting)– Experiencing severe domestic violence

• Predictors From Follow Up Assessment– Having a high score on the CAGE questionnaire

(indicating a higher likelihood of alcohol dependence)

– Experiencing severe domestic violence

Predicting CPS Outcomes• Predicting 1 Year CPS Recidivism• 93 Cases With Complete Data• 25% Explained Variance• Met Other Model Quality Criteria• Risk Factors

– Pregnant at the Time of Intake– Having a Higher Final CAGE Questionnaire

Score– Having Low Parenting Skills at Intake

• No Protective Factors

Predicting CPS OutcomesRisk & Protection for 1 Year CPS Referral

4.5

2.2

8.2

-5.0

0.0

5.0

10.0

15.0

Pregnant at Intake Low Total APPI Score* Final CAGE Points

Factor

Od

ds

Ra

tio

RISK

PROTECTION

* = Not Statistically Significant

Dynamic Risk

• Assessment Results Are Not Very Stable Over Time– Pretest – Posttest Correlations Are Weak To

Modest

• Risk and Protective Factors That B & B Clients Experience Are Dynamic Rather Than Static– B & B Home Visitors Must View Assessment

As An Ongoing Process

Dynamic Risk

Chronbach's Alpha and Test-Retest Reliability

Internal Consistency Test-Retest Reliability

pre post r

AAPI-2 0.76 0.81 .48 - .67

MSSI 0.71 0.70 .41 - .51

CES-D 0.82 0.84 .32 - .34

CPS Recidivism Results

• B & B participation was associated with lower aggregate CPS recidivism rates

• B & B participation was associated with a delayed onset of CPS recidivism

• Given the broad ecologies of child maltreatment, these results can be viewed as promising

Discussion Questions• What challenges have you encountered in

conducting quasi experimental evaluation research, and how have you addressed these?

• Since California has a county administered child welfare system what efforts can we make to encourage uniformity of data for greater generalizability of results?

Discussion Questions• What are the trade offs between using

matched (case control) designs versus designs that rely on inclusion and exclusion algorithms for comparison groups?

• Since California and it's counties, like so many other states and locales, are facing deep budget shortfalls, how can we continue to promote evidence based practice as a priority?

Dissemination Plan

• Incorporate feedback about our study from this symposium into a final manuscript.

• Submit the manuscript for publication.

• Peer reviewed journals being considered include:– Children and Youth Services Review;– Research on Social Work Practice.

Follow Up Contact

Ed Byrnes, Ph.D.

Email: [email protected]

(509) 455-3422

Michael Lawson, M.S.

Email: [email protected]