Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post Signs of Safety

Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post

Signs of Safety

Dr Mary Salveron, Professor Leah Bromfield and Professor Fiona ArneyBritish Association for the Prevention and Study of Child Abuse and Neglect

(BASPCAN)12-15th April 2015, University of Edinburgh

Overview• Signs of Safety Post-Doctoral Research

Program• Evaluation approaches• Signs of Safety Theory of Change• Signs of Safety Administrative Data Study• Limitations and Discussion

Signs of Safety Post-Doctoral Research Program

• 3 year research collaboration between ACCP and WA Department for Child Protection and Family Support

• To examine the impact and implementation of the Signs of Safety child protection practice model/approach on children, parents and practitioners

Signs of Safety Post-Doctoral Research Program

2013• Theory of Change• Implementation• Children’s Study

2014• Practitioner Study• Administrative Data

Study• Parent Study

2015• Aboriginal Staff and

Practitioner Study• Triangulation Paper

Signs of Safety• Home-grown, locally developed approach to child protection

practice• Origins in Solution-focused brief therapy • Used in 12 countries and spreading across the world• Designed to provide child protection practitioners with

straight forward methods and tools to tackle child abuse in partnership with parents and professionals/service providers

• Limited evidence base

Administrative Data Study• Use Departmental data from ASSIST (known as

‘administrative data’)• Examine the impact of Signs of Safety on outcomes for

children• Compare these to outcomes for children pre Signs of

Safety• Essentially we want to know if Signs of Safety is making

a difference at the highest level (substantiations, entry into care) (Department KPIs)

HypothesesCompared to children pre Signs of Safety, there will be:1. A decrease in the number of children coming into care of

Department2a. A decrease in proportion of children with substantiations

following an unsubstantiated notification2b. A decrease in proportion of children with substantiations

following a substantiated notification3. An increase in proportion of Safety and Wellbeing

Assessments (SWA) outcomes proceeding to family support or child centred family support

Limitations• Retrospective study• Use of administrative data that was not designed for

evaluative purposes• Outcomes are multiply determined and influenced by:– Changing policy and practice context– Evolving Signs of Safety child protection practice framework– Family and other service factors– Demands on the system and resourcing

Methodology• Retrospective (follow forward) historical

control using Departmental administrative data from 3 consecutive years pre-Signs of Safety implementation (2005-2007) and 3 consecutive years post-Signs of Safety implementation (2011-2013)

• First notifications

Cohort 1 – Pre SoS(2005-2007)

Cohort 2 – Post SoS (2011-2013)

Number of Children (Pre-birth to 18 years)

3,011 7,562

Gender 1425 boys, 1565 girls 3383 boys, 3813 girlsAboriginality 854 Aboriginal, 1738 non-

Aboriginal419 not recorded

1602 Aboriginal, 1928 non-Aboriginal

4032 not recorded

Notifications 4,340 12,003Investigations/Safety and Wellbeing Assessments

2,391 (55% of notifications) 8,050 (67% of notifications)

Substantiations 962 (40% of investigations) 2,160 (27% of investigations)

Sample and Context

FindingsHypothesis 1: There will be decrease in the number of children coming into

care of DepartmentThere was an increase in the number and rate of of children coming into care of Department at a population level

Cohort 1 (pre-SoS) Cohort 2 (post-SoS)All children 2.06 per 1000 children in

2005 (347 children)3.89 per 1000 children in 2011 (647 children)

Aboriginal children 0.3 per 1000 children in 2005 (147 children)

0.49 per 1000 children in 2011 (268 children)

FindingsHypothesis 1: There will be decrease in the number of children coming into

care of DepartmentThere was a decrease in the proportion of children who were notified who subsequently entered care

Cohort 1 (Percentage of children who were first notified and came into care)

Cohort 2 (Percentage of children who were first notified and came into care)

All children 3011 children with a notification367 who were first notified and came into care (12.1%)

7562 children with a notification647 who were first notified and came into care (8.5%)

Aboriginal children 3011 children with a notification147 Aboriginal children who were first notified and came into care (4.8%)

7562 children with a notification268 Aboriginal children who were first notified and came into care (3.5%)

FindingsHypothesis 2a: There will be a decrease in proportion of children with

substantiations following an unsubstantiated notificationThere was an increase in the proportion of children with substantiations following an unsubstantiated notification between Cohort 1 and Cohort 2

Number of children with substantiations following an unsubstantiated notification

Number of children with substantiations following an unsubstantiated notification

All children 1,271 children with unsubstantiated notification99 children (7.8%) had a substantiated notification

4,578 children with unsubstantiated notification523 children (11.4%) had a substantiated notification

Aboriginal children 370 children with unsubstantiated notification44 children (11.9%) had a substantiated notification

976 children with unsubstantiated notification181 children (18.5%) had a substantiated notification

FindingsProportion of all children with substantiations following an unubstantiated

notification by primary harm typePrimary Harm Type Cohort 1 Cohort 2

Emotional Abuse 5 out of 105 children (4.8%)

66 out of 1053 children (6.3%)

Neglect 20 out of 296 children (6.8%)

114 out of 1067 children (10.7%)

Physical Abuse 18 out of 492 children (3.7%)

19 out of 861 children (2.2%)

Sexual Abuse 9 out of 441 children (2%)

35 out of 1324 children (2.6%)

FindingsProportion of Aboriginal children with substantiations following a

unsubstantiated notification by primary harm typePrimary Harm Type Cohort 1 Cohort 2

Emotional Abuse 5 out of 44 children (11.4%)

9 out of 142 children (13.4%)

Neglect 7 out of 116 children (6%)

46 out of 193 children (23.8%)

Physical Abuse 5 out of 119 children (4.2%)

2 out of 85 children (2.4%)

Sexual Abuse 3 out of 109 children (2.8%)

4 out of 126 children (3.2%)

FindingsHypothesis 2b: There will be a decrease in proportion of children with

substantiations following a substantiated notificationThere was an increase in the proportion of children with substantiations following an substantiated notification between Cohort 1 and Cohort 2

Number of children with substantiations following an unsubstantiated notification

Number of children with substantiations following an unsubstantiated notification

All children 854 children with substantiated notification100 children (11.7%) had a substantiated notification

1,779 children with substantiated notification319 children (17.9%) had a substantiated notification

Aboriginal children 304 children with substantiated notification52 children (17.1%) had a substantiated notification

501 children with substantiated notification117 children (23.4%) had a substantiated notification

FindingsProportion of all children with substantiations following a substantiated

notification by primary harm typePrimary Harm Type Cohort 1 Cohort 2

Emotional Abuse 13 out of 112 children (11.6%)

59 out of 529 children (11.2%)

Neglect 30 out of 276 children (10.9%)

82 out of 575 children (14.3%)

Physical Abuse 15 out of 273 children (5.5%)

17 out of 315 children (5.4%)

Sexual Abuse 4 out of 223 children (1.8%)

18 out of 379 children (4.7%)

FindingsProportion of Aboriginal children with substantiations following an

substantiated notification by primary harm typePrimary Harm Type Cohort 1 Cohort 2

Emotional Abuse 5 out of 49 children (10.2%)

19 out of 178 children (10.7%)

Neglect 17 out of 115 children (14.8%)

39 out of 217 children (18%)

Physical Abuse 9 out of 89 children (10.1%)

6 out of 69 children (8.7%)

Sexual Abuse 3 out of 64 children (4.7%)

6 out of 46 children (13%)

FindingsHypothesis 3: There will be an increase in proportion of Safety and

Wellbeing Assessments (SWA) outcomes proceeding to family support or child centred family support

Hypothesis cannot be answered as no comparable fields for Cohort 1 and Cohort 2.

Limitations and understanding the context

• Dramatic increase in children coming into contact with the Department

• Policy and practice changes – Legislation• Entry in to out-of-home care – Early

intervention services, placement prevention

OTHER IMPORTANT PEOPLE (Carers, Relatives, Friends, Non-Government

Practitioners/Staff) ARE ENGAGED

Figure 1: Signs of Safety Results Logic

Children and young people are SAFE (long term sustained safety)

Children and young people are SAFE from imminent/immediate harm

3

Future-focused | Working relationships based on partnership | SIGNS OF SAFETY PRINCIPLES | Critical Inquiry | Recognising what is working well

Signs of Safety Tools for Children and Young People

Signs of Safety Mapping

(incl Three Columns)

Creating Safety Networks

Workforce Wellbeing/Resilience

CHILDREN AND YOUNG PEOPLE ARE ENGAGED

2

PARENTS ARE ENGAGED

1

SHARED UNDERSTANDING

SAFETY

PLAN

CASE PLAN

SHARED AGREEMENT

Children and Community Services Act 2004 (WA), Western Australia Child Protection Policy

Context

Children participateNoticing what works Families are socially connected

SIGNS OF SAFETY DISCIPLINES

Signs of Safety IMPLEMENTATION(e.g. training, coaching, leadership, appreciative inquiry)

Signs of Safety practiced with fidelity

Children’s perspective acts as

circuit breaker

4

Key Messages• Importance of articulated theory of change – What is Signs of

Safety intended to do?• Limitations of administrative data as evidence in a changing

context• Problems related to the use of blunt indicators and

directionality of hypotheses (surveillance effects, enhanced decision making)

• Signs of Safety fidelity, implementation and dose• Best outcomes for children may not be measured by

administrative data

Dr Mary SalveronPost-Doctoral Fellow (Signs of Safety)Australian Centre for Child Protection

University of South Australia+61 8 83022980 / 0400 863 784

[email protected]/childprotection