Justice Data Lab: Facing the Third Sector How can we develop the capacity of third sector organisations to engage with data? Scottish Universities Insight.

Justice Data Lab:Facing the Third Sector

How can we develop the capacity of third sector organisations to engage with data?Scottish Universities Insight Institute

Georgina Eaton & Tillie Paul, Ministry of Justice

4th February 2015

What will be covered

• Aims and history of the Justice Data Lab

• How does the Justice Data Lab work and how can it be accessed?

• Findings to date

• Receptions of the service

• Challenges and successes

Aim of the Justice Data Lab

Launched in April 2013

..to improve the evidence base on successful rehabilitation..

..by giving organisations working with offenders secure and legal access to aggregate re-offending data

..enabling them to better assess the impact of their work on re-offending

Why are we doing the Justice Data Lab?

In 2012 we identified that charitable organisations in particular found it difficult to access re-offending data on their clients…

… this meant that they could not understand how effective their services were at rehabilitating offenders…

… and they were therefore unable to understand how their services could be improved, or have the evidence for further funding

It soon became clear that there was intense interest in this initiative from both public and private sector organisations too

Project timeline

2013 2014

April 2013:

One year pilot goes live

Justice Data Lab Pilot

December 2012: Announcement that Justice Data Lab would be piloted

October 2013:

First publication of Justice Data Lab products

January 2012:

Ministerial Approval to look into the feasibility of the Justice Data Lab

2012: Feasibility

Spring 2014:

Pilot extended for further year and announcement of improvements to the service

December 2011:

NPC approach MoJ about Data Lab idea

2012

How does the Justice Data Lab work?

Individual level data sent to MoJ

Provider organisation

MoJ

Analysis and Matching

Aggregate data return

Accessing the Justice Data Lab service

• Data upload template requirements:

Sample size must be at least 60 needed for robust analysis Matching variables including:

– full name– date of birth– gender– intervention start date

• Thorough information on the service they provide

• Send securely (CJSM or gsi account)

What does the Justice Data Lab not do?

X We won’t disclose individual level data

X We currently only give re-offending related outcomes

X There are ethical and practical considerations for assessing interventions aimed at certain groups of offenders (e.g. sex offenders, vulnerable persons) that mean that a Justice Data Lab analysis is not appropriate

Processing and matching the data

• Match to Police National Computer for demographics, criminal histories

• Find correct sentence and re-offending follow-up period in MoJ administrative datasets

• Link to employment and benefits data - Data share with DWP/HMRC

• Create matched control group of similar offenders who have not had the intervention from MoJ administrative datasets

Processing and matching the data

• Propensity Score Matching to match individuals in the two groups to each other

• Test differences in re-offending for the groups

• Statistical significance testing to determine whether there is a true difference between the groups

What is provided to Justice Data Lab users?

• One year re-offending rate

• Frequency of re-offending

• Time to re-offending

• Information on characteristics of both the treatment and control groups

0%

10%

20%

30%

40%

50%

60%

WYJS Participants (82offender records)

Matched ControlGroup (41,403 offender

records)

On

e ye

ar p

rove

n r

e-o

ffen

din

g r

ate

The best estimates for the one year proven re-offending rate for offenders who received an intervention from WYJS, and a matched

control group.

Cumulative findings to date

Of the 121 reports published so far:

• 27 reports indicated statistically significant reductions in re-offending on the one year proven re-offending rate

• 87 reports indicated insufficient evidence to draw a conclusion about the effect on the one year proven re-offending rate

• Of these 87, 11 reports detail statistically significant reductions in the frequency of re-offending

• 7 reports indicated a statistically significant increase in re-offending on the one year proven re-offending rate

0 10 20 30 40 50 60 70 80 90

Accommodation

Arts

Education / Learning

Employment

Problem Solving

Mentoring

Health and Wellbeing

Relationship Building

Restorative Justice

Youth Interventions

Number of Published Requests

Inte

rven

tion

Typ

eJustice Data Lab results by intervention type, published to date (the effect on the one

year proven re-offending rate)

Statistically significant increases Statistically significant decreases Total number of published findings

7

22

32

26

1

6

9

114

3

0

5

10

15

20

25

30

35

40

45

Educational Institution Private Public Voluntary Community Sector

Num

ber o

f Pub

lishe

d R

eque

sts

Sector Type

Justice Data Lab results by sector type, published to date (the effect on the one year proven re-offending rate)

Inconclusive Result Significant Decrease in Re-Offending Significant Increase in Re-Offending

Specific reports – Blue Sky

• Short term, full-time employment contracts. Aims to move them into onward full-time employment elsewhere.

• The one year proven re-offending rate for 72 offenders employed by Blue Sky was 31%, compared with 43% for a matched control group of offenders with similar characteristics.

• A reduction in re-offending between 1 and 23 percentage points.

The best estimates for the one year proven re-offending rate for offenders who received

an intervention from Blue Sky, and a matched control group.

Specific reports – Prisoners Education Trust

Prisoners Education Trust submitted data relating to offenders who had a grant for Open University, distance learning courses, or Art and Hobby materials between 2002 and 2010. We carried out one overall analysis

and four sub-analyses, the results are in the table below.

Treatment Significant Estimate of impact Reduction in frequencyGrant type Size Difference? on re-offending of re-offending? Open University 805 Y 2% to 8% YAccredited 152 N inconclusive inconclusiveUnaccredited 76 Y 1.4% to 21% YArts Materials 173 Y 0.3% to 14% YOverall 3,085 Y 5% to 8% Y

Specific reports – Prisoners Education Trust

“We have already changed our approach to funding different course

types as a result of the findings.”

“Our charity and our funders know that learning in prison works – but now we have the evidence to prove

it with this robust, hard-edged report carried out by MoJ

statisticians.”

Rod Clark, Chief Executive of Prisoners Education Trust

Reception of reports

Survey issued to organisations who have used the Justice Data Lab service showed that:

• The expectations around the Data Lab had been met, although our customer service could be variable

• The Justice Data Lab is a useful service for the third sector, helping to provide information on re-offending and impact

• More information on outcomes (severity, re-incarceration rates, employment and benefits) would be helpful

• Results had been used to understand / demonstrate impact internally and externally

Challenges

• First time the sector has had transparency about their effectiveness

• Claims about success vs. evidence

• Understanding technical aspects / statistical literacy

• Availability and quality of internal and external data

• Resources for MoJ and organisations

• Legalities for organisations

40% ↓ re-offending

Successes

• Positive feedback from users

• Organisations using results to change their services

• Continual demand for service

• Engaging with users

• Service developments

• Passing on experience to others

•

Award Winners!

• Government Finance Insight Award 2014

“Judges were impressed by the use of data in an innovative way, and the presentation of complex data in a way that is understood by all.”

• Royal Statistical Society Excellence in Official Statistics Award 2014

“Judges were impressed by the use of statistical techniques to assess success (or failure) in a critical area and by the exceptionally close way

MoJ statisticians had worked with their users, mainly non-statisticians.”

How can we develop the capacity of third sector organisations to engage with data?

• Data collection

• Central data system – Charity Log?

• Data manager/analyst

• Access to data through government/local authorities

• Open data

• Asking for help – Academics, Government Statisticians and Think Tanks

• Don’t be scared of data!!

Contact Details

Email: [email protected]

Accessing the Justice Data Lab service:https://www.gov.uk/government/publications/justice-data-lab

Published reports:www.gov.uk/government/collections/justice-data-lab-pilot-statistics

Government Statistical Service:https://gss.civilservice.gov.uk

Propensity Score Matching

Scale of propensity score

-1 0 1

Control offenders

Treatment offenders

Key

Matched offenders

Control offenders are matched to treatment offenders if the control offender’s propensity score are within a specified range away from the treatment offender’s propensity score.