Using Behavioural Insights and Randomised Controlled Trials in public policy Dr Rory Gallagher CabinetOffice Behavioural Insights Team
Using Behavioural Insights and Randomised
Controlled Trials in public policy
Dr Rory Gallagher
CabinetOfficeBehavioural Insights Team
CabinetOfficeBehavioural Insights Team
Behavioural insights (BI) offers new evidence
and frameworks for influencing behaviour
1. Regulation
2. Incentives
3. Information
Behavioural Insights
CabinetOfficeBehavioural Insights Team
BI can help change the way we think about and
approach policy problems
“Our government will find intelligent ways to encourage, support and enable people to make better choices for themselves.”
CabinetOfficeBehavioural Insights Team
• ‘Gold Standard’ in evaluation
• Widely used in medicine, international development
and business, but less so in public policy
• Integral part of BIT’s methodology
RCTs are a good way of evaluating whether our
(BI) interventions are working
.
CabinetOfficeBehavioural Insights Team
Lots of myths about using RCTs in publioc
1. Pointless
2. Expensive2. Expensive
3. Unethical
4. Difficult to run
CabinetOfficeBehavioural Insights Team
• Understand what works
– Evidence-based policy; VfM; prove efficacy
• Context is everything
Why we use RCTs
• Context is everything
– Lab vs field; different policy areas/locations/people
• Details matter
– Small details = big effects; co-design
CabinetOfficeBehavioural Insights Team
Multi-arm trials can highlight important nuances
35.1
35.9
37.2
39.0% paying tax debts after 23 days
33.6
35.1
Control (8,558) UK Norm (8,300) Local Norm (8,403) Debt Norm (8,779) Local + Debt Norm
(8,643)
CabinetOfficeBehavioural Insights Team
Different measures also reveal nuances
5.00%
10.40%
15.40%
Proportion Leaving a Legacy Gift
£3,300 £3,100
£6,610 Size of Legacy Gifts
5.00%
Control Just Ask Passion Ask Control Just Ask Passion Ask
CabinetOfficeBehavioural Insights Team
21
35 35
25
30
35
40
Response rate of Doctors to HMRC letters
Don’t always produce the results you expect
4
21
0
5
10
15
20
Generic HMRC
letter
Tax Health Plan
letter
BI letter BI letter + social
norms
CabinetOfficeBehavioural Insights Team
Don’t always produce the results you want!
22
28
25
20
25
30
% Discontinuing council tax discount
0
5
10
15
Original letter (control) New BI letter New BI letter + signature up
front
CabinetOfficeBehavioural Insights Team
• Operational constraints– Policy interventions?
– Buy-in
– Sample size
– Data sharing
• Political constraints
Constraints and trade-offs
• Political constraints– Established programmes
– Transparency - ‘nil’ results/ ‘failures’
– Time frames
• Trade-offs– Who/how randomise
– No. of arms
– Impact (‘bundle’) vs specific causal effect
CabinetOfficeBehavioural Insights Team
• Randomisation
• Design
Things to watch out for
• Implementation
• Measurement
CabinetOfficeBehavioural Insights Team
BIT paper: Test, Learn, Adapt
BI and RCTs in action:
Two case studiesTwo case studies
CabinetOfficeBehavioural Insights Team
What is this letter about?
CabinetOfficeBehavioural Insights Team
Actions/consequences unclear
Is the
heading
clear? What is a
distress
Contact info
on the back
page
distress
warrant?
Lots of possible
consequences –
diluting effect
Key action –
how contact?
Primes
appeals
Is this info
necessary?
CabinetOfficeBehavioural Insights Team
Court fines
CabinetOfficeBehavioural Insights Team
Court fines
£9
£13
£11£12
£8
£11
£9£10
10
12
14
16
Av
era
ge
am
ou
nt
pa
id (
£) Trial 1
Trial 2
£4
£9 £8 £9
0
2
4
6
8
10
No text Standard Personal Amount Personal +
Amount
Av
era
ge
am
ou
nt
pa
id (
£)
Text condition
CabinetOfficeBehavioural Insights Team
Test, Learn, Adapt – 9 steps
1. Intervention - test effect of text messages on payment of court fines
5 arms = control, standard, personal, amount, personal+ amount
2.Outcome - % individuals paying court fines by the deadline date
3. Unit - randomise at individual level
(rather than different courts or local authority areas)
4. Numbers - to demonstrate a 10% increase in payment rates, we needed a
sample size of 2,000 people.
5. Randomisation - individuals randomly allocated to 5 conditions
6. Administration - Courts Service sent text messages to the individuals
randomly assigned to receive them
7-9. Test, Learn, Adapt - 3 trials, impressive results.
HMCTS exploring national roll-out.
CabinetOfficeBehavioural Insights Team
Getting people into employment
CabinetOfficeBehavioural Insights Team
Getting people into employment
1. Cut down process 2. Commitments 3. Strengths identification
CabinetOfficeBehavioural Insights Team
Getting people into employment
69%
66%65%
70%
80%% off-flow from benefits after 13 weeks
55% 55%57%
56%58%
61%
53%
40%
50%
60%
Feb Mar April May June July Aug
CabinetOfficeBehavioural Insights Team
Pragmatic stepped wedge randomised trial:
job centres across the county
Job
CentreReadiness
Implementation
Order
(TBC)
On-flow
per
month
Off-flow at
13 weeks
Months of trial
0 1 2 3 4 5 6 7 8 9 10 11 12 etc....
1
Ready
Harlow 608 51.70% *M1 *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....
2 Chelmsford 727 54.00% *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....
3 Colchester 734 52.80% *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....
4 & 5Canvey & 235 & 55.5% &
*M4 *M5 *M6 *M7 *M8 *M9 etc....
UNCLASSIFIED
4 & 5
Almost
Ready
Canvey &
Rayleigh
235 &
308
55.5% &
57.3%*M4 *M5 *M6 *M7 *M8 *M9 etc....
6 & 7Clacton &
Harwich
441 &
129
48.8% &
47.0%*M5 *M6 *M7 *M8 *M9 etc....
8 Grays 683 50.50% *M6 *M7 *M8 *M9 etc....
9
Not ready
Basildon 901 53.50% *M7 *M8 *M9 etc....
10 & 11Braintree &
Witham
444 &
177
55.2% &
52.2%*M8 *M9 etc....
12 Southend 930 50.20% *M9 etc....
Dark grey = current jobseeker process (control)
Blue = new jobseeker process (intervention)
*MX = measurement of 13 week off flow for cohort which began jobseeker claim in month X of trial (in bold = first month of outcome measure for that cluster)
CabinetOfficeBehavioural Insights Team
• BI is having major impacts
• Provides new frameworks and approach - humility
• RCTs increasing - driving ‘what works’
• Test, Learn, Adapt: agile implementation
Conclusions
• Test, Learn, Adapt: agile implementation
• Context, Details, People … matter
• Early days for BI & RCTS- long way to go, just scratching surface
• Next phase = sustainability and segmentation
• Future enablers = ‘big’ data, digital, transparency, VfM
http://blogs.cabinetoffice.gov.uk/behavioural-insights-team