1 Do the Claims for Spending Billions on Crime Reduction Initiatives Stand up Do the Claims for Spending Billions on Crime Reduction Initiatives Stand.

11

Do the Claims for Spending Billions Do the Claims for Spending Billions on Crime Reduction Initiatives Stand on Crime Reduction Initiatives Stand

upupRadical Statistics Conference 2006Radical Statistics Conference 2006

Paul MarchantPaul [email protected]@leedsmet.ac.uk

22

AimsAims A look at knowing ’What Works’.A look at knowing ’What Works’.

Things worth encouraging:Things worth encouraging: ‘‘Good Statistics’ Good Statistics’

(as many arguments are statistical in (as many arguments are statistical in nature)nature)

Transparency in design and reportingTransparency in design and reporting ‘‘Investigative Statistics’Investigative Statistics’ Scientific scepticismScientific scepticism Research which is sufficiently sound in order Research which is sufficiently sound in order

to properly justify spending on major to properly justify spending on major programmesprogrammes

33

Science Science Acquiring knowledge using data from Acquiring knowledge using data from

observation and experiment.observation and experiment. An inherently uncertain matter……Statistics!An inherently uncertain matter……Statistics! Not all results given, are the same. Therefore Not all results given, are the same. Therefore

there is the need to synthesise. there is the need to synthesise. Science is a public matter: not just because of Science is a public matter: not just because of

the impact of the products of science, but the impact of the products of science, but also because of the need to check work. also because of the need to check work.

Need open access, 'to pretty much Need open access, 'to pretty much everything', so that the work can be everything', so that the work can be replicated and checked (data, methods, clear replicated and checked (data, methods, clear complete reports).complete reports).(Need protocols to be published in advance.)(Need protocols to be published in advance.)

44

A scientific answer.…A scientific answer.…

is never something like x = 1.23is never something like x = 1.23 nor is it just x = 1.23 nor is it just x = 1.23 ± 0.45± 0.45 but rather also how it was derived, what but rather also how it was derived, what

assumptions and approximations are assumptions and approximations are involved, so that outsiders can scrutinize.involved, so that outsiders can scrutinize.

Just because assumptions are not Just because assumptions are not mentioned does not mean they are not mentioned does not mean they are not being made!being made!

55

Some quotations Some quotations ““The combination of some data and an The combination of some data and an

aching desire for an answer does not aching desire for an answer does not ensure that a reasonable answer can be ensure that a reasonable answer can be extracted from a body of data.” extracted from a body of data.” John Tukey John Tukey

““While every data set contains noise, some While every data set contains noise, some data sets may contain signals. Therefore data sets may contain signals. Therefore before you can detect a signal within any before you can detect a signal within any given data set you must first filter out the given data set you must first filter out the noise.”noise.”

Donald J. Wheeler in Understanding Variation: Donald J. Wheeler in Understanding Variation: the key to managing chaos. Pub SPCthe key to managing chaos. Pub SPC

66

Time Variation in CrimeTime Variation in Crime

It appears that little is known about It appears that little is known about how crime varies on the small scale. how crime varies on the small scale. Therefore it is difficult to be clear if Therefore it is difficult to be clear if any changes are due to a crime any changes are due to a crime reduction intervention. reduction intervention.

77

The Randomised Controlled The Randomised Controlled TrialTrial

(A truly marvellous scientific (A truly marvellous scientific invention)invention) Note to avoid ‘bias’:Note to avoid ‘bias’:

Allocation is best Allocation is best made tamper-proof. made tamper-proof. (e.g. use (e.g. use

‘concealment’)‘concealment’) Use multiple Use multiple

blinding of:blinding of: patients, patients, physicians, physicians, assessors, assessors, analysts …analysts …

Population

Take Sample

Randomise to 2 groups

Old Treatment

Compare outcomes (averages) recognising that

these are sample results and subject to sampling variation when applying back to the population

New Treatment

88

Counts of those cured and not Counts of those cured and not cured under the two cured under the two

treatmentstreatmentsCured Not

CuredNew Treatment a b

Control(Standard treatment)

c d

99

The Odds Ratio The Odds Ratio

OR = Ratio of the odds of cure under the two OR = Ratio of the odds of cure under the two treatmentstreatments

== Odds of cure under new treatmentOdds of cure under new treatment = = a/ba/b Odds of cure under old treatment c/d Odds of cure under old treatment c/d

== adadbcbc

If If OR > 1 the new treatment is betterOR > 1 the new treatment is betterOR < 1 the old treatment is betterOR < 1 the old treatment is better

Cure Not

New a b

Old c d

1010

But …But …

… … there is there is sampling variabilitysampling variability..Consider a table: Consider a table:

With OR = With OR = 60604545 = 1.23 = 1.23 404055 55 is not good evidence for a difference in is not good evidence for a difference in

treatment effectiveness. treatment effectiveness. The numbers are small and the sample OR = The numbers are small and the sample OR =

1.23 could be due to chance, when in fact 1.23 could be due to chance, when in fact the population OR=1.0 the population OR=1.0

CureCure NotNot

6060 4040

5555 4545

1111

Sampling VariationSampling Variation

Sampling Variation is given by the (asymptotic) Sampling Variation is given by the (asymptotic) standard error of ln(OR)standard error of ln(OR)

(S.E (ln(OR)) )(S.E (ln(OR)) )22= Var(ln(OR))= Var(ln(OR))== 11 + + 11 + + 11 + + 1 1

a b c d a b c d If the events are If the events are statistically independentstatistically independent..The sample ln(OR) is distributed about the The sample ln(OR) is distributed about the

population ln(OR) in a ‘Normal’/Gaussian population ln(OR) in a ‘Normal’/Gaussian fashion with its standard deviation given by fashion with its standard deviation given by that calculated from the s.e. (5% resides that calculated from the s.e. (5% resides outside 1.96 s.e.)outside 1.96 s.e.)

1212

Crime counts before and afterCrime counts before and afterin the two areasin the two areas

Examine the Cross Product Ratio (CPR) = Examine the Cross Product Ratio (CPR) = a/ba/b c/d c/d

If it is convincingly >1 then lighting works against crime.If it is convincingly >1 then lighting works against crime.

Before After

Treatment Area(lighting increased)

a b

Comparison Area(lighting stays the same)

c d

1313

Lighting and crime Lighting and crime

Much justification for exterior lighting Much justification for exterior lighting is made on the basis of crime is made on the basis of crime reduction.reduction.

(e.g. The Institution of Lighting (e.g. The Institution of Lighting Engineers)Engineers)

There seem to be many ‘theoretical There seem to be many ‘theoretical suggestions’ why lighting might suggestions’ why lighting might increase or decrease crime. increase or decrease crime.

1414

Light has a good press!Light has a good press!

‘…‘…God said, let there be light and God said, let there be light and there was light. And God saw the there was light. And God saw the light, that it was good….’ Genesis light, that it was good….’ Genesis Chapter 1 Verses3/4.Chapter 1 Verses3/4.

It seems ‘wicked’ to question the It seems ‘wicked’ to question the benefit of lighting.benefit of lighting.

However there is a ‘dark side’; However there is a ‘dark side’; lighting’s environmental impacts, lighting’s environmental impacts, possible health impacts.possible health impacts.

1515

My ‘Interest’My ‘Interest’ “…“…Paul Marchant, statistician at Leeds Paul Marchant, statistician at Leeds

Metropolitan University who argues that statistics Metropolitan University who argues that statistics used in the Home Office Study 251 could equally used in the Home Office Study 251 could equally be used to show that street lighting actually be used to show that street lighting actually increases levels of crime. This is an argument increases levels of crime. This is an argument which the APPLG, alongside the ILE, would hope to which the APPLG, alongside the ILE, would hope to show as utterly absurd. Of course it is worth show as utterly absurd. Of course it is worth noting that Paul Marchant is also an astronomer noting that Paul Marchant is also an astronomer as well as being a statistician, and that this may as well as being a statistician, and that this may lead to some bias in his interpretation of the lead to some bias in his interpretation of the statistics he refers to.”statistics he refers to.”

P56 of the March/April 2004 issue of the Lighting P56 of the March/April 2004 issue of the Lighting JournalJournal,, the magazine of the Institution of Lighting the magazine of the Institution of Lighting Engineers.Engineers.

APPLG= The All-Party Parliamentary Lighting GroupAPPLG= The All-Party Parliamentary Lighting GroupILE= The Institution of Lighting EngineersILE= The Institution of Lighting Engineers

1616

Odds ratio.1 1 10

Study % Weight Odds ratio (95% CI)

3.82 (2.28,6.40) Birmingham 0.7

1.72 (1.17,2.52) Stoke-on-Trent 1.7

1.44 (1.17,1.77) Dudley 6.3

1.39 (1.04,1.86) Atlanta 3.3

1.38 (0.97,1.97) Fort Worth 2.2

1.37 (1.06,1.77) Milwaukee 4.2

1.35 (1.23,1.48) Bristol 36.3

1.24 (0.95,1.64) Kansas City 4.0

1.14 (0.62,2.08) Dover 0.9

1.02 (0.75,1.40) Harrisburg 3.4

0.98 (0.88,1.11) New Orleans 24.4

0.94 (0.79,1.12) Portland 10.9

0.75 (0.48,1.19) Indianapolis 1.8

1.22 (1.15,1.29) Overall (95% CI)

Odds ratio.1 1 10

Study % Weight Odds ratio (95% CI)

3.82 (2.28,6.40) Birmingham 0.7

1.72 (1.17,2.52) Stoke-on-Trent 1.7

1.44 (1.17,1.77) Dudley 6.3

1.39 (1.04,1.86) Atlanta 3.3

1.38 (0.97,1.97) Fort Worth 2.2

1.37 (1.06,1.77) Milwaukee 4.2

1.35 (1.23,1.48) Bristol 36.3

1.24 (0.95,1.64) Kansas City 4.0

1.14 (0.62,2.08) Dover 0.9

1.02 (0.75,1.40) Harrisburg 3.4

0.98 (0.88,1.11) New Orleans 24.4

0.94 (0.79,1.12) Portland 10.9

0.75 (0.48,1.19) Indianapolis 1.8

1.22 (1.15,1.29) Overall (95% CI)

Odds ratio.1 1 10

Study % Weight Odds ratio (95% CI)

3.82 (2.28,6.40) Birmingham 0.7

1.72 (1.17,2.52) Stoke-on-Trent 1.7

1.44 (1.17,1.77) Dudley 6.3

1.39 (1.04,1.86) Atlanta 3.3

1.38 (0.97,1.97) Fort Worth 2.2

1.37 (1.06,1.77) Milwaukee 4.2

1.35 (1.23,1.48) Bristol 36.3

1.24 (0.95,1.64) Kansas City 4.0

1.14 (0.62,2.08) Dover 0.9

1.02 (0.75,1.40) Harrisburg 3.4

0.98 (0.88,1.11) New Orleans 24.4

0.94 (0.79,1.12) Portland 10.9

0.75 (0.48,1.19) Indianapolis 1.8

1.22 (1.15,1.29) Overall (95% CI)

Odds ratio.1 1 10

Study % Weight Odds ratio (95% CI)

3.82 (2.28,6.40) Birmingham 0.7

1.72 (1.17,2.52) Stoke-on-Trent 1.7

1.44 (1.17,1.77) Dudley 6.3

1.39 (1.04,1.86) Atlanta 3.3

1.38 (0.97,1.97) Fort Worth 2.2

1.37 (1.06,1.77) Milwaukee 4.2

1.35 (1.23,1.48) Bristol 36.3

1.24 (0.95,1.64) Kansas City 4.0

1.14 (0.62,2.08) Dover 0.9

1.02 (0.75,1.40) Harrisburg 3.4

0.98 (0.88,1.11) New Orleans 24.4

0.94 (0.79,1.12) Portland 10.9

0.75 (0.48,1.19) Indianapolis 1.8

1.22 (1.15,1.29) Overall (95% CI)

HORS251HORS251 StudyStudy STATA metanSTATA metanOdds RatioOdds Ratio(95% CI)(95% CI) % Weight% Weight

Forest Plot from Meta-Forest Plot from Meta-analysisanalysis

1717

The Confidence IntervalsThe Confidence IntervalsThe confidence intervals of individual The confidence intervals of individual

studies, which go to give the combined studies, which go to give the combined result, are calculated as though the events result, are calculated as though the events come from independent random samples. come from independent random samples.

So that:So that:

Var (ln(CPR))Var (ln(CPR)) = = 11 + + 11 + + 11 + + 1 1 a b c da b c d

But this can not be!But this can not be!

1818

Correlations Within Correlations Within Crime is committed by criminals!Crime is committed by criminals! Look at the data!Look at the data! Most studies are just Most studies are just

counts of crime in the 2 areas, before counts of crime in the 2 areas, before and after. But one that is not is Bristol.and after. But one that is not is Bristol.

If the independent random samples If the independent random samples assumption were correct, the variance assumption were correct, the variance of the count would be expected to be of the count would be expected to be approximately equal to the mean approximately equal to the mean count. But it is not. It is an order of count. But it is not. It is an order of magnitude higher.magnitude higher.

1919

The Bristol Study (Shaftoe The Bristol Study (Shaftoe 1994)1994)

1990.01989.01988.01987.01986.0

1500

1400

1300

1200

1100

1000

900

800

700

600

Year

No.

Cri

mes

1311

1464

1160

1237

14241374

142915171414

840840846

727

648

833838

685630

Number of Crimes Reported(in half-year periods)

Brighter Lighting

Control

New lighting introduced from July 87 to March 89 as marked onthe time axis.

Shaftoe said ‘no discernable lighting benefit’ but HORS251 says Shaftoe said ‘no discernable lighting benefit’ but HORS251 says z=6.6 ! Note: had the data for the year immediately prior to the z=6.6 ! Note: had the data for the year immediately prior to the introduction of the relighting, i.e. periods 2 and 3, been used introduction of the relighting, i.e. periods 2 and 3, been used rather than unnaturally using periods 1 and 2 which leaves a rather than unnaturally using periods 1 and 2 which leaves a gap of ½ year, the effect found would have been half of that gap of ½ year, the effect found would have been half of that claimed. (Shows large variability.)claimed. (Shows large variability.)

2020

Overdispersion (1)Overdispersion (1)

The Bristol (and Birmingham) studies show The Bristol (and Birmingham) studies show large variation over time when the light level large variation over time when the light level is constant. The variance is many times the is constant. The variance is many times the mean; mean; ’’OverdispersionOverdispersion’. ’.

(The(The large ≈ 60 heterogeneity statistic, Q, given by the meta-analysis of the 13 studies, also suggests this. A large Q shows that there is an inconsistency between, within study variation and between study variation)

2121

Overdispersion (2)Overdispersion (2)

The problem for HORS251 is that the The problem for HORS251 is that the confidence intervals around the effect size confidence intervals around the effect size must therefore be substantially increased. must therefore be substantially increased.

Also because the underlying overdispersion Also because the underlying overdispersion is not properly known for individual is not properly known for individual studies, we can not say what weights we studies, we can not say what weights we must use, as these will must use, as these will notnot be the same as be the same as used in the original incorrect HORS251 used in the original incorrect HORS251 meta-analysis.meta-analysis.

2222

Examine Overdispersion in Comparison Examine Overdispersion in Comparison AreasAreas

Study NameStudy Name DDobsobs = = ss22//xx AtlantaAtlanta 58.359458.3594

MilwaukeeMilwaukee 13.754713.7547

PortlandPortland 7.70177.7017

Kansas CityKansas City 9.74499.7449

HarrisburgHarrisburg 1.57481.5748

New OrleansNew Orleans 46.681046.6810

Fort WorthFort Worth 0.29340.2934

IndianapolisIndianapolis 0.04000.0400

DoverDover 4.76474.7647

BristolBristol 44.711644.7116

BirminghamBirmingham 1.53061.5306

DudleyDudley 4.44204.4420

Stoke-on-TrentStoke-on-Trent 0.00830.0083

DDobsobs are extremely are extremely variable and right variable and right skewed. The skewed. The arithmeticarithmetic mean is mean is 15 for these 15 for these comparison areas. comparison areas. (Larger still if the mean (Larger still if the mean includes weighting by includes weighting by number of crimes.)number of crimes.)

Calculated from the Calculated from the before and after before and after counts in the counts in the comparison areascomparison areas

2323

What observed overdispersions What observed overdispersions are expected if repeated samples are expected if repeated samples

taken?taken?It can be shown:It can be shown: for a Normal(for a Normal(μμ, , σσ22) such that ) such that σσ22 = = κμκμ, (i.e. the , (i.e. the

variance = the overdispersion factor (k) variance = the overdispersion factor (k) the the mean (mean (μμ) )) )

that the sampling distribution of the observed that the sampling distribution of the observed overdispersion koverdispersion kobsobs (= s (= s22//xx ) is (provided k<< ) is (provided k<<μμ) ) approximatelyapproximately

Chi-squared 1df, scaled by k.Chi-squared 1df, scaled by k.This can be written equivalently asThis can be written equivalently asGamma (scale =2k, shape=1/2) Gamma (scale =2k, shape=1/2) This is a right skewed distribution with arithmetic This is a right skewed distribution with arithmetic

mean = k. mean = k.

[If the before-after correlation = ρ, then k is replaced by k(1-ρ). [If the before-after correlation = ρ, then k is replaced by k(1-ρ). This is the effective overdispersion, relevant for a before-after This is the effective overdispersion, relevant for a before-after study, i.e. the quantity which is of interest. Indeed sstudy, i.e. the quantity which is of interest. Indeed s2 2

estimates estimates σσ22(1-ρ).](1-ρ).]

2424

The CDF of the HORS251 The CDF of the HORS251 overdispersionsoverdispersions

Sample overdispersion for each study's comparison area

Perc

ent

100806040200

100

80

60

40

20

0

Shape 0.5Scale 29.78N 13

HORS251 Empirical CDF and Gamma with shape=1/ 2 and scale=fitted

k=15k=15

2525

Other crime data for Other crime data for confirmationconfirmation

I used a data set of burglary count I used a data set of burglary count data from 124 anonymised small data from 124 anonymised small areas. The data was from a project areas. The data was from a project described in Tilley et al. 1999. described in Tilley et al. 1999.

Has counts of similar size to those in Has counts of similar size to those in HORS251.HORS251.

2626

Overdispersion

Perc

ent

120100806040200

100

80

60

40

20

0

Shape Scale N0.5 20.24 1180.5 21.26 116

VariableODB23thouODB12thou

CDF Overdispersion of burglary data (both available intervals: 1-2, 2-3)

Interval 1 to 2

Interval 2 to 3

Burglary data from 124 areas

k=10 both

2727

What to conclude about What to conclude about overdispersionoverdispersion

Both the HORS251 and the burglary data show Both the HORS251 and the burglary data show great overdispersion of 10 or more.great overdispersion of 10 or more.

In the case of HORS251 and the CI is (7.9, 38.7).In the case of HORS251 and the CI is (7.9, 38.7).(Similar results are obtained with using the quasi-Poisson (Similar results are obtained with using the quasi-Poisson

Generalised Linear Model in R)Generalised Linear Model in R)

A big problem in HORS251 is essentially confusion about A big problem in HORS251 is essentially confusion about ‘Unit ‘Unit of Analysis’of Analysis’. (It is Area, not Crime-event). (It is Area, not Crime-event)

2828

The Dudley Study (1)The Dudley Study (1) Used a household crime survey. Painter and Used a household crime survey. Painter and

Farrington 1997. in Situational Crime Prevention: Farrington 1997. in Situational Crime Prevention: Successful case studies.Successful case studies. (Also there’s the Stoke on (Also there’s the Stoke on Trent Study) Trent Study)

Question “Did you experience crime in the past year Question “Did you experience crime in the past year if so how many?”if so how many?”

Two areas supposedly matched; one has lighting Two areas supposedly matched; one has lighting increased, the other stays the same. The household increased, the other stays the same. The household survey was carried out before and after the new survey was carried out before and after the new lighting introduced. Households were planned to be lighting introduced. Households were planned to be linked before and after, but did linked before and after, but did notnot happen (nor in happen (nor in Stoke)! Stoke)!

The reports of the studies make great claims of The reports of the studies make great claims of success of the effectiveness of lighting. Claimed by success of the effectiveness of lighting. Claimed by the Institution of Lighting Engineers as 'Proof' that the Institution of Lighting Engineers as 'Proof' that lighting is effective against crime. lighting is effective against crime.

2929

The Dudley Study (2):The Dudley Study (2):Some Problems uncoveredSome Problems uncovered

Eventually I was given some of the data. Eventually I was given some of the data. (With a limited number of background (With a limited number of background variables however.)variables however.)

Markedly different crime rates at the start Markedly different crime rates at the start between the 2 areas.between the 2 areas.

One tailed testing used to claim a One tailed testing used to claim a statistically significant effect.statistically significant effect.

Overdispersion: Variance of the number of Overdispersion: Variance of the number of crimes per household is much larger than crimes per household is much larger than the mean, therefore Poisson methods are the mean, therefore Poisson methods are inappropriate. inappropriate.

3030

The Poisson Model is Inappropriate.The Poisson Model is Inappropriate.(See below, the distribution of crime counts for households.)(See below, the distribution of crime counts for households.)

0.2

.4.6

Pre

dic

ted P

r(y=

k)

from

pois

son/O

bserv

ed P

r(y=

k)

from

pois

son

0 2 4 6 8 10Count

Predicted Pr(y=k) from poisson Observed Pr(y=k) from poisson

Fitted and observed for Poisson Model

3131

The Dudley Study (3):The Dudley Study (3):Some problems uncovered cont.Some problems uncovered cont.

Differential loss to follow up.Differential loss to follow up. Old people are much less prone to experience crime Old people are much less prone to experience crime

and their number is much reduced due to loss to and their number is much reduced due to loss to follow-up in the comparison area. So the relative follow-up in the comparison area. So the relative composition changes during the experiment. composition changes during the experiment.

Results are very sensitive to the loss or addition of Results are very sensitive to the loss or addition of just one personjust one person

But importantly there is But importantly there is correlation between correlation between householdshouseholds, giving extra overdispersion , giving extra overdispersion (variability).(variability).

Essentially it’s a Essentially it’s a non-randomised two-cluster non-randomised two-cluster trialtrial..

3232

Spatial Correlation (1)Spatial Correlation (1)

An expression can be derived for the variance of An expression can be derived for the variance of

ln(CPR) for a household survey, before and ln(CPR) for a household survey, before and after, intervention-comparison study, i.e. of after, intervention-comparison study, i.e. of the Dudley type. This includes, in addition to the Dudley type. This includes, in addition to the variability between households, the variability between households, bothboth: :

(1)(1) correlations within households between times. correlations within households between times.

(2)(2) correlations correlations betweenbetween households at any one households at any one time. time.

3333

Spatial Correlation (2)Spatial Correlation (2)

What you get basically is the expression you What you get basically is the expression you would get if you ignored the correlation would get if you ignored the correlation between households at one time, i.e. between households at one time, i.e. ignored the spatial correlation, multiplied ignored the spatial correlation, multiplied by the ‘Design Effect’, Deff. (Just as in by the ‘Design Effect’, Deff. (Just as in clustered surveys / trials)clustered surveys / trials)

Deff=(1+(n-1) ρDeff=(1+(n-1) ρss))

ρρss = the spatial correlation = the spatial correlation

n = the number in a cluster, i.e. arean = the number in a cluster, i.e. area

3434

Spatial Correlation (3)Spatial Correlation (3) The spatial correlation was not taken into The spatial correlation was not taken into

account in the Dudley and Stoke analyses thus account in the Dudley and Stoke analyses thus ignoring the fact that neighbours ‘share risk’.ignoring the fact that neighbours ‘share risk’.

An expression for the variance of the logarithm An expression for the variance of the logarithm of the Cross Product Ratio CPR is:of the Cross Product Ratio CPR is:

1 1 1 1 1 1ln 2 tVar CPR λDeff r

a b c d ab cd

3636

The response given to my The response given to my pointing out that pointing out that

overdispersion exists (1)overdispersion exists (1)The expression for the household survey type The expression for the household survey type

study, that I give above, study, that I give above, but without Deffbut without Deff was used on the original Dudley Poisson was used on the original Dudley Poisson result to give a variance adjustment of only result to give a variance adjustment of only about 3 about 3 (i.e. just (i.e. just λ estimate)λ estimate). This . This overdispersion adjustment was then applied overdispersion adjustment was then applied in the meta-analysis for in the meta-analysis for allall 13 studies. 13 studies.

See Addendum to HORS251 (added in Sept. See Addendum to HORS251 (added in Sept. 2003), with which I most profoundly disagree, 2003), with which I most profoundly disagree, ….even though my name is mentioned!….even though my name is mentioned!

3737

The response given to my The response given to my pointing out that pointing out that

overdispersion exists (2)overdispersion exists (2)Additionally, Farrington and Welsh (2004), Additionally, Farrington and Welsh (2004),

following my own short article in the BJC, following my own short article in the BJC, cite the cite the geometricgeometric mean of the s mean of the s22//xx of of the 13 studies of HORS251 to justify a the 13 studies of HORS251 to justify a small value. small value.

However, as I have indicated here, it is the However, as I have indicated here, it is the arithmeticarithmetic mean which is appropriate, mean which is appropriate, showing that the overdispersion is much showing that the overdispersion is much bigger, with a value of something like 15.bigger, with a value of something like 15.

3838

The response given to my The response given to my pointing out that pointing out that

overdispersion exists (3)overdispersion exists (3) Farrington and Welsh justify their Farrington and Welsh justify their

overdispersion estimate because if one overdispersion estimate because if one divides the original heterogeneity statistic, divides the original heterogeneity statistic, Q=60, by their favoured estimate an Q=60, by their favoured estimate an ‘acceptable’ heterogeneity statistic results! ‘acceptable’ heterogeneity statistic results! Note it is usual to use Q to uncover anomalies Note it is usual to use Q to uncover anomalies in the data rather than remove them! The in the data rather than remove them! The larger value of overdispersion, 15, would larger value of overdispersion, 15, would indicate excessive homogeneity Q indicate excessive homogeneity Q revisedrevised = = 60/15 as might result from publication bias. 60/15 as might result from publication bias. (There is no register for study protocols, which (There is no register for study protocols, which would guard against publication bias).would guard against publication bias).

3939

Lack of Equivalence between Lack of Equivalence between AreasAreas

Invariably it is the most crime-ridden Invariably it is the most crime-ridden area that gets the lighting, whereas the area that gets the lighting, whereas the relatively crime-free ‘relatively crime-free ‘control control ’ is not re-’ is not re-lit. So there is lack of equivalence at lit. So there is lack of equivalence at the start. One effect of this is to allow the start. One effect of this is to allow ‘‘regression towards the meanregression towards the mean’’ to to operate. (see later) operate. (see later)

The name The name ‘Control‘Control Area’ is a misnomer. Area’ is a misnomer. ‘‘ComparisonComparison Area’ is a better name. Area’ is a better name.

4040X The before measurement

Y T

he a

fter

mea

sure

men

t

Cloud ofDataPoints

Line of Equality

0 10050

0

50

100

Line of mean of Y for a given X

4141

The response given to the lack of The response given to the lack of equivalence between the 2 areas. equivalence between the 2 areas. (RTM)(RTM) ‘‘Regression towards the mean’ (RTM) has Regression towards the mean’ (RTM) has

not been acknowledged to be a problem, not been acknowledged to be a problem, after I pointed it out.after I pointed it out.

The burglary data shows RTM nicely. The burglary data shows RTM nicely. Splitting the data for the 124 areas into 2, Splitting the data for the 124 areas into 2, above or below the mean burglary rate in above or below the mean burglary rate in the first year, exhibits a tendency in the the first year, exhibits a tendency in the following year for the high burglary rate following year for the high burglary rate group to show a fall and the low burglary group to show a fall and the low burglary rate group to show a rise.rate group to show a rise.

4242

RTM Example from the Burglary RTM Example from the Burglary Data Data

Rate1 Above Its Mean or Not (1=Above the Mean)Incr

ease

in B

urg

lary

Count

goin

g fro

m P

eriod 1

to 2

*10

200

100

0

-100

-200

-300

-400

-71.0455

5.875

Increase in Burglary Count period 1 to 2 Depending on whether the rate was above the mean at period 1, or not

(Seen in period 2 to 3 also. And using rate, rather than (Seen in period 2 to 3 also. And using rate, rather than count)count)

4343

Regression Towards the Mean RTM: Regression Towards the Mean RTM: Seeing effects which aren’t thereSeeing effects which aren’t there

A statistical novice might interpret the fact A statistical novice might interpret the fact that the high burglary rate group shows that the high burglary rate group shows a reduction in burglaries (-71), as a reduction in burglaries (-71), as opposed to the low rate group (+6), as opposed to the low rate group (+6), as evidence of ‘something important going evidence of ‘something important going on’ rather than just what is expected on’ rather than just what is expected when you have correlated data. RTM when you have correlated data. RTM follows from correlation. (As Francis follows from correlation. (As Francis Galton discovered more than a century Galton discovered more than a century ago, in the 1880s.)ago, in the 1880s.)

4444

The response given to the lack of The response given to the lack of equivalence between the 2 areas. equivalence between the 2 areas.

(RTM)(RTM) Farrington and Welsh (2006) claim that RTM is a Farrington and Welsh (2006) claim that RTM is a

not problem because the effect in crimes counted not problem because the effect in crimes counted in 250 Police ‘Basic Command Units’ going from in 250 Police ‘Basic Command Units’ going from 2002/3 to 2003/4 showed only small effect. This is 2002/3 to 2003/4 showed only small effect. This is hardly surprising as the areas and hence the hardly surprising as the areas and hence the number of crimes counted are an order of number of crimes counted are an order of magnitude larger than in HORS251 so the year to magnitude larger than in HORS251 so the year to year correlation is high. Note Wrigley (1995) “This year correlation is high. Note Wrigley (1995) “This tendency for correlation coefficients to increase in tendency for correlation coefficients to increase in magnitude as the size of the areal unit involved magnitude as the size of the areal unit involved increases has been known since the work of increases has been known since the work of Gehlke and Biehl (1934)”.Gehlke and Biehl (1934)”.

4545

Bristol study revisitedBristol study revisitedA lighting benefit effect with p=0.011 is claimed in A lighting benefit effect with p=0.011 is claimed in

reply to me. But this depends on an arbitrary, reply to me. But this depends on an arbitrary, specific regression model that requires the specific regression model that requires the variance to be the same in both areas and to variance to be the same in both areas and to include a linear time trend, include a linear time trend, identicalidentical in both areas, in both areas, but which is not ‘statistically significant’.but which is not ‘statistically significant’.

On the other hand, a model which just uses the crime On the other hand, a model which just uses the crime count in the comparison area as predictor of crime count in the comparison area as predictor of crime in the re-lit area (in the spirit of HORS251) shows in the re-lit area (in the spirit of HORS251) shows no stat. sig. effect.no stat. sig. effect.

Does the data really look as if there is such a clear Does the data really look as if there is such a clear effect, i.e. one which would only occur 1 time in effect, i.e. one which would only occur 1 time in 100, when there is in fact no lighting benefit? 100, when there is in fact no lighting benefit?

4646

The Bristol data againThe Bristol data again

1990.01989.01988.01987.01986.0

1500

1400

1300

1200

1100

1000

900

800

700

600

Year

No.

Crim

es

1311

1464

1160

1237

14241374

142915171414

840840846

727

648

833838

685630

Number of Crimes Reported(in half-year periods)

Brighter Lighting

Control

New lighting introduced from July 87 to March 89 as marked onthe time axis.

It seems to me that this hardly presents clear evidence for lighting benefit! There’s a big problem of model uncertainty.

4747

Cost benefit analysesCost benefit analyses

Cost benefit analysis has been done based on very Cost benefit analysis has been done based on very few studies by lighting and crime researchers (and few studies by lighting and crime researchers (and gives a highly favourable result for lighting). gives a highly favourable result for lighting). However doing this only compounds the problem. As However doing this only compounds the problem. As an unknown, unproven benefit/harm is being an unknown, unproven benefit/harm is being compared with uncertain costs.compared with uncertain costs.

We need to get much better information to do such We need to get much better information to do such an exercise properly otherwise it tends to look an exercise properly otherwise it tends to look ‘scientific’ to the eye of a novice, when in fact it ‘scientific’ to the eye of a novice, when in fact it isn’t, because of flimsy data and method. isn’t, because of flimsy data and method.

4848

‘‘Researchvertising’Researchvertising’

Unsurprisingly HORS251 and the Unsurprisingly HORS251 and the Dudley Study are used by the Dudley Study are used by the lighting industry to promote its lighting industry to promote its wares. Also ‘responsibilities under wares. Also ‘responsibilities under the crime and disorder act’ are the crime and disorder act’ are invokedinvoked

4949

My take on lighting and My take on lighting and crimecrime

It may be that lighting reduces crime, or may be it It may be that lighting reduces crime, or may be it increases crime. We have to look at the evidence as increases crime. We have to look at the evidence as given. The conclusion, at present, is: We do not given. The conclusion, at present, is: We do not know....yet we ought to know!know....yet we ought to know!

Note, I know of no scientific trials of exterior Note, I know of no scientific trials of exterior 'Security' lighting. So no one knows if this works. 'Security' lighting. So no one knows if this works.

We ought to take a ‘Popperian’ view and entertain We ought to take a ‘Popperian’ view and entertain the possibility of light being ineffective or worse, the possibility of light being ineffective or worse, against crime.against crime.

Of course we all need light at night, to see by. Of course we all need light at night, to see by. (Those concerned about light pollution are basically (Those concerned about light pollution are basically talking ‘lamp-shades’). However there is no sound talking ‘lamp-shades’). However there is no sound evidence we need light to protect us from crime, in evidence we need light to protect us from crime, in spite of claims.spite of claims.

5050

Car AlarmsCar Alarms

It seems that there is little evidence It seems that there is little evidence that car alarms prevent cars being that car alarms prevent cars being broken into etc. broken into etc.

But they do disturb people’s sleep!But they do disturb people’s sleep! Attempt in New York to get them Attempt in New York to get them

banned, (rely on passive methods of banned, (rely on passive methods of risk reduction instead.)risk reduction instead.)

5151

Wider problems of inappropriate Wider problems of inappropriate

methodsmethods The costs of crime and attempts at its reduction The costs of crime and attempts at its reduction

are large.are large. Similar problems probably exist for the Similar problems probably exist for the

evaluation of other area-based crime reduction evaluation of other area-based crime reduction interventions, too. (They certainly do for interventions, too. (They certainly do for HORS252 on CCTV where the same methods as HORS252 on CCTV where the same methods as HORS251 are used on 18 studies, Q=270. HORS251 are used on 18 studies, Q=270. However no effect of CCTV is claimed). However no effect of CCTV is claimed). Problems seem to be encouraged by the Problems seem to be encouraged by the ‘Maryland Scientific Methods Scale’ which ‘Maryland Scientific Methods Scale’ which seems to suggest that weaker designs, than seems to suggest that weaker designs, than RCTs, might suffice. RCTs, might suffice.

We do need to have proper evidence to decide We do need to have proper evidence to decide ‘what works’ in crime and in all spheres . ‘what works’ in crime and in all spheres .

5252

Crime Prevention.….an Crime Prevention.….an ‘Art’ ?‘Art’ ?

Or is it more like 17Or is it more like 17thth Century Century medicine?!medicine?!

5353

Some considerations for Some considerations for evaluation of anythingevaluation of anything

Interventions are expensive, as are the Interventions are expensive, as are the consequences, so their effects need to be consequences, so their effects need to be researched to a very high standard…e.g. the researched to a very high standard…e.g. the necessity of using randomisation. necessity of using randomisation.

Also the target population could be depleted Also the target population could be depleted through poorly conducted studies.through poorly conducted studies.

Evaluation needs to be done right.Evaluation needs to be done right. Caution is needed with systematic Caution is needed with systematic

reviewing/meta-analysis as it involves moving reviewing/meta-analysis as it involves moving away from the primary sources.away from the primary sources.

It is possible to do ‘roll out’ of a programme in a It is possible to do ‘roll out’ of a programme in a way that is amenable to proper scientific way that is amenable to proper scientific evaluation.evaluation.

5454

Much can be borrowed from Much can be borrowed from

methods in health researchmethods in health research For example in area-based crime reduction For example in area-based crime reduction The Methods of Cluster Randomised Trials a appropriate.The Methods of Cluster Randomised Trials a appropriate.See e.g. See e.g. Ukoumunne et al (1999) Ukoumunne et al (1999) andandCampbell et al (2004)….CONSORT extension for Cluster Campbell et al (2004)….CONSORT extension for Cluster Randomised Trials.Randomised Trials.

Also, post-implementation surveillance is highly desirable for Also, post-implementation surveillance is highly desirable for any programme.any programme.

(Note; it is problematic enough to determine What Works in (Note; it is problematic enough to determine What Works in healthcare where the ‘unit’ is ‘person’, through e.g. healthcare where the ‘unit’ is ‘person’, through e.g. dissemination bias.)dissemination bias.)

5555

High standards are needed for High standards are needed for

evidenceevidence NeedNeed Adequate funding to provide quality research, as the Adequate funding to provide quality research, as the

costs of rolling out programmes on a national scale costs of rolling out programmes on a national scale are huge. Need to be aware of the costs of are huge. Need to be aware of the costs of implementing ineffective or counterproductive implementing ineffective or counterproductive programmes.programmes.

Effective random allocation for experiments.Effective random allocation for experiments. Blinding where possible, e.g. of assessors.Blinding where possible, e.g. of assessors. Pre-publication of protocols.Pre-publication of protocols. Open-access in order to check any work. Open-access in order to check any work.

(As, for example, reports/papers may confuse (As, for example, reports/papers may confuse standard deviations and standard errors or, as above, standard deviations and standard errors or, as above, not recognise correlated data.) not recognise correlated data.)

Let’s ultimately have the raw data!Let’s ultimately have the raw data!

5656

ConclusionConclusion There is no good scientific evidence to show that There is no good scientific evidence to show that

night time lighting reduces crime. (In spite of the night time lighting reduces crime. (In spite of the claims).claims).

The effect of some other crime reduction methods is The effect of some other crime reduction methods is also questionable, because of inherent weakness of also questionable, because of inherent weakness of research methods used, and needs examination. research methods used, and needs examination. (Look at the variation in comparison areas.). (Look at the variation in comparison areas.). Variation in crime is poorly understood.Variation in crime is poorly understood.

High quality studies, sound in design through to High quality studies, sound in design through to conclusion, are necessary in important and costly conclusion, are necessary in important and costly matters matters

Statisticians have a vital role in finding ‘what works’.Statisticians have a vital role in finding ‘what works’. Eminence in any subject does not guarantee the Eminence in any subject does not guarantee the

correctness of statistical pronouncements.correctness of statistical pronouncements.

5757

Final points Final points Do be investigative. Don't just mind your own Do be investigative. Don't just mind your own

business.business.--------------------------------------------------------------------------------------------------------------------Stephen Senn quote “Trust nobody, check everything” Stephen Senn quote “Trust nobody, check everything”

from ‘Fear and Loathing in Pharmaceutical Statistics’, from ‘Fear and Loathing in Pharmaceutical Statistics’, RSS2002RSS2002

HG Wells quote “Statistical thinking will one day be as HG Wells quote “Statistical thinking will one day be as necessary for efficient citizenship as the ability to necessary for efficient citizenship as the ability to read and write”. read and write”.

The RSS magazine Significance which aims to The RSS magazine Significance which aims to encourage ‘statistical thinking’ June 2005 p62 has an encourage ‘statistical thinking’ June 2005 p62 has an article by me ‘Evaluating area-wide crime reduction article by me ‘Evaluating area-wide crime reduction measures’.measures’.

5858

ReferencesReferencesCampbell MK, Elbourne DR, Altman DG for the CONSORT

Group (2004) CONSORT statement: extension to cluster randomised trials. BMJ 328 702-708. http://bmj.bmjjournals.com/cgi/reprint/328/7441/702

Farrington D.P. and Welsh B.C. (2002) The Effects of Improved Street Lighting on Crime: A Systematic Review, Home Office Research Study 251, http://www.homeoffice.gov.uk/rds/pdfs2/hors251.pdf

Farrington D.P. and Welsh B.C. (2004) Measuring the Effects of Improved Street Lighting on Crime: A reply to Dr. Marchant The British Journal of Criminology 44 448-467 http://bjc.oupjournals.org/cgi/content/abstract/44/3/448

Farrington D.P. and Welsh B.C. (2006) How Important is Regression to the Mean in Area-Based Crime Prevention Research?, Crime Prevention and Community Safety 8

Marchant P.R. (2004) A Demonstration that the Claim that Brighter Lighting Reduces Crime is Unfounded The British Journal of Criminology 44 441-447 http://bjc.oupjournals.org/cgi/content/abstract/44/3/441

5959

References continuedReferences continuedMarchant P.R. (2005) What Works? A Critical Note on the Evaluation of Crime

Reduction Initiatives,Crime Prevention and Community Safety 7 7-13 www.extenza-eps.com/extenza/loadHTML?objectIDValue=63645&type=abstract

Painter, K. and Farrington, D. P. (1997) The Crime Reducing Effect of Improved Street Lighting: The Dudley Project, in R.V. Clarke ed., Situational Crime Prevention: Successful case studies 209-226 Harrow and Heston, Guilderland NY.

Shaftoe, H (1994) Easton/Ashley, Bristol: Lighting Improvements, in S. Osborn (ed.) Housing Safe Communities: An Evaluation of Recent Initiatives 72-77, Safe Neighbourhoods Unit, London

Tilley N., Pease K., Hough M. and Brown R. (1999) Burglary Prevention: Early Lessons from the Crime Reduction Programme, Crime Reduction Research series Paper1 London Home Office

Ukoumunne, O. C., Gulliford, M. C., Chinn, S., Sterne, J. A. C., Burney, P. G. J., and Donner, A. (1999). Evaluation of Health Interventions at Area and Organisation level, British Medical Journal, 319 376-379 http://bmj.bmjjournals.com/cgi/content/full/319/7206/376

Wrigley N., Revisiting the Modifiable Areal Unit Problem and Ecological Fallacy Wrigley N., Revisiting the Modifiable Areal Unit Problem and Ecological Fallacy pp49-71 in Gould PR, Hoare AG and Cliff AD Eds Diffusing Geography: pp49-71 in Gould PR, Hoare AG and Cliff AD Eds Diffusing Geography: Essays for Peter Haggett Essays for Peter Haggett

6060

Further material written in this area Further material written in this area

I wrote ‘Failing to measure any effect of increased I wrote ‘Failing to measure any effect of increased lighting on crime: A reply to Profs. Farrington and lighting on crime: A reply to Profs. Farrington and Welsh’ It was sent to the Home Office in Dec. 2004Welsh’ It was sent to the Home Office in Dec. 2004

www.imresearch.org/praxiscentre/Papers/RevReplyToFW1www.imresearch.org/praxiscentre/Papers/RevReplyToFW1B.pdfB.pdf

Dr. Barry Clark of the Astronomical Society of Victoria, Dr. Barry Clark of the Astronomical Society of Victoria, Australia has written much on the oft repeated but Australia has written much on the oft repeated but seemingly dubious claim that lighting reduces crime.seemingly dubious claim that lighting reduces crime.

http://www.asv.org.au/index.php?http://www.asv.org.au/index.php?option=com_content&task=view&id=33&Itemid=76option=com_content&task=view&id=33&Itemid=76

6161

Appendix: Other Useful Resources to Appendix: Other Useful Resources to assist in evidence based workassist in evidence based work

1.1. RCTs designing and writing upRCTs designing and writing upThe CONSORT statement (CONSORT = The CONSORT statement (CONSORT = Consolidated Reporting Of Randomised Trials) is Consolidated Reporting Of Randomised Trials) is very useful for reporting results and also thinking very useful for reporting results and also thinking about design. about design.

www.consort-statement.orgwww.consort-statement.org 2.2. The The QUORUMQUORUM statement: Quality Reporting of statement: Quality Reporting of

Meta-analyses.Meta-analyses.3.3. The The American Association for Public Opinion American Association for Public Opinion

ResearchResearch www.aapor.orgwww.aapor.org has useful information has useful information about Standards and Best Practices in survey about Standards and Best Practices in survey methodsmethods