Institute for Policy Research Evidence for Agile Policy Makers The Contribution of Transformative Realism Graham Room 14 February 2014 Victoria University.

Institute for Policy Research

Evidence for Agile Policy Makers

The Contribution of Transformative Realism

Graham Room

14 February 2014

Victoria University Wellington


Evidence for Policy

Assess impacts of different interventions by systematic review of evidence

‘What Works Centres’:•National Institute for Health and Clinical Excellence

.

•Education Endowment Foundation

•Centre for Local Economic Growth


X

Y




The Hierarchy of Evidence and Rigour

1a Evidence from a systematic review of randomized controlled trials

1b Evidence from at least one randomized controlled trial

II a Evidence from at least one study without randomization

II b Evidence from at least one other type of quasi-experimental study

III Evidence from non-experimental descriptive studies (e.g., case control studies, correlation studies)

IV Expert consensus


Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective

‘Impact’ (medical) versus ‘engagement’ (social policy)

Interventions unfold in institutionally complex environments

Rarely is it the ‘same’ intervention that is delivered across all contexts

Pawson’s Realist Critique


Pawson’s contingencies of intervention


Darwin’s Tree of Life


‘Transformative realism’

A

A1 A2

A3

B

B1 B2

B3


EBPM: Evidence of impact of a given intervention: universal and unconditional

Pawson: Impact of a given intervention under a range of contingencies

Transformative realism: Dynamic synergies among interventions

Three Paradigms of Evidence-Based Policy-Making


The Choice of ParadigmsAssess the scope

and strength of any dynamic synergies between the policy intervention and the policy ecosystem -

using evidence from other settings?

Strong

Apply paradigm 3

Weak

Apply paradigm 2

They are active and

agile and this is important

for the effectiveness

of the intervention

How active and agile are the subjects of the intervention?

How important is it for the effectiveness of the

intervention that they should apply and adapt it to their

circumstances?

Apply paradigm 1

Inactive


Paradigm 1: Well-evidenced generalisations about impact (universal truths?)

Paradigm 2: Well-based theory as to out-turn under specific local contingencies

Paradigm 3: Need real-time watching and learning-by-doing

Types of Knowledge


• How to tune social developments in real time?• Where to probe and where to avoid?• What signs of tipping points to avoid or approach? • What data in real time? Big Data?• Ian Stewart (1997), Does God Play Dice?)

• Tuning a complex system (engineering and medicine)• Can this approach be applied to social systems?

Real-Time Watching and Learning-by-Doing


• How to ‘read’ the ‘sea conditions’?• Networks of ‘navigators’ – tracking how policy ecosystem is developing and

deciding what ‘tuning’ is needed • Elizabeth Eppel, Amanda Wolf, David Taylor: Complex Policy Implementation: The

Role of Experimentation and Learning• ‘Ordinary’ situations - rule of thumb • Anomalous (extraordinary) situations – examine carefully and devise a specific

strategy • What thresholds to take as triggers for concern?

Real-Time Watching and Learning-by-Doing


• Danger of de-politicising policy• What works for whom?• Corporate and other interests try to shape our

policy futures: is there any shared national interest?

• Good research on impact is necessary but not sufficient for good policy

Concluding Reflections


Behavioural Insights Team (2012), Test, Adapt, Learn, UK Cabinet Office

Eppel, E., Turner, D., & Wolf A. (2011). ‘Complex policy implementation: The role of experimentation and learning’. In B. Ryan & D. Gill (Eds.), Future State: Directions for public management in New Zealand (pp. 182-212).

Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective

Room, G. (2011), Complexity, Institutions and Public Policy: Agile Decision-Making in a Turbulent World

Room, G. (2013), 'Evidence for Agile Policy Makers :The Contribution of Transformative Realism', Evidence and Policy, 9(2): 225-244.

Stewart, I (1997), Does God Play Dice?

References

Institute for Policy Research Evidence for Agile Policy Makers The Contribution of Transformative Realism Graham Room 14 February 2014 Victoria University.

Documents

policy research evidence

policy research paradigm

policy research ebpm

policy ecosystem

policy research x y

evidencebased policy

national institute

complex policy implementation