1 UK POLITICAL STUDIES ASSOCIATION (PSA) ANNUAL CONFERENCE MANCHESTER APRIL 14-16, 2014 TRAINING TO DELIVER: HOW AND WHAT DO REGULATORS LEARN ABOUT ORGANIZATIONAL REFORM? Claire A. Dunlop(*), Jonathan Kamkhaji(**) and Claudio M. Radaelli(***), University of Exeter, UK 1 [email protected]; [email protected]; [email protected]UNDER REVIEW – PLEASE DO NOT CITE WITHOUT PERMISSION ☺ Contemporary regulators face an increasing array of initiatives designed to boost the effectiveness of policy delivery and cut administrative burdens. While a good deal of analytical attention is given to these governance tools, we know much less about how regulators themselves understand and learn about them. This paper outlines presents a quasi-experiment to assess the effects of training on local government inspectors’ understandings of the Primary Authority (PA) scheme. Established by the Department for Business Innovation and Skills’ (BIS) Better Regulation Delivery Organisation (BRDO) in 2009, PA partnerships are legally binding agreements that provide businesses operating across England with a single point of regulatory contact. These PA inspectors provide advice and reduce duplication of inspections and paperwork. The scheme is a complex one, and marks a significant departure from the existing inspection framework. This study explores the impact of training on inspectors’ understandings of the scheme itself and its underlying rationale concerning regulatory burdens. The findings suggest that, regardless of training, regulatory innovations like the PA scheme, are well understood among local authority inspectors. But, where training may make a difference it is in aspects of regulatory reform which are contentious or could be taken as counter-intuitive to professional norms. 1 (*) Senior Lecturer in Political Science, University of Exeter (UK) and corresponding author, (**) PhD Candidate in Political Science and Research Associate, University of Exeter (UK), (***) Professor of Political Science, and Director, Centre for European Governance, University of Exeter (UK)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
UK POLITICAL STUDIES ASSOCIATION (PSA) ANNUAL CONFERENCE
MANCHESTER APRIL 14-16, 2014
TRAINING TO DELIVER: HOW AND WHAT DO REGULATORS LEARN ABOUT
ORGANIZATIONAL REFORM?
Claire A. Dunlop(*), Jonathan Kamkhaji(**) and Claudio M. Radaelli(***), University of
UNDER REVIEW – PLEASE DO NOT CITE WITHOUT PERMISSION ☺
Contemporary regulators face an increasing array of initiatives designed to boost the
effectiveness of policy delivery and cut administrative burdens. While a good deal of
analytical attention is given to these governance tools, we know much less about how
regulators themselves understand and learn about them. This paper outlines presents a
quasi-experiment to assess the effects of training on local government inspectors’
understandings of the Primary Authority (PA) scheme. Established by the Department for
Business Innovation and Skills’ (BIS) Better Regulation Delivery Organisation (BRDO) in 2009,
PA partnerships are legally binding agreements that provide businesses operating across
England with a single point of regulatory contact. These PA inspectors provide advice and
reduce duplication of inspections and paperwork. The scheme is a complex one, and marks
a significant departure from the existing inspection framework. This study explores the
impact of training on inspectors’ understandings of the scheme itself and its underlying
rationale concerning regulatory burdens. The findings suggest that, regardless of training,
regulatory innovations like the PA scheme, are well understood among local authority
inspectors. But, where training may make a difference it is in aspects of regulatory reform
which are contentious or could be taken as counter-intuitive to professional norms.
1 (*) Senior Lecturer in Political Science, University of Exeter (UK) and corresponding author, (**) PhD Candidate in Political Science and
Research Associate, University of Exeter (UK), (***) Professor of Political Science, and Director, Centre for European Governance,
University of Exeter (UK)
2
INTRODUCTION
Regulatory reform is a constantly changing activity, where economic needs, policy
paradigms and fads combine to re-define purposes and approaches (Lodge, 2008). In the
UK, regulatory reform has mutated across the decades. Governments often prefer to keep
existing instruments but, at the same time, they layer new ones on top of existing
arrangements. By doing so, they modify the overall approach to reform. Behind this process
of layering, there is also a slow, and deeper, process of changing regulatory paradigms.
Notably in 2005, the UK government acknowledged that while instruments were still
important, there was no point in perfecting instruments without attempting to change the
underlying regulatory culture. This ‘cultural turn’ was put centre stage by the publication of
the landmark ‘Hampton Review’ on Reducing Administrative Burdens: Effective Inspection
and Enforcement (HMT, 2005). Hampton posits risk-based regulation as the key mechanism
for burden reduction, and exhorted policymakers to develop new institutions and
programmes to embed this idea.
Of course, in this cultural turn, we are not speaking of a phenomenon unique to the UK. But,
the UK was the European country that most clearly turned to the ambitious plan to change
regulatory culture, and to target a specific component of implementation processes, that is,
the activity of local authority inspectors and those they regulate. This switch to delivery,
culture and institutions provides the motivation for our project.
While a good deal of analytical attention is given to regulatory policy instruments, and
governmental interest in enforcement and inspections has been foregrounded in the
academic literature grounded in theories of responsive regulation (Grabosky and
Braithwaite, 1986) and regulatory discourse (Black, 2002), we know much less about how
street-level regulators – i.e. inspectors – learn about institutional innovations and their
permeability to attempts to shape their understanding and attitudes. In particular, little is
known about the role of training in helping frontline inspectors deliver ‘better regulation’.
To address this gap, this paper provides evidence regarding the effect of training on a hard
to reach experimental population – regulatory inspectors at the forefront of Hampton’s
burden reduction challenge. Specifically, it reports on a quasi-experiment designed to assess
the effects of training on local authority inspectors’ understandings of the Primary Authority
(PA) scheme. Established by the Department for Business, Innovation and Skills’ (BIS) Better
Regulation Delivery Office (BRDO)2 in 2009, the PA scheme represents the central response
of policymakers to the challenge laid down by Hampton. Its aim is stimulate risk-based
regulation by facilitating closer collaboration between regulators and regulated. PA
partnerships are statutory agreements between local authorities and the businesses they
2 BRDO is a non-departmental public body accountable to the Department for Business, Innovation and Skills (BIS) through the Better
Regulation Executive (BRE). Created on 1 April 2012, the Better Regulation Delivery Office (BRDO) took over the running of the PA scheme
from its predecessor the Local Better Regulation Office (LBRO) which was established in 2007. As the principle functions of this body have
not altered with this new name, and nothing relating to the specifics of the PA scheme are different, we opt for the simplicity here of using
the current acronym of BRDO throughout.
3
regulate. Crucially, they aim to provide businesses with a single point of regulatory contact.
The PA scheme is underpinned by the principle of burden reduction; whereby local
authority inspectors across England who are part of PAs provide advice and reduce
duplication of inspections and paperwork. The ultimate aim is to ensure consistency of
regulation and proportionate risk management.
The scheme is a complex one, and marks a significant departure from the existing inspection
framework. Using a quasi-experiment, we aim to capture the effects of training
interventions, designed and delivered by the BRDO, on inspectors’ knowledge of the PA
scheme and its underlying rationale concerning the redistribution of regulatory burdens and
risk. But, why adopt a quasi-experimental approach? This design offers the best hope to
avoid the endogenity problem whereby the impact of training becomes impossible to
disentangle from the impact of the wider reform process that has occurred since Hampton.
By surveying two groups of trained and untrained inspectors we can better address the
counterfactual – what understanding would regulators’ possess in the absence of training?
While laboratory experiments have historically been a ‘neglected … stepchild’ in public
management research (Bozeman, 1992: 290), quasi-experiments fair even worse and are
often considered the ‘ugly sister’ of the experimental method. Indeed Campbell, who knew
well their potential messiness, memorably dubbed them ‘queasy experiments’ (1988: 322).
This not without good reason; as the introduction to this special issue notes the absence of
randomization can pose serious threats to internal validity: can we be sure the intervention
has caused the effect we observe? Yet, as the special issue editors also observe ‘… if
randomization can never be sacrificed, many research questions need to remain
unanswered’. And so, for those involved in quasi-experiments the aim must be to mitigate
the problems created by the absence of randomization as far as possible while celebrating
the potential gains on external validity this design entails.
In our case, a quasi-experiment was a necessity. Our interest is in regulators’
understandings of a complex public sector initiative, and specifically in the separating out
the impact of training from the wider better regulation reforms that have been successfully
understood by and embedded in regulators’ everyday working lives over the past decade
(CIEH, 2011; NAO, 2008). This is a hard to reach group, yet it is the only appropriate
population for this question. For sure, the classic student population or representative
cross-section of citizens would be easier to reach and randomize, but the trade-offs on
validity are unpalatable in this case. The internal validity achieved through experimenting
with a random sample of students or ordinary citizens would be meaningless given that such
non-specialist populations have not been exposed to the Hampton agenda in the same
intensity as regulators, and would be unlikely to actually understand the PA scheme3.
Experimenting with a specific population also avoids the biases associated with particular
3 Indeed, it is only after intensive reading of the PA literature, interviews with the scheme’s designers and attendance at a conference and
the two of the training interventions that the authors can claim to understand the complexities of the scheme!
4
groups established by psychometric research (for example see Slovic, 2000 for a collection
of papers).
A further reason to find beauty in the quasi-experimental design is the opportunities it
opens up to work with policymakers directly. While relaxing the requirements of
randomization and control over the intervention associated with lab, field and survey
experiments reduces internal validity (see table 1 of this special issue’s introduction), the
greater flexibility of quasi designs enable researchers to access policy actors and address
research questions that are important to academic and useful to public managers.
The paper is structured as follows. Section one describes the PA scheme in more depth, and
discusses the link between training and learning in the public sector. Section two details our
hypotheses about the potential impact of training. Section three presents our quasi-
experimental design, the survey’s sample and administration. Section four describes the
experimental conditions and the two sets of outcome variables we measured – vignettes
used to ascertain inspectors’ comprehension of the PA scheme, and positional questions
used to gauge their views of regulatory burden reduction and risk management. Section five
presents and discusses the findings about these two sets of outcome variables. We conclude
by drawing out our findings’ implications for those who design training for regulators and
explore the possible future directions for research.
1. EVALUATING TRAINING IN THE PUBLIC SECTOR
Employee training is considered a central component for high performance human resource
practices (Blume et al, 2010; Combs et al, 2006; Pfeffer 1998; Kelman, 2005), and
competence at continuous learning a core employee attribute (Maurer and Weiss, 2010).
But, what is the impact of such training? Empirically, we know that when compared with no-
training or pre-training situations, training has an overall positive effect on job-related
behaviour as employees learn (see Arthur et al, 2003 meta-analysis of 1152 effect sizes).
But, the majority of this evidence concerns training in private sector organisations. What of
the public sector? As the largest service provider in the world, the public sector is one of the
main investors in such capacity building. Public sector employees – particularly street-level
bureaucrats – face an on-going challenge of keeping up-to-date with the new policy
initiatives they are required to implement. But, there is an added challenge; they must also
understand the deeper values and ethos’ that underpin those initiatives – what van Wart
calls ‘the art of value management for practitioners ….’ (1998: 319). Training is the main way
that public practitioners are supported to understand the nuts-and-bolts of programme
innovation, and engage in discussion about the deeper values that drive new policy delivery
systems. Yet, we know relatively little about the impact of all this training. The research in
this paper offers an evaluation of the extent to which training helps public officials learn.
5
Specifically, we are interested in the impact of training on public officials’ understanding of
‘better regulation’. To explore this, we conduct a quasi-experiment with local government
inspectors in England to examine how training affects both their general attitudes toward
burden reduction in the delivery of trading standards, environmental health and licensing
enforcement services, and their specific understanding of a policy innovation designed to
reduce regulatory burdens on businesses – the Primary Authority (PA) scheme.
Primary Authority enables local authorities to form partnerships with the businesses they
regulate. Established in the UK4 by the 2008 Regulatory Enforcement and Sanctions Act
(RESA) and launched in April 2009, the PA scheme is overseen by the Department for
Business, Innovation and Skills’ (BIS) Better Regulation Delivery Office (BRDO). The BRDO
administers the scheme offering a web-based IT system to record partnerships that have
been formed and advice given; operates a dispute resolution procedure and, importantly,
provides training courses for local authorities and businesses that are interested in forming
or have already formed a PA partnership.
What is a PA partnership, and why form one? The PA scheme responds to concerns raised
by businesses that while local authorities should apply environmental health, licensing and
trading standards in a similar way, businesses operating across multiple local authorities can
experience contradictory advice, duplicated worked and the absence of an effective way to
resolve disputes when councils disagree. With the advent of the PA scheme, businesses
operating across council boundaries5 can form a statutory partnership with a single local
authority to cover all environmental health and trading standards legislation, or specific
functions such as food safety, petroleum licensing and age-restricted sales of gambling6. The
inspectors of that local authority then become the ‘primary authority’ for that business. The
aim here is to provide businesses with reliable, accessible and consistent advice – reducing
business costs, inspiring confidence and driving-up local regulation standards for consumers,
the environment and workers (BRDO, 2009). It is estimated that £48 million per year will be
saved across UK business through PA (BRDO, 2010: 5).
The core mechanism that underpins these new arrangements is the provision of information
from a single source – where a single local authority provides its partner business with
advice to support regulatory compliance and standards that then hold across other local
authorities. Once legally nominated by the BRDO, partnerships are automatically recognised
by all local regulators. The primary goal is the elimination of information complexity and
inconsistency of interpretation that results when the same business gets different advice
4 The PA scheme operates differently for different parts of the UK, but the broad description given here holds for all four jurisdictions. The
specifics of the scheme in Scotland and Northern Ireland are defined in The Co-ordination of Regulatory Enforcement Order 2009
http://www.opsi.gov.uk/si/si2009/uksi_20090669_en_1 5 Following the Enterprise and Regulatory Reform Act 2013, on 1 October 2013 the PA scheme became accessible to businesses that do
not trade across local authority boundaries, but do share an approach to compliance with at least one other business that trades in a
different local authority. These shared approaches will usually be co-ordinated by a business' trade association or its franchisor. These are
known as co-ordinated partnerships. 6 For a comprehensive list of areas covered by PA see http://www.bis.gov.uk/brdo/primary-authority/about-pa/changes-to-
pa#sthash.iArMvLCl.dpuf
6
from different inspectors in different authorities. Allied to that, those authorities that
become primary authorities also become a resource for other so-called ‘enforcing
authorities’ ‘… by providing valuable intelligence on businesses’ operations through advice
and the development of inspection plans’ (BRDO, 2009: 4). These optional ‘inspection
plans’7, agreed with the regulators of the primary local authority, exist to guide enforcing
authorities on what to take account of when carrying out inspections or dealing with
apparent non-compliance. The advice and agreements contained within inspection plans
must be respected by all enforcing regulators. If problems arise, the primary authority
coordinates enforcement action to ensure consistent treatment for the business. Where
disputes arise, the BRDO operates a resolution procedure8.
Though the PA scheme builds on earlier initiatives, it involves fundamental changes in the
relationships both between the regulated and regulators, and between regulators
themselves. At its heart is a shift in the nature of how responsibility is shared and risk
managed across regulatory parties. Before PA, the management of regulatory risk was
dispersed across as many local authorities as a business operated in. And so, multi-site
businesses with large geographical footprints were required to seek advice from a wide
range of inspectors. The result, at best, was the duplication of activities and, at worst,
multiplication. In the PA model, the aim is that consistency is increased and costs reduced
by locating responsibility in a partnership between primary authority and business, where
enforcing authorities must take advice from and work with their colleagues inside the
partnership. Proportionate risk management is the ultimate outcome (BRDO, 2010).
Though uptake at the start of the scheme was slow, interest has picked up considerably. In
October 2011, 389 businesses had forged partnerships with 69 local authorities of the 433
of the UK covering 4800 premises; by October 2013, this had grown to 880 businesses in
partnership with 112 authorities covering 70, 000 premises9. Yet, little is known about the
PA scheme beyond government and practitioner communities. In particular, we have no
analysis of how local authority regulators are trained in this complex scheme, and how
effective those training exercises are.
Why does this matter? While we know that inspectors understand, broadly support and are
implementing much of the Hampton agenda (CIEH, 2011; NAO, 2008), we know nothing
about the impact of training on their understandings of specific Hampton-inspired
initiatives. Getting to grips with this question is not simply a matter of academic enquiry for
those interested in better regulation. Evaluating the impact of training matters for public
managers, more generally. While training is treated as central to capacity building in
7 Inspection plans and any formal advice given to a business by a primary authority must be made available to enforcing authorities
through the IT system administered by the BRDO. There is also a new legal duty that the enforcing authority provide feedback to primary
authorities on inspection activity undertaken, where the primary authority requests such feedback http://www.bis.gov.uk/brdo/primary-
authority/about-pa/changes-to-pa#sthash.iArMvLCl.dpuf 8 This mechanism is only used in exceptional cases. Indeed, so far the BRDO has not been required to break the deadlock between a
primary authority or its business partner and an enforcing authority. 9 http://www.bis.gov.uk/brdo/primary-authority#sthash.eQkdMJsK.dpuf
7
government (PWC, 2007, 2013), in an era of austerity, the budgets and time allocated to
training are under intense pressure (Smith, 2012)10
. In this context, producing an evidence
base that delineates training effects appears prescient. Training must prove its worth: ‘… we
have entered a new era in which both achieving useful results and proving that they add
value to the organization and our shared society are required’ (Kaufman and Guerra, 2001:
319).
What impact(s) are we looking for when we evaluate training? One conceptualisation of
training evaluation dominates the organizational learning and human resources literatures:
the Kirkpatrick model. Kirkpatrick (1959) outlines four levels of training evaluation; each one
distinguished by its focus and level of complexity. At its most basic level, level 1, training
monitors participants’ reactions. This usually takes the form of short check box and Likert
scales questionnaires handed out at the end of training sessions to measure participants’
reactions to training programmes (Guerra-López 2008). These so-called ‘smile sheets’ are
equated with measuring customer satisfaction. Level 2 evaluates what has been learned.
This second level of training effect has three possible learning components: knowledge,
attitudes and skills. The importance of evaluating learning in our case is clear – incorrect
understanding of a policy programme, or the persistence of attitudes that run counter to
the ethos of burden reduction, undermines the adoption and smooth running of PA
partnerships. Most commonly, measuring learning takes the form of pre- and post-test
questionnaires. Where possible, the ideal is to establish a quasi-experiment that compares a
trained group with an untrained control where the test items are closely matched to the
actual objectives of the training intervention. Beyond reactions and learning, training
evaluations can be designed for two further purposes which concern the transfer of training
to organizational behaviour (at level 3) and organizational outcomes (at level 4) (see Blume
et al 2010 comprehensive meta-review of the transfer literature).
Our quasi-experiment explores Kirkpatrick’s second level; we are interested in the learning
affected by training in knowledge and attitudes. What did local authority inspectors learn
about the PA scheme in particular, and attitudes toward regulatory burden reduction more
generally, by participation in training provided by BRDO? Before we can answer these, we
must outline the training interventions and hypotheses explored.
2. TRAINING AND HYPOTHESES
Two levels of training are available for local authority inspectors who are either interested in
creating, or already part of, a PA scheme11
. Both courses are provided by the BRDO at
venues around England – principally London and Birmingham. It should be noted that
10
In 2011/12 days given for ‘off the job’ training for local authority officers in England reduced from 1.4 days to 0.9 days per employee
(Keeling, 2013). 11
A separate training course is available for businesses interested in forming a PA partnership.
8
membership of a PA scheme does not require any training to have been completed.
Provided since 2010, both courses comprise a one day workshop for a maximum of 15
inspectors.
The training format at each level follows the same structure. An expert trainer, who has
successfully established PA partnerships, uses a slide presentation and group work to
outline the scheme and its underlying philosophy. The group work uses worked examples of
best practice cases of established PA partnerships. Participants at both levels are given
hypothetical stories or vignettes which are used as the focus for questions that reveal
participants’ comprehension of the scheme and its approach to regulation.
The content of the two training levels moves from the general to specific. Level 1 offers an
introduction to the scheme; while level 2 is designed for those who want a deeper insight
into how to build a successful PA partnership. Though there is no requirement that
participants in either level of training are already part of a PA partnership, the specificity of
the courses means that participants are at least interested in establishing one. We want to
explore whether those trained at either level display a deeper understanding of the scheme
itself and its better regulation philosophy than those who are untrained. Since the
completion of level 1 training is not a pre-requisite for participation in level 2, we cannot
assume that those trained at level 2 have a deeper knowledge and understanding than
those at level 1. And so, we combine the two groups to create a single experimental cohort
to be contrasted with our untrained control.
We explore two specific hypotheses that build on the training literature discussed in section
1 that posits a link between training and learning:
H1. Participation in training increases awareness and understanding that regulatory risk
can be decomposed for other professionals – i.e. it increases the understanding of the
operational aspects of the PA scheme.
H2. Participation in training leads to a greater awareness of the multiple identities
implied by becoming a PA – where inspectors become regulators and advisors – i.e. it
moves professional identities of inspectors toward business’ needs.
Some more detail on the content of the training interventions, and what participants are
expected to learn, is in order. Level 1 training – ‘Becoming a Primary Authority’ (summarised
in box 1) – outlines the main aspects of the scheme focussing in particular on the status of
the advice given by a primary authority to its business partner and the differentiation of the
roles assumed by primary and enforcing authorities in the management of risk and
enforcement. In terms of the wider regulatory ethos underpinning the scheme, level 1
emphasizes effective regulation as the product of partnerships between regulators and
regulated, and explores the value of PA for businesses and citizens. Communication,
dialogue, burden reduction, proportionate risk management and flexibility are all themes
9
that recur in this session. Level 2 training – ‘Making an Impact Through Your Partnership’ –
covers all of these themes, and also offers a more detailed account of managing a
partnership and building a working relationship between primary authority and businesses
(see summary in box 2). Again, the emphasis on proportionate risk management is strong.