-
FP7 Project: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
SPREE. All rights reserved. This report was produced by SPREE
consortium with the financial assistance of the European Union and
is available on SPREE website free of charge to visitors for
non-commercial use, provided that the European Union is duly
credited. The contents of this document are the sole responsibility
of the SPREE Consortium and can under no circumstances be regarded
as reflecting the position of the European Union. SPREE -
Servicizing Policy for Resource Efficient Economy - is a project
within the Seventh Programme and is co-funded by the European Union
(Grant Agreement 308376). The project duration is from July 1, 2012
June 30, 2015.
Methodology Development for the Evaluation of
Policy Instruments to Promote Servicizing
-Deliverable 3.1-
Elah Matt, Moshe Givoni, Bryan Epstein (Tel Aviv University)
Eran Feitelson (Jerusalem Institute for Israel Studies, Hebrew
University)
With contributions from:
Surrey University, UK
Oxford University, UK
The Finnish Environment Institute, Finland
University of Santiago de Compostela, Spain
Lund University, Sweden
Jerusalem Institute for Israel Studies, Israel
Tel Aviv University, Israel
Ben-Gurion University of the Negev, Israel
February 2013
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
2
Contents
List of Tables
....................................................................................................................................
3
Executive summary
..........................................................................................................................
4
1. Introduction
.................................................................................................................................
6
2. Policy evaluation in the public policy literature
.............................................................................
8
2.1 What is a policy evaluation?
.......................................................................................................
9
2.2 Who evaluates, why, and how?
................................................................................................
10
2.3 What is evaluated?
...................................................................................................................
13
3. Evaluation of policy instruments in the SPREE project
................................................................
15
3.1 Impact evaluation of policy instruments
...................................................................................
17
3.1.1 Economic impacts
...............................................................................................
17
3.1.2 Environmental impacts
.......................................................................................
18
3.1.3 Social impacts
.....................................................................................................
20
3.1.4 Potential effect on decoupling
............................................................................
21
3.2 Evaluating Implementation
.......................................................................................................
22
3.2.1 Implementation feasibility: financial, technical,
technological and pre-condition
considerations
..................................................................................................
22
3.2.2 Institutional settings
...........................................................................................
24
3.2.3 Political legitimacy
..............................................................................................
25
3.2.4 Social acceptability
.............................................................................................
26
3.2.5 Flexibility & Reflexivity
.......................................................................................
27
4. Summary and next steps
............................................................................................................
28
4.1 Using the evaluation framework in the SPREE project
..............................................................
30
References
.....................................................................................................................................
32
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
3
List of Tables
Table 1: Aspects of administrative evaluation
........................................................................................
14
Table 2: Evaluation criteria proposed by policy instruments
perspectives ............................................. 15
Table 3: Main criteria for the evaluation of policy instruments
..............................................................
16
Table 4: Examples of environmental impact indicators
..........................................................................
19
Table 5: Summary of evaluation methodology for the SPREE
project..................................................... 29
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
4
Executive summary
Policy evaluation is an inherently political activity,
influenced by the interests, values, beliefs and norms
of the evaluator. Positivist policy evaluations, based largely
on quantitative research methodologies that
were advocated during the 1960s and 1970s, were therefore over
time complemented by more
qualitative research methodologies. Today, many policy
evaluations rely on a mix of qualitative and
quantitative methods.
In this deliverable, we aim to construct a broad methodological
framework for the evaluation of existing
policies that promote servicizing. Broadly, servicizing can be
defined as a transaction where value is
provided through the provision of services or functions, rather
than sale and purchase of products. We
distinguish between servicizing systems, which induce economic,
environmental and social impacts, and
policy instruments, which may facilitate the uptake and
functioning of these systems. In this deliverable,
we focus on the evaluation of the latter. This task promotes two
of the SPREE project's objectives. First,
the evaluation of individual policy instruments is a necessary
first step in building policy packages to
promote servicizing. This evaluation will be carried out in the
context of the specific case studies chosen
in each of the three SPREE sectors (mobility, water and
agri-food) using quantitative Agent Based
Modelling framework, and qualitative methods. Second, the
guidelines outlined in this document will
also inform the evaluation of servicizing systems at later
stages of the project.
Our evaluation framework comprises two levels of analysis. In
the first level, we suggest criteria for
evaluating the impacts of individual policy instruments in terms
of their economic, environmental and
social impact, as well as decoupling effect. In the second level
of analysis, we introduce criteria to
examine the "implementability" of policy instruments in terms of
their financial, technical and
technological feasibility and any preconditions necessary for
their uptake, institutional settings, political
legitimacy, social acceptability, flexibility and reflexivity of
these instruments.
The suggested methodological framework comprises a list of
criteria that inform a multi-criteria-
decision type analysis. Each criterion is further divided into a
number of guiding questions that can be
utilized in the ex-post evaluation of existing policy
instruments, as well as ex-ante evaluation of
proposed policy instruments. The evaluation methodology is
developed in parallel to the ex-post policy
analysis on the national levels (task 2.5 of WP2), with
reciprocal knowledge transfer between them. In
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
5
addition, the evaluation methodology developed is to be used in
the next steps of the project and
further tailored to each of the sectors which are the core focus
of the SPREE project. It should also
support the development of the Agent Based Modelling framework
and tools and to construct policy
packages ('Servicizing Policy Packages") to promote servicizing
as a way to achieve resource efficient
economy.
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
6
1. Introduction
In a world where multiple economic, environmental and social
crises dominate the public agenda, there
is a need for new thinking on these matters. In the SPREE
project, it has been argued that a shift
towards a service-based economy can result in the improvement of
economic, environmental and social
circumstances, and ultimately to a decoupling of economic growth
from environmental degradation.
The SPREE project aims to examine this assumption. Another
objective of the project is to propose
policy packages, defined as a combination of individual policy
instruments aimed at addressing one or
more policy goals (Givoni et al., 2013), which could facilitate
and promote the shift to servicizing and
consequently contribute to decoupling and resource efficiency.
Before policy packages can be
assembled, a wide range of individual policy instruments must be
considered and evaluated. The aim of
this deliverable is therefore to outline a methodological
framework that will allow for the evaluation of
policy instruments to promote servicizing to be carried out
later in the project, and in the three sectors
considered - the water, agri-food and mobility sectors.
Policymaking and hence policy evaluation are undertaken in
increasingly complex and uncertain
realities. A shift from one-way governing systems towards
two-way governance systems between those
governing and those governed has been purported (Kooiman, 1993).
The often-conflicting interests of a
range of public and private policy actors must therefore be
taken into consideration when prescribing
and evaluating policy instruments. Moreover, increasing
uncertainties regarding, for example, the
effects of global climate change, the volatility of financial
markets and geo-political instability pose
further challenges to public policymaking. In light of these
challenges, policy evaluation has become a
growth industry, undertaken by academics, governmental and
non-governmental organisations, private
consultancy firms and lobby groups (Huitema et al., 2011). As
such, policy evaluation is an inherently
socio-political practice shaped by the interests, values and
beliefs of policy evaluators and participants
in the policy process (Guba and Lincoln, 1989). The challenge
for policy evaluators is therefore to strive
for objectivity in policy evaluation, while acknowledging the
inherent complexities and value-laden
nature of this practice. In this context, the aim of this
deliverable is to develop a general methodological
framework for the evaluation of policy instruments to promote
servicizing.
It is perhaps prudent to provide definitions here for some key
terms used throughout this deliverable,
notably Methodology, Servicizing and Policy instruments.
Methodology can be understood as 'the tasks,
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
7
strategies, and criteria governing scientific inquiry' (Gerring,
2012: 6). This is in contrast to the definition
of a method, which can be understood as 'a specific procedure
for gathering and/or analyzing data'
(ibid). In this deliverable, we are therefore concerned with the
construction of a broad methodological
framework, rather than prescribing specific tools to carry out
the evaluation. The methods for the
implementation of this framework will be chosen by the project
partners, in particular within task 3.5:
"Development of data collection methods".
Servicizing can be defined as a transaction where value is
provided through services or functions and
not just through selling products (a more account has been
developed in the context of task 2.3:
"Establishing a common understanding of the notion of
servicizing"). Servicizing systems include both
the environmental and societal systems that support/facilitate
or act as barriers to servicizing solutions
and provide the context for the desired implications and effects
of servicizing solutions. In order to
promote servicizing - the development and success of servicizing
systems, policy instruments can be
utilised, individually or preferably as policy packages.
Policy instruments can be defined as the techniques or means
through which states attempt to
maintain their goals (Howlett, 2011: 22). More broadly, policy
instruments can be understood as an
identifiable method through which collective action is
structured to address a public problem (Salamon,
2002: 19). This definition acknowledges the involvement of
public and private actors in shaping the
choice, design, and implementation of policy instruments. In
this view, policy instruments are
determined by, and determine which actors will be involved in
the policy process, what role they will
play, and how their (often conflicting) interests will shape the
policy process (Salamon, 2002: 10-11).
Therefore, in order to understand and evaluate public policies,
we need to evaluate policy instruments
and their impact and implementabiltiy in practice (Eliadis et
al., 2005: 5).
Our evaluation methodology comprises two levels of analysis. The
first examines the economic,
environmental and social impacts of policy instruments to
promote servicizing and their potential
effects on decoupling. Second, we examine the implementation
feasibility (implementabiltiy) of
individual policy instruments. We measure implementabiltiy
through the consideration of financial,
technical and technological feasibility, and necessary
preconditions for the implementation of the policy
instrument. We further identify institutional settings,
political legitimacy, social acceptability and
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
8
flexibility as important determinants of the implementabiltiy of
policy instruments. For each of these
categories we provide a number of guiding questions to inform
the instrument's evaluation.
The methodological framework proposed in this document is
intended for use throughout the
remainder of the SPREE project. It will be used for evaluating
policy instruments to promote servicizing
in the mobility, agri-food and water sectors. Although the
stated aim of this task is to propose a
methodology for the evaluation of existing policy instruments,
the framework will also provide the basis
for ex-ante evaluation of policy instruments, and for assessment
of their inclusion in policy packages to
promote servicizing, adopting both qualitative and quantitative
methods (including the Agent Based
Modelling). The methodology will need to be adapted to the three
sectors examined in the SPREE
project, and further refined to in order to evaluate servicizing
systems in different countries and
localities. These objectives will be pursued at later stages of
the project.
This deliverable continues as follows. Section 2 provides a
review of the existing public policy literature
on policy evaluation. The section answers questions about what
is policy evaluation, who evaluates,
why, and what is evaluated. The section concludes that the
evaluation of policy instruments requires
the use of a broad methodological approach, and a mix of
qualitative and quantitative research
methods, supporting the choice of the methodological approach of
the SPREE project. Section 3 then
outlines evaluation criteria in two categories. First, the
evaluation of impacts, or the effectiveness
dimension of policy instruments is outlined. Second, the
implementabiltiy aspect of policy instruments
is explored. Section 4 concludes this deliverable with a summary
of the main findings, the proposed
evaluation criteria.
2. Policy evaluation in the public policy literature
In order to develop the methodological approach of this study,
it is important to first outline the
concept of policy evaluation. This section attempts to answer
the following questions: what is policy
evaluation; when is policy evaluated; who evaluates policies and
why; how are policies evaluated; and
what is evaluated. The section concludes that policy evaluation
is an inherently political practice, which
requires the application of a mix of qualitative and
quantitative research methods.
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
9
2.1 What is a policy evaluation?
Definitions of policy evaluation abound and vary. Broadly,
policy evaluation can be understood as
learning about the consequences of public policy (Dye, 1987:
351). Policy evaluation can be
conceptualised as either an ongoing activity throughout the
policymaking process, or as a separate
stage of the policy cycle. The purpose of policy evaluation is
to assess whether the policy objectives
pursued were (or will be) met, and the means employed to achieve
these objectives (Howlett, Ramesh
and Perl, 2009: 178).
Three different types of policy evaluation can be discerned.
Understanding policy evaluation as a
discrete stage of the policy cycle, results in the
conceptualisation of two types of policy evaluation. Ex-
ante policy evaluation implicates the assessment of policy
instruments before they are chosen, designed
and implemented. This practice aims to inform policymakers of
the possible consequences of various
policy instruments before these are adopted. Ex-ante evaluation
measures include environmental
impact assessment, strategic environmental assessment and
regulatory impact assessment (see for
example Turnpenny et al., 2009; Persson and Nilsson, 2006).
Ex-post policy evaluation occurs following
the implementation of the policy instrument, and often (but not
exclusively) ahead of its reformulation
or termination. Ex-post evaluation investigates 'how a public
policy has actually fared in action
(Howlett, Ramesh and Perl, 2009: 178).
Continuous evaluation occurs at all stages of the policy
process. Continuous evaluation involves
investigating a policy program to obtain all information
pertinent to the assessment of its performance,
both process and result (Wollmann, 2007: 393, emphasis added).
Also termed formative evaluation,
this type of evaluation is undertaken intermittently at all
stages of the policy cycle, and particularly
during policy implementation (Parsons, 1995: 546-7). Formative
evaluation requires analysis of the
extent to which a program is being implemented and the
conditions that promote successful
implementation (Palumbo, 1987: 40, see also Parsons, 1995: 547).
This description corresponds with
the two evaluation dimensions presented in the introduction to
this deliverable, and outlined in section
3. These are the considerations of social, economic,
environmental and decoupling impacts alongside
considerations of the implementability of policy instruments.
Continuous evaluation challenges the
earlier perceptions of evaluation as a distinctive stage of the
policy cycle (Hill, 2013: 298). There are
clear benefits to this approach, such as ensuring the assessment
of public policies not only before or
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
10
after implementation but on a regular basis, maintaining dynamic
policy implementation, and dealing
with uncertainties and unexpected consequences. However, in
practice, continuous evaluation remains
the least implemented type of policy evaluation.
These notions of policy evaluation and particularly the
conception of ex-ante and ex-post evaluation,
assume that policymaking is a linear process defined by the
discrete stages of the policy cycle. However,
this is rarely the case, as policymaking is never a rational,
linear process. As Parsons (1995: 79) notes:
The idea of dividing policy-making in such a way greatly
overstates the rational nature of policy-making and gives a false
picture of a process which is not a conveyor belt in which
agenda-setting takes place at one end of the line and
implementation and evaluation occurs at the other.
In practice, policymaking in general, and policy evaluation
specifically, seldom occur in a linear fashion.
Still, the stagist model of policymaking provides a useful
heuristic device 'with which to orientate and
situate a particular academic contribution' (Givoni et al.,
2013: 5). We therefore maintain the distinction
between ex-ante and ex-post evaluation, although in practice
policy evaluation can be understood as a
continuous process.
2.2 Who evaluates, why, and how?
Policy evaluation is conducted by governments and other
interested policy actors, who wish to
influence the policymaking process (Howlett, Ramesh and Perl,
2009: 178). Policy evaluators include
academic scholars, independent research institutes, consultancy
firms, international and national
governmental bodies, non-governmental organisations and business
groups (Huitema et al., 2011). Each
of these actors has their own interests, values and beliefs that
may shape the outcomes of the
evaluation process.
Two opposing views on the execution of policy evaluation exist.
The rational model supports evidence-
based policymaking. In this view, policy evaluation should
objectively assess the achievements of a given
policy instrument (or policy packages), creating a natural
feedback cycle to subsequent policy
improvement (Hill, 2013: 296). However, this ideal seldom occurs
in practice, and there is rarely a
systematic connection between evaluation and policy improvement
(Hill, 2013: 298). Interpretivist
views of policymaking therefore conceive policy evaluation as a
political tool which shapes all (or some)
stages of the policymaking process. These two conflicting views
subsequently lead to diverging
methodological perspectives on policy evaluation.
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
11
The public policy evaluation literature evolved since the 1960s,
in line with trends in the broader public
policy literature (Wollmann, 2007). From the 1960s through to
the early 1980s, the evaluation literature
was largely positivist, and based on two predominant
assumptions. First, it was thought that in order to
assess policy outcomes, it is necessary to identify in advance
policy goals and political intentions.
Second, in order to identify objective casual relation between
observed change and policy program,
there was a need for quantitative, quasi-experimental research
design (Wollmann, 2007: 398).
Evaluation was viewed as a neutral, technical exercise in
determining the success (or failure) of
government efforts to deal with policy problems, and was
therefore seen as an objective, systematic
and empirical process (Howlett, Ramesh and Perl, 2009: 178).
However, this practice was fraught with difficulties. For
example, it is difficult to identify goals and
objectives that will serve as a measuring stick. Policy goals
often come in bundles and are difficult to
translate into measurable indicators. Empirical data to fill
indicators is difficult to get, and the more
meaningful an indicator is, the more difficult it is to obtain
viable data (Wollmann, 2007: 398). Side-
effects and unintended consequences are hard to trace and
measure. Further, methodologically robust
research design can often not be applied as the large N small
amount of variables conditions for
quantitative research are often replaced with small N large
amount of variables", necessitating
qualitative research methods (ibid). Positivist analysis proved
difficult to conduct in the study of public
policies, in which real-world experiments can rarely be
conducted. The analysis of policy impacts and
their distribution therefore proved to be a challenging task for
the positivist researcher, since the
distinction between who gets and who does not get the benefit of
a new policy can rarely be made
(Hill, 2013: 296).
From the early 1980s, the second phase of evaluation studies was
influenced by constructivist-
interpretative thinking. Scholars began questioning the premises
that policies should be evaluated
according to policy goals and political intentions. Instead, a
plurality of perspectives, interests and
values were recognised by these perspectives. For example,
Stufflebeam (1983) suggested the CIPP
model, based on the evaluation of Context, Input, Process and
Product. This line of research
acknowledged that developing adequate and acceptable measures
for evaluating policy was more
contentious and problematic than was previously believed
(Howlett, Ramesh and Perl, 2009: 179). This
literature also acknowledged that evaluation could be employed
to conceal some governmental
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
12
shortcomings. Thus, evaluation was increasingly perceived as a
political tool in the hands of policy
actors aiming to influence policymaking processes (Howlett,
Ramesh and Perl, 2009: 179). Extreme post-
positivist thinking therefore maintained that since similar
conditions could be interpreted differently,
there is no definitive way of determining the correct evaluation
mode (Howlett, Ramesh and Perl,
2009: 179). Instead, the prevailing interpretation of evaluation
is determined by political conflicts and
compromises among the various actors (ibid). Evaluation was thus
understood as a political activity,
with little systematic connection between evaluation and policy
improvement (Hill, 2013: 298).
Accordingly, qualitative evaluation methods were developed
during this period.
The third wave of policy evaluation, since the 1990s, was
inspired by the New Public Management
literature, and the idea of creating more efficient public
administration. Increasingly, scholars and
practitioners are combining positivist and post-positivist views
on policy evaluation. This perspective is
captured by, Howlett, Ramesh and Perl (2009: 179):
[It] is not to suggest that policy evaluation is irrational or a
purely political process, devoid of genuine intentions to assess
the functioning of a policy and its effects. Rather, it served as a
warning that we must be aware that relying solely on formal
evaluation for drawing conclusions about a policys relative success
or failure will yield unduly limited insights about policy outcomes
and their assessment. To get the most out of policy evaluation, the
limits of rationality and the political forces that shape it must
also be taken into account, without going so far as to believe that
the subjective nature of policy assessments allows no meaningful
evaluation to take place (Howlett, Ramesh and Perl, 2009: 179).
Thus, policy evaluation needs to rely on both qualitative and
quantitative research methods in order to
provide an encompassing understanding of the outcomes of
policies, but also to capture the inherently
political and subjective nature of this process.
According to Wollmann (2007: 397), evaluation research faces two
main conceptual and methodological
tasks. The first is the conceptualisation of the intended and
unintended consequences that are to be
assessed. These consequences are the dependent variables.
Secondly, evaluation research needs to
determine "whether and how the observed changes are causally
linked to the policy and measure under
consideration". These instruments are the independent variable.
In the bid to answer these questions,
the positivist and interpretivist camps have come closer over
the years, and it is now widely agreed that
[T]here is no king road in the methodological design of
evaluation research; instead, one should acknowledge a pluralism of
methods. The selection and combination of the specific set and mix
of methods depends on the evaluative question to be answered, as
well as the time frame and financial and personnel resources
available (Wollmann, 2007: 398)
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
13
In summary, from the positivist approaches to policy evaluation
that prevailed in the 1960s and 1970s,
policy evaluation studies evolved first towards
constructivist-interpretivist approaches based on
qualitative research methods, and more recently towards a mix of
qualitative and quantitative research
methods.
We now aim to answer the questions at the beginning of this
section, with respect to the SPREE project.
The evaluators employed under the SPREE project are the project
consortium, which includes academic,
governmental, and non-governmental organisations, from a range
of EU Member States and Associated
Countries. The evaluation is undertaken under the European
Commission's Seventh Framework
Programme (FP7), with the aim of promoting economic growth,
environmental resource-efficiency and
social cohesion through the promotion of servicizing systems (as
discussed above). Our research
methodology is mixed in its nature; combining both positivist
and constructivist elements. As such, we
suggest the utilisation of mixed research methods, including
both quantitative and qualitative
techniques. The following section goes on to examine in more
detail what is actually being evaluated by
different evaluation approaches.
2.3 What is evaluated?
As mentioned above, policy evaluation is an inherently political
practice, influenced by the interests and
values of the evaluator. The selection of evaluation criteria
and the perspectives of different policy
actors can therefore lead to different conclusions regarding
policy outcomes (Hill, 2013: 297). Public
policy scholars created numerous categorisations of evaluation
criteria. These studies were often
concerned with the process, content, or outcomes of policies and
policy instruments, either separately
or in combination. For example, administrative evaluation, as
conceived by Howlett, Ramesh and Perl
(2009: 186), addresses these three aspects, as summarised in
Table 1. Similarly, Marsh and McConnell
(2010) provide detailed criteria for policy evaluation, based on
the examination of process, content and
outcome. These criteria, they claim, are constructed in order to
overcome the barrier between
positivists and constructivists evaluation approaches.
Fischer (1995) outlined four considerations that affect
evaluation efforts. First, positivist verification
examines whether the policy's stated objectives were met (in our
case the objectives are mainly
achieving absolute decoupling through servicizing). Second,
validation of the policy in terms of its
relevance to the problems it aims to address will affect the
evaluation outcomes. Third, vindication
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
14
examines whether the policy contributes to society as a whole,
while social choice examines the
ideological concerns about the policys objectives. These
considerations, Fischer argues, influence the
choice of divergent evaluation criteria, and therefore may lead
to different results of the policy
evaluation process. Not only the choice of indicators to assess
a specific impact, or criterion will affect
the evaluation outcome, but of course, and first of all the
choice of what is evaluated (before deciding
on how it will be evaluated). Thus, evaluation can consider
economic impacts and appear successful
only since environmental impacts are not considered.
Table 1: Aspects of administrative evaluation
Type of evaluation What is being evaluated
Process evaluation Organisational methods: Can policymaking
processes be streamlined and made more efficient? Implementation
divided into tasks and evaluated
Effort evaluation Program input: how much effort did government
put into achieving policy goals in monetary terms
Performance evaluation Policy outputs: What is the policy
producing?
Efficiency evaluation Economic efficiency: Can policy goals be
achieved more efficiently by different means?
Effectiveness evaluation Is the performance adequate? Are policy
goals met?
Source: Howlett et al. (2009: 186)
From the above, it can be seen that research on policy
evaluation often focuses on one or more
evaluation aspects, including the context, process, content or
outcome of public policies. However,
many studies evaluating policy instruments hold in common their
focus on the outcome dimension of
evaluation. These approaches often examine the economic, social
and political impacts of policy
instruments, as summarised in Table 2. The common themes
discerned from this table are the
consideration of effectiveness in reaching stated objectives;
economic and resources efficiency; social
legitimacy and equity and political acceptability.
In the methodology developed in the following section, we
address the outcome dimension of policy
evaluation by suggesting criteria for assessing policy impacts.
We also offer some means for appraising
the process and content dimensions of policy evaluation (and to
a lesser extent the context dimension)
through the examination of implementabiltiy.
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
15
Table 2: Evaluation criteria proposed by policy instruments
perspectives
Policy instrument
attributes
Sabatier and Pelkey
(1987)
Salamon (1989) Linder and Peters
(1989)
Treiblock (1994)
Theoretical
approach
Advocacy- coalition
framework
Policy
implementation
Public management
Policy design Public choice
Valuation criteria 1. Economic
resources
2. Political costs
3. Economic costs
4. Efficacy
1. Efficiency
2. Political support
3. Administrative
feasibility and
equity
4. Supply and
targeting
effectiveness
1. Resource-
intensiveness
2. Political risk
3. Financial and
ideological
4. Targeting
1. Efficiency
2. Political
considerations
3. Distributional
considerations
Source: Landry and Varone (2007: 110)
3. Evaluation of policy instruments in the SPREE project
In the remainder of this deliverable, we construct a
methodological framework for evaluating policy
instruments and their impact in the context of the SPREE
project. All the policy instruments that will be
evaluated in the course of the project will be expected to
promote and support, directly or indirectly,
servicizing systems and a shift to servicizing. Whether these
policy instruments will also support
resource efficiency, decoupling and desired social outcomes
depends on whether the servicizing system
they support contribute to these aspects (different
methodologies have been developed for this
purpose within task 3.1: "Methodology development for measuring
economic and environmental
impacts of servicizing activities" and task 3.2: "Methodology
development for measuring social impacts
of servicizing activities").
In assessing and evaluating policy instruments two dimensions
are considered. First, the extent to which
the policy instrument, by promoting servicizing, affects the
economic, environmental and social
performances of a particular servicizing system. In this
respect, there is similarity between the
evaluation of servicizing systems (by using the methodologies
developed within WP3) and the
evaluation of policy instruments to promote them, outlined in
this deliverable. The second dimension
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
16
we consider is the extent to which a given policy instrument can
be implemented. The consideration of
these two dimensions will inform the evaluation of policy
instruments, and their role in facilitating a
transition to servicizing.
Table 3: Main criteria for the evaluation of policy
instruments
The methodological framework outlined in this deliverable covers
both of the above dimensions. First,
the effectiveness of servicizing policy instruments - policy
instruments to promote servicizing, in terms
of economic, environmental, social and decoupling impacts is
considered. Effectiveness represents the
degree to which a policy intervention influences the
corresponding policy objective in a desirable
manner. Second, the implementabiltiy of servicizing policy
instruments is examined. Implementabiltiy is
considered in terms of technical, technological and financial
feasibility, preconditions, institutional
settings, political legitimacy, social acceptability,
flexibility and reflexivity. These criteria are summarized
in Table 3, and discussed in more detail below. While the
evaluation criteria are treated as discrete
categories, in practice there is an overlap between them. Where
possible, we have tried to indicate
these overlaps, but have adhered to distinct evaluation criteria
in order to facilitate the evaluation work
which will be conducted later in this project. We suggest a
series of questions to guide the evaluation of
the different criteria. Based on these questions, specific
indicators will be selected and considered for
the actual, quantitative and qualitative, evaluation of policy
instruments later in this project.
Evaluation dimension Evaluation criterion
Impact
(effectiveness)
Economic impacts
Environmental impacts
Social impacts
Absolute decoupling effect
Implementabiltiy Feasibility
(financial, technical, technological and pre-condition
considerations)
Institutional settings
Political legitimacy
Social acceptability
Flexibility and Reflexivity
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
17
3.1 Impact evaluation of policy instruments
As mentioned above, the first part of this evaluation
methodology examines the economic,
environmental, social, and decoupling impacts of policy
instruments that promote servicizing. These
impacts are outlined in more detail below.
3.1.1 Economic impacts
Much work has been done on the economic impacts of environmental
legislation and technological
innovation (see for example Jaffe et al., 2005). Indeed,
environmental economists have over the past
decades created detailed methodologies for economic evaluation,
based largely on cost-benefit
analyses. In the SPREE project, our ambitions are more modest.
We relate the concept of economic
impacts to the economic effects of policy instruments on
governments, service providers, consumers
and other affected parties, or in other words, the value chain
of the examined system. We suggest a
series of guiding questions, which will inform an economic
analysis. Economic impact can be considered
to include the impact on economic growth, employment and changes
in the costs and benefits to
affected parties.
In the context of the SPREE project, and especially the
quantitative analysis, the focus is on the service
providers and produces, consumers, and on the chain/sector
level. To assess the economic impacts on
these actors the following questions can be used to guide the
evaluation.
For service providers and producers:
1. Does the policy instrument creates additional profits, and
reduces risks for the service provider?
2. Is innovation stimulated (dynamic efficiency)?
3. What are the costs in terms of compliance, investment, and
marketing?
4. Are intellectual property rights secured?
5. Are environmental externalities internalised?
For consumers:
1. What are the costs for consumers associated with the policy
instrument?
2. Are there any utility gains in terms of reduced costs, time
savings and/or improved quality of life
(see also social impacts)?
3. What are the likely short and long-term impacts on consumer
demand and consumption behaviour?
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
18
For the chain/sector level:
1. What are the overall profits and added value in the system
(e.g. knowledge gained, incomes
secured, expertise developed, networks constructed etc.)?
2. What are the labour employment consequences of the change in
the system?
3.1.2 Environmental impacts
Servicizing systems and the policy instruments that support
their uptake are often intended to reduce
the negative environmental impacts of economic development. One
of the main arguments in favour of
the creation of servicizing systems is that gains can be made in
resource-use efficiency. Material
products are treated as capital assets rather than consumables.
Servicizing may thus create incentives
to maximize resource-use efficiency and product durability,
whilst minimizing costs to the producer,
consumer and the environment.
Environmental assessment of servicizing solutions demonstrates
that, on average, single-digit-factor
improvement was achieved in various case studies, including:
washing centres (Weaver, Jansen et al.
2000; Hirschl, Konrad et al. 2003), car sharing (Sperling,
Shaheen et al. 2000), ski rental services (Hirschl,
Konrad et al. 2001), and integrated pest management
(Schmidt-Bleek and Lehner 1998). The relatively
modest improvements stem from a lack of focus on the systemic
changes that need to be made in order
to deliver services in a more sustainable way. At the same time,
quantitative methods for evaluating
environmental impacts of servicizing solutions are still in
short supply. Studies employing life-cycle
analysis (LCA) and similar methods for evaluating servicizing
solutions still yield diverging results, owing
to differences in the evaluation systems initial conditions,
boundaries and functional units chosen. The
evaluation of environmental impacts therefore constitutes an
important challenge to be addressed
throughout the SPREE project.
In the SPREE evaluation of environmental impacts associated with
the implementation of policy
instruments to promote servicizing, we distinguish between two
evaluation criteria. Maximising
resource-use productivity and minimising environmental impacts.
Resource-use oriented indicators
represent the amounts of resources used, while
environmental-impact based indicators relate to the
environmental impacts of servicizing systems, including the
production and consumption of goods and
services. Some examples of these indicators are outlined in
Table 4, with regards to materials, energy
and water resources. Some of these indicators can be evaluated
through LCA. However, we
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
19
acknowledge that it may not be possible to include all the
suggested indicators in case-specific
evaluations. For example, it may be difficult to assess the
impacts of a given policy on air pollution.
Likewise, current LCA techniques have not accounted for some
resource-efficiency indicators, such as
metal depletion. Therefore, the factors suggested in Table 4 are
more for illustration while specific
indicators for this category will be selected later in the
project.
Table 4: Examples of environmental impact indicators
Resource Resource use-oriented Environmental impact-oriented
Materials consumption of biotic and abiotic resources, Domestic
material consumption, imports
terrestrial acidification, freshwater eutrophication,
photochemical oxidant formation, particulate matter formation
Energy cumulative energy demand (CED), total renewable and
non-renewable energy use
Greenhouse Gas emissions, air pollution, carbon footprint
Water Water consumption and water abstraction Water exploitation
index1
Source: Adapted from BIO (2012: 8)
To guide the evaluation of changes in resource-use and
environmental impacts of a specific policy
instrument, several questions should be considered. These
include:
1. Does the policy instrument promote better resource use
associated with servicizing?
2. Does the policy instrument ensure better product durability
and life-time maximization, and
therefore greater resource-use efficiency, and positive
environmental impact?
3. Are incentives created for reduction, reuse, and
recycling/remanufacturing of material resources?
4. Does improved efficiency/reduced cost lead to a rebound
effect and therefore possibly to an overall
increase in resource consumption?
5. Can the overall environmental impacts of the policy
instrument be discerned, with respect to the
criteria suggested above? (resource use, GHG emissions, carbon
footprint etc.)
Due to some trade-offs between different environmental impacts,
it is crucial to evaluate multiple
environmental impacts of servicizing systems based on an LCA
methodology (as explained in detail in
Task 3.1). In the case of the SPREE project, indicators are
identified for each of the case-study sectors.
Examples of these indicators include inventory data, such as
greenhouse gas emissions, energy use,
1The European Environment Agency suggests a water exploitation
index whereby water abstraction should stay below 20%
of available renewable freshwater sources (BIO, 2012: 13). This
indicator may not be applicable in non-EU countries (for example,
Israel), and needs to be better defined through expert
consultation.
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
20
water use, and waste generated, as well as impact indicators
such as Global Warming Potential (GWP),
acidification, etc.
3.1.3 Social impacts
Within the SPREE project, we examine the effect of policy
instruments on the social impacts of
servicizing systems and we examine the effect of policy
instruments that promote servicizing on the
behaviour of consumers and businesses. Different types of social
impacts can be distinguished among
them are quality of life, the function of well-being (Diener,
2005), access to resources, job opportunities
and equity issues.
The concept of quality of life refers to the extent to which a
persons life is desirable, often in relation to
some external factors such as income or environment (Diener,
2005). However, the measurement of
quality of life is not always based solely on objective
measurements such as GDP, life expectancy and
education. For example, Pukeliene and Starkauskiene (2011)
differentiate between individual quality of
life and societal quality of life. Individual quality of life is
explicit and considers how well individuals live,
societal quality of life is implicit and relates to the
stability of society. For the purpose of this task, and in
order to simplify the evaluation procedure, we aim to evaluate
individual quality of life, based on the
measurements of objective indicators.
The perception of wellbeing is more subjective. Wellbeing can be
understood as a function of how
people feel and operate on a personal and societal level,
together with an overall self-evaluation of a
persons life (Diener, 2005; Michaelson et al., 2012). Community
wellbeing has been defined by Forjaz
et al. (2011: 734) as satisfaction with the local place of
residence taking into account the attachment to
it, the social and physical environment, and the services and
facilities. More broadly, Hall et al.(2010)
define societal well-being as the sum of human well-being
together with the condition of the
ecosystem, which provides the necessary resources and ecosystem
services to maintain well-being.
Thus, there is clearly an overlap between the notions of quality
of life, and well-being. However, for
methodological purposes we distinguish between these
concepts.
In evaluating the social impacts of policy instruments to
promote servicizing, equity must also be
considered. While policy instruments that promote servicizing
may improve the quality of life and the
well-being of society as a whole, they may have negative impact
upon certain individuals, or they might
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
21
not improve quality of life and wellbeing equally across
society. Hall and his colleagues (2010) argued
that when considering the sum of human well-being, it is
important to consider inter-generational and
intra-generation equity. Although average material well-being
can rise in a society, if the distribution of
well-being is skewed, progress may not be sustainable or
equitable (Hall et al., 2010). For example,
some women may feel less comfortable travelling on public
transport late at night than travelling in
their own car, and thus could be negatively affected by measures
to improve public transport a
servicizing system. Disabled users, who previously owned
specially-adapted equipment, may struggle to
benefit from services geared towards the wider population and
which do not take into consideration
their particular needs. Therefore, considerations of equity and
fairness must be taken into account
when measuring the social impacts of policy instruments to
promote servicizing (these aspects have
been studied within task 3.2).
Some guiding questions for evaluating the social impacts of
servicizing and policy instruments that
promote servicizing include:
1. Are there visible changes in quality of life which can be
attributed to the policy instrument (e.g.
changes in GDP, life expectancy, environmental quality,
education and health services)?
2. Is peoples perceived sense of individual wellbeing and
personal satisfaction affected by the
instrument (e.g. Are people more satisfied/happier with their
life; see also Table 5)?
3. Is the distribution of costs and benefits among different
societal groups equitable?
4. Are certain societal groups likely to be adversely affected
by the policy instrument? Can these
groups be compensated?
5. Have there been changes in the accessibility to local
amenities and services (local well-being,
Morrison, 2010: 1046)?
6. Have there been changes to the affordability of social
services, including accommodation, food,
utility bills, health care and education?
3.1.4 Potential effect on decoupling
In the SPREE project, we ultimately aim to evaluate whether the
synergetic economic, environmental
and social effects of servicizing systems create a decoupling of
economic growth from environmental
impacts, together with positive social effects. We are
interested in absolute decoupling, whereby the
overall negative environmental effects decrease, while economic
growth continues. Some guiding
questions for evaluation include:
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
22
1. Is there any evidence that continued economic growth has
resulted in lower overall environmental
impacts?
2. Can observable changes in environmental quality be related to
the economic impact associated with
the policy instrument?
3. Are there specific relational effects among environmental and
economic impacts (e.g. impact on
emissions of greenhouse gases and employment)?
4. Can positive social impacts be observed in relation to
economic/environmental decoupling?
3.2 Evaluating Implementation
The previous section examined criteria for evaluating the change
in the impacts of servicizing systems as
a result of policy instruments to promote servicizing systems.
In this section, we turn to examine factors
that may affect the implementation of these instruments. It is
common knowledge that the impact of
policy instruments depends on their implementation in practice
(Parsons, 1995: 547). Or, as Rist (1998:
158-159) observes:
Successful policy implementation involves the translation of
policy intent into policy action. While the failure to craft a
thoughtful policy involving the choice of appropriate instruments
can reasonably predict that the policy will not have the intended
effects, so also can the failure to appropriately implement the
policy ensure that the intended effects do not occur. Intention
without execution leads nowhere.
Thus, changes in the economic, environmental and social impacts
of policy instruments depend on the
implementation of these instruments. Still, most studies on
policy instruments have not adequately
addressed their implementation in practice (Jordan et al. 2011:
546). In the remainder of this section,
we introduce some of the factors that may influence the
implementabiltiy (that is the likelihood of
successful implementation) of policy instruments. These factors
include feasibility, institutional settings,
political legitimacy, social acceptability and flexibility in
dealing with uncertainty. These considerations
and their evaluation are discussed in some detail below. The
suggested criteria must not be read as a
comprehensive review of the vast literature on policy
implementation, but should be seen as providing
general guidelines for the empirical work which will be
undertaken later in this project.
3.2.1 Implementation feasibility: financial, technical,
technological and pre-condition considerations
In order to ensure the successful implementation of policy
instruments, they must be feasible
financially, technically and technologically. As put by
McDonnell and Elmore (1987: 146),
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
23
The selection of policy instrument depends on the constraints a
policymaker faces and the resources available either to diminish
the force of those constraints or to enhance the effectiveness of a
given instrument. In simplest terms, identifying resources and
constraints is how policymakers assess what is feasible, given how
they define a policy problem.
We therefore need to understand the material constraints under
which policy instruments are chosen
and designed. These include financial limitations on both public
bodies and the private actors, who are
responsible for the implementation in practice of policy
instruments. A policy instrument cannot be
implemented if there are no funds to implement it, if it is not
financially viable, or it could only be
partially implemented (e.g. unfinished project). Equally, policy
instruments need to be technically
feasible, so that their implementation is not only financially
viable, but also relatively straight-forward
to undertake in terms of technical feasibility. For example,
financial resources might exist to build a new
high-speed train line, but the expertise to build the required
infrastructure, or the boring machines for
the required tunnels are not available.
Another consideration that must be taken into account when
evaluating the implementabiltiy of policy
instruments to promote servicizing is whether the instrument is
technologically feasible. Some policy
instruments are adopted in order to promote technological
innovation (technology-forcing
instruments). For example, the European Union car emissions
standards have over several decades
promoted the uptake of new technologies, such as catalytic
converters in the 1980s and 1990s (Arp,
2000; Wurzel, 2002), and more recently the uptake of hybrid cars
and other low-carbon car innovations,
following EU legislation on car CO2 emissions. However, when
assessing technological feasibility it does
not suffice to examine solely the available technology. Rather,
as suggested by Unruh (2000), we must
also understand societal and institutional factors, as well as
interdependencies with other technologies
and infrastructure in order to adequately address this
criterion. The matter of technological feasibility
will need to be carefully monitored during the project and the
implementation of this methodological
framework, as some of these matters are in the heart of the
SPREE project and the promotion of a
resource-efficient servicizing economy.
One of the main elements in formulating policy packages is
accounting for the relations between
individual policy instruments. An important relation is
precondition relation, relating to a situation
whereby the successful implementation of one policy instrument
entirely depends upon the prior
successful implementation of another. In other words, a policy
instrument will only work if its
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
24
preconditions have been implemented (Givoni et al., 2013;
Taeihagh et al., forthcoming). In this respect
it is important to identify, when characterising and evaluating
a policy instrument whether it has any
preconditions.
From the above, we can discern several guiding questions
relating to the implementation feasibility of
policy instruments. These questions include:
1. What is the cost of implementing the policy measure?
2. Are there sufficient financial resources to support the
implementation of policy instruments? Can
funds be raised or borrowed (e.g. from the private sector)?
3. Is the implementation of the policy instrument technically
achievable? Is the instrument relatively
easy to implement in practice?
4. Does the instrument require the prior implementation of other
policy instruments (pre-conditions)?
5. Is technological innovation required for the implementation
of a policy instrument? If so, are there
barriers and time delays before the innovation can be used?
3.2.2 Institutional settings
The implementabiltiy of policy instruments is influenced by the
institutional settings within which policy
problems are defined, instruments are chosen and designed, and
later implemented. In the context of
the SPREE project, we adopt a definition of institutions which
conceives these as 'organizations that are
expected to initiate policy actions' (Gysen et al., 2006: 99).
Institutions in this context refer to the public
organisations within which publicly-elected officials and
politicians, bureaucrats and administrators
operate. These aspects were not well addressed in the public
policy literature until recently (Parsons,
1995: 158), but are detrimental to the implementabiltiy of
policy instruments to promote servicizing.
Particularly, institutional settings are thought to prohibit
policymakers from taking action in new
directions and departing from the status-quo and policymaking
norms (McDonnell and Elmore, 1987:
146).
Therefore, when evaluating the implementabiltiy of policy
instruments, we need to carefully examine
the institutional settings within which these instruments are
implemented. This examination can be
multi-levelled where necessary, and include the evaluation of
local, national, regional and international
institutions, as appropriate. Furthermore, questions of
coordination among vertical and horizontal
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
25
levels of government (and governance) need to be addressed here
(see e.g. Jordan and Schout, 2006).
Some guiding questions that can be asked in this context
include:
1. Who are the main institutional actors?
2. At what spatial level do these actors operate in and what
level(s) they have influence on or control
over (on scale from local to global)?
3. Do the institutional settings and culture facilitate or act
as a barrier to implementation?
4. Does coordination among horizontal and vertical levels of
government required?
5. Are institutional actors held accountable for the
implementation?
3.2.3 Political legitimacy
Questions of political legitimacy are important at all stages of
the policy cycle, and affect the
implementabiltiy of policy instruments. Foremost, under this
heading we address issues of democratic
legitimacy. Schmidt (2013) conveniently distinguishes between
output legitimacy, input legitimacy, and
throughput legitimacy, giving examples of EU governance. In her
conception, output legitimacy refers to
the effectiveness of policy instruments for the people. This
notion relates to social acceptability,
discussed in more detail in the following section. Input
legitimacy refers to the 'responsiveness to citizen
concerns as a result of participation by the people' (Schmidt,
2013: 2, emphasis in original). Throughput
legitimacy refers to the efficacy, accountability, inclusiveness
and transparency of the policymaking
process. Schmidt's distinction addresses the main criteria for
evaluating political legitimacy, namely -
the accountability of politicians, citizen participation, and
transparency (Huitema et al., 2011: 5).
To these criteria we add the consideration of the relative power
of policy actors vis--vis one another.
This evaluation of power relations can shed light on the
democratic legitimacy of various policy
instruments. Issues of equity and distributional effects also
fall under this category, as does the legality
of policy instruments. Thus, in evaluating the political
legitimacy of policy instruments, we need to
address the following questions:
1. Are policymakers held accountable for the implementation of
the policy? Is corruption mitigated?
2. Is transparency and provision of information promoted?
3. Is public participation safeguarded?
4. Who are the main policy actors? What is their relative power
vis--vis each other? (power analysis).
Do certain actors have greater lobbying, and even
veto-power?
5. How are the costs and benefits associated with the policy
instrument distributed between policy
actors? Is the distribution of these impacts equitable?
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
26
6. Do any of these policy actors have veto power over
implementation? Have they acted to
promote/hinder the uptake of the policy instrument?
7. Is legitimacy ensured? Have public and private policy actors
accepted the policy instrument (See e.g.
Mickwitz, 2003: 427).
In order to answer questions of political legitimacy, we need to
consult a range of sources. Often non-
governmental organisations and lobby groups provide good
information and critique of these aspects of
political legitimacy. Other sources that may be consulted
include mass-media, policymakers and policy
experts.
3.2.4 Social acceptability
The issue of social acceptability constitutes an important
evaluation criterion, outlined here and
discussed in greater detail later in this project. Definitions
of acceptability abound, yet there is no
consensus as to the meaning of this term. Perhaps the most
concise definition to date stems from the
work of Vlassenroot and his colleagues (2010). The authors make
an important distinction between the
terms acceptability and acceptance, based on time and experience
of the individual. In their view,
[A]cceptance entails beliefs and attitudes, based on their
behavioral reactions after the introduction of a measure.
Acceptability describes the prospective judgment, based on
attitudes and beliefs about a measure, without experience, to be
introduced in the future (Vlassenroot et al., 2010: 176, emphasis
added).
In this view, acceptance can be understood as an ex-post
evaluation criterion, while acceptability can be
perceived as an ex-ante criterion. However, we wish to diverge
from this understanding, and define
acceptability as a criterion for both ex-post and ex-ante
evaluation.
The evaluation of social acceptability naturally correlates with
other evaluation criteria suggested in this
deliverable. For example, there is a correlation between
acceptability and impacts such as fairness and
equity (Bristow et al. 2010, Cools et al. 2011, Eriksson et al.
2006 and 2008, Schade and Schlag 2000 and
2003). The measurement of acceptability is often based on
qualitative surveys or user choice
preference, coupled with a quantitative, statistical analysis of
collected data. The most popular model
for measuring acceptability was expressed by asking participants
to evaluate and choose between
different policy scenarios or measures. The question To what
extent are you in favor of or against the
implementation of this policy measure/policy package? (rated on
a scale from -3 to 3) (Eriksson et al.,
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
27
2008: 1122) seems a useful starting point for the evaluation of
acceptability. However, when evaluating
social acceptability we will need to ascertain and refine this
question.
3.2.5 Flexibility & Reflexivity
Policy evaluation takes place in increasingly complex and
uncertain environments (Mickwitz, 2003). As
such, evaluation needs to ascertain whether policy instruments
are flexible enough to address these
dynamic conditions, and the often-conflicting interests of a
multitude of policy actors. In this context,
the last few decades have seen a growing advocacy of more
flexible policy instruments. This trend is
perhaps most evident in the field of environmental policy, where
"new environmental policy
instruments", which allow policy actors greater flexibility in
complying with environmental regulations,
have been strongly promoted (Jordan et al., 2003). For example,
various economic policy instruments,
including permit and emission trading schemes have allowed
economic actors greater flexibility in
complying with environmental legislation.
When addressing uncertainty, we need to examine whether the
precautionary principle has been
applied, whether the instrument is adaptable, and whether it
addresses possible economic,
environmental and social changes.
To this, we add the consideration of reflexivity. When dealing
with scientific, social, environmental and
economic uncertainties, policy evaluations need to ascertain
whether a policy instrument and policy
goal can be redefined in accordance with changing knowledge and
background conditions. Reflexivity
can therefore by defined as 'willingness to question formal
policy goals' (Huitema et al., 2011: 15). This
criterion, Huitema and his colleagues argue, relates more to the
evaluation procedure per-se, rather
than to the evaluated instrument. Therefore, when conducting our
evaluation, we need to evaluate not
only the implementation and impact of the policy instrument, but
also question whether the proposed
policy goal promotes servicizing and ultimately a
resource-efficient economy.
In this context, questions to be asked include:
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
28
1. Can the policy instrument be revised and adapted to changing
and uncertain conditions (see
Mickwitz, 2003: 426)?
2. How does the policy instrument address uncertainty and
unpredictability?
3. Does the implementation of the policy instrument can
increase, or result in, lock-in effects and
irreversibility effects?
4. Are instruments flexible enough to encourage lower compliance
costs? (Jaffe et al., 2005: 165).
5. Is there a reflexive evaluation of policy goals? Do the
policy goals promote a servicised, resource-
efficient economy? (Huitema et al. 2011)
These are only some tentative questions that we are interested
in addressing in the SPREE project. We
hope that the questions listed in this section will stimulate
debate among the project partners, and
inform the methodological approach of this project, as well as
the choice of methods.
4. Summary and next steps
In this deliverable, we introduced some current debates and
trends in the public policy evaluation
literature. We noted that largely quantitative research methods
that were advocated in the 1960s and
1970s were later replaced by calls for the employment of
constructivist, qualitative research methods.
Increasingly, these approaches are combined, and a mix of
quantitative and qualitative research is often
encouraged in the public policy evaluation literature. This is
also the approach adopted in the SPREE
project.
In this light, we propose a methodological framework to evaluate
policy instruments that promote
servicizing. We distinguish between two dimensions in the
evaluation. The first refers to the social,
economic, environmental and decoupling impacts of these
instruments the effectiveness of the policy
instruments. The second dimension examines the implementabiltiy
of these instruments in terms of
their feasibility, institutional constraints, political
legitimacy, social acceptability, flexibility and
reflexivity. The outcome of the framework proposed is a list of
guiding questions to inform the
evaluation of individual policy instruments later in this
project. This list of questions is summarised in
Table 5.
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
29
Table 5: Summary of evaluation methodology for the SPREE
project
Evaluation
dimension
Criterion Summary of guiding questions
1. Impact
Economic 1. What are the implications for economic growth?
2. What are the consequences for employment?
3. What are the costs and benefits to service providers,
consumers
and the chain/sector level?
Environmental 1. What are the impacts in terms of resource-use
efficiency?
2. What are the environmental impacts, in terms of air water
and
soil quality, and climate change?
Social 1. Are there observable objective changes in quality of
life (e.g.
employment, wages, environmental quality)
2. Are there subjective changes in well-being and happiness?
3. Is the distribution of social impacts equitable?
Decoupling 1. Does continued economic growth result in overall
reduction in
negative environmental impact?
2. Are there any observable positive relational effect among
economic and environmental impacts?
3. Can social impacts be associated with any observable
decoupling
trends?
2. Implementabiltiy Feasibility 1. Is implementation financially
viable?
2. Is implementation technically feasible?
3. Are there any necessary preconditions for implementation?
4. Are technological solutions in place/needed for
implementation?
Institutional
settings
1. Are there institutional barriers/enablers to
implementation?
2. What spatial scales do these institutions operate on?
3. Is there vertical and horizontal coordination among these
institutions?
4. Are these institutions held accountable?
Political
legitimacy
1. Are politicians held accountable?
2. Has public consultation and participation been promoted?
3. Is transparency guaranteed
4. How is power distributed among policy actors?
5. Is the policy equitable?
Social
acceptability
1. Is the policy instrument accepted by the general public?
Flexibility 1. Is instrument flexible enough to address
uncertainties?
2. Can it be modified in light of any unexpected changes?
3. Is there reflexivity in evaluating the policy goals?
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
30
4.1 Using the evaluation framework in the SPREE project
The analysis in the SPREE project essentially compares between
three systems: one based on the selling
of products, one based on selling of services, and one where the
selling of services is supported with
various policy instruments (through the implementation of policy
packages). Such comparison will be
within the mobility, agri-food, and water sectors and through
specific case studies chosen in each
sector. The 'impact' dimension of the evaluation (section 3.1)
can be used here to guide the comparison
between the three states.
The methodological framework developed and described in this
deliverable is however aimed at
evaluating policy instruments to assess the degree to which they
can support the transition from a
product system to a service system and evaluate how they can
favourably alter the impacts (economic,
environmental and social) of the service system to facilitate
absolute decoupling. This requires attention
also to the implementability of the policy instruments.
Moreover, it is generally accepted, also in the
SPREE project, that the implementation of individual policy
instruments in isolation from one another is
unlikely to result in achieving the defined policy goals -
resource efficient economy achieved through a
shift to servicizing. To increase the likelihood of achieving
the defined policy goals, policy packages must
be designed and implemented. To create policy packages, the
characteristics of each policy instrument
proposed and considered must be fully understood before it is
selected for inclusion in a package. This is
done by applying the methodological framework described above.
To apply the framework in practice,
the list of guiding questions need to be transformed into
measurable indicators and data on these
indicators need to be collected.
The methodological framework will be used to inform both
quantitative and qualitative evaluation
within the SPREE project. Qualitative evaluation will be based
on the expertise within the SPREE
partners and by involving various stakeholders in the SPREE
projects, for example through several
workshops. A central element in the SPREE project is the
development of an Agent Based Modelling
(ABM) tool for quantitative analysis of servicizing and its
impacts, especially on decoupling. This tool
will also be used for a quantitative evaluation of policy
instruments and policy packages. In this respect,
the methodological framework described in this deliverable will
inform the development of the ABM to
allow the evaluation of policy packages. This requires that the
criteria listed in Table 6 will be included
in the ontology which is the basis for developing the ABM. For
example, when evaluating policy
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
31
instruments or packages, we would like to test if the
implementation of a policy instrument by the
agent 'government' affected the economic outcome of each
'consumer' and/or 'business' agent and to
what degree, in order to assess the effect on the system as a
whole. This effect of the policy instrument
depends also on its successful implementation, the likelihood of
which should ideally be also simulated
and examined using the ABM and based on the criteria outlined in
Table 6. Similarly, when evaluating
whether absolute decoupling has been achieved in the modelled
system, we would like to see if the
policy packages considered increase or decrease decoupling by
recording changes in the relevant
indicators that will represent the decoupling criterion.
The design of a policy package starts by considering many and
wide range of policy instruments that
could potentially promote servicizing and influence the impacts
of servicizing systems. The evaluation
framework proposed here will be used to assemble an inventory of
around 100 or more measures.
Based on the same evaluation framework and using qualitative
evaluation methods the most promising
30 or so instruments will be selected, and might be assembled
into a policy package. At this point
quantitative evaluation methods will be employed and an analysis
of the instruments and/or policy
packages will be carried out using the ABM. It is envisaged that
the process will end with returning to
use qualitative evaluation methods in the final assessment of
'Servicizing Policy Packages' in the three
sectors.
This document has encapsulated the crucial criteria for
evaluating policy instruments, and eventually
promoting a shift towards a resource-efficient, service-based
system. These criteria and the
methodological framework developed are the starting point for
assessing and promoting servicizing.
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
32
References
Abdallah, S., Mahony, S., Marks, N., Michaelson, J., Seaford,
C., Stoll, L., & Thompson, S, 2011 Measuring
our progress: The power of well-being, London, New Economics
Foundation
Arp, H, 2002 "Technical regulation of politics: the interplay
between economic interests and environmental policy goals in EC car
emission legislation", in: Jordan, A (ed), Environmental Policy in
the European Union: Actors, Institutions and Processes, London,
Earthscan, pp. 256-274.
BIO Intelligence Service, Institute for Social Ecology and
Sustainable Europe Research Institute, 2012, Assessment of resource
efficiency indicators and targets, Final report prepared for the
European Commission, DG Environment, available at:
http://ec.europa.eu/environment/enveco/resource_efficiency/pdf/report.pdf,
accessed: 12.2.2013
Bristow A, Wardman M L, Zanni A M, Chintakayala P K, 2010,
"Public Acceptability of Personal Carbon
Trading and Carbon Tax", Ecological Economics, 69, pp.
1824-1837
Cools, M., Brijs, K., Tormans, H., Moons, E., Janssens, D.,
Wets., G, 2011, "The Socio-cognitive Links
between Road Pricing Acceptability and Changes in
Travel-behavior", Transportation Research A, 45, pp.
779-88
Diener, E, 2005, "Guidelines for national indicators of
subjective well-being and ill-being", Social indictors network
news, 84, 4-6. Eliadis F P, Hill M M, Howlett M, 2005,
"Intorduction", in Designing government: from instruments to
governance Eds F P Eliadis, M M Hill, M Howlett, McGill-Queen's
University Press, Montreal, pp 3-18 Eriksson L, Garvill J, Norlund
A, 2006, Acceptability of travel demand management measures: the
importance of problem awareness, personal norm, freedom, and
fairness Journal of Environmental Psychology, 26, pp. 15-26
Eriksson L, Garvill J, Norlund A, 2008, "Acceptability of single
and combined transport policy measures: the importance of
environmental and policy specific beliefs" Transport Research Part
A, 42, pp. 1117-128 European Commission 2012 Life cycle indicators
for resources: development of life cycle based macro-level
monitoring indicators for resources, products and waste for the
EU-27. European Commission, Joint Research Centre, Institute for
Environment and Sustainability, available at:
http://lct.jrc.ec.europa.eu/pdf-directory/LBNA25517ENN.pdf ],
accessed, 12.2.2013
Fischer F, 1995, Evaluating public policy, Nelson-Hall
Publishers, Chicago
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
33
Forjaz, M. J., Prieto-Flores, M. E., Ayala, A.,
Rodriguez-Blazquez, C., Fernandez-Mayoralas, G., Rojo-
Perez, F., & Martinez-Martin, P, 2011, "Measurment
properties of the Community Wellbeing Index in
older adults" Quality of life research, 20(5), 733-743
Gerike R, Gehlert T, Richter F, Schmidt W, 2008, "Think
Globally, Act Locally - Reducing Environmental
Impacts of Transport" European Transport 38, pp. 61-84
Givoni M, Macmillen J, Banister D, Feitelson E, 2013 "From
policy measures to policy packaging"
Transport Reviews (in press) Guba, E G., Lincoln, Y S, 1989,
Fourth generation evaluation, Sage Publications, London Gysen J,
Bruyninckx H, Bachus K, 2006, "The Modus Narrandi A Methodology for
Evaluating Effects of Environmental Policy" Evaluation 12 95-118
Hall, J, Giovannini, E., Morrone, A., & Ranuzzi, G, 2010, A
framework to measure the progress of societies, OECD Statistics
Working Papers. Paris: OECD Publishing Hill M, 2013, The public
policy process sixth edition (Pearson, Harlow) Hirschl, B, Konrad,
W, 2001, New concepts in product use for sustainable consumption. 7
European Roundtable for Cleaner Production, Lund, International
Institute for Industrial Environmental Economics at Lund
University.
Hirschl, B, Konrad W, 2003, "New concepts in product use for
sustainable consumption" Journal of Cleaner Production 11(8):
873-881.
Howlett M, 2011 Designing public policies: principles and
instruments (Routledge, London) Howlett M, Ramesh M, Perl A, 2009
Studying public policy: policy cycles and policy subsystems, Oxford
University Press, Oxford Huitema, D, Jordan, A, Massey, E, Rayner,
T, van Asselt, H, Haug, C, Hildingsson, R, Monni, S, Stripple, J,
2009, The evaluation of climate policy: theory and emerging
practice in Europe, Policy Science, 44, pp. 179-198 Jaffe, A B,
Newell, R G, Stavins, R N, 2005, A tale of two market failures:
technology and environmental policy, Ecological Economics, 54, pp.
164-174
Jordan, A, Benson, D et al. 2011, "Policy instruments in
practice" The Oxford handbook of climate change and society, J. S.
Dryzek, R. B. Norgaard and D. Schlosberg (eds), Oxford, Oxford
University Press, pp. 536-549. Kooiman J, 1993, "Social-political
governance: introduction", in Modern governance: new
government-society interactions Ed J Kooiman, SAGE, London, pp
1-19
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
34
Marsh D, McConnell A, 2010, Towards a framework for establishing
policy success, Public Administration,88(2), pp. 564-583
Landry R, Varone F, 2005, The choice of policy instruments:
confronting the deductive and the interactive approaches, in
Eliadis F P, Hill M M, Howlett M, Designing government: from
instruments to governance, McGill-Queen's University Press,
Montreal, pp. 106-131 McDonnell, L. M. and R. F. Elmore,
1987,"Getting the Job Done: Alternative Policy Instruments"
Educational Evaluation and Policy Analysis, 9(2), pp. 133-152
Michaelson, J., Mahony, S., & Schifferes, J, 2012, Measuring
Well-being, New Economics Foundation, London Mickwitz P, 2003, "A
framework for evaluating environmental policy instruments: context
and key concepts", Evaluation, 9(4), pp. 415-436
Mickwitz P, Birnbaum M, 2009, Key insights for the design of
environmental evaluations, in M. Birnbaum & P. Mickwitz (Eds.),
Environmental program and policy evaluation: Addressing
methodological challenges. New Directions for Evaluation, 122,
105112. Morrison, P S, 2010, "Local expressions of subjective
well-being: The New Zealand experience" Regional Studies, 45(8),
1039-1058 Palumbo D J, 1987, The politics of program evaluation,
Sage, London Parkin J, Wardman M, Page M, 2007, "Models of
perceived cycling risk and route acceptability" Accident Analysis
and Perception, 39, pp. 364-371 Parsons D W, 1995 Public policy: an
introduction to the theory and practice of policy analysis, Edward
Elgar, Aldershot Persson A, Nilsson M, 2006, Towards a framework
for ex post SEA: theoretical issues and lessons from policy
evaluation, in Emmelin L (ed), Effective environmental assessment
tools critical reflections on concepts and practice, Blekinge
Institute of Technology,Research Report No 2006:03, Report No 1
from the MiSt-programme Rist R C, 1995, Choosing the right policy
instrument at the right time: the contextual challenges of
selection and implementation, in Belmanns-Videc M L, Rist R C,
Vedung E (eds), Carrots, sticks & sermons policy instruments
& their evaluation, Transaction Publishers, London Salamon L M,
2002, "The new governance and the tools of public action: an
introduction", in The tools of government: a guide to the new
governance Eds L M Salamon, O V Elliott, Oxford University Press,
Oxford, pp 1-47
-
FP7 PROJECT: SPREE
SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY
35
Schade J, Schlag B, 2000, Acceptability of Urban Transport
Pricing .Rep. no.PL97-2258, European
Commission 4th Framework - Transport RTD, AFFORD
Schade J, Schlag B, 2003, "Acceptability of Urban Transport
Pricing Strategies, Transportation Research F, 6, pp. 45-61 Schmidt
V A, 2013, "Democracy and legitimacy in the European Union
revisited: input, output and throughput" Political Studies, 61, pp.
2-22 Schmidt-Bleek, F. and F. Lehner, 1998, Das MIPS-Konzept.
Weniger Naturverbrauch - mehr Lebensqualitt durch Faktor 10 (The
MIPS-Concept. Fewer consumption of natural resources - more quality
of life through Factor 10), Munich, Droemer
Sperling, D., S. Shaheen, et al., 2000, Carsharing - Niche
market or new pathway? Berkeley, University of California
Stahel, W, 1997, "The Service Economy: 'Wealth without Resource
Consumption'?", Philosophical Transactions A, Royal Society, London
355(1728), pp. 1309-1319.
Stavins, R N 1995, "Transaction costs and tradable permits",
Journal of Environmental Economics and Management, 29, pp.
133-148
Taeihagh A., Givoni M., Baares-Alcntara