November 2015 Evaluation policy 2015–2020
November 2015
Evaluation policy 2015–2020
2 / Evaluation policy 2015–2020
© Commonwealth of Australia, 2015.
Evaluation policy 2015–2020 is licensed by the Commonwealth of Australia for use under a Creative Commons By Attribution 4.0 Australia licence with the exception of the Coat of Arms of the Commonwealth of Australia, the logo of the agency responsible for publishing the report, content supplied by third parties, and any images depicting people. For licence conditions see: http://creativecommons.org/licenses/by/4.0/
This report should be attributed as ‘Evaluation policy 2015–2020, Commonwealth of Australia 2015’.
The Commonwealth of Australia has made all reasonable efforts to identify content supplied by third parties using the following format ‘© Copyright, [name of third party] ’.
Image credits: (Front cover, L–R) Snow pit sampling (Alison McMorrow), Iron grass area (Andrew Tatnell), Water testing (Dragi Markovic), Lord Howe Island diving (Paradise Ink),Water level measurement (Mark Mohell), Western Swamp Tortoise (Mark Mohell).
(Back cover, L–R) Water quality sampling (Michael Marriott) © Michael Marriott and Department of the Environment, Fuel testing (Michelle McAulay), Water quality testing (Dragi Markovic), Monitoring groundwater depth using three nested piezometers (Dragi Markovic), Suburban Jobs Program (Andrew Tatnell), Vegetation survey (John Baker) © John Baker and Department of the Environment.
Title Evaluation policy 2015–2020 Version 1.0
Date of effect 1 November 2015 Current to December 2020
Endorsed by Executive Board
Business owner Evaluation team, Policy Analysis and Implementation Division (PAID)
Contact [email protected]
3
Contents
Overview 4Why is evaluation important? 4
Performance monitoring and periodic evaluations 4
How to use this policy 5
Section 1: Evaluating policies, programmes and regulations 7The Department’s evaluation principles 7
Planning monitoring and evaluation 8
Performance monitoring 8
Periodic evaluations 9
Section 2: Evaluating departmental performance 11Activity performance plans 12
Biannual performance report 13
Annual performance statement 13
Section 3: Implementation of this policy 14Implementation and evaluation 14
Governance 14
Evaluation support services 15
4 / Evaluation policy 2015–2020
OverviewEvaluation is the systematic assessment of the efficiency, effectiveness and appropriateness of policy, programme or regulatory interventions.
Put simply, it is about reflecting on what we do, why we do it, whether we are achieving what we set out to, and whether there are better ways of doing it.
The Evaluation Policy 2015–2020 sets out the Department of the Environment’s approach to evaluation. Through implementing this policy, we aim to improve the efficiency and effectiveness of our interventions.
Why is evaluation important?The Department is committed to evaluating its policies, programmes and regulations to help continuously improve what we do and demonstrate our accountability.
> Adaptive management: Effective evaluation can help demonstrate whether interventions are achieving their objectives and identify areas that need improvement. It provides critical intelligence that helps to identify and manage risks, make evidence-based decisions, guide resource allocations, and ultimately improve social, economic and environmental outcomes.
> Accountability: Evaluation forms the basis for demonstrating the Department’s accountability to the Minister, Government, Parliament and the Australian public by assessing the extent to which we have efficiently and effectively achieved our purposes.
In testing new ideas and learning what works and what doesn’t, evaluation also contributes to the evidence base for future policy development.
Performance monitoring and periodic evaluationsEvaluation involves two distinct but closely related activities: performance monitoring and periodic evaluations. Both of these activities are concerned with assessing the performance of government interventions; the key differences are when they are undertaken and the depth of analysis.
> Performance monitoring occurs on a regular basis (such as monthly or annually) and involves collecting quantitative and/or qualitative data and reporting against performance measures. Performance measures should be chosen to be relevant to, and inform, decision making.
> Periodic evaluations provide a deeper analysis of the effectiveness, efficiency and appropriateness of an intervention. Evaluations draw on performance monitoring results and generally involve additional targeted data collection, including through consultation with community and business groups.
5
It is important that evaluation requirements are considered early in the design phase for policies, programmes and regulations, so that data collection processes can be established. Typical data collection methods, which support performance monitoring and evaluations, include:
• Direct measurement—Measuring changes in the environment, such as temperature, air quality or vegetation condition.
• Data analysis—Extracting quantitative data from existing datasets to analyse relationships between variables (with methods such as regression analysis).
• Surveys—Collecting information on community and business views or attitudes.
• Benchmarking—Comparing performance on key dimensions with similar activities.
• Peer reviews—Performance assessments undertaken by experts or industry or community leaders.
How to use this policyThe policy outlines the Department’s approach to measuring performance and evaluation, provides advice on planning, scaling, scheduling and transparency, using and communicating findings, and sets requirements for “significant interventions”.
Section 1: Evaluating policies, programmes and regulations
• Creating a programme logic model
• Developing monitoring and evaluation plans
• Monitoring performance through developing effective performance measures
• Planning and conducting a periodic evaluation
Section 2: Evaluating Departmental Performance
• Implementing the requirements of the Public Governance, Performance and Accountability Act 2013
• Preparing activity performance plans
• Setting departmental performance measures for the Portfolio Budget Statements and Corporate Plan
• Reporting against performance measures for the biannual performance report and Annual Performance Statement
Section 3: Implementation of this policy
• Implementation and evaluation of the policy
• Governance of evaluation within the Department
6 / Evaluation policy 2015–2020
Figure 1: Scope of the Evaluation Policy
Outcome (e.g. Outcome 1:
“Conserve, protect and sustainably
manage Australia’s biodiversity...”)
Programme (e.g. Programme 1.1: Sustainable Management of
Natural Resources and the
Environment)
Performance planning Overarching strategy Streamlined programme objectives Performance measures (deliverables, output and outcome indicators) Schedule of planned evaluations
Intervention (e.g. Green Army Programme, National
Landcare Programme, Reef 2050, 20 Million Trees, etc.)
Purpose (“Conserve, protect
and sustainably manage Australia’s
terrestrial and marine
biodiversity...”)
Project (e.g. 20 Million Trees Grant Round 1,
Green Army mid-term evaluation)
Enhanced Commonwealth Performance Framework (Dept of Finance)
Key components: PBS and Corporate Plan Annual Performance Statements New performance methodology
Evaluating departmental performance (SECTION 2)
Departmental activity performance planning, evaluation and reporting (DAPPER) framework
Key components: Activity performance plans Biannual performance report Annual performance statement
Implementation of the policy (SECTION 3)
Key components: Implementation and evaluation Governance Evaluation support services
Evaluating policies, programmes and regulation (SECTION 1)
Key components: Evaluation principles Planning for evaluation Performance monitoring Periodic evaluations
External Internal
Activity (e.g. Sustainable management of
natural resources )
PBS Corporate Plan
7
Section 1: Evaluating policies, programmes and regulationsThe Department is committed to evaluating our approaches to continually improve how we work and ensure accountability.
While there is no one-size-fits-all approach to evaluation and it should be tailored to the intervention in question, there are better practices that should be followed, particularly when evaluating significant policy, programme or regulatory interventions.1
This section sets out principles and guidelines for ensuring we undertake appropriate and effective evaluation, and to help us adapt and learn.
1 For the purposes of this policy a significant intervention is defined as: a policy/strategy that has national scope; a programme with a budget of greater than $5 million over its life; or a regulation with a Regulatory Burden Measure costing of greater than $2 million.
2 The Institute of Public Administration Australia’s Policy Professional Capability Standard identifies “Design, Implement and Evaluate” as one of five key policy capability domains.
The Department’s evaluation principlesThese seven principles underpin evaluation in the Department:
1) Integrated: Evaluation is part of our core business, not an ad hoc or compliance activity.2 Evaluation plans should be integrated into the design of new interventions, and evaluation should inform policy decisions. Consistent performance metrics establish clear connections between evaluation at the departmental, divisional and team levels.
2) Fit-for-purpose: The scale of effort and resources devoted to evaluation should be proportional to the value, impact and risk profile of the intervention. Significant interventions should be subject to comprehensive independent evaluation.
3) Timely: Periodic evaluations should be timed to support and inform the decision-making process. Monitoring should occur at a frequency that allows officers and managers to adjust policy settings in response to performance feedback.
4) Evidence-based: Departmental evaluation should rely on the best available data and employ robust research and analytical methods. Baseline measurements should be taken and performance measures should remain consistent to allow trend data to be collected. Periodic evaluations should aim to establish the impact of an intervention.
5) Independent: Evaluation should have an appropriate degree of independence from the responsible policy area. Evaluation should be undertaken by an independent party, or at least by an evaluation team that includes independent members.
6) Transparent: The Department’s default position is that the processes and results of evaluation should be widely communicated internally and made publicly available in a timely manner to strengthen confidence and understanding in the allocation of public resources.
7) Outcomes-focussed: While evaluation should look at how much we did and how well we did it, greater effort should be made to evaluate what difference we are making and whether conditions are improving. Measuring long-term environmental outcomes can be challenging, so it is often best to also measure intermediate or proxy outcomes.
8 / Evaluation policy 2015–2020
Adaptive management
The evidence generated through monitoring and evaluation activities is relevant at all stages of policy development. The model outlined in Figure 2 of integrating evaluation into the way we develop, implement and manage policy, programme and regulatory interventions should underpin the way we work.
While the requirements in this section apply to significant interventions, evaluation practices can be applied to smaller scale projects and at a sub-intervention level. For example, proponents could be required to apply the principles of adaptive management to scoping and monitoring environmental projects for natural resource management grants or regulatory offsets.
Planning monitoring and evaluationEvaluation requirements should be considered during the design and planning phases of a policy, programme or regulatory intervention. A robust evaluation planning process ensures you are well set up to collect the data necessary to refine and adapt your intervention and demonstrate its impact and value. If planning is delayed, evaluation activities can be complicated, costly and less effective.
A key first step in planning monitoring and evaluation is to develop a programme logic model. Programme logic a simple, visual method of displaying, communicating and testing the assumed causal links between the intended short-term, intermediate and long-term outcomes of an intervention.
• Guides to Developing a programme logic model and Planning monitoring and evaluation can help you set up an effective monitoring system, scope and schedule periodic evaluations, and decide the resources you need for evaluation.
• If the outcomes of a policy, programme or regulation are uncertain, consider running a policy trial. The Department’s Behavioural Insights team in PAID can provide assistance.
Significant intervention requirements
� Use the Department’s Monitoring and evaluation plan template.3
� Enter planned periodic evaluations into the Departmental Evaluation Schedule.
Performance monitoringPerformance monitoring involves collecting quantitative and qualitative data and reporting against predetermined performance measures on a regular basis to inform decision making and ensure accountability. A set of measures should be developed covering each of the following four questions:
1) How much did we do? (quantity)—e.g. number of grants awarded
2) How well did we do it? (quality)—e.g. timeliness of assessments
3) What difference did we make? (impact)—e.g. environmental improvements in project areas compared to baseline
4) Are conditions improving? (context/long-term outcome)—e.g. regional/national trends in environmental conditions
EVIDENCE Collect and analyse data
Develop decision support tools
DESIGN Understand the problem Identify policy options
Evaluate benefits and costs Choose best option
PLAN Programme logic Monitoring and evaluation plan Key evaluation
questions Performance
measures
IMPLEMENT Monitor and report against performance
measures Refine intervention
ADAPT Conduct periodic
evaluations Communicate
findings Continuously
adapt intervention
3 For smaller-scale projects, the evaluation components embedded in the project management framework should meet most needs.
Figure 2: Stylised evaluation and policy cycle
9
Figure 3: Types of performance measures
• Make use of existing data sources where possible, and try to use metrics that are consistent with other interventions in your policy area so results can be aggregated and reported at a higher level. Where data gaps exist for important policy questions, consideration should be given at the earliest stage to data collection.
• The Planning monitoring and evaluation guide provides advice on how to develop a set of performance indicators for regular monitoring.
Significant intervention requirements
� As part of your Monitoring and Evaluation plan, develop a set of indicators addressing the four questions listed above (output, quality, impact and context) and monitor and report against them at agreed intervals.
Periodic evaluationsPeriodic evaluations offer an opportunity to go beyond regular performance monitoring to consider deeper questions about how well a policy, programme or regulation is performing and why. There are two main purposes of periodic evaluations: formative and summative.
> Formative evaluations occur in the earlier stages of an intervention (usually a couple of years in) and focus on assessing how an intervention is working and what refinements need to be made.
> Summative evaluations are conducted towards the end of an intervention, both to document its achievements for accountability purposes and to inform future policy decisions about successor interventions.
Quantity
How much did we do? How well did we do it?
What difference did we make?
Perf
orm
ance
mea
sure
s
Are conditions improving?
% %
%
% %
# # #
# #
%
Quality
Impact Context (Long-term outcome)
Output indicators
Outcom
e indicators
10 / Evaluation policy 2015–2020
At a minimum, a formative evaluation should be undertaken a few years into an intervention and a summative evaluation towards the end of the intervention.
Evaluations are generally run as in-depth research projects and can take several months to complete. They require the establishment of a small evaluation team, which will often include external experts. In contrast to monitoring, evaluations involve
additional targeted data collection and analysis, and, importantly, consultation with community and business groups affected by the intervention.
In applying the Department’s evaluation principles consideration should be given to scaling and prioritising evaluation proportionate to the size, significance or risk profile of the intervention (see Table 1 below for guidance on scaling evaluations).
Cha
ract
eris
tics
of
inte
rven
tion
Low risk
Limited resources
Similar to previous successful projects
Not widely publicised
Low strategic priority
Medium risk
New or untried project that requires testing and/or stakeholder input
Moderate public profile and expectations
Moderate political significance
High risk
Resource intensive
Controversial
High public profile and expectations
Politically significant
Like
ly c
hara
cter
isti
cs
of e
valu
atio
n
Informal process
Limited data requirements
Low resource allocation
Can be completed internally
Limited consultation
Low key release
Greater level of data collection and analysis
Multiple evaluation points during development and implementation
Regular progress reporting
May involve external evaluators
Formal process
High resource allocation
External evaluators
Extensive consultation
Central agency involvement
Wide public release
Table 1: Scaling evaluation to intervention characteristics
Evaluation findings should be communicated widely, except where there are overriding “public interest” concerns against disclosure. Externally facing evaluation reports should be “digital by design” and link directly to data sources where possible.
The Evaluation team in Policy Analysis and Implementation Division (PAID) maintains a library of evaluation documents on the intranet (the Evaluation Document Library). Staff are encouraged to communicate evaluation findings by delivering presentations internally and externally, including through the Evaluation Community of Practice.
To improve our performance and share learnings, results from periodic evaluations of significant policies, programmes and regulations are reported internally to the Executive Board twice a year through the activity performance report (see Section 2: Evaluating departmental performance). Findings may also be
reported externally to the Minister and Parliament and publicly through annual performance statements.
• The Conducting periodic evaluation guide provides advice on how to plan, manage and conduct a periodic evaluation.
Significant intervention requirements
� Consult the Evaluation team in PAID prior to finalising the evaluation brief (or terms of reference).
� Ensure the team undertaking the evaluation and the steering committee include independent members.
� Consultation with community and business groups affected by the intervention must form part of the evaluation design.
� Provide a copy of the evaluation report to the Evaluation team in PAID.
11
Section 2: Evaluating departmental performance The Evaluation Policy introduces a new departmental activity performance planning, evaluation and reporting (DAPPER) framework.
The framework, which applies at the Corporate Plan “activity” level, has been developed to support the implementation of the Department of Finance’s enhanced Commonwealth performance framework under the Public Governance Performance and Accountability Act, which came into effect on 1 July 2015.
The DAPPER framework introduces two new performance elements:
Activity performance plans: In February to April each year, the Policy Analysis and Implementation Division (PAID) will coordinate the development of activity performance plans, outlining objectives, contextual indicators, performance measures and scheduled evaluations for Corporate Plan activities. The information in the plans forms the basis for the performance information presented in the Department’s Portfolio Budget Statements and Corporate Plan.
Biannual performance report: In February and August each year, PAID will present a biannual performance report to the Executive Board, outlining results against performance measures in activity performance plans and a summary of recent evaluation outcomes. These reports will be used as the basis for preparing the Department’s Annual Performance Statement, which is due in October.
Key requirements of the enhanced Commonwealth performance framework (with references to relevant sections of the PGPA Act):
> Achievement of purposes: The Secretary must govern the Department in a way that promotes the achievement of its purposes (section 15). ‘Purposes’ are defined in the PGPA Act as the ‘objectives, functions and role’ of the Department.
> Corporate Plan: By August each year, the Department must prepare a four-year Corporate Plan (section 35). The Corporate Plan outlines the Department’s purposes, the activities it will pursue to achieve its purposes and how their success will be measured. Strategic planning and performance elements of the Corporate Plan include: the Department’s operational context, statement of purposes; planned performance; capability; and risk oversight and management.
> Performance records: The Department must measure and assess its performance in achieving its purposes (section 38), and records must be kept about its performance (section 37). The DAPPER framework has been designed to meet this requirement.
> Annual performance statements: At the end of the financial year, the Department must prepare an annual performance statement (section 39) and report on the results of planned performance outlined at the beginning of the reporting cycle in the Corporate Plan. The annual performance statement is published in the Department’s Annual Report.
> Audit of performance: At the request of the Minister or Finance Minister, the Australian National Audit Office may examine and report on the Department’s annual performance statements (section 40).
12 / Evaluation policy 2015–2020
The DAPPER framework has been designed to minimise administrative burden on the Department. Performance information developed for activity performance plans and biannual performance reports will flow directly through to the preparation of corporate documents (i.e. Portfolio Budget Statements, Corporate Plan, Annual Performance Statement), meaning divisions will only need to review performance information for accuracy rather than producing new information for multiple documents (as outlined in Figure 4 below).
At the same time, performance information developed for the DAPPER framework will cascade though division and branch plans to individual performance agreements. Staff, sections, branches and divisions are encouraged to align internal planning and performance reporting with the departmental framework, and use the evaluation approaches outlined in Section 1: Evaluating policies, programmes and regulations.
Figure 4: DAPPER cycle
A key aim of the biannual reporting cycle is that focussing on high-level, outcomes-based performance measures will encourage the establishment and maintenance of a stable set of performance indicators for the Department. Following the completion of one cycle, the biannual performance reporting process will be evaluated in late 2016 as part of the initial review of the Evaluation Policy (see Section 3: Implementation of this policy).
Activity performance plansActivity performance plans are the primary performance planning document for the Department’s policies, programmes and regulations. The information outlined in the plans forms the basis for the Department’s biannual performance report to the Executive Board and external performance reporting to Parliament through the Portfolio Budget Statements, Corporate Plan and Annual Performance Statement.
Timing: Performance planning will occur in February to April each year, prior to the finalisation of the Portfolio Budget Statements. While activity performance plans are living documents reviewed at regular intervals (particularly prior to finalising the Corporate Plan and Portfolio Additional Estimates Statements), it is expected that modifications to plans will generally be minor and only reflect significant policy changes. Performance measures should remain stable to ensure performance can be monitored consistently over time.
13
Key elements of activity performance plans
> Objectives: Intended results of an activity (and corresponding Budget programme) in contributing to the broader outcomes expressed in its purposes statement.
> Performance measures: Used to observe progress and measure actual results compared to expected results. Includes information on: methodology for data collection; relevant data sources; existing baseline data; performance targets; and timeframes.
> Scheduled evaluations: Details of any scheduled evaluations over the coming financial year and forward estimates period, including information on timing and key stakeholder groups.
Biannual performance reportPAID will coordinate a biannual performance report to the Executive Board, drawing on results reported by divisions against performance measures outlined in activity performance plans, including a summary of any evaluations undertaken over the previous half year.
The report will provide a concise yet comprehensive overview of the Department’s performance to inform decision making processes. It will also assist the Department in meeting the PGPA Act requirements to measure and assess its performance in achieving its purposes (section 38) and keep records about its performance (section 37).
Timing: The report will be compiled and presented to the Executive Board twice a year, in February and August. The February report is timed to coincide with performance planning for the following year; with the August report forming a key input into the Department’s Annual Performance Statement for the Annual Report.
Annual performance statementPAID will prepare the Department’s Annual Performance Statement based on divisional reporting through the biannual performance report. The statement will be developed in line with the requirements under the Public Governance, Performance and Accountability Rule 2014. The draft statement will be circulated to divisions for final input and clearance prior to finalisation.
Under the PGPA Act, the Departmental Audit Committee is responsible for advising the Secretary on the preparation of, and reviewing, the annual performance statements. PAID will work with the Committee to facilitate this process.
Timing: The Department’s Annual Performance Statement will be compiled from August to September and must be published with the annual report by 15 October each year.
14 / Evaluation policy 2015–2020
Section 3: Implementation of this policyThe policy comes into effect from November 2015 and will be evaluated biennially until 2020.
Implementation and evaluationBroad consultation on the draft evaluation policy was undertaken within the Department in early 2015, and a baseline evaluation of departmental evaluation capacity was conducted in mid 2015.
These evaluation activities lead to changes in the final policy; in particular, the introduction of requirements for significant interventions.
At the same time, the activity performance plan component of the DAPPER framework (outlined in Section 2: Evaluating Departmental Performance) was trialled for the 2015–16 financial year. Recommendations from the evaluation of this trial have also been incorporated into the policy.
The policy will be implemented by PAID through undertaking actions and facilitating compliance with requirements outlined in Section 1 and Section 2.
The policy will be evaluated three times.
• A formative evaluation will be undertaken by December 2016, following finalisation of the 2015–16 annual performance statements (marking the first full year of compliance with the Commonwealth performance framework).
• A mid-cycle evaluation will be conducted by December 2018.
• A summative evaluation will occur in 2020 to assess the continuing need for the policy and potentially inform a revised policy.
GovernanceGovernance of evaluation in the Department is largely decentralised, with divisions responsible for the conduct of most evaluations. Elements managed centrally are the evaluation policy and toolkit, and the DAPPER framework. Table 2 below outlines departmental governance arrangements for evaluation.
Position/group Roles and responsibilities
Secretary Promote achievement of the Department’s purposesApprove Corporate Plan and Annual Performance Statement
Executive Board Approve Evaluation Policy, Corporate Plan and Annual Performance Statement
Governance and Performance Committee
Oversee preparation of DAPPER framework outputs (activity performance plans, biannual performance report, annual performance statement)Endorse Evaluation Policy, toolkit and capacity building activities
Departmental Audit Committee Review approach to selecting performance measuresProvide independent assurance of Annual Performance Statement
Division heads Responsible for most evaluations, reporting to relevant Deputy SecretaryClear divisional input to DAPPER framework outputs (activity performance plans, biannual performance report, annual performance statement)
Policy Analysis and Implementation Division (PAID)
Develop and implement Evaluation Policy and toolkitCoordinate DAPPER framework outputs (activity performance plans, biannual performance report, annual performance statement)Facilitate capacity building and communication of evaluation outcomes
Science Division Provide advice and support for the development of data collection and analysis approaches
Table 2: Evaluation governance arrangements
15
Evaluation support servicesPAID is responsible for developing and implementing this policy and also provides support and guidance to line areas to assist them with evaluation activities. The support services PAID can provide are outlined in Table 4.
PAID is responsible for... PAID can advise line areas on...
Developing and implementing the Evaluation Policy
Coordinating the Department’s performance planning and reporting
Disseminating evaluation tools and guidance
Facilitating capacity building and communication of evaluation outcomes
Participating in evaluations of significant departmental interventions
Maintaining the evaluation library and schedule of evaluations
Working Science Division to improve the rigour of evaluation methods
Developing a programme logic model
Preparing a monitoring and evaluation plan
Developing effective performance measures
Implementing performance monitoring and reporting
Scoping and conducting a periodic evaluation
Applying the departmental evaluation principles
Table 4: Support services provided by PAID
PAID’s capacity building support includes:
> Evaluation toolkit and training: PAID has a dedicated evaluation intranet site which provides guides and tools for staff working on performance monitoring, evaluation and reporting activities. In addition, in collaboration with line areas, PAID arranges training and runs seminars on evaluation for departmental staff.
> Community of practice: The Department has an active evaluation community of practice, which is open to staff at all levels with an interest in evaluation. The community of practice meets quarterly to share evaluation outcomes, insights and intelligence. The aims of the community are to foster linkages between evaluation activities across different divisions and to build staff capability in evaluation.
environment.gov.au
CO
R36
0.09
15