Evaluation at NRCan: Information for Program Managers Strategic Evaluation Division Science & Policy Integration July 2012.

Evaluation at NRCan:Information for Program Managers

Strategic Evaluation DivisionScience & Policy Integration

July 2012

2

.

Purpose

• The purpose of this document is to provide program managers with an overview of the evaluation function at NRCan.

• The TBS Policy on Evaluation (April 2009) requires that all direct spending, including all G&C Programs, be evaluated every five years

– most program managers will find themselves participating in an evaluation at some point.

3

.

What is Evaluation?

• Evaluations are the systematic collection and analysis of evidence on the outcomes of programs to make judgments about

– their relevance;

– performance; and

– alternative ways to deliver them or to achieve the same results.”

• Evaluations must be neutral, and evidence-based.

• An evaluation is not the same as an audit.

Evaluations AuditsFocus on whether we are doing the right things, and the extent to which a program is achieving its expected outcomes, in a cost-effective manner.

Internal audit is looking at financial management, processes, controls and risk.

Make assessments on the relevance & performance of programs

Identify strengths and weaknesses in the management control framework

4

.

Why do Evaluations?

• The objective of evaluation is to create a comprehensive and reliable base of evidence to support:

– policy and program improvement;

– expenditure management;

– Cabinet decision-making; and

– public accountability.

• Evaluations are often required to support TB submissions and Memoranda to Cabinet.

• They are also a critical source of evidence for Strategic Reviews, to support resource reallocation (next review will occur in 2014).

5

.

Background: Evaluation Stakeholders

NRCan DM

Canadian Public

Program Managers TBS

Evaluation Reports

Sector ADMs

Cabinet

Strategic Review

Policy & Program Improvement

Expenditure Management/ Cabinet Decision-making

Parliament

Accountability/ Public Reporting

The challenge is to meet the diverse information needs of many stakeholders for decision-making AND public accountability in a timely fashion.

6

.

The Evaluation Cycle

Evaluation Planning

Approvals/Posting

(2-3 months)

Evaluation Assessment

(1-3 months)

Report &Recommendations

(1-4 months)

Implementing Change

Contracting(1-2 months)

Field Work/Analysis(6-8 months)

Management Responses(1 month)

Including planning and approvals/posting, large

evaluations typically take 12-18 months to complete.

Some phases may overlap.

7

.

Evaluation Planning

• NRCan must evaluate all direct program spending, including all ongoing grant and contribution programs, every five years.

• NRCan has developed a five-year Evaluation Plan based on PAA units that is updated annually and approved by the Evaluation Committee.

• In most cases, the evaluation of an individual program will be conducted within the scope of a broader evaluation of a PAA unit.

• The current plan summary appears on the Strategic Evaluation Internet site at: http://nrcan.gc.ca/evaluation/plans-eng.php.

8

.

Phases of an Evaluation

1. Evaluation Assessment (1-3 months)

– research and planning to understand the programs

– develop the Terms of Reference

– obtain approval from the Evaluation Committee

2. Contracting (1-2 months)

– Consultants are often used to supplement in-house staff. Their roles will vary by project.

3. Fieldwork or Data Collection/Analysis (6-8 months)

– develop a detailed methodology report

– methodologies: key informant interviews; focus groups; file/document/literature reviews; surveys; case studies; and data and economic analysis

– analyse information collected from these multiple lines of evidence to develop conclusions

9

.

Phases of An Evaluation, cont’d

4. Reporting & Development of Recommendations (2-4 months)

– prepare preliminary findings and discuss with programs

– draft report

– address comments and revisions

– develop recommendations

5. Management Responses (1 month)

– obtain ADM-approved management responses and action

plans to the recommendations

6. Approvals/Posting of report (2-3 months)

– recommendation by the Evaluation Committee

– approval by the DM

– translation, ATIP review, media lines, release on Internet

10

.

Evaluation Questions and Issues

• Evaluations address relevance and performance.

• Relevance issues focus on:

– continued need for program;

– alignment with government priorities; and

– alignment with federal roles and responsibilities.

• Performance issues focus on effectiveness:

– achievement of expected outcomes; and

– demonstration of efficiency and economy.

• Evaluators work with program managers to develop more detailed evaluation questions relevant to their program.

11

.

Roles and Responsibilities

• Under the TBS Evaluation Policy, Deputy Ministers are responsible for the evaluation function.

• NRCan’s Departmental Evaluation Committee – an ADM-level Committee – is chaired by the DM

• NRCan’s Head of Evaluation – who is also the DG of Planning and Performance Management Reporting – reports to the Evaluation Committee.

12

.

The Role of the Evaluation Division

• The Strategic Evaluation Division (SED) is responsible for:

– Proposing a five year departmental evaluation plan to the Evaluation Committee, and updating it annually;

– Managing and conducting evaluation studies, including managing contracts and deliverables when consultants are used and issuing reports in a timely manner.

• Additionally, SED will help program managers develop their performance measurement strategies, with the goal of ensuring that good data is collected to support future evaluations.

– Evaluation will work with your team to develop objectives, a logic model, a performance measurement framework and evaluation requirements.

• NRCan's Strategic Evaluation Division is also responsible for reviewing and providing advice on the accountability and performance provisions in Cabinet documents (Memoranda to Cabinet (MCs) and TB Submissions).

13

.

The Role of Program Managers

• Program managers are key to conducting evaluations.

• They are responsible for developing, implementing and monitoring ongoing performance measurement – the foundation of evaluation.

• Additionally, during an evaluation, they must be actively involved in:

– explaining how their programs work;

– contributing to evaluation planning, including identifying more detailed evaluation questions;

– providing performance measurement information on resources used, activities undertaken and results achieved;

– providing detailed documentation (see next slide) and suggestions on potential interviewees, case studies etc.;

– participating in working groups to review questionnaires, preliminary findings, draft evaluation reports, etc.;

– developing management responses and action plans for their ADMs and implementing them after the evaluation.

14

.

Key Documents for An Evaluation

• In preparation for an evaluation, program managers will be asked to provide key documents as early as possible, including:

– Legislation, Regulations, MCs, TB Submissions

– RMAFs, RBAFs or Performance Measurement Frameworks

– references in budgets, SFTs, DPRs, RPPs

– briefing notes, reports --including annual and project reports, studies, databases

– websites and communications products

– five-years of financial expenditures for the PAA unit including

G&C expenditures, O&M, and salaries

15

.

Questions and Assistance

• If you have any questions on evaluation, or wish evaluation assistance in developing performance measurement information or Cabinet documents, please contact:

• The Director of Strategic Evaluation

(613) 996-9649

• Electronic copies of this document, completed evaluation reports and the Terms of Reference for the Evaluation Committee are available at:

http://nrcan.gc.ca/evaluation/index-eng.php