Office of the Auditor General of Canada The State of Program Evaluation in the Canadian Federal Government Glenn Wheeler Director, Results Measurement.

Office of the Auditor General of Canada

The State of Program Evaluation in the Canadian Federal Government

Glenn Wheeler

Director, Results Measurement


October 17, 2001


Outline of Presentation

1. Background on Canadian System of Government

2. About the Office of the Auditor General of Canada

3. The Auditor General’s Approach to Evaluation

4. The Auditor General’s Audits of Program Evaluation

5. Emergence of Performance Reporting within the

Context of Program Evaluation

6. Lessons Learned from the Canadian Experience


1. Background on Canadian System of Government

Division of Powers

- a sovereign democracy:

- federation of 10 provinces and 3 territories

- a federal Parliament and 13 provincial/territorial legislatures

- federal government - authority in all matters not specifically reserved for provincial/territorial governments

- provincial/territorial governments - authority for education, hospitals, property and local government


Federal Government

- over 100 departments and agencies responsible for areas such as employment, national defence, fisheries, agriculture and finance

- each department and agency has a minister who reports to Parliament

- budgets approved annually


2. About the Office of the Auditor General of Canada

History

1878: Office of the Auditor General established

1931: Clarification of Auditor General’s role

1950: beginning of expansion of Auditor General’s mandate

1977: new legislation - the Auditor General Act

1994: legislation amended to allow Auditor General to allow for tabling of up to 3 reports per year in addition to annual report

1995: legislation amended to establish a Commissioner of the Environment and Sustainable Development within the Office


Current Office

- new Auditor General, Sheila Fraser, for 10 year term

- headquarters in Ottawa with 5 regional offices

- staff of 520 auditors and support staff

Role

- The Auditor General aids accountability by conducting independent audits of federal government operations

- these audits provide members of Parliament with objective information to help them examine the government’s activities and hold it to account


The Auditor General’s Role


Vision

- We are committed to making a difference for Canadians by promoting, in all our work for Parliament, answerable, honest and productive government that reflects a commitment to sustainable development

Mission

- The Office conducts independent audits and examinations that provide information, advice and assurance to Parliament. We promote accountability and best practices in government operations.


Auditor General Act

Section 7 (2) of the Auditor General Act requires the Auditor General to call attention to anything he considers to be of significance and of a nature that should be brought to the attention of Parliament, including cases in which he has observed:

a) accounts have not been properly maintained or public money has not been fully accounted for;

b) essential records have not been maintained to ensure proper allocation of revenue and to ensure that expenditures have been made only as authorized;

c) money has been spent other than for the purposes for which it was appropriated by Parliament;


d) money has been expended with due regard to economy and efficiency;

e) satisfactory procedures have not been established to measure and report on the effectiveness of programs; or

f) money has been expended without due regard to effects on the environment.


3. The Auditor General’s Approach to Program Evaluation

– Types of audits

– - attest audits

– - compliance audits

– - value-for-money audits

– A value-for-money audit is a systematic and objective examination of government activities. Its scope includes the examination of economy, effficiency and cost-effectiveness of activities. It also examines procedures to measure effectiveness. The subject of the audit can be a program, sectoral activity, or a government-wide sectional area.

– A program evaluation is a disciplined assessment of government programs and activities. It is based on independent, systematic measurement and analysis, carried out to meet expectations set in policy and standards, and publicly reported.


Interpretation

- although value-for-money audits and program evaluations are similar, they are different in some ways

- the Auditor General does not conduct program evaluations but rather value-for-money audits which examine procedures used by departments to measure effectiveness (i.e., to undertake program evaluations)


What the Auditor General expects when undertaking audits on program evaluation

Value-for-money audits examine how departments plan and undertake program evaluations of their programs, specifically:

1) the department’s evaluation policy

2) responsibility for program evaluation within the department

3) the organization of the evaluation branch within the department

4) resources (people and funds) dedicated to program evaluation

5) the number and quality of program evaluations completed

6) the mechanisms used to ensure quality of program evaluations

7) the relevance and timeliness of program evaluations

8) how program evaluation findings are used


Why the Auditor General considers program evaluation to be a crucial function in the federal government

1) to provide information to allow senior management to better manage their programs

2) to provide evidence for departments concerning whether to continue funding existing programs or re-allocate these funds to other programs

3) to provide information to managers as these seek funding to renew programs

4) to encourage accountability to Parliament for the results of programs

5) to enhance public debate surrounding policy and program options

6) to allow government to determine priorities

7) to encourage transparency in departments


4. The Auditor General’s Audits of Program Evaluation

Audits undertaken on the program evaluation function across the federal government -

1978 - first audit of evaluation function

1983 - update on implementation of 1978 recommendations

1993 - major government-wide audit on the state of program evaluation in the federal government

1996 - examination of program evaluation in the context of a changing public sector environment

2000 - reporting performance to Parliament


Federal Government Policy on Program Evaluation

1977 - original policy

1991 - revised policy

1994 - review policy (review, internal audit and program

evaluation

1995 - revised expenditure management system

2001 - new evaluation policy


1993 Audit - Key Observations

- program evaluation branches poorly funded, and often subsumed within internal audit banches

- departments did not plan for evaluation work

- many program evaluations of smaller programs; fewer evaluations of larger more important programs

- program evaluations often focussed on operational issues and less likely to challenge the existence of a program

- limited use of program evaluation results

- program evaluation results not linked to resource allocation or accountability reporting



- program evaluation did not consistently provide effectiveness information for government decision making and broader accountability

- more larger programs were being evaluated than in 1993, but undertaken in response to external demands

- scope and quality of program evaluation remained a concern


5. Emergence of Performance Reporting in Context of Program Evaluation

Two key developments

1) revised expenditure management system

- in 1995 the federal government announced that under its revised expenditure management system departments would be required to submit an annual “Performance Report” to Parliament

2) creation of new agencies

- since 1997 three new agencies have been created to streamline key activities previously undertaken by many departments (Canadian Food Inspection Agency, Parks Canada, Canada Customs and Revenue Agency)

- new agencies required to submit annual reports

- legislation for the three agencies states that the Auditor General annually shall “provide an assessment of the fairness and reliability of the information about the Agency’s performance”



- department focus too much on listing program activities and outputs and to little on reporting outcomes or results

- reporting makes too little use of evaluation findings on the results that activities are accomplishing and on how programs are contributing to outcomes


6. Lessons Learned from the Canadian Experience6. Lessons Learned from the Canadian Experience

Organizations must:

1) have a unit in place with resources (people and funds) and expertise to undertake program evaluations

2) have a reporting structure in place so that the manager of the program evaluation unit reports directly to the head of the organization or its senior management committee

3) consider constructing a policy formalizing roles and responsibilities with respect to program evaluation

4) plan to undertake program evaluations. Plans should cover a one-year period at a minimum, and several years preferably

5) have a plan which prioritizes which programs should be evaluated based on their importance and level of risk


6) have mechanisms built into the process to ensure the quality of program evaluations and to ensure that they are timely

7) ensure that program evaluation results are utilized by senior management in making decisions regarding funding and whether programs should be redesigned or discontinued

8) follow up with the management of programs evaluated to determine whether evaluation recommendations are being implemented

9) periodically review the performance of their program evaluation units to determine the value that program evaluation adds to the organization

Office of the Auditor General of Canada The State of Program Evaluation in the Canadian Federal Government Glenn Wheeler Director, Results Measurement.

Documents

office slide

auditor generals role

expansion of auditor

auditor generals approach

local government slide

results measurement

federal parliament

productive government