Archived [2009-03-31] - Policy on Evaluation

This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Archived

This page has been archived on the Web.

Effective date

This document contains the entire text of the policy as revised on April 1, 2001. This policy replaces Chapters 1-1, 1-2, 3-1 and 3-2 of the "Review" volume of the Treasury Board Manual dated July 31, 1994.

Preface

This policy supports the generation of accurate, objective and evidenced-based information to help managers make sound, more effective decisions on their policies, programs and initiatives and through this provide results for Canadians.

This policy clarifies the role of evaluation within the government's management framework.

The policy is based on three fundamental principles:

  • first, that achieving and accurately reporting on results is a primary responsibility of public service managers;
  • second, that rigorous and objective evaluation is an important tool in helping managers to manage for results; and
  • third, that departments with the support of the Treasury Board Secretariat, are responsible to ensure that the rigour and discipline of evaluation are sufficiently deployed within their jurisdictions.

Managing for Results

Managing for results is the prime responsibility of public service managers. As outlined in the management framework for the federal government, Results for Canadians, public service managers are expected to define anticipated results, continually focus attention towards results achievement, measure performance regularly and objectively, and learn and adjust to improve efficiency and effectiveness.

Managers must be accountable for their performance to higher management, to ministers, to Parliament and to Canadians.

Evaluation - like internal audit, risk management capacity and other management tools - helps managers to operate effectively in this environment. Evaluation can support managers' efforts to track and report on actual performance and help decision-makers objectively assess program or policy results. This distinguishes evaluation from internal audit - a function that provides assurances on a department or agency's risk management strategy, management control framework and information, both financial and non-financial, used for decision-making and reporting.

Evaluation as a Management Tool

Evaluation has two main purposes:

  • to help managers design or improve the design of policies, programs and initiatives; and
  • to provide, where appropriate, periodic assessments of policy or program effectiveness, of impacts both intended and unintended, and of alternative ways of achieving expected results.

Evaluation operates in a complex environment that involves partnerships with other federal organisations, with other levels of government, with the private sector or with not-for-profit entities. In addition, as the delivery of programs is devolved to other jurisdictions, the evaluation of underlying policies increases in importance. Evaluation should contribute to improvements in policy, as well as program design and delivery.

In this milieu, departments should embed the discipline of evaluation into the lifecycle management of policies, programs and initiatives to:

  • develop results-based management and accountability frameworks for new or renewed policies, programs and initiatives;
  • establish ongoing performance monitoring and performance measurement practices;
  • evaluate issues related to the early implementation and administration of the policy, program or initiative, including those that are delivered through partnership arrangements (formative or mid-term evaluation); and
  • evaluate issues related to relevance, results and cost-effectiveness.

Departments should undertake an appropriate balance of evaluation work. To achieve this balance, evaluators should develop a strategically focussed plan that is based on assessments of risk, departmental priorities and priorities of the government as a whole.

Effective Deployment of Evaluation

The Government of Canada is committed to becoming a learning organisation. Evaluation supports this aim by helping to find out what works and what does not, and by identifying cost-effective, alternative ways of designing and improving policies, programs, and initiatives.

The success of evaluation depends on a number of important factors. It requires clarity of roles, application of sound standards, ongoing support for rigorous, professional practice, and developing a conducive environment where managers embed the discipline of evaluation into their work. The organisational positioning of evaluation should reflect the unique needs of the department. Evaluation discipline should be used in synergy with other management tools to improve decision-making. Heads of evaluation should work to ensure that evaluation in their organisation is healthy in respect of all these factors.

The Treasury Board Secretariat, in its role as the government's management board, should support the practice of evaluation in departments by providing advice on best practices, setting standards, monitoring the evaluation capacity in departments and using the products of evaluation to inform decision-making at the centre.

Policy Objective

To ensure that the government has timely, strategically focussed, objective and evidence-based information on the performance of its policies, programs and initiatives to produce better results for Canadians.

Policy Statement

It is government policy that departments and agencies embed evaluation into their management practices:

  • to help design policies, programs, and initiatives that clearly define expected results and that embody sound performance measurement, reporting and accountability provisions at their outset; and
  • to help assess in a rigorous and objective manner the results of government policies, programs, and initiatives, including their impacts both intended and unintended, and alternative ways of achieving expected results.

Application

This policy applies to all organisations considered to be departments within the meaning of section 2 of the Financial Administration Act.

Policy Requirements

Deputy heads must establish an appropriate evaluation capacity, tailored to the needs and resources of the department, to evaluate policies, programs and initiatives - including those of an inter-organisational nature. To that end they should:

  1. appoint a senior head of evaluation to conduct strategically focussed evaluation studies and work with managers to embed the discipline of evaluation into departmental management practices;
  2. establish an evaluation committee and designate a senior departmental executive to chair it; and
  3. ensure that, in addition to accountability practices internal to the department, the Treasury Board Secretariat is given access to evaluation plans and early warning of evaluation findings that indicate major concerns respecting the management or effectiveness of policies, programs and initiatives.

Departmental heads of evaluation must provide leadership and direction to the practice of evaluation in the department and, to that end, should:

  1. ensure that strategically focussed evaluation plans - founded on assessments of risk, departmental priorities, and priorities of the government - appropriately cover the organisation's policies, programs and initiatives;
  2. work with managers to help them enhance the design, delivery and performance measurement of the organisation's policies, programs, and initiatives;
  3. conduct evaluation studies of policies, programs and initiatives in accordance with established plans;
  4. inform senior management and the appropriate departmental players (e.g. internal audit) without delay of any findings that indicate major concerns respecting the management or effectiveness of policies, programs, or initiatives;
  5. make completed evaluation reports available to the Treasury Board and to the public with minimal formality in both official languages; and
  6. apply the evaluation standards appended to this policy.

Departmental managers must manage for results and to that end, should:

  1. draw on the organisation's evaluation capacity, where appropriate, and ensure that they have reliable, timely, objective and accessible information for decision-making and performance improvement; and
  2. incorporate approved evaluation findings and measures for improvement into priority setting, planning, reporting and decision-making processes.

Treasury Board Secretariat must provide central direction for evaluation in the Government of Canada and, to that end, should:

  1. establish a Centre of Excellence to provide leadership, guidance and support to the practice of evaluation;
  2. use evaluation results where appropriate in decision-making at the centre;
  3. set standards; and
  4. monitor evaluation capacity in the government.

Monitoring

The Treasury Board Secretariat will monitor this policy to ensure its success in meeting its intended objectives.

To monitor the policy, the Treasury Board Secretariat, in consultation with its stakeholders, will develop a results-based management and accountability framework that sets out, among other things, the objectives and key results to be achieved and the performance measurement strategy.

The Treasury Board Secretariat will conduct an evaluation of this policy within five years of the coming into force of the policy.

In monitoring the implementation of this policy, the Treasury Board Secretariat will be guided by the requirements of the Evaluation Standards for the Government of Canada, appended to this document.

References

Financial Administration Act

Access to Information Act

Privacy Act

Results for Canadians: A Management Framework for the Government of Canada

Official Languages Act

The Social Union Framework Agreement

Transfer Payments Policy

Study of Program Evaluation in the Federal Government

Managing for Results

Modernisation of Comptrollership in the Government of Canada

Communications Policy of the Government of Canada

Alternative Service Delivery Policy Framework

Cancellation

This policy and related standards supersede Chapters 1-1, 1-2, 3-1 and 3-2 of the "Review" volume of the Treasury Board Manual dated July 31, 1994.

Enquiries

Enquiries about this policy should be directed to the Planning, Performance and Reporting Sector of the Treasury Board Secretariat.

Appendix A - Definitions

Approved (or completed) (approuvé (ou réalisé)) - when referring to an evaluation product means approved by the organisation's evaluation committee.

Department (or departmental) (Ministère (ou ministériel)) - includes a department, agency or any other organisation covered by this evaluation policy.

Inter-organisational (Inter-organisationnels) - refers to inter-departmental, inter-governmental, or other relationships, including those with the private or not-for-profit sectors.

Monitor (Surveiller) - means to observe and oversee and if necessary provide warning.

Performance (Rendement) - means actual achievements measured against defined goals, standards or criteria.

Performance measurement (Mesure du rendement) - is the ongoing monitoring of the results of a program, policy or initiative, and in particular, progress towards pre-established goals.

Results (Résultats) - relate to what was achieved. They are the collection of impacts and outcomes associated with a program, policy or initiative.

Results-based management and accountability frameworks (Cadres de gestion et de responsabilisation axé sur les résultats) - are a mandatory requirement for all transfer payment programs (Reference: Policy on Transfer Payments, June 2000) and are commonly required by Treasury Board in the approval of new or renewed programs and initiatives. They generally include:

  • a clear statement of the roles and responsibilities of the main partners involved in delivering the policy, program or initiative;
  • a clear articulation of the resources to be applied and the objectives, activities, outputs and key results to be achieved, along with their linkages;
  • an outline of the performance measurement strategy, including costs and performance information (key indicators) that will be tracked;
  • the schedule of major evaluation work expected to be done; and
  • an outline of the reporting provisions as appropriate for funding recipients and those for the department, including parliamentary reporting.

Appendix B - Evaluation Standards for the Government of Canada

Introduction

The government's management framework, Results for Canadians, commits managers to sharpening their citizen focus, to applying sound public service values, to managing for results and to spending in a responsible manner.

Managers have a responsibility to monitor the performance of policies, programs and initiatives - to make sound decisions and to report on performance to higher departmental authorities, to Parliament and to the Canadian public.

Evaluation is a management tool that can operate throughout the lifecycle of a policy, program or initiative. It can help managers design and implement reliable performance measurement systems. It can also periodically assess effectiveness in achieving objectives, impacts, both intended and unintended, continued relevance and alternative ways of achieving expected results.

Treasury Board policy is that departments are responsible for establishing and supporting evaluation capacity that is appropriate to their needs and is capable of fulfilling these responsibilities in a timely and professional manner.

These standards provide clear expectations for evaluation in an environment, where policies, programs and initiatives often involve partnerships with other federal organisations, with other levels of government or with the private or not-for-profit sectors. Evaluation should contribute to improvements in policy, as well as in programs and on a departmental as well as inter-organisational basis.

By using common criteria, the standards provide departments with a basis for improving the quality of evaluation practice. They also provide the Treasury Board Secretariat with a basis for monitoring the implementation of its evaluation policy.

Evaluation Standards

Evaluation Planning and Issues

Standard

The department must apply the discipline of evaluation to assess the performance of its policies, programs and initiatives, both departmental and inter-organisational, taking into account its priority concerns as well as those of its partners and the government as a whole.

Guidance

Evaluators should develop a strategically focussed plan that is based on assessments of risk, departmental priorities and reporting requirements, and priorities of the government as a whole.

The full range of evaluation issues should be considered at the planning stage of an evaluation:

  • does the policy, program or initiative continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need? (relevance);
  • is the policy, program or initiative effective in meeting its objectives, within budget and without unwanted outcomes? (success); and
  • are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches? (cost-effectiveness).

Evaluators should address issues that are needed for accountability reporting, including those involving key performance expectations (a) identified and conveyed to the Treasury Board or (b) resulting from Cabinet decisions requesting evaluation information.

Competency

Standard

The person or persons carrying out evaluations, or evaluation related work, must possess or collectively possess the knowledge and competence necessary to fulfil the requirements of the particular evaluation work.

Guidance

Evaluators should possess or ensure the provision of content knowledge appropriate for the evaluation and continuously strive to improve their methodological and practice skills. Evaluators should possess the knowledge, skills and experience in:

  • the application of sound research design able to answer the chosen questions;
  • the collection and analysis of reliable quantitative and qualitative data; and
  • the development of valid, credible and unbiased conclusions and recommendations.

Objectivity and Integrity

Standard

Individuals performing evaluation work must be free from impairments that hinder their objectivity and must act with integrity in their relationships with all stakeholders.

Guidance

Evaluators should:

  • accurately represent their level of skills and knowledge; and
  • declare any matter that could compromise the objectivity of either evaluator or stakeholder before embarking on an evaluation project or at any point during the project.
  • evaluators should be accountable for their performance and their products and be responsible for:
  • ensuring that their work addresses the priority concerns and accountability requirements of departmental management and the government;
  • conferring with stakeholders on decisions such as confidentiality, privacy, communications and ownership of findings and reports;
  • ensuring sound fiscal decision-making so that expenditures are accounted for and clients receive good value for their dollars; and
  • completing evaluation work within a reasonable time as agreed to with the clients.

Consultation and Advice

Standard

Evaluation work must incorporate sufficient and appropriate consultation and, where appropriate, apply the advice and guidance of specialists and other knowledgeable persons.

Guidance

Evaluators should consult major stakeholders in the conduct of their work.

Where appropriate, peer review groups should be organised to review evaluation products to improve their quality and enhance the sharing of best practices.

Measurement and Analysis

Standard

Evaluation work must produce timely, pertinent and credible findings and conclusions that managers and other stakeholders can use with confidence, based on practical, cost-effective and objective data collection and analysis.

Guidance

Evaluation products should be made available at the most appropriate time to aid management decision-making.

Evidence should be sufficient in relation to the decision-making context. Evaluation findings should be relevant to the issues addressed and follow from the evidence.

Evaluation products should be demonstrably useful to managers in improving performance and reporting on results achieved.

In planning and conducting the evaluation, the department should comply with government policies related to information collection, use, preservation and dissemination.

Reporting

Standard

Evaluation reports must present the findings, conclusions and recommendations in a clear and objective manner.

Guidance

Evaluation reports should be written so that senior managers and external readers can readily focus on and understand the important issues being reported. They should:

  • be concise and clearly written;
  • include only information that is needed for a proper understanding of the findings, conclusions and recommendations;
  • present the conclusions and recommendations so that they flow logically from evaluation findings;
  • clearly expose the limits of the evaluation in terms of scope, methods and conclusions; and
  • satisfy, where applicable, Cabinet, Treasury Board submission or external reporting requirements.

Additionally, evaluation reports should:

  • provide the reader with appropriate context by describing the objectives and timing of the work, the policy, program or initiative that was evaluated, how it fits into the overall operations of the organisation, and its importance;
  • provide an accurate assessment of the results that have been achieved by the policy, program or initiative;
  • contain clear and actionable recommendations, and timing for management action; and
  • provide relevant analysis and explanation of the exposure to risks for any significant problems identified and in respect of key recommendations.