Archived [2012-04-01] - Directive on the Evaluation Function

This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

1. Effective Date

1.1 The directive takes effect April 1, 2009. Departments have until March 31, 2013 to fully implement sections 6.1.1 (a) and 6.1.3 (b) (iii) of the directive.

1.2 Transitional considerations:

1.2.1 Pending the full implementation of section 6.1.3 (b) (iii), but no later than March 31, 2013, heads of evaluation will ensure that:

  1. departmental evaluation plans that they develop demonstrate progress towards achieving coverage of all departmental direct program spending (excluding ongoing programs of grants and contributions) over five years; and,
  2. approved departmental evaluation plans that they submit to the Treasury Board of Canada Secretariat as per section 6.1.3 (d) of this directive, and that do not demonstrate evaluation coverage of all direct program spending over the ensuing five-year period, use a risk-based approach to planning coverage of direct program spending (excluding ongoing programs of grants and contributions.) Departmental evaluation plans that use a risk-based approach to planning coverage of direct program spending will include a written, risk-based rationale to explain the department's evaluation coverage and non-coverage choices.

2. Application

2.1 This directive applies to departments as defined in section 2 of the Financial Administration Act, with the exception of the Office of the Governor-General's Secretary, and the staffs of the Senate, House of Commons, Library of Parliament, Office of the Senate Ethics Officer and Office of the Conflict of Interest and Ethics Commissioner.

2.2 Application of this directive to small departments and agencies as defined in Annex A of the Policy on Evaluation is deferred until otherwise directed by the President of the Treasury Board, except for section 6.3, which shall apply as of the effective date of the directive.

2.3 The provisions in paragraphs 6.1.3 b (vi) and (viii) and 6.3.1(a) that provide for the Secretary of the Treasury Board to request specific evaluation coverage, do not apply with respect to the Office of the Auditor General, the Office of the Privacy Commissioner, the Office of the Information Commissioner, the Office of the Chief Electoral Officer, the Office of the Commissioner of Lobbying, the Office of the Commissioner of Official Languages and the Office of the Public Sector Integrity Commissioner. Further, sections 7 and 8 of this directive, which adopt all requirements of sections 7 and 8 of the Policy on Evaluation, apply to these organizations with the exception of paragraphs 7.3 and 8.2 of the policy. The deputy heads of these organizations are solely responsible for monitoring and ensuring compliance with this directive within their organizations, as well as for responding to cases of non-compliance in accordance with any Treasury Board instruments that address the management of compliance.

3. Context

This directive:

3.1 supports the objectives of the Policy on Evaluation by operationalizing requirements for departmental staff involved in evaluation. This directive establishes the specific requirements to enable the production of credible, timely, neutral, and cost-effective evaluations to support policy, program and expenditure decision-making;

3.2 promotes collaboration between the evaluation function and program managers to help improve the design, delivery, performance and performance measurement of the organization's policies and programs;

3.3 supports the principle that heads of evaluation, as primary departmental experts in evaluation, have final decision-making authority on technical issues, subject to the decision-making authority of deputy heads;

3.4 is consistent with and supports requirements outlined in the Policy on Transfer Payments and in section 42.1 of the Financial Administration Act;

3.5 outlines the roles and responsibilities of heads of evaluation and program managers, and establishes the requirements for core competencies for heads of evaluation and evaluators; and,

3.6 is to be read in conjunction with the Policy on Evaluation; and Standard on Evaluation for the Government of Canada; the Financial Administration Act and the Management, Resources and Results Structure Policy.

3.7 The Treasury Board has delegated to the President of the Treasury Board the authority to issue, amend and rescind the Directive on the Evaluation Function and to approve any exception to the directive.

4. Definitions

Definitions used in the interpretation of this directive can be found in Annex A of the Policy on Evaluation.

5. Directive Statement

5.1 Objective

The objective of this directive is to clarify the responsibilities of departmental staff involved in evaluation so that departmental evaluation functions work effectively to support the evaluation information needs of Canadians, Parliamentarians, Ministers, central agencies and deputy heads.

5.2 Expected results

The expected results of this directive are that the evaluation information needs of Canadians, Parliamentarians, Ministers, central agencies and deputy heads are supported through:

5.2.1 adequate, regular and systematic evaluation coverage of direct program spending;

5.2.2 departmental evaluation functions and evaluations that are considered credible by, Canadians, Parliamentarians, Ministers, central agencies and deputy heads; and,

5.2.3 departmental evaluation functions that produce appropriate information to support decision making and public reporting, in a timely manner.

6. Requirements

6.1 Responsibilities of heads of evaluation

6.1.1 Competencies of heads of evaluation and evaluators

Heads of evaluation are required to:

  1. meet competency requirements, as specified by the Treasury Board of Canada Secretariat; and,
  2. ensure that the person or persons involved in designing, conducting or managing evaluations has, or collectively have the appropriate training, required competencies and experience in evaluation.

6.1.2 Managing the evaluation function

In managing the evaluation function in departments, heads of evaluation consult appropriately with program managers, stakeholders, and peer review or advisory committees during evaluation project design and implementation. In this context, heads of evaluation are responsible for:

  1. managing an efficient and effective evaluation unit that ensures evaluations are conducted in a neutral, cost-effective manner;
  2. ensuring that evaluation work and persons involved in designing, conducting and managing evaluations are informed of and adhere to the Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada;
  3. consulting with the departmental head of communications on evaluation work that could potentially be considered “public opinion research” as defined in the Policy on Government of Canada Communications;
  4. supporting a senior committee of departmental officials (referred to as the Departmental Evaluation Committee, as defined in Annex B of the Policy on Evaluation) that is assigned the responsibility for guiding and overseeing the evaluation function; and,
  5. issuing evaluation reports (and other evaluation products, as appropriate) directly to the deputy head and the Departmental Evaluation Committee in a timely manner.

6.1.3 Evaluation coverage and quality

Heads of evaluation are responsible for:

  1. developing a rolling five-year departmental evaluation plan (hereafter referred to as the departmental evaluation plan), and updating the plan annually;
  2. ensuring that their departmental evaluation plans:
    1. align with and support the departmental Management, Resources and Results Structure;
    2. support the requirements of the Expenditure Management System, including strategic reviews;
    3. include all direct program spending, excluding grants and contributions;
    4. include all ongoing grant and contribution programs for which their department is responsible, as required under section 42.1 of the Financial Administration Act;
    5. include the administrative aspect of all major statutory spending;
    6. include programs that are set to terminate automatically after a specified period of time, if requested by the Secretary of the Treasury Board following consultation with the affected deputy head;
    7. include elements of the Government of Canada Evaluation Plan that apply to the department; and,
    8. include specific evaluations, if requested by the Secretary of the Treasury Board following consultation with the affected deputy head.
  3. identifying and recommending to the deputy head and the Departmental Evaluation Committee a risk-based approach for determining the evaluation approach and level of effort to be applied to individual evaluations comprised in the five-year departmental evaluation plan, and the appropriate level of resources required to conduct individual evaluations included in the plan;
  4. submitting the departmental evaluation plan, as approved by the deputy head, to the Treasury Board of Canada Secretariat at the beginning of each fiscal year, along with confirmation from the deputy head as required by section 6.1.7 of the Policy on Evaluation;
  5. implementing the approved departmental evaluation plan while ensuring the timely completion of individual evaluations, in accordance with the Standard on Evaluation for the Government of Canada; and,
  6. ensuring that all evaluations that are intended to count toward the coverage requirements of subsections “a”, “b”, or “c” of section 6.1.8 of the Policy on Evaluation, include clear and valid conclusions about the relevance and performance of programs by addressing all core issues outlined in Annex A.

6.1.4 Performance measurement

Developing, implementing and monitoring ongoing performance measurement strategies for programs is the responsibility of program managers. In that context, heads of evaluation are responsible for:

  1. reviewing and providing advice on the performance measurement strategies for all new and ongoing direct program spending, including all ongoing programs of grants and contributions, to ensure that they effectively support an evaluation of relevance and performance;
  2. reviewing and providing advice on the accountability and performance provisions to be included in Cabinet documents (Memoranda to Cabinet, Treasury Board submissions);
  3. reviewing and providing advice on the performance measurement framework embedded in the organization's Management, Resources and Results Structure; and,
  4. submitting to the Departmental Evaluation Committee an annual report on the state of performance measurement of programs in support of evaluation.

6.1.5 Dissemination of evaluation reports

Heads of evaluation are required to:

  1. submit complete evaluation reports, management responses and action plans (in electronic format) to the Treasury Board of Canada Secretariat immediately upon approval of these reports by the deputy head; and,
  2. make approved evaluation reports, along with the management responses and action plans, available to the public with minimal formality and post them on departmental websites in both official languages in a timely manner following their approval by the deputy head.
  3. ensure that evaluation reports posted on departmental websites respect the Access to Information Act, Privacy Act, and the Government Security Policy.

6.2 Responsibilities of program managers

Program managers are responsible for:

6.2.1 developing and implementing ongoing performance measurement strategies for their programs, and ensuring that credible and reliable performance data are being collected to effectively support evaluation;

6.2.2 developing and implementing a management response and action plan for all evaluation reports in a timely and effective manner; and,

6.2.3 consulting with the head of evaluation on the performance measurement strategies for all new and ongoing direct program spending.

6.3 Responsibilities of heads of evaluation of small departments and agencies

6.3.1 Heads of evaluation in small departments and agencies are required to:

  1. ensure coverage of the administrative aspect of major statutory spending and of direct program spending excluding grants and contributions, as appropriate to the needs of the department or agency, and to undertake specific evaluations if requested by the Secretary of the Treasury Board following consultation with the affected deputy head;
  2. ensure that all ongoing programs of grants and contributions are evaluated every five years, as required under section 42.1 of the Financial Administration Act;
  3. consult with the departmental head of communications on evaluation work that could potentially be considered “public opinion research” as defined in the Policy on Government of Canada Communications;
  4. ensure that evaluation work and persons involved in designing, conducting and managing evaluation activities are informed of and adhere to the Standard on Evaluation for the Government of Canada;
  5. ensure that all evaluations that are intended to count toward the coverage requirements of subsection “a” of section 6.2.5 of the Policy on Evaluation include clear and valid conclusions about the relevance and performance of programs by addressing all core issues outlined in Annex A;
  6. identify and recommend to the deputy head a risk-based approach for determining the evaluation approach and level of effort to be applied to individual evaluations;
  7. submit complete evaluation reports, management responses and action plans (in electronic format) to the Treasury Board of Canada Secretariat immediately upon approval of these reports by the deputy head; and,
  8. make approved evaluation reports, along with the management responses and action plans, available to the public with minimal formality and post them on departmental websites in both official languages in a timely manner following their approval by the deputy head.
  9. ensure that evaluation reports posted on departmental websites respect the Access to Information Act, Privacy Act, and the Government Security Policy.

7. Monitoring and Reporting

7.1 The monitoring and reporting requirements outlined in section 7 of the Policy on Evaluation apply to this directive.

8. Consequences

8.1 The consequences of non-compliance with this directive are identified in section 8 of the Policy on Evaluation.

9. References

10. Enquiries

Please direct enquiries about this directive to:

Senior Director
Centre of Excellence for Evaluation
Treasury Board of Canada Secretariat
222 Nepean Street, 4th Floor
Ottawa, Ontario K1A 0R5
Telephone: 613-952-7447
Fax: 613-946-6262

Annex A - Core Issues to be Addressed in Evaluations

Evaluations can be conducted to serve a variety of purposes and, consequently, can be designed to answer a multiplicity of questions through many different approaches. Evaluations that are intended to count toward the coverage requirements of subsections “a”, “b”, or “c” of Section 6.1.8 or subsection “a” of Section 6.2.5 of the Policy on Evaluation will address value for money by including clear and valid conclusions about the relevance and performance of programs. To address value for money, evaluations will be required to assess all core issues identified below (as appropriate, departments may choose to address additional issues in their evaluations).

Subject to the requirement to address all core issues, departments have the flexibility to determine the evaluation approach and level of evaluation effort in accordance with the program's risks and characteristics, and the quality of performance information available for each individual program.

Core Issues
Relevance
Issue #1: Continued Need for programAssessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians
Issue #2: Alignment with Government PrioritiesAssessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes
Issue #3: Alignment with Federal Roles and ResponsibilitiesAssessment of the role and responsibilities for the federal government in delivering the program
Performance (effectiveness, efficiency and economy)
Issue #4: Achievement of Expected Outcomes Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes
Issue #5: Demonstration of Efficiency and EconomyAssessment of resource utilization in relation to the production of outputs and progress toward expected outcomes