Guide to Developing a Risk-Based Departmental Evaluation Plan

1. INTRODUCTION

• 1.1 Purpose of this Guide
• 1.2 Organization of this Guide

2. GETTING STARTED

• 2.1 Why do Risk-based Planning?
• 2.2 What Goes in a Plan?
• 2.3 Who Should Be Involved?
• 2.4 When to Start the Planning Process
• 2.5 Integrated Evaluation and Internal Audit Plans

3. DEVELOPING A RISK-BASED DEPARTMENTAL EVALUATION PLAN

• 3.1 Preliminary Assessment of Evaluation Universe
• 3.2 Confirm Department's and Government-Wide Priorities
• 3.3 Establish Priorities Based on Risk-Based Approach
• 3.4 Present The Draft Plan
• 3.5 Approve, Publish and Distribute

4. MONITORING THE PLAN

5. TBS' CRITERIA FOR REVIEWING EVALUATION PLANS

6. FOR MORE INFORMATION

APPENDIX A: SAMPLE TABLE OF CONTENTS

APPENDIX B: SAMPLE EVALUATION PROJECT SUMMARY

1. INTRODUCTION

The Treasury Board of Canada's Evaluation Policy ^[1] requires departments to ensure that "a strategically focused evaluation plan – founded on an assessment of risk, departmental priorities, and the priorities of the government – appropriately covers the organization's policies, programs and initiatives". Once approved, a copy of this plan should be forwarded to Treasury Board of Canada Secretariat (TBS).

TBS reviews these plans to ensure an overall strategic and risk-based approach to evaluation efforts.   It also uses the plan to actively monitor departmental evaluation commitments so that results information is made available, on a timely basis, for evidence-based executive decision-making within departments and central agencies.

In addition to fulfilling a policy requirement, there are a number of benefits to undertaking a risk-based approach including:

1.1   Purpose of this Guide

This Guide provides advice to departments ^[2] in developing and implementing a risk-based departmental evaluation plan. It identifies issues to consider and provides criteria used by the TBS' Centre of Excellence for Evaluation (CEE) to review the quality and completeness of evaluation plans.   

Other related guidance worth consulting includes "Good Practices Guide to Developing Evaluation Plans" (May2003) ^[3].

1.2 Organization of this Guide

This Guide was developed to provide an overview of the evaluation planning process. It offers general information on how to develop a plan including sample templates and presentation tables.

Section 2.0 provides guidance on getting started with a focus on purpose, who to involve, and timing.   Section 3.0 suggests two approaches to risk-based evaluation planning. Departments are encouraged to adapt these approaches to best suit their needs and unique circumstances. Section 4.0 discusses issues related to the use or implementation of the plan. Finally, Section 5.0 outlines TBS' review criteria.

2.0 OVERVIEW

2.1 Why do Risk-based Planning?

The purpose of a risk-based evaluation plan is to ensure evaluation activities are strategically integrated into the department's overall management activities and resources are allocated to evaluation activities that can provide the greatest value-added to the organization.   For example, projects or areas within a department that are high-risk may benefit from well-timed evaluation activities.  

The evaluation plan also informs senior management of the scope of the department's evaluation activities and the limitations or impact of that scope. There should be sufficient information presented in the plan to help senior management to determine whether the evaluation function is effectively supporting the strategic business objectives of the organization.

The best evaluation plan is one that directly supports the information or decision-making needs of the Deputy and his/her executive. It is more than a list of projects. A good evaluation plan is a tool to:

Once approved, the evaluation plan serves as a record of the evaluation unit's contribution to results-management and departmental accountability.

2.2 What Goes Into a Plan?

At a basic level, a department's evaluation plan should present the Evaluation Unit's intended contributions to the department's business objectives, its goals, work schedules, staffing plans and financial budgets.   In addition to the primary evaluation activities of the Unit (i.e., planning, producing and reporting on evaluation studies), consideration should be given to the variety of services that the Unit provides. For some departments, this may include services such as:

The plan should also present those activities related to the management of the function (i.e., quality assurance, human resource planning and administration, monitoring, etc.).

Finally, the CEE recommends including other relevant information to help inform senior executives and managers of the importance and potential impact of departmental evaluation activities.   This critical "background piece" can be presented by outlining the overall context of evaluation practice within the department.   Questions to consider include:

A sample Table of Contents is found in Appendix A.

2.3   Who Should Be Involved?

There are a number of individuals and organizational units that have a vested interest in evaluation activities and should be involved in the planning process including:

• Deputy Head
• Senior Executives
• Audit and Evaluation Committee
• Program Managers
• Central Agencies (e.g., TBS Program Sector, CEE)
• Evaluation Unit.

The Audit and Evaluation Committee has a particularly important role to play in the process. Comprised of the Deputy Head and other members of the department's senior executive, this Committee should ensure quality, relevance and completeness of the plan and, ultimately, hold the Evaluation Unit accountable for delivering on its commitments.

For many departments, applying a risk-based approach to evaluation planning is a new undertaking.   Hence, consideration should be given to working with risk management experts with a good understanding of the role, importance and practice of evaluation (including evaluation standards) as well as a solid understanding of the departmental context (i.e., strategic objectives, corporate risks, departmental priorities, etc.).

2.4   When to Start the Planning Process

The planning process should be tied to the departmental annual planning, budgeting, reporting and appraisal processes.   This will ensure that resource requirements for departmental evaluation activities will be available on a timely basis.

In most cases, the evaluation planning process should start in late-fall or winter and be completed by early spring for the subsequent year's evaluation activities (see Figure 1 ).   Approved plans should be forwarded to the CEE no later than end of the first quarter of the fiscal year (i.e., June).

To best support the budget and planning process, large and medium sized departments should develop a multi-year plan, which lays out projects for the current year and potential projects for future years.   A mid-year progress review is appropriate to update priorities and changes in the environment (i.e. change in government priorities and new or emerging risk areas, etc.).

For small agencies, which may conduct few evaluations (if any) a year, an annual planning and review process is recommended.

Figure 1: Locating Evaluation Planning in the Business Cycle

2.5 Integrated Evaluation and Internal Audit Plans

In some departments, joint evaluation and internal audit plans are prepared.   While certain aspects of planning for internal audit may differ from those of evaluation planning, the CEE recommends that departments consider an integrated approach and joint approval process.  

An integrated audit and evaluation plan ensures that not only an overall assessment of the adequacy and effectiveness of a department's control processes are examined (through appropriate audit coverage) but that success, relevance and cost-effectiveness of a department's overall activities are also assessed (through appropriate evaluation coverage).   The two functions are mutually supportive and can contribute greatly to the overall effectiveness and efficiency of a department meeting its business objectives.

Integrating the audit and evaluation planning process can also minimize the potential for duplication of efforts and improve coordination between audit and evaluation functions.   This coordination should not only take place during the planning process but also in the reporting and monitoring of audit and evaluation activities. This will ensure both audit and evaluation results are leveraged to support the broader needs of the department (i.e., executive decision-making).

In departments where evaluation and internal audit plans are separate, efforts should be made to ensure that the plans are prepared and approved in a coordinated manner.

3. DEVELOPING A RISK-BASED DEPARTMENTAL EVALUATION PLAN

The following section outlines an overall approach to developing a risk-based departmental evaluation plan.   This approach should be adapted to meet the unique needs of department or agency and should therefore not be construed as the only approach acceptable to TBS in the development of risk-based evaluation plans.

There are five main steps in developing a risk-based evaluation plan:

3.1 Preliminary Assessment of Evaluation Universe

As a first step, Evaluation Unit staff should develop a preliminary list of potential evaluation projects and issues. This is the first step in identifying the "evaluation universe".  

A department's "evaluation universe" should include what is "evaluable" in terms of the department's programs, policies and initiatives and what evaluation work has already been completed.   A department's Management, Resources and Results Structure (and the associated Program Activity Architecture), is a good starting point for identifying and collecting this information.

Other sources of information to identify potential evaluation projects and issues include:

The Evaluation Unit may also consider the following:

For each potential project, a general description of the program, policy, or initiative should be presented and the objective(s) of the evaluation identified. This information will help support the review, selection and presentation of final projects.   An example of how projects can be presented is found in Appendix B.

3.2 Confirming Departmental and Government-Wide Priorities

Linking evaluation activities to departmental and government-wide objectives and priorities and therefore the information needs of the senior executives is a critical step in the planning process. It begins with confirming executive information requirements by engaging senior management, in particular the Deputy Head, in the planning process. A key step in this process is meeting with the Deputy Head to determine his or her priorities for the upcoming year as well as his/her views on the corporate risks facing the organization.

Concurrently, a call letter soliciting evaluation needs could also be sent to all executives   - Assistant Deputy Ministers (ADMs) and Regional Directors General (RDGs).   If necessary, individual meetings can be conducted with key ADMs who express a clear and significant need for evaluation-related activities.

It is also recommended that TBS (i.e., Program Sector Analysts and the CEE) be consulted to confirm government-wide priorities, identify other strategic issues, and discuss issues of timing, quality assurance and capacity.

Upon completion of this consultation process, the Evaluation Unit should have a complete list of all potential evaluation projects and issues for the current and future years.

3.3 Establish Priorities Using a Risk-Based Approach

After identifying potential projects, the Evaluation Unit should undertake a risk-based priority setting exercise to identify the evaluation projects to be included in the proposed plan. There are two ways of going about this – top down or bottom-up.

In the top-down approach , the Evaluation Unit uses the department's Corporate Risk Profile to identify those areas in which evaluation can be used to mitigate risk. For those departments that have advanced risk management practices and a strong evaluation function, this approach is ideal because the risk assessment process is already complete and being used to support both strategic planning and executive decision-making.   Considerable time and effort is therefore saved.

While this approach strategically links evaluation into the management practices of the department, it does not necessarily ensure that government-wide priorities are taken into consideration or existing commitments to central agencies (i.e., renewal commitments) will be included in the plan.   A priority setting exercise is still required for these projects and can follow the bottom-up approach suggested next.

For those departments, that do not have advanced risk management practices, a bottom-up approach is recommended. The bottom-up approach concentrates on developing a risk matrix or table to assist in strategically assessing the level of risk associated with potential evaluation projects.

To begin, departments should identify what constitutes a risk factor in their context. Common risk factors worth considering include:

• Materiality
• Renewal commitments
• Public/media/parliamentary/ministerial interest
• Health and safety to the public or environment
• Information needs of senior management/executive
• Policy or delivery complexity
• Past evaluation or performance findings.

Table 1 provides an example of how to translate these risk factors into a table to help determine the level of risk.

Table 1: Sample Risk Table to Determine Priorities

For each proposed evaluation project, a risk rating (high, medium or low) or priority level is given to the each of these factors and the overall risk is determined.   To establish the individual as well as overall ratings, consultation with key stakeholders should be undertaken as well as professional judgement applied. Table 2 provides an example of how to present the results of this rating exercise.

Table 2: Sample Risk Rating Table

It is important to note that not all risks or issues are of equal significance. Additional "weights" can be assigned to risk factors, as required. For example, evaluations that support policy requirements such as those associated with the renewal of a transfer payment program are mandatory. Hence, while a project may be considered   "low" risk (in all other risk categories), it would be given an overall rating of "high" if a renewal commitment exists in that given year.

In both approaches, it is important to note that some programs, policies or initiatives may not present themselves as high-risk, it does not mean that there are not problems or challenges with these programs, policies or initiatives.   In fact, the results of evaluation activities may lead to a greater understanding of risk areas within the organization or a program or initiative.   Hence, the final list of proposed evaluation projects should be based upon an agreed upon proportion of high-risk and medium/low-risk programs, policies or initiatives (e.g. 70% high-risk to 30% medium/low-risk).   The relative proportions are at the discretion of the department.

3.3.1 Other Considerations

In addition to risk, potential evaluation projects should also be analyzed based on an assessment of coverage, evaluation resources available and current capacity of the Evaluation Unit.

Coverage can be assessed on a number of fronts including:

• Risks addressed;
• Priorities and strategic objectives addressed;
• Expenditures addressed; and/or
• Organizational units, sectors or types of transactions covered.

TBS recommends a balanced coverage to ensure that the department is achieving full value of its evaluation investment.

In some cases, Evaluation Units may need to conduct evaluability assessments to determine the level of scope, detail and timing of an evaluation. This should be built into the planning process and Evaluation Plan.   Adjustments should be made to the Plan as assessment results are brought forth.

In departments where evaluation resources or capacity is limited, consideration should be given to strategically focusing evaluation activities.   For example, economies may be realized by combining evaluations or select evaluation activities of projects similar in scope, purpose or target audience.   This should be explored when reviewing the feasibility of the overall plan.

Finally, consideration must be given to the costs and sources of funds for evaluation activities.   This information is required in drafting the evaluation plan so as to inform executives of the relative costs for specific and overall evaluation activities and, if necessary, adjust resource allocations.

3.4 Present The Draft Plan

Once a prioritized list of evaluation projects has been developed, it should be shared with other key areas of the organization (in particular audit) and external stakeholders to ensure issues are adequately addressed and appropriately coordinated.   A draft plan should then be presented to the Evaluation Committee.

Briefing materials associated with the plan should focus on communicating how the Evaluation Unit is allocating scarce resources in a manner that provides maximum value-added to the department in achieving its business objectives.   It is equally important to inform the Committee members where the plan is deficient due to resource limitations and the potential impact of these deficiencies.

Once the Committee has been briefed and input/feedback obtained, a final detailed plan with proposed schedules and detailed budgets by project should be prepared.

3.5 Approve, Publish and Distribute

Once the Evaluation Committee approves the evaluation plan, a detailed workplan for the year (or period covered) should be prepared.   The detailed workplan should include critical timelines for all projects (at a minimum expected start and end dates), resource requirements by project (e.g. O& M and FTEs) and project authorities (who in the Evaluation Unit will oversee the proposed projects or services).

The final plan should be translated into both official languages and forwarded to the Deputy Minister for signature.   A copy of the signed plan should be provided to TBS (i.e., the CEE) and to affected stakeholders.   Finally, the signed plan (or an abbreviated version) should be widely distributed within the Department in particular to managers at levels one lower than ADMs.

4.0   MONITORING THE PLAN

The detailed workplan not only supports the implementation of the plan but also helps the department monitor its commitments. To further support monitoring, the department should:

By monitoring (and if necessary adjusting) the plan, the department will ensure that commitments are being fulfilled and appropriate action is taken to address unexpected changes in the environment such as new policies or priorities, new or emerging risks, increases/decreases in departmental budgets, unexpected human resource changes, etc.

5.0 TBS' CRITERIA FOR REVIEWING EVALUATION PLANS

The CEE reviews departmental evaluation plans to ensure an overall strategic approach to evaluating government priorities.   The following criteria are used to support this process:

Once reviewed, CEE monitors the implementation of evaluation plans to ensure departments are fulfilling their commitments.

6.0 FOR MORE INFORMATION

For more information or to provide comments on this guide, contact:

Centre of Excellence for Evaluation
Treasury Board Secretariat
L'Esplanade Laurier
300 Laurier Avenue West
Ottawa, Ontario. Canada.   K1A 0R5

Tel:   (613) 952 1746
Fax:   (613) 946 6262

APPENDIX A: SAMPLE TABLE OF CONTENTS

1. INTRODUCTION

  1.1   Context

2. APPROACH

  2.1   Approach to Developing the Evaluation Plan

3. YEAR IN REVIEW

  3.1   Evaluation Activities Conducted in Year XXXX-XXXX and Contribution to Departmental Performance

  3.2   Evaluations to be Carried Forward

4. CURRENT YEAR

  4.1   Proposed Evaluation Activities for Year XXXX-XXXX

  4.2   Proposed Timeline for Evaluation Activities for Current Year

  4.3   Resource Requirements

5. FUTURE YEAR EVALUATION SCHEDULE

Appendix A: Departmental Evaluation Universe (Optional)

Appendix B: Proposed Evaluation Projects for Current Year – Detailed Project Summaries

APPENDIX B: SAMPLE PROJECT SUMMARY

[1] April 2001.

[2] As with the Evaluation Policy, this guidance applies to all organisations considered to be departments within the meaning of Section 2 of the Financial Administration Act.

[3] This guidance can be found on the Centre of Excellence for Evaluation website at Good Practices Developing Evaluation Plans.