We are currently moving our web services and information to Canada.ca.

The Treasury Board of Canada Secretariat website will remain available until this move is complete.

Evaluation Guidebook for Small Agencies


Archived information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject à to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Section Two: Defining Evaluation

Section Two: Defining Evaluation

 

This section

  • defines evaluation;
  • describes different types of evaluation;
  • compares evaluation to other accountability activities such as performance measurement, internal audit, and management or operational reviews; and
  • places evaluation in the context of other government initiatives such as the Management Accountability Framework (MAF) and Management Resources and Results Structure (MRRS).

2.1  What is Evaluation?

Evaluation can be defined as the systematic collection and analysis of information on the performance of a policy, program, or initiative to make judgments about relevance, progress or success and cost-effectiveness, and/or to inform future programming decisions about design and implementation.

RBM E-Learning Tool ; TBS Web site

Note that evaluation

  • is periodic (has a "cycle");
  • can cover policies, programs or initiatives within a small agency;
  • involves judgment about a policy, program or initiative's merit or worth (based on systematic and high quality data);
  • focusses on how and why results are achieved;
  • looks at intended and unintended effects; and
  • attempts to address future options and strategies for improvement.

Evaluation provides a periodic opportunity to take an in-depth look at how a program, policy or initiative is doing. The primary focus is usually on being able to bring about improvements to facilitate the achievement of results or to determine the degree to which the program, policy or initiative led to the achievement of desired results (i.e., attribution).

RBM E-Learning Tool ; TBS Web site

2.2  Why Evaluation?

Following the release of the document Results for Canadians: A Management Framework for the Government of Canada in April 2001, TB released an Evaluation Policy "to ensure that the government has timely, strategically focussed, objective and evidence-based information on the performance of its policies, programs, and initiatives to produce better results for Canadians."

 Evaluation has the following two main purposes:

1.  Improvement: to help managers design or improve policies, programs, and initiatives; and

2.  Accountability: to provide, where appropriate, periodic assessments of policy, program, or initiative effectiveness, of impacts both intended and unintended, and of alternative ways of achieving expected results.

Within the small agency context, both improvement and accountability objectives are relevant. Evaluations can assist small agencies

  • to become more focussed on results in their decision making;
  • by providing valuable performance information for Departmental Performance Reports and as required by the Management Resources and Results Structure (e.g., a program's relevance and achievements with respect to strategic objectives); and
  • by satisfying elements of the Management and Accountability Framework (e.g., results and performance, accountability, risk management, policy and programs).

2.3  Evaluation and the Program Development Cycle

Ideally, planning and evaluation are interdependent processes. Evaluation can and should be built into the planning process for agencies. Combined with the feedback from performance measurement activities, information from evaluations can help to guide planning within the agency.

Evaluation and the Program Development Cycle

2.4  Overview of Tasks in Conducting Evaluations

The illustration below provides an overview of the tasks required to conduct an evaluation. This guide examines these tasks in more detail in Sections 4, 5, and 6.

Overview of the tasks required to conduct an evaluation

 

2.5  Types of Evaluation

There are two main types of evaluations: formative and summative.

Formative Evaluation

This type of evaluation examines the effectiveness of implementation for facilitating improvement. A formative evaluation may be conducted in mid-cycle of the program, policy or initiative (i.e., within two years). Its intent is to provide information to improve the program, policy or initiative.

Formative evaluations should be used judiciously. They may not be required for ongoing programs. If compliance audits include operational questions, then a formative evaluation may not be needed. Formative evaluations may also target particular aspects of a program – for example, the performance measurement system.

Formative evaluations focus on the following:

  • management issues of how the policy, project or initiative is being implemented and delivered;
  • how risk is being managed;
  • if the performance measurement system is generating valid and reliable performance data;
  • verifying if adjustments are necessary; and
  • to what extent progress toward the achievement of the desired results is occurring.

Where "full formative evaluations" are undertaken, there is an expectation that outputs, early results, validation of program logic, and the likelihood of long-term results achievement are assessed. [4]

All evaluations, whether formative or summative, should address the Expenditure Review Committee's questions. (See Appendix D.)

 

Summative Evaluation

This type of evaluation examines impacts in order to make a decision about overall effectiveness. They have primarily an accountability function and are generally conducted towards the end of the cycle.

Summative evaluations focus on the following:

  • Relevance: Does the program continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need?
  • Success: The degree to which desired results have been achieved and the extent to which the policy, program or initiative has contributed to the achievement of results.
  • Cost-effectiveness: Are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?

All evaluations, whether formative or summative, should address the Expenditure Review Committee's questions. (See Appendix D.)

Another type of evaluation often referred to within the federal context is a Horizontal Evaluation . These are evaluations of initiatives that involve the co-ordinated activities of several federal departments or agencies. See Appendix B for more details.

2.6  Other Accountability Activities

As previously mentioned, evaluation is one type of activity or tool that can be used by managers to demonstrate accountability. This section contains brief descriptions of other tools (e.g., performance measurement, audit), a discussion of how to determine when each is most appropriate, and how they can be used in conjunction with one another.

2.6.1  Performance Measurement

Performance measurement is the regular collection of information for monitoring how a policy, program, or initiative is doing at any point in time. It generally focuses on providing operational performance information to program managers. For many small agencies, one of the first tasks in developing an evaluation function is to develop a performance measurement strategy that encompasses periodic evaluation.

Evaluation and performance measurement are two complementary activities. Evaluation provides accountability by ensuring performance measurement is on track and indicators are appropriate. In turn, performance measurement data are often important data sources for periodic evaluations.

Small Agency highlights Small agencies vary in terms of the relative emphasis they place on performance measurement and evaluation. Some may rely more heavily on ongoing performance measurement, while others may place more emphasis on periodic evaluations. The most important thing is to determine what are an agency's information requirements with respect to decision making, and then ensure that the performance measurement and evaluation strategy is meeting these requirements.

2.6.2  Internal Audit

Internal audit is a function that provides assurances on a department or agency's risk management strategy, management control framework and information, both financial and non-financial, used for decision making and reporting.

The internal audit function does the following:

  • assesses the adequacy of internal control;
  • verifies compliance with established rules, regulations or procedures;
  • assesses the risk of each of the above;
  • systematically reviews operations to ensure conformance with policies, strategies and plans;
  • systematically reviews management practices and controls;
  • reviews information for validity (e.g., financial, operational and management information); and
  • defines policies, projects, initiatives that are managed by an entity (audit domain).

Small Agency highlights In many small agencies , the same person or small groups of people may be responsible for planning, implementing, and managing performance measurement, evaluations, audits, and reviews.

 

"When used in combination, performance measurement, audit and evaluation serve as an effective means to monitor the performance of an initiative throughout its life-cycle."

— RBM E-Learning Tool; TBS Web site

Audit and evaluation

2.6.3  Review

Reviews are often conducted in response to a pressing or immediate need of management. As such, the emphasis is usually on quick generation of sufficient information to inform decision making or reassure senior management of the dimensions of a problem or situation. The methodology used to gather information is usually secondary to developing an adequate answer in a timely fashion (i.e., evaluation or audit protocols and approaches are not adhered to). Although they are useful to address targeted issues, reviews or special studies do not conform to external reporting requirements, project control processes, or standards which delineate a discipline such as audit or evaluation.

2.7  Evaluation in Context

When would you conduct an evaluation?

You would conduct an evaluation when you need information about how and why results were achieved, the extent to which something has been implemented, whether your initiative is relevant, the extent to which the intended impacts have been achieved, what unintended impacts have resulted, and when you want to consider future options or strategies for improvement.

When would you conduct an audit?

You would conduct an internal audit when you need information about the adequacy of internal controls, risk management strategies, extent of compliance with rules, regulations or procedures, management practices and controls, and the extent to which financial, operational, or management information is valid.

When would you conduct a review?

You would conduct a review when there is a pressing need for management information that may be more limited in scope than an evaluation (or outside the scope of an evaluation) and when time or resources do not permit the rigour expected in an evaluation.

Evaluation has a long history in the federal government. More recently, evaluation is perceived as a key tool for public service managers as they address the requirements of recent approaches to management in the public sector. Some of these are briefly described below to situate evaluation in its current context. More detailed information on each of these approaches or initiatives is available directly from TBS. [5] All of the initiatives described below will continue to have an impact on managers in small agencies as they develop capacity in areas such as evaluation, performance measurement, and internal audit.

2.7.1  Results-based Management

Results-based management is a comprehensive approach to management aimed at improving performance through achieving better results.

Shortly after the federal government introduced Results for Canadians in 2000, the Results-based Management and Accountability Framework (RMAF) was introduced. The RMAF is a tool used to plan, monitor, evaluate, and report on the results of a program. The RMAF integrates the evaluation function within the context of results-based management. It is also a link to the Management Resources and Results Structure (see below). RMAFs will be discussed in more detail in Section 4 of this guidebook.

2.7.2  Management Accountability Framework (MAF)

The MAF defines and clarifies management expectations. It is a set of ten statements that summarizes TBS's expectations for modern public service management.

The MAF aims to

  • improve management practices and stewardship of resources across government;
  • align management expectations to the vision of Results for Canadians ; and
  • represent management as a broader integrative function.

The MAF will be used in the following ways:

  • as a basis of dialogue between TBS and departments or agencies;
  • as an assessment tool of organizational health;
  • as input for assessing deputy minister performance; and,
  • for framing future reporting on management.

A graphical representation of the Management Accountability Framework follows.

Management Accountability Framework: Ten Essential Elements of Modern Managemen
Management Accountability Framework: Ten Essential Elements of Modern Management

2.7.3  Management Resources and Results Structure (MRRS)

The MRRS replaces the Planning, Reporting and Accountability Structure (PRAS) policy. In accordance with MAF expectations, MRRS supports governance and strategic direction, accountabilities and results and performance. The new policy is directed to the organizational level and encourages the alignment of programs, resources, and management practices with expected results.

The Program Activity Architecture (PAA) is an element of the MRRS and comprises

  • clearly defined and appropriate Strategic Outcomes; and
  • a complete program inventory that links all agency programs and program activities so that they roll up to these strategic outcomes.

Over time, an integrated MRRS should also include

  • performance measures for each level of the agency's architecture; and
  • a governance structure that defines decision making and accountability by outcome and by program.

The Expenditure Management Information System (EMIS) will provide a common framework that aligns information on priorities, plans, actual expenditures, and results.

2.7.4  Where Does Evaluation Fit In?

With its focus on the "how" and "why" of results, evaluation is a key tool in managing for results. As a part of management practices, evaluation can help to design and improve programs. Evaluation should be used in conjunction with other management tools to improve accountability and decision making.

Evaluations are a critical tool for demonstrating results and performance and as such help to support the other elements contained in the MAF. Moreover, through the inclusion of results and performance as a key management expectation, the MAF makes an explicit commitment to evaluation.

Evaluation can provide timely information on impacts and relevance with respect to strategic objectives of an agency. Hence, evaluation will address some of the information needs as required by MRRS. Annual evaluation plans (organizational level) and RMAFs (program level) provide links to the MRRS.

The graphic below places evaluation within the context of other government initiatives.

Evaluation in Context

Evaluation in Context

Key References

Canadian Evaluation Society. Project In Support of Advocacy and Professional Development: Evaluation Benefits, Outputs, and Knowledge Elements , September 16, 2002.

Goss Gilroy Inc. Review of Approaches in Other Jurisdictions. Lessons Learned from International Experience with Performance Measurement/Evaluation in Small Agencies , 2003.

National Science Foundation. User-Friendly Handbook for Project Evaluation.

Treasury Board Canada Secretariat Web site RBM E-Learning Tool .

Treasury Board Canada Secretariat Web site. Management Accountability Framework

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

Treasury Board of Canada Secretariat. Preparing and Using Results-based Management and Accountability Frameworks , April 2004.

Treasury Board of Canada Secretariat. Results for Canadians , 2000.

 



Date modified: