Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - Review of Policy Monitoring

Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.

1.0 Executive Summary

The successful monitoring of Treasury Board (TB) policies is critical in today's environment of rapid change and challenges in addressing policy compliance and effectiveness.  It is essential that effective policy monitoring practices be in place at the Treasury Board Secretariat (TBS) to ensure the success of TB policies.  This review was requested by the TBS Internal Audit and Evaluation Committee to provide TBS management with an independent study of leading practices in the area of policy monitoring.  The review was designed to assess the current status of policy monitoring within TBS and provide practical examples of leading practices.  The recommendations made in this report will provide guidance for the Policy Review and Reporting Project as it proceeds to streamline TB policies and reporting requirements.

Leading Practices Research

The review has researched leading practices for policy monitoring as documented in literature.  Our research indicates that the critical success factors for implementing effective monitoring and reporting include the following:[1]

  • Effective working relationships with departments to ensure provision of information to the central agency;
  • Strong intelligence gathering skills on policy outcomes and processes;
  • Capacity to understand, process, and utilize information based on robust internal analytical processes;
  • Internal information-sharing linking strategic priorities and policy development to monitoring information;
  • Clear understanding of the central agency's objectives and the ability to promote these to departments; and
  • Ongoing evaluation of the information from departments and assessing its consistency with information prepared and processed by the central agency.

Proposed TBS Framework for Policy Monitoring (the "Framework") 

The review also studied policy monitoring practices at TBS by reviewing seventeen TB policies.  Based on our findings, leading practices have been presented in a framework to provide a tool for TBS to effectively analyze policy outcomes and monitoring activities in a consistent and systematic manner. 

There are four levels of focus in the Framework:

1. Level of Risk Risk is the overriding factor that impacts all elements of policy monitoring.  Policies that have greater risk need to be emphasized to ensure they have effective monitoring procedures in place.  The continued effectiveness of these procedures should be evaluated on an on-going basis.

2. Policy Objectives and Outcomes Policies are designed to support the achievement of an organization's objectives and the management of risks.  The following elements are fundamental requirements for an effective policy to achieve its desired outcomes:

  • Policy objectives are clear and concise;
  • Outcomes are well defined and measurable, with performance targets and indicators[2]; and
  • Roles and responsibilities are well defined and communicated to departments.

3. Process Enablers Policy monitoring "process enablers" are mechanisms that assist TBS and departments with information sharing and reporting in order to achieve TBS policy objectives.  Policy monitoring is effective where process enablers are systematically applied to develop and implement policy and monitor outcomes.

The Framework defines seven process enablers.  The one that appears to be the most effectively applied is TBS and departmental working groups and committees.  The process enabler that appears to be the least effectively applied at TBS is committed funding, due to insufficient resources.

4. Monitoring and Reporting Activities Monitoring and reporting requires multiple approaches and activities for information gathering and follow-up.  Information assists TBS and departments in setting priorities and compiling outcome gap analysis.  Structured reporting relationships are required between TBS and departments to report and analyze outcomes on a timely and systematic basis.

The Framework defines ten policy monitoring and reporting activities.  The one that appears to be the most effectively applied is TBS' ongoing advice to departments.  The activities that have been under-utilized or are the least effectively applied are the use of key performance indicators, ongoing reporting, analysis and follow-up by TBS, and dependence on internal audit and evaluation reports.

The Framework has been presented to and discussed with the representatives of the TBS policy centres, and other TBS staff we interviewed in the study.  The Framework was well received and there was a general consensus that adoption of the Framework will lead to improved policy monitoring.

Summary of Findings Related to Selected TB Policies

Each element of the Framework was compared to the results from the sample of policies studied.  We grouped the sample of policies by level of risk, and charted the level of monitoring identified in comparison to a desired level.  From this exercise, we made three observations.

  1. Generally, TB policies identified as high risk need to enhance their level of monitoring and reporting activities;
  2. Generally, policies supporting medium risk areas have adequate levels of monitoring; and
  3. Generally policies seen as low risk areas are not receiving the minimum threshold levels of monitoring.

Areas for Improvement

Our study has revealed that certain elements described in the Framework have been adopted within TBS.  We also observed that policy areas generally understand the level of risk, a fundamental component of the Framework.  However there is an inconsistent application of the monitoring elements throughout TBS and certain practices are more effective as TBS monitoring tools.

The report recommends that TBS should adopt the Policy Monitoring Framework as a tool for a systematic approach to defining policy monitoring and reporting for policies that are developed in the future.  A portfolio of the Framework's process enablers and policy monitoring and reporting activities should be selected and tailored to each policy area based on the level of risk.

Recommendations are also made for improvement in the use of nine of the elements of the Framework.  Of these, TBS should initially focus on the following four areas.

1. Key Performance Indicators 

We recommend that TBS continue to establish key performance indicators (KPIs) as a key requirement for policy development and monitoring.  KPIs should be practical and cost-effective to measure.

2. TBS Committed Funding

We recommend a more effective use of TBS resources in conducting monitoring of policy outcomes.  Steps would include:

  • Monitoring high-risk policy areas Given the large number of TBS policies, an initial approach to adopt the Framework would be to focus resources on high-risk policy areas.  In most cases, structured monitoring may occur at the policy framework level rather than at the individual policy level.
  • Key elements to monitorGiven the current TBS resource constraints, key performance indicators, policy outcomes, follow-up of departmental action items and implementation are the key elements to focus on.

3. Ongoing Reporting and Analysis

We recommend that TBS adopt a structured reporting approach to policy monitoring including:

  • Reporting by policy outcomes (policy implementation, policy results);
  • Pre-determined reporting schedule leveraging existing departmental reporting tools and processes;
  • Improved analytical processes and dissemination;

TBS providing more meaningful and value added feedback to departments comprising government wide trends and benchmarks as well as departmental comparisons.

4. Actionable Items

We recommend that consideration be given to improving mechanisms for information sharing within TBS and to convert information into actionable items for correction including:

  • Departmental intelligence gathering from multiple TBS in-house sources through the TBS business planning team mechanism;
  • Structured department and TBS reporting of action items against policy outcomes;
  • Structured escalation of deficiencies in action items to the TBS executive level; and
  • Ongoing reporting between the Secretary and departmental deputy heads, using such mechanisms as the Management Accountability Framework.

Management response

Due to the nature of the report, management has chosen to provide an overall response to the findings and recommendations.  See the management response.