ARCHIVED - Review of Policy Monitoring

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

4.0 Areas for Improvement

Throughout the study, we observed that certain elements described in the Framework have been adopted within TBS.  We found that certain practices are more effective than others as TBS monitoring tools.  We also observed that policy areas generally understood the level of risk, a fundamental component of the Framework.  However there is an inconsistent application of the monitoring elements throughout TBS.

4.1 Analysis of Effectiveness

After completing our interviews with the representatives of the TBS policy centres and departments, we summarized our understanding of the effective monitoring practices, areas for consideration, level of risk, and constraints to monitoring for each of the policies in the sample.  We met again with the TBS policy centre representatives to confirm our understanding, to present the Framework and to discuss it in relation to the policy.

Next we completed an analysis by assessing the effectiveness of each of the Framework elements for each of the 13 policies in the sample that were identified as medium to high risk.  We combined the assessments of effectiveness for these individual policies and generated a relative effectiveness rating for each of the Framework elements.  The overall rating of effectiveness for any element is influenced by the number of policies in which the element is present and by the assessed effectiveness of the element in each of the policies.

We have illustrated the results of this analysis in the bar chart in Figure 4.  Each bar represents one of the leading practices defined previously in the report by Framework element and is measured against our assessment of a low, medium or high effectiveness rating.

Display full size graphic

There are a number of elements of the Framework that appear to be effectively used by TBS.  For example, as illustrated by the bar chart in Figure 4, "Working Groups and Committees" and "Advice to Departments" have been assessed as effective practices.  There are some other elements in the Framework, such as "Intelligence Gathering" and "Integration with Key TBS Financial Processes", which we consider less important for TBS to address than others.  We will not discuss the elements in these two groups further in the report; however, summary observations, including TBS effective practices, are presented in a table in Section 4.2.10.

We observed that some of the other elements included in the Framework, such as "Committed Funding" are not consistently applied and provide opportunities for improvement.  We consider other elements of the Framework, such as "Key Performance Indicators" to be very important and require special attention.  In the next section, we set out our observations and discuss our recommendations for improvement for the eight Framework elements that fall into these two groups.

4.2 Areas for Improved Policy Monitoring

The Framework has defined nineteen elements for effective policy monitoring.  We have identified nine of these elements with areas for improvement, including eight that were selected from the chart in Figure 4, plus level of risk.  The nine elements are:

• Level of Risk
• Policy Objectives and Outcomes
• Key Performance Indicators
• Committed Funding
• Ongoing Reporting and Analysis
• Effective Implementation
• TBS and Departmental Executive Leadership/Sponsorship
• Actionable Items
• Audit and Evaluation

These are described in the following sections.  Recommendations are included in the areas for consideration.

4.2.1 Level of Risk

Observation

There is an absence of systematic and formalized ranking and monitoring of the level of risk associated with the policies. 

Historically the number of policies has continued to grow in number without any significant elimination or consolidation of less relevant or outdated policies.  As a result, TBS has directed each of its business lines to review the number and content of policies and improve upon their effectiveness.

Areas for Consideration

Risk management is a one of the four levels of focus in the Framework for Policy Monitoring.  This is consistent with the TBS Management Accountability Framework and modern management practices.

We recommend that TBS establish a standard set of criteria for evaluating risks associated with each policy area.  Criteria would include the impact on departments and departmental risk areas.  Additional steps for consideration include:

• Ensure that risk assessments for policies are reviewed on an ongoing basis;
• Ensure that the level of policy risk determines the appropriate levels of monitoring; and
• Ensure that the policy monitoring capacity (resources and skills) is aligned with the level of monitoring desired for moderate to high-risk areas in both TBS and departments.

4.2.2 Policy Objectives and Outcomes

Observation

Generally the policy objectives and monitoring requirements supporting the TBS policies we studied are broad statements without clarity of outcomes or success criteria.  New policies being developed are focusing attention on improved clarity around outcomes, key performance indicators and roles and responsibilities.  For example, the policy on Continuous Learning adopted a different policy format from traditional TBS policies by presenting departmental commitments, expectations, timeframes and providing a detailed description of roles and responsibilities.

Areas for Consideration

As part of TBS' policy renewal effort, we suggest creating improved policy objectives and outcomes that reflect the following:

• Roles, responsibilities and accountabilities for policy outcomes of TBS and departments are well‑defined in the policy;
• TBS provides accurate outcome analysis, identifying outcome gaps and trends across departments;
• TBS capacity exists to prepare outcome gap reports, summaries and analysis.  Information will be aligned with results from other planning and reporting processes (e.g., Management Accountability Framework) prepared by TBS and departments; and
• Policy outcomes including policy implementation schedules are clearly communicated and actionable.

4.2.3 Key Performance Indicators

Observation

The establishment and monitoring of key performance indicators that support policy outcomes was the most significant gap from the sample of policies studied.  Improvements in data and information capture infrastructure and tracking of performance indicators has been an area of increased focus for TBS policies under development (e.g., Employment Equity uses performance indicators to measure performance).

Areas for Consideration

We recommend that TBS continue with the establishment of key performance indicators (KPIs) as a key requirement for policy development and monitoring.  Steps for consideration include:

• Performance criteria are pre-established and analyzed with policy outcomes;
• Departmental KPI reporting is practical and systematic (minimize manual and ad hoc reporting requirements);
• TBS assists departments to develop a standard set of KPIs per policy area; and
• Where practical, KPIs are aligned to executive performance.

4.2.4 Committed Funding

Observation

A common issue raised by TBS interviewees was that limited resources adversely impacts TBS' ability to conduct policy monitoring and reporting.  Furthermore, TBS has faced staff retention issues and its ability to attract top talent in policy areas has been a challenge.

In the case of most policies studied, A-Base funding provides resources to the policy centres within TBS.  Although we did not study the level of resources assigned by policy, it is apparent that the amount of resources and consequently the approach for monitoring varies considerably within TBS.  In some cases the variation in monitoring can be explained by the level of risk associated with the policy.  However, in others, the relationship between risk and resources was not apparent.

In the case of two policy areas (Modern Controllership - Integrated Risk Management and Federal Contaminated Sites Accelerated Action Plan- Contaminated Sites Management Policy) implementation is supported by project funding.

Areas for Consideration

We recommend that TBS assign resources to monitoring activities by adopting the following strategies:

• Monitoring at the Policy Framework Level.  In the future, structured monitoring may occur at the policy framework level rather than at the individual policy level.  This will facilitate increased monitoring in higher risk areas.
• Capacity/Resources for Monitoring.  Given the large number of TBS policies, an initial approach to adopt the Framework would be to focus resources on high-risk policy areas.
• Key Monitoring Elements to Focus.  Given the current TBS resource constraints, key performance indicators, policy outcomes, follow-up of departmental action items and follow-up of implementation are important elements to focus on for the initial adoption of the Framework.

4.2.5 Ongoing Reporting and Analysis

Observation

TBS continues to evaluate its reporting requirements with departments through various initiatives including the Management Accountability Framework.

Reporting tends to be statistical with limited follow-up by TBS.  Reporting practices varied considerably within the sample of policies studied.  These include the following:

• Annual reporting of statistics, financial indicators and volumes (Prevention and Resolution of Harassment, Access to Information, Contracting, Contaminated Sites, Continuous Learning);
• Formal TBS analysis and feedback to departments (Employment Equity, Official Languages);
• Policy implementation status reporting (Government Security Policy, Integrated Risk Management, Transfer Payments); and
• Action item tracking and follow-up (Contracting).

Qualitative analysis was minimal and was generally attributed to lack of TBS resources.

Areas for Consideration

We recommend adoption of additional ways to enhance monitoring and reporting including the following:

• Structured Reporting Approach – Adoption of a structured approach to reporting for each policy framework would include the following elements:
  • Reporting by policy outcomes (policy implementation, policy results);
  • Pre-determined reporting schedule (quarterly, yearly); and
  • Leveraging existing departmental reporting tools and processes.
• Analytical Processes and Dissemination – TBS has a significant number of data gathering processes and information sources within its organization (Program Sectors, Centres of Excellence, Policy Centres, departmental executive reporting).  More meaningful and value added feedback to departments would include leading practices, government-wide trends, analysis, departmental comparisons and benchmark reporting.

4.2.6 Effective Implementation

Observation

Our study identified a number of effective practices for policy implementation which include the following:

• Data Gathering on Progress of Policy Implementation – Policies, once rolled-out, require ongoing post implementation support and monitoring.  For example, Integrated Risk Management will continue to conduct surveys gather information and prepare TBS reports on implementation progress.
• Implementation Tools – Certain sampled policies have supported implementations with a number of tools and techniques:
  • Communication tools/website (Integrated Risk Management);
  • Implementation guidelines, checklists, audit guidelines and training material (Privacy Impact Assessment, Integrated Risk Management, Contracting); and
  • Mid-term reports (Government Security Policy and Continuous Learning).
• Pilot implementation – A number of policies have been designed, developed and implemented initially as a pilot in key stakeholder departments, followed by a broader rollout to other departments (Integrated Risk Management Framework, Privacy Impact Assessments).  This approach allows for the application of lessons learned, and refinements to the policies, support tools and guidelines in an accelerated manner prior to the full rollout of policy.

Areas for Consideration

In addition to the effective TBS implementation tools referred to above, we recommend TBS consider more consistent adoption of the following practices to assist in monitoring policy implementation:

• Implementation Working Team – A sub-team to the leadership team is involved from design through implementation and support.
• Well-defined and Timely Implementation Milestones – Implementation targets are clearly defined in the policy outcomes.  These will include the following:
  • Establishment of key performance indicators supporting policy outcomes and objectives; and
  • Departmental implementation milestones, which include tailoring the policy to address department specific issues, departmental training and reporting on policy outcomes.
• Monitoring Policy Implementation – Progress is measured against the milestones including the following:
  • Ongoing review of implementation progress by department; and
  • Independent audit/evaluation of implementation and effectiveness of policy at a predetermined milestone (e.g., mid-term report).
• Implementation Support – Roll-out of the policy is accelerated with implementation supported with the following change management elements:
  • Training material;
  • Guidelines and checklists to assist departments with implementation;
  • Lessons learned and best practices from pilot and early adopters of policy;
  • Provision for TBS subject matter experts available to support implementation.

4.2.7 TBS and Departmental Executive Leadership/Sponsorship

Observation

From the policies studied, we observed several practices demonstrating executive leadership from TBS and key stakeholder departments that supported policy direction, development, and monitoring.  The Clerk of the Privy Council also provides leadership in some areas in terms of establishing government-wide management priorities.  The leading practices we observed include:

• Performance management agreements require the achievement of policy outcomes (Integrated Risk Management Framework, Employment Equity);
• TBS Secretary conducts follow-up of compliance to policy implementation and deliverables (Integrated Risk Management);
• TBS Secretary and deputy heads reporting through the Management Accountability Framework process; and
• TBS executive sponsors are active in the design and implementation of policy (Privacy Impact Assessments).

Areas for Consideration

For high-risk policy areas, we recommend that TBS develop an escalation protocol and procedures within TBS and with departments to report performance exceptions to TBS executives.  This will require establishment of ongoing reporting and analysis of policy outcome gaps.

4.2.8 Actionable Items

Observation

The study revealed that TBS is challenged by resource constraints impacting active follow-up and monitoring of policy outcomes and performance.  Follow-up and reporting does not appear to be consistent and systematic.

Areas for Consideration

We recommend that consideration be given to improve the mechanisms for information sharing within TBS and to convert information into actionable items for correction including:

• Departmental intelligence gathering from multiple TBS in-house sources such as the TBS portfolio teams;
• Structured department and TBS reporting of action items against policy outcomes;
• Structured escalation of deficiencies in action items to TBS executive level; and
• Ongoing reporting between Secretary and departmental deputy heads.

4.2.9 Audit and Evaluation Monitoring and Reporting

Observation

A majority of the policies studied make reference to reliance on the conduct of departmental internal audits as the key, and in many cases, sole identified monitoring mechanism.  However, these specific responsibilities are not consistent with the role of internal audit.  The Policy on Internal Audit affirms the function of internal audit as a provider of assurance services to departmental management.  Audit engagements are structured to the specific needs of the organization, as determined through risk assessments and consultation with senior management.  While internal audits provide assurance that the appropriate monitoring and control frameworks are in place, they should not be considered the primary mechanism for monitoring and reporting on individual policies.  Internal audit plans are department focused and therefore do not provide consistent coverage across the government.

Evaluations are distinct from audits.  While audit provides assurance services, evaluation is a management tool that has two main purposes:

• To help managers design or improve the design of policies, programs and initiatives; and
• To provide, where appropriate, periodic assessments of policy or program effectiveness, of impacts both intended and unintended, and of alternative ways of achieving expected results.

There are nine policies in the study sample with a specific requirement for departmental internal audits or evaluations of the policy.  However, Contracting, Transfer Payments and Government Security Policy are the only policies where we found that internal audits or evaluations are being completed.

We observed that information from departmental audits is not easily shared and disseminated in a meaningful way within TBS.  Internal audit reports are circulated across TBS, and are reviewed independently in many areas, but there does not appear to be a centralized approach to assessing the impact of the information contained the audit reports.

Area for Consideration

We recommend that the requirement for internal audits of compliance of specific policies be discontinued as a generic policy requirement since it does not reflect the way in which internal auditing is planned and conducted.  The risk-based audit approach undertaken by TBS and departments does not align with the specific policy monitoring required for TBS and departments.  Internal audit should be viewed as a provider of assurance over and above the other monitoring and reporting mechanisms adopted for policies.  If appropriate, the policy could refer to the role of internal audit by making reference to the Policy on Internal Audit.

We also recommend that consideration be given to identify emerging risks and share audit results including:

• Integration of Policy Centre Analysis with Centres of Excellence Reviews of Audits and Evaluations – Formal monitoring of policy risk is conducted by Policy Centres and is communicated to the TBS Centre of Excellence for monitoring purposes.
• Knowledge sharing of departmental audit and evaluation findings – Ongoing communication of audit results is channeled through TBS Centres of Excellence.  Formal analysis and assessment is shared with Policy Centres and Program Sectors through existing TBS channels (e.g., portfolio teams and TBS communities of interest).
• Collaborative Approach to Follow-up of Results – Follow-up of departmental audit plans and results and identification of emerging areas of policy risk is a shared responsibility of Centres of Excellence, Program Sectors and Policy Centres.  TBS portfolio teams may be a key enabler that can be re-established to address issues and strategies for follow-up.

4.2.10 Other Observations

In developing the Framework, a number of effective TBS monitoring activities from select policies were identified and are included as key elements. These are briefly listed in the table below.  

Element of Framework	TBS Policies	TBS Effective Practice
Existing Legislation External Bodies	Employment Equity Official Language Access to Information	Audit by External Body (legislative) Annual Reporting to Parliament Performance review and "report cards"
Effective Working Groups/Committees	Federal Contaminated Sites Management Policy Integrated Risk Management (IRM) Official Languages	Contaminated Sites Working Committee Implementation Council (IRM) Departmental Advisory Committee on Official Languages
Integration with Key TBS Financial Processes	Contaminated Sites Management Policy Privacy Impact Assessment	Program funding from Federal Contaminated Sites Accelerated Action Plan TB Project Submissions
Executive Level Reporting	Employment Equity Official Language Access to Information	TBS Annual Reports to Parliament TBS Reports to President of Treasury Board Departmental reports to Parliament
Communications	Integrated Risk Management	Websites
Intelligence gathering	Various	TBS Program/Policy Forum
Knowledge Sharing Tools	Privacy Impact Assessment Integrated Risk Management Contracting	Best practices Training material Implementation guidelines Audit Criteria
Advice to departments	Government Security Policy	TBS Subject Matter Expertise
Strong Relationships with departments	Contaminated Sites Management Policy Integrated Risk Management	Smaller communities and stakeholders

Management Response

The senior management of Treasury Board Secretariat (TBS) acknowledges the necessity of having effective policy monitoring practices in place at TBS and generally agrees with the recommendations stemming from this review, which aim to improve these practices at TBS. 

Leading practices in policy monitoring activities at TBS were identified and presented in a Policy Monitoring Framework ("the Framework") as a tool for TBS to achieve a systematic approach to defining policy monitoring and reporting for policies that are developed in the future.  It was suggested that adoption of the Framework would lead to improved policy monitoring. 

We agree to take steps towards adopting the critical elements of the proposed Policy Monitoring Framework ("the Framework") outlined in the report including, as a priority, four of the nine recommended areas of focus plus one additional area:

• Key Performance Indicators
• TBS Committed Funding
• Ongoing Reporting and Analysis
• Actionable Items, and
• Levels of Risk.

On the issue of risk, we agree with the general proposals for evaluating the risks associated with each policy area, including the needs for periodic risk assessments to determine appropriate policy monitoring levels, and for ensuring the alignment of necessary capacity (resources and skills) to accomplish policy monitoring.  Specific attention will be paid to the way in which high-risk policies are monitored, and how structured monitoring can be built in to the policy framework.

In the context of the present TBS Executive review of strategic business re-engineering initiatives (of which the TB Management Policy Review is one), we will review the internal distribution of resources to support the necessary monitoring of high-risk policies, and recommend necessary incremental investments or resource reallocations to that end.  These decisions will be made during this fiscal year.

As recommended by the Review of Policy Monitoring, and consistent with the principles of the current TB Management Policy Review, we will continue to consider and implement improvements in the areas of:

• Policy Objectives and Outcomes
• Effective Implementation
• TBS and Departmental Executive Leadership/Sponsorship, and
• Audit and Evaluation.

In general, we will ensure that policy monitoring is conducted in a way consistent with the new approach to strategic oversight and risk assessment being adopted by TBS – including through related assessments of departmental and government-wide management and expenditure performance, based on the Management Accountability Framework.  We will address the identified requirements for policy monitoring during the course of the TB Management Policy Review, whose main timelines for implementation are as follows:

• Content review for procurement and real property policies – by March 31, 2005
• Content review for all other policies – by March 31, 2006
• Full implementation for all policies – by March 31, 2007.

General Remarks

The recommendations made in the review are meant to provide guidance to the TB Management Policy Review, as it proceeds to streamline TB policies and reporting requirements.  Through the work of the TB Management Policy Review, a coherent and consistent approach to these recommendations will be taken across the policy suite. 

The TB Management Policy Review will examine ways in which policy monitoring and reporting requirements can be better linked to the Management Accountability Framework and the Expenditure Management Information System.  This will help to ensure better knowledge capacity, better knowledge transfer and less redundancy in the reporting of results.

The review suggests that "the requirement for internal audits of compliance of specific policies be discontinued as a general policy requirement since it does not reflect the way in which internal auditing is planned and conducted."  Recognizing that risk-based audit planning practices are accepted across government, we appreciate the argument for discontinuing general policy requirements for internal audits of compliance within policies.  We will look at possibilities for discontinuing the audit requirements articulated in specific policies, on a policy-by-policy basis.  However, existing requirements will not be discontinued outright until other options for providing similar oversight capacity are identified and in place.  Until then, the requirement for internal audits of compliance with specific policies will remain.

We would suggest that specific audit requirements within policy instruments should be contemplated for circumstances of generalized risk related to the policy, and in consultation with TBS internal audit policy experts.  In the context of broader reviews (e.g. to strengthen compliance and sanctions related to the Financial Administration Act), we are presently exploring the range of mechanisms for ensuring effective compliance with policy and legislation.  The resulting recommendations will influence how we address audit requirements in the TB policy suite.

• Previous Page
• Table of Contents
• Next Page