ARCHIVED - Review of Policy Monitoring

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

3.0 Review Results

Introduction

The review was designed to provide senior management with an assessment of leading practices in the area of policy monitoring that will act as a support tool to TBS' Policy and Reporting Review Project.

Our research into external leading practices included "thought leadership" in the areas of corporate governance, performance monitoring and reporting, and management control frameworks.  We consulted the work of the Canadian Institute of Chartered Accountants (Study of Control Criteria), Conference Board of Canada, the Committee of Sponsoring Organizations of the Treadway Commission and author Dr. John Carver. 

Practices from New Zealand's Pacific People's Monitoring Framework provided valuable insight into monitoring frameworks.

Further, the review team has identified effective practices within TBS as a result of studying the sample of policies from each TBS business line.

The identified leading practices have been consolidated and are presented in the following sections of the report.  They are introduced in our proposed model - TBS Framework for Policy Monitoring (Section 3.1).  Each of the key monitoring elements of the framework is then defined.  The definitions are followed by our findings from the sample of TBS policies studied (Section 3.2).  In Section 4.0, we present our analysis of TBS current monitoring practices against the leading practices of the framework.  The areas for improvement identified in the comparison are set out for consideration.

3.1 Summary of Leading and Effective Practices

Our research indicates that the critical success factors for implementing effective monitoring and reporting include the following:^[5]

• Effective working relationships with departments to ensure provision of information to the central agency;
• Strong intelligence gathering skills on policy outcomes and processes;
• Capacity to understand, process, and utilize information based on robust internal analytical processes;
• Internal information-sharing linking strategic priorities and policy development to monitoring information;
• Clear understanding of the central agency's objectives and the ability to promote these to departments; and
• Ongoing evaluation of information received from departments and assessing its consistency with information prepared and processed by the central agency.

Based on this information, we developed the framework in Figure 1 to provide a tool for TBS to develop an effective approach to policy monitoring that can be implemented in a consistent and systematic manner.  It highlights the importance of obtaining information about policy performance from departments and to effectively analyze policy outcomes.

The four levels of focus of the TBS Framework for Policy Monitoring (the Framework) as illustrated in Figure 1 are: Level of Risk; Policy Objectives; Process Enablers; and Monitoring and Reporting Practices.

Display full size graphic

The following explanations describe the four levels of the framework.

1. Level of Risk 

An organization needs to identify its significant internal and external risks on an ongoing basis so that it can react to or initiate changes in an appropriate and timely manner.  Monitoring needs to be structured according to the risk of not meeting the desired policy outcomes.^[6]

Risk is the overriding factor that impacts all elements of policy monitoring.  Policies that have greater risk need to be emphasized to ensure they have effective monitoring procedures in place.  The continued effectiveness of these procedures should be evaluated on an on-going basis.

2. Policy Objectives and Outcomes

Policies are designed to support the achievement of an organization's objectives and the management of risks.  The following elements are fundamental requirements for an effective policy to achieve its desired outcomes:

• Policy objectives are clear and concise;
• Outcomes are well defined and measurable, with performance targets and indicators^[7]; and
• Roles and responsibilities are well defined and communicated to departments.

3. Policy Monitoring Process Enablers

Policy monitoring "process enablers" are mechanisms that assist TBS and departments with information sharing and reporting in order to achieve TBS and departmental policy objectives.

Policy monitoring is effective where process enablers are systematically applied to develop and implement policy and monitor outcomes^[8].  Process enablers seek to:

• Provide an information stream between stakeholders to support policy development in strategic areas;
• Provide legislation that defines roles and responsibilities;
• Provide mechanisms through which groups work collaboratively to meet policy objectives;
• Encourage central agencies and departments to establish processes to monitor policy implementation; and
• Ensure funding and staffing capacity is available to support policy effectiveness and monitoring.

4. Monitoring and Reporting Practices

Monitoring and reporting requires multiple approaches and activities for information gathering and follow-up.  Three policy performance areas requiring monitoring include:

• Implementation Effectiveness – Strong relationships, the availability of subject matter experts and sharing of best practices between TBS and departments facilitate policy implementation and contribute to continuous improvement in the policy area.
• Policy Compliance – Information gathered assists TBS and departments in monitoring policy compliance, identifying gaps and initiating corrective adjustments to TB policies. 
• Outcome Monitoring – Information gathered assists TBS and departments in setting priorities and compiling outcome gap analysis.  Structured reporting relationships are required between TBS and departments to report and analyze outcomes on a timely and systematic basis.

TBS Framework for Policy Monitoring

This study identified seven leading "Process Enablers" and ten leading "Monitoring and Reporting" practices that have been incorporated into our proposed TBS Framework for Policy Monitoring illustrated in Figure 2.  The following sections further define the "Process Enablers" and "Monitoring and Reporting" elements of the Framework.

Display full size graphic

3.1.1 Policy Monitoring Process Enablers

The Framework's seven leading process enablers and their desired characteristics are:

1.  TBS and Departmental Executive Leadership/Sponsorship – TBS and departmental executives' expectations and accountabilities for achieving policy objectives and monitoring performance are well understood and aligned.  TBS and departmental leadership teams are committed to and focused on policy improvement.  TBS and departmental leadership is well informed, and follows up on the results of actions taken.

2.  Effective Implementation – Policy development and implementation is effective where policy outcomes are well defined, policy schedule and milestone targets are established and these are agreed upon and followed-up by executives.  Support tools, such as training material, guidelines, checklists, and lessons learned accelerate the rollout of the policy.

3. Committed Funding – Adequate resources for monitoring are committed to provide skilled staff and tools for information gathering, key performance indicator monitoring and action item follow-up.  Monitoring activities are designed as an integral part of the organization (A-Base budgets) taking into consideration the risks to achieving objectives and the interrelationship with other monitoring elements.

4. Enabling Legislation – The existence of prescriptive legislation (e.g., Employment Equity Act) with well-defined requirements for annual reports to Parliament, audits, and roles of departmental officials is an effective enabler.  The legislation provides additional authority that focuses on compliance to government priorities.

5. Oversight by External Bodies – The existence of external bodies (e.g., Commissioner of Official Languages) empowered by legislation to conduct independent audits, and follow-up of complaints is an effective enabler.  External bodies provide the independent focus that can bring valuable feedback to TBS and departments.

6. Working Groups/Committees – Formalized working groups, chaired and co-chaired by primary stakeholders, are effective policy enablers.  These formal groups have well-defined deliverables for policy design, implementation and follow-up.  For optimal effectiveness, executive leadership teams from TBS and departments sponsor policy initiatives from design through to implementation and policy support.

7. Integration of Policy Monitoring with Key TBS Financial Processes – Transaction monitoring by TBS (e.g., Privacy Impact Assessments) is a detailed approach to monitoring specific policy requirements.  It is effective in identifying cases of non-compliance or issues identified during a submission review.  In order to be effective, sufficient TBS resources must be in place to handle the volume of submissions.

These seven process enablers represent a basket of effective practices to strengthen the achievement of the policy objectives.  In developing each policy, an optimal combination of these practices should be selected.  The appropriate combination or portfolio will vary for each policy area based on the assessed level of risk and on other factors.

3.1.2 Monitoring and Reporting Practices

The Framework's ten leading policy monitoring and reporting practices and their desired characteristics are:

1.  Key Performance Indicators Reporting (KPIs) – Organizations use a mixture of quantitative and qualitative performance targets to measure progress towards achieving the defined policy objectives.  These targets are measurable and aligned with the objectives.^[9]  Externally derived benchmarks are also used to help ensure targets are aligned with external best practices.  Where performance criteria have pre-established targets and are analyzed against policy outcomes, then the performance indicators provide early warning signals as to whether the targets have not been met.  It is critical to the success of KPIs that departmental KPI reporting is practical and supported by existing reporting systems.

2.  Executive Level Reporting – Executive level reporting and communicating of progress against policy outcomes enables TBS and departments to complete their priority setting and decision-making activities.  The enhancements that are being made to management reporting between TBS and departments through various initiatives, such as the Management Accountability Framework, place greater focus on outcomes and performance indicators.  The terms of executiveperformance management agreements also endorse achievement of policy outcomes.

3.  Strong Relationships with Departments – Developing and maintaining good working relationships between TBS and departments to ensure provision of information is necessary for ongoing monitoring.^[10]  TBS plays an effective advisory role with the departments where it contributes the following:

• Policy area subject matter expertise;
• Effective knowledge sharing, including posting best practices on websites, and providing training material, guidelines and checklists; and
• Both proactive and responsive guidance to departments in the application of policies.

4.  Ongoing Reporting and Analysis – Departments and TBS each have timely and reliable information to evaluate compliance to policy objectives and to measure against expectations, with performance primarily monitored by the departments.  Leading departments have the capacity to understand, process, and utilize information based on robust internal analytical processes.  Central agencies also monitor the information provided by departments and assess its consistency with information prepared and processed by the central agency.

5.  Audit and Evaluation Monitoring and Reporting – Monitoring is best conducted using multiple approaches including ongoing reporting and analysis.  Independent audits and evaluations conducted by internal audit and evaluation and by external bodies provide a fresh focus on the effectiveness of the policy as well as the effectiveness of monitoring.^[11]  However, audit and evaluation engagements are structured to the specific needs of the departments, as determined through risk assessments and consultation with senior management.  They are not driven by individual policy requirements.  Therefore, these activities are, at best, viewed as supplementary assessments of the adequacy of the monitoring and reporting processes in place and not as a primary tool for providing feedback on performance in meeting the policy objectives.

6.  Advice to Departments – A good understanding of policy objectives and priorities and the ability to promote these to departments is a critical success factor for effective policy development, deployment and monitoring.^[12]  The central agency positions itself as the "trusted advisor" or subject matter expert working collaboratively and in partnership with departments.  The central agency takes a proactive role in identifying and addressing issues in addition to reacting to problems as they arise.

7.  Intelligence Gathering – Monitoring the external environment provides valuable information on the state of the internal environment.  Changes in external conditions (such as changes in regulations, societal trends, and program and political changes) can have a significant effect on departments' and TBS' ability to meet their objectives.

Intelligence gathering is facilitated by various tools and techniques such as:

• Information sharing groups (TBS portfolio teams); and
• Financial planning and reporting tools.

8.  Knowledge Sharing Tools and Processes – Sharing best practices and lessons learned from stakeholder departments and TBS promotes effective policy implementation.  Information obtained and shared provides a range of approaches to addressing policy concerns and the achievement of policy outcomes.  Building a culture of knowledge capture and sharing further enhances the TBS' leadership and "trusted advisor" role with departments.

9.  Communication – Open communication both creates and depends on trust.  A high level of trust encourages people to ensure that everything of importance is known by others.  Communication processes support common values and achievement of objectives.  For monitoring to be effective, there are communication processes capable of supporting two-way, open communication of timely, relevant and reliable information.  These include:

• Broad Base Communication Practices:  Ongoing communication around policy issues, development and improvements (e.g., bulletins and websites).
• Performance Reporting:  Structured information gathering (e.g., surveys, formal meetings) and dissemination of results from action plans and follow-up of policy outcomes.

10.  Actionable Items - A well established reporting and follow-up protocol between departments and TBS is in place to implement corrective action for policy outcome gaps. 

These ten policy monitoring and reporting activities represent a basket of effective practices to strengthen the feedback mechanisms for assessing the achievement of policy outcomes.  In developing each policy, an optimal combination of these practices should be selected.  The appropriate combination or portfolio will vary for each policy area based on the assessed level of risk and on other factors.

3.1.3 Adoption of the Framework for Policy Monitoring

The Policy Monitoring Framework has been presented to and discussed with the representatives of the TBS policy centres, and other TBS staff we interviewed in the study.  The Framework was well received and there was a general consensus that adoption of the Framework will lead to improved policy monitoring.

Recommendation

We recommend that TBS adopt the Policy Monitoring Framework as a tool for taking a systematic approach to defining the policy monitoring approach for policies that are developed in the future.

We also recommend that TBS adopt a portfolio of the Framework's process enablers and policy monitoring and reporting practices that is tailored to each policy area and level of risk to improve policy monitoring.

See Management Response

3.2 Summary of Findings Related to Selected TB Policies

Seventeen TB policies were selected for review with key TBS and departmental representatives responsible for the selected policy areas.  The purpose was not to critique or comment on specific policy issues.  Rather, the objective was to identify and discuss general trends and issues in policy monitoring including:

• Level of risk – The sample of policies selected included higher risk and politically sensitive areas as well as routine and stable policy areas.  We discussed the level of risk associated with each policy with TBS and departmental policy centre representatives. 
• Approach to policy monitoring – The approach to and level of monitoring activities varied across all policies and our analysis assessed effective levels of monitoring.  These also were discussed with key policy stakeholders.

Based on our analysis, we have grouped the seventeen policies by their level of risk and our assessment of the level of monitoring activity and presented the results in Figure 3.  The level of risk is presented on the horizontal axis (Low/Medium/High) and the level of monitoring is presented on the vertical axis (minimum/moderate/enhanced).  Also presented in Figure 3, is the "Desired State", which is the recommended level of monitoring associated with each level of risk.  Effective policy monitoring requires a balance between TBS monitoring activities and the level of risk associated with the policy area.

Display full size graphic

Following are three observations with respect to the level of monitoring being conducted from the policies studied in the review.

1. Strategic Impact – Enhanced Level of Monitoring – Generally TBS policies identified as high risk areas need to enhance their level of monitoring and reporting activities to address their strategic significance. These policies in a number of cases have less monitoring than policy areas with medium risk.

2. Operational Excellence – Moderate Level of Monitoring – Policies supporting medium risk areas generally have adequate levels of monitoring.  From our study, it would appear that some policy areas with medium risk are over monitored while other medium risk policies are not sufficiently monitored.

3. Operational Stability – Minimum Level of Monitoring – Policies that are seen as low risk areas require a minimum level of TBS monitoring to ensure stability is maintained.  Generally, low risk policies were not receiving minimum threshold levels of monitoring.

• Previous Page
• Table of Contents
• Next Page