ARCHIVED - Evaluation of the Treasury Board Submission Process
This page has been archived.
Archived Content
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
Evaluation of the Treasury Board Submission Process
Table of Contents
List of Appendices
- Appendix A: Logic Model
- Appendix B: Treasury Board Submission Process Evaluation Matrix
- Appendix C: Documents Reviewed
- Appendix D: List of Interviewees
- Appendix E: Interview Guides
- Appendix F: ADM Working Session Questionnaire
- Appendix G: Survey Results
- Appendix H: Description of Tools
- Appendix I: Description of EXCO Measures
- Appendix J: Definitions
List of Tables
Table 1: "Standard" vs. "Complex" Submissions- Table 2: Summary of the Evaluation Issues and Questions
- Table 3: List of Interview Group
- Table 4: Survey Response Rates
- Table 5: Assessment of Understanding of the Treasury Board Submission Process
- Table 6: Views on Drafts Submitted That Did Not Require a Treasury Board Submission
- Table 7: Assessment of the Submission Review Process
- Table 8: Survey Respondents' Views on Quality of Secretariat Advice/Interpretation - Review Stage
- Table 9: Comparison of Perceptions of Time Required to Review Treasury Board Submissions
- Table 10: Program Analysts' Perception of Change in Overall Quality of Submissions
- Table 11: Assessment of Documents Submitted for Review
- Table 12: Proportion of Time Analysts Spent on Stages of the Submission Process
- Table 13: Assessment of the Efficiency of the Treasury Board Submission Process
- Table 14: Assessment of Proposed Measures' Potential to Improve Efficiency of the Treasury Board Submission Process
List of Acronyms
| Acronym | Meaning |
|---|---|
| ADM | Assistant Deputy Minister |
| ARLU | Annual Reference Level Update |
| BOSR | Budget Office Systems Renewal |
| CFO | Chief Financial Officer |
| CIO | Chief Information Officer |
| COE | Centre of Expertise |
| DM | Deputy Minister |
| EMIS | Expenditure Management Information System |
| EMS | Expenditure Management Sector |
| ES-0# | Economics and Statistics (employee classification) |
| EX-0# | Executive (employee classification) |
| EXCO | Executive Committee |
| FAA | Financial Administration Act |
| FTE | Full-Time Equivalent |
| GIC | Governor in Council |
| GOS | Government Operations Sector |
| HR | Human Resources |
| MAF | Management Accountability Framework |
| MC | Memorandum to Cabinet |
| OIC | Order in Council |
| PDG | Program Directors Group |
| RDIMS | Records, Documents and Information Management System |
| STS | Submission Tracking System |
| Secretariat | Treasury Board of Canada Secretariat |
Acknowledgements
The Internal Audit and Evaluation Bureau (IAEB) of the Treasury Board of Canada Secretariat would like to thank all the individuals who contributed to this evaluation-interviewees, survey respondents, working session participants, and the Secretariat's Evaluation Committee-for their valuable input and support. The work of Secretariat officials involved in the validation of findings, particularly the Program Directors Group and assistant secretaries from the program sectors, was also appreciated. Finally, IAEB would like to thank the consultants, Government Consulting Services (Public Works and Government Services Canada), who undertook the evaluation, and Mark Schacter, who collaborated in writing the report.
Executive Summary
Introduction
The Treasury Board of Canada Secretariat (Secretariat) is the administrative arm of Treasury Board, a Cabinet committee whose ministers render decisions on the government's management and expenditures. The Secretariat supports Treasury Board ministers in this function and also fulfills the statutory responsibilities of a central government agency. When a federal organization is seeking approval or authority from Treasury Board for an initiative that it would not otherwise be able to undertake or that is outside its delegated authorities,1 it prepares a Treasury Board submission in consultation with the Secretariat.
This evaluation assessed the relevance, effectiveness, and economy of the Treasury Board submission process. Given that the Secretariat is the "owner"; of the Treasury Board submission process, the evaluation paid close attention to the role it plays in contributing to the success of the process. It should be noted that the scope of this evaluation was limited to the current process used to support Treasury Board ministers and did not include an assessment of the process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation; however, the review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore.
The evaluation framework used multiple lines of evidence and complementary research methods to ensure the reliability and validity of the data. The following research methods were used:
(i) document review; (ii) interviews with stakeholders and experts; (iii) a working session with assistant secretaries from the Secretariat and assistant deputy ministers from selected federal organizations; (iv) review of administrative, financial, and statistical data; and (v) a survey of analysts at the Secretariat and in federal organizations. A limitation of the methodology was that Treasury Board ministers could not be interviewed because data collection was undertaken during an election period.
The evaluation covered all steps of the current submission process involving the Secretariat:
- the support that the Secretariat provides to federal organizations;
- guidance developed by the Secretariat, such as that found in A Guide to Preparing Treasury Board Submissions (the Guide);
- the communication of decisions regarding submissions; and
- the tracking of conditions placed on approved submissions.
The evaluation did not include federal organizations' internal processes for preparing Treasury Board submissions. Furthermore, the evaluation covers only "Part A" submissions.2
This is the first formal evaluation of the Treasury Board submission process; therefore, it was not possible to anticipate, at the outset, all the evaluation issues to be addressed and lines of inquiry to be pursued. Because the data-gathering exercise uncovered evaluation issues that were not initially apparent, new lines of evidence were introduced, such as additional document reviews, additional interviews, and validation sessions with Secretariat senior management. The research for the evaluation was conducted between November 2008 and March 2009.
Conclusion
The current Treasury Board submission process is relevant, appears to be effective, and has key strengths; however, some important opportunities exist to improve and enhance its effectiveness and efficiency, which should be addressed through a combination of action and further investigation.
Strengths:
- The submission process is relevant to the government's good public management needs. Determining whether there are other effective and efficient alternatives to accomplish the same outcomes was outside the scope of the evaluation.
- There is widespread belief among stakeholders that the submission process is successful at ensuring that federal organizations present submissions that comply with legislative authorities and Treasury Board policies.
- There is widespread belief among stakeholders that the submission process is successful at ensuring that federal organizations put forth appropriate submissions.
- Treasury Board decisions appear to reflect the advice of the Secretariat to a high degree, which is indicative of Treasury Board's confidence in the quality of that advice.3
Opportunities for improvement:
- There is a need to finalize the logic model for the Treasury Board submission process to support performance measurement, clear communication of the process's ultimate outcome, and further study.
- A majority of federal organizations are satisfied with the support and advice they receive from the Secretariat's program analysts; however, there are specific areas of concern (e.g. consistency of advice).
- The evidence on the quality of draft Treasury Board submissions is mixed.
- The Secretariat does not account for the costs of the submission process.
- The Secretariat's efforts to educate federal organizations and its own staff about the details of the submission process appear to be falling short of their intended results.
- Rather than a shared, centralized system, analysts at the Secretariat use their own private filing systems to manage documentation related to Treasury Board submissions.
Recommendations
Opportunities to improve the Treasury Board submission process fall into three areas:
- finalizing the logic model for the Treasury Board submission process;
- deepening the Secretariat's understanding of certain aspects of the process itself and practices related to it and implementing improvements to the process wherever possible; and
- providing the Secretariat's analysts with more and better tools to enhance the effectiveness and economy of the submission process.
Logic model for the Treasury Board submission process
1. The Secretariat should finalize the logic model for the Treasury Board submission process to support performance measurement, clear communication of the process's ultimate outcome, and further study.
Deeper understanding of the submission process and related practices, leading to improvements
2. The Secretariat should examine in detail, and address where necessary, survey results relating to the advice its analysts provide to federal organizations during the submission process.
3. Secretariat should examine how human resources issues, especially turnover among program analysts, affect the Treasury Board submission process and knowledge management.
4. The Secretariat should account for the costs of managing the submission process-at least for the department if not for the entire public administration.
Development of more and better tools for the submission process
5. The Secretariat should review, and improve where necessary, its professional development, training, and outreach activities related to the Treasury Board submission process.
6. The Secretariat should explore options for a more systematic approach to information and knowledge management in the Treasury Board submission process.
1. Introduction
The Treasury Board is a Cabinet committee of the Queen's Privy Council of Canada. It was established in 1867 and given statutory powers in 1869. The Financial Administration Act (FAA) provides the authority for the Treasury Board to exercise responsibilities in areas relating to general administrative policy in the federal public administration, the organization of the federal public administration, financial management, the review of annual and longer term expenditure plans and programs of federal organizations, the management and development of lands by federal organizations, human resources management in the federal public administration, the terms and conditions of employment, internal audit, and other matters determined by the Governor in Council (GIC).
The Treasury Board is headed by a President, whose formal role is to chair the Treasury Board. He carries out his responsibility for the management of the government by translating the policies and programs approved by Cabinet into operational reality and by providing federal organizations with the resources and the administrative environment they need to do their work.
The Treasury Board of Canada Secretariat (Secretariat) is the administrative arm of the Treasury Board and has a dual mandate: to support the Treasury Board itself, which is a committee of ministers, in rendering decisions on the government's management and expenditures and to fulfill the statutory responsibilities of a central government agency. With respect to the Treasury Board submission process, the Secretariat provides advice and support to Treasury Board ministers in fulfilling their responsibilities as outlined in the FAA and their role of ensuring value for money in government spending through oversight of federal organizations' financial management functions. The Secretariat makes recommendations and provides advice to the Treasury Board on policies, directives, regulations, and program expenditure proposals involving the management of the government's resources. Treasury Board's responsibilities for the general management of the government have an impact on initiatives, issues, and activities that cut across all policy sectors managed by federal organizations and organizational entities.
A Treasury Board submission is required when a federal organization is seeking approval or authority from Treasury Board for an initiative that it would not otherwise be able to undertake or that is outside its delegated authorities. Treasury Board submissions can relate to one or more of the Treasury Board responsibilities outlined in the FAA.
The Treasury Board submission process has never been evaluated; as a key line of business of the Secretariat, it was important to do so. This evaluation assessed the relevance, effectiveness, and economy of the Treasury Board submission process. Given that the Secretariat is the "owner" of the Treasury Board submission process, the evaluation paid close attention to the role played by the Secretariat in contributing to the success of the process. It should be noted that the scope of this evaluation was limited to the current process used to support Treasury Board ministers and did not include an assessment of the process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation; however, this review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore.
The purpose of the evaluation was to assess the relevance, effectiveness, and economy of the Treasury Board submission process, with a focus on the following issues:4
- Relevance-Does the Treasury Board submission process address a demonstrable need, is it appropriate to the federal government, and is it responsive to the needs of Canadians?
- Effectiveness-To what extent has the Treasury Board submission process achieved its expected outcomes?
- Economy-Does the Treasury Board submission process consume the minimum amount of resources required to achieve its expected outcomes?
The research for this evaluation was conducted between November 2008 and March 2009.
2. Background
(a) Change Agenda
The Change Agenda, an initiative launched in 2007, is the Secretariat's plan for building management excellence across the federal government by refocusing its relationship and the manner in which it conducts business with other federal organizations. Through the Change Agenda, it is expected that the Secretariat will demonstrate leadership in management excellence, while taking a more strategic and less transactional approach to federal organizations. The efficiencies gained from such an approach would allow for a greater focus on higher value-added activities, would improve relationships with federal organizations, and would result in better advice to Treasury Board ministers. Although not a subject of this evaluation, the Change Agenda is integral to the Treasury Board submission process because it addresses how the Secretariat's business is conducted.
(b) Treasury Board submissions
Treasury Board submissions are one of the three main documents, besides Memoranda to Cabinet (MC) and GIC submissions, that support the formal decision-making process in government.
Legislation, Treasury Board policies, or other Cabinet decisions usually establish the requirements for Treasury Board approval.5 While Treasury Board submissions may relate to any of the Treasury Board responsibilities outlined in the FAA, typical examples of submissions include seeking:
- authority to allocate resources previously approved by Cabinet or included in the federal budget;
- authority to make grants or contributions or approval of terms and conditions of grant and contribution programs;
- recommendations of approval of orders in council (OIC) with resource or management implications;
- authority to carry out a project or initiative the costs of which would exceed a minister's delegated authority;
- authority to enter into a contract above or outside a federal organization's or minister's authority; or
- exemption from a Treasury Board policy.
It should be noted that the Treasury Board submission process has many links to the government's budget, Estimates, and supply processes. For instance:
- Federal organizations are required to develop Treasury Board submissions to obtain approval of the specific authorities and resources required to implement initiatives announced in the federal budget.
- Key elements of many Treasury Board submissions are requests for spending authority to be included in the Main and Supplementary Estimates, which are the mechanisms for seeking supply from Parliament.
- The recent renewal of the expenditure management system and the associated Treasury Board submissions requesting authority to carry out the results of the strategic review exercises and reallocate resources from low-priority and low-performing programs to higher priorities are also linked to the budget, Estimates, and supply processes.
Treasury Board submissions are divided into "Part A" and "Part B" submissions. All Treasury Board submissions (including strategic reviews) are considered "Part A" submissions with the exception of OICs requiring Treasury Board recommendation. "Part B" submissions are therefore those with OIC attached. As a result, the process, procedures, and relevant actors for "Part A" and "Part B" submissions differ. While OICs are legislative instruments, Treasury Board submissions are a mechanism used by a legal body, Treasury Board.
(c) Current Treasury Board submission process
What is generally referred to as the "Treasury Board submission process" in fact incorporates three phases: pre-submission, submission, and post-approval (depicted in Figure 1).
The pre-submission phase begins when a federal organization contacts the Secretariat either to seek guidance on whether a submission is required or to give notice that it intends to make a submission. Even when a federal organization is simply giving notice of its intent to put forth a submission, the Secretariat may provide its advice on whether or not the submission is in fact necessary. Strictly speaking, there is no Treasury Board submission at this point. Nevertheless, the evaluation-reflecting general practice within government-treats the related activities as being part of the submission process.
The submission phase covers the period between when a decision is taken by the federal organization to prepare a submission and the point at which the submission is considered at a Treasury Board meeting. This phase includes the draft submission stage in which program analysts, along with their managers, act as the single window to provide advice to representatives of federal organizations, including advice from Centre of Expertise (COE) analysts. It also includes the final submission stage in which a précis is prepared and discussed at the Strategy Committee, a senior governance body chaired by the Secretary of the Treasury Board. The Strategy Committee, which is part of Secretariat-wide due diligence and oversight, enables the provision of considered, coherent, and consolidated advice to Treasury Board ministers. The committee meets frequently and reviews all submissions and associated risk assessments and advice.
The submission is then presented to the Treasury Board for consideration and approval. At this point, Secretariat advice on risks and considerations is also presented to ministers.
The post-approval phase covers actions taken by the Secretariat following approval of a submission, such as communicating the decision to the federal organization, communicating conditions (if any) attached to the submission, and monitoring these conditions.
Figure 1: Treasury Board Submission Process
As noted earlier, the Change Agenda is integral to the Treasury Board submission process. The Secretariat in fact performs several of the roles described in the Change Agenda throughout the submission process. In the pre-submission phase, the Secretariat assumes its enabler role when providing the federal organization with guidance and advice. In the submission phase, the Secretariat challenges the submission. However, once a submission is ready to be presented to Treasury Board, the Secretariat shifts to its support role, providing the best advice and recommendations to Treasury Board ministers. This shift in the Secretariat's role is necessary in the Canadian federal government model6 because the Secretariat both assists the client department in drafting the submission and ensures that the assistant secretary presenting the submission to Treasury Board is well-equipped with the best possible advice to support the recommendation.
The Treasury Board submission process is a deliberate and iterative process, at times involving intensive discussion and negotiation between the Secretariat and federal organizations as well as discussion within the Secretariat itself. Given the Secretariat's single-window approach, much of the process is coordinated by the Secretariat program sector that is responsible for the federal organization's portfolio7 and remains the primary point of contact.
While the program analyst plays the lead role in relation to the client department, many Secretariat sectors, apart from the related program sector (e.g. the Expenditure Management Sector (EMS), the Office of the Comptroller General (OCG), and various policy centres), typically get involved in reviewing draft submissions and formulating advice to Treasury Board ministers. Analysts from these sectors, COE analysts, play a substantially different role than program analysts: they review and provide feedback on the sections of a submission that pertain to their area of expertise and policy mandate.
Following review of the submission, the program analyst drafts a précis, which is essentially the Secretariat's opinion and advice meant for Treasury Board ministers. The précis includes a summary of the federal organization's request, risk assessment against pre-established risk criteria (which are described in the Guide8), and a recommendation to Treasury Board to either approve, not approve, or approve with conditions the proposals put forth in the submission.
Upon consideration of the Treasury Board submission and Secretariat advice, Treasury Board ministers may attach further conditions to a submission beyond those proposed by the Secretariat or may request follow-up from the federal organization or the Secretariat on a particular issue. For example, a condition may relate to the time frame in which to conduct a formal evaluation of a program.
How risk is handled in the Treasury Board submission process
The Guide states that federal organizations should "provide details of specific risks that need to be managed, measures proposed to mitigate those risks, and any residual risks." Beyond this, it becomes the responsibility of Secretariat analysts9 to assess submissions based on a standardized risk framework that includes the following general risk categories:
- complexity of implementing the proposed program;
- robustness of the organization's structures, accountabilities, and management regimes to successfully deliver the proposal;
- previous issues encountered or current issues that will be raised by proceeding with the proposal; and
- financial risk of proceeding with the proposal.
Before preparing the précis, the Secretariat assigns a risk rating-low, medium, high, or very high-to each of these categories as well as an overall risk rating to the submission. The risk ranking is discussed by Secretariat senior management at the strategy meeting and adjusted if required. It serves to frame Secretariat recommendations and advice to Treasury Board ministers.
(d) Treasury Board submission process - One element in a larger process
The Treasury Board submission process is one element in a larger process that involves federal organizations obtaining approval to undertake new initiatives or to continue with existing ones. The following description of this larger process is simplified and does not pretend to cover all cases; however, it is adequate for setting the Treasury Board submission process within the larger context.
Federal organizations undertake their own process of policy development through research and consultation, both internally and interdepartmentally, and ensure that any proposed initiative supports one or more of its substantive policy objectives and is consistent with its mandate. At the end of this developmental process, the sponsoring minister seeks approval for the new initiative. The primary document used for this purpose is the MC. The MC is the vehicle through which the minister proposes and explains the organization's new initiative to Cabinet, including related options and considerations, and obtains its approval.
The MC gives Cabinet an overview of the proposed new initiative's objectives and financial implications and of the links between the recommended course of action and the government's policies and objectives. It includes a communications plan. Central agencies such as the Privy Council Office (PCO), the Department of Finance Canada, and the Secretariat all have roles to play in the MC process.
PCO is responsible for the MC process and it advises the federal organization on, among other things, whether the aims of the initiative described in the MC are consistent with government priorities. The Department of Finance Canada supports PCO by providing advice and guidance on the MC's fiscal information to ensure that it fits with the government's fiscal priorities. The Secretariat, in its supporting role, helps ensure that the implementation requirements are realistically stated, that the MC includes appropriate accountability and transparency provisions, that it is consistent with Treasury Board policy, and that it makes appropriate linkages between the proposal and other existing programs and federal organizations.
A Cabinet policy committee considers the MC and recommends whether the proposal should be referred to Cabinet for ratification. The MC may either be fully approved as presented, approved in principle, postponed pending clarification, or approved with changes. After the MC has been ratified by Cabinet, the sponsoring federal organization develops a Treasury Board submission to obtain the specific authorities10 needed to implement Cabinet's decision.
The fundamental distinction between an MC and a Treasury Board submission is that, while an MC focuses primarily on the policy rationale and overall funding for a new policy or program initiative, the Treasury Board submission provides details on program design, specific costs, expected results and outcomes, and program delivery and implementation. The Treasury Board submission transforms policy rationale and objectives into a program that will achieve those objectives. It details how the federal organization will carry out the policy initiative; why the proposed method of implementation is the best one; how the proposal ensures accountability and transparency; what the expected outcomes and deliverables are, as per the organization's Management, Resources, and Results Structure (MRRS); and how the federal organization will conduct monitoring, performance measurement, and evaluation to ensure the program is meeting its policy objectives. This includes progress reports on outcomes, projected efficiency, timelines, and cost targets. The Treasury Board submission may also seek approval for expenditure authority, contract authorities, or transfer payment authorities.
3. Scope of the Evaluation
The scope of this evaluation was limited to the current submission process and did not include an assessment of the process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation; however, this review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore.
The evaluation covered all steps of the current submission process involving the Secretariat:
- the support that the Secretariat provides to federal organizations;11
- guidance developed by the Secretariat, such as that found in A Guide to Preparing Treasury Board Submissions (the Guide);
- the communication of decisions regarding submissions; and
- the tracking of conditions placed on approved submissions.
The evaluation did not include federal organizations' internal processes for preparing Treasury Board submissions. Furthermore, the evaluation covers only "Part A" submissions. MCs, GIC submissions, and the federal budget, though outside the scope of the evaluation, were considered to the extent that they provided contextual information.
Canada's Economic Action Plan
The evaluation's planning and data collection phases were already completed at the time Canada's Economic Action Plan was announced in late January 2009. As a result, implementation of this stimulus package could not be included within the scope of the evaluation.
4. Contextual Issues
(a) Volume and complexity of submissions
Between 2004 and 2009 (September to August calendar), an average of 804 Treasury Board submissions were made each year. It should be noted that the Secretariat itself usually accounts for the largest number of Treasury Board submissions among all federal organizations given that it acts as both a department and the management board of the federal public service. As such, some of the Secretariat's submissions are to seek approval or amendments to Treasury Board policies or are related to Treasury Board's role as employer.
As shown in Table 1, the Secretariat's program sectors, with the exception of the International Affairs, Security and Justice Sector, reported a higher incidence of "standard" submissions than "complex" ones.12 Though in the minority, figures for complex submissions are nevertheless significant.
| Program Sector | % Standard | % Complex |
|---|---|---|
| Social and Cultural | 54.9 | 34.3 |
| Economic | 48.7 | 38.1 |
| International Affairs, Security and Justice | 41.7 | 53.2 |
| Government Operations (program side) | 66.0 | 29.9 |
| Overall Average | 54.2 | 38.4 |
Evaluation participants described standard submissions as follows:
- clear and straightforward;
- low risk;
- having a relatively simple policy context;
- written by federal organizations that have a track record of relatively problem-free submissions; and
- involving stakeholders with a relatively common set of interests and priorities.
Complex submissions were said to involve one or more of the following:
- wide-ranging proposals;
- multiple or unclear accountabilities;
- requests for exemptions from government policies;
- significant legal issues;
- outstanding policy issues;
- horizontal initiatives involving multiple federal organizations;
- complicated governance mechanisms;
- relatively large amounts of public funding;
- sensitive media considerations; and/or
- implications for a range of regions and/or jurisdictions across the country.
(b) Logic model for the Treasury Board submission process
No logic model had previously been developed for the Treasury Board submission process. A logic model is essential to this (and to any) evaluation, because it describes the consensus among key stakeholders on the intended outcomes of the Treasury Board submission process. A consensus view on outcomes is the touchstone against which the evaluation must be conducted. Relevance, effectiveness, and economy cannot be assessed in a vacuum; rather, these terms only have meaning in relation to the ultimate outcome of the submission process.
It was therefore necessary for the evaluation team to develop a logic model that would provide the necessary foundation for the evaluation. Consultations were held to gather information on the intended outcomes of the Treasury Board submission process. The evaluation team interviewed 19 individuals from areas across the Secretariat who deal with Treasury Board submissions, including analysts and senior analysts, directors, and an assistant secretary. In addition, two working sessions were held with Secretariat analysts to develop and validate the logic model and evaluation matrix that had been developed by the evaluation team. The consensus among this group was that the ultimate outcome of the submission process was to support Treasury Board in making decisions that contributed to ensuring that programs and spending are aligned with the government's priorities.
Consultation with senior management provided another perspective. Participants in the Assistant Deputy Minister (ADM) Working Session observed that ensuring alignment of new initiatives with the government's priorities is the intended outcome of the MC process rather than the Treasury Board submission process. They suggested that the purpose of the Treasury Board submission process was to provide federal organizations with the resources and authorities needed to achieve the social and economic outcomes desired by Cabinet and to ensure that the programs delivered demonstrated value for money. This proposed ultimate outcome appears to be consistent with the Secretariat's documentation and communications about its expected results. The statement is consistent with the Secretariat's strategic outcome: "Government is well managed and accountable, and resources are allocated to achieve results." This ultimate outcome is also consistent with the expected results outlined in the Secretariat's 2008-09 Departmental Performance Report (DPR): "allocation of resources so that they are aligned with the government's priorities and responsibilities, thereby ensuring that federal programs are effective and efficient and provide value-for-money." The link to broader outcomes (e.g. social and economic) is captured in the DPR's reference to "alignment of government's priorities and responsibilities."
Given that the Treasury Board submission process is not limited to either a single Secretariat subactivity or one organization within the Secretariat, instead crossing a number of organizations, including the program sectors, EMS, and many policy Centres of Excellence, it is perhaps not unexpected that a logic model for the process did not exist before the evaluation and that the ultimate outcome of the Treasury Board submission process was therefore not defined. That said, the evaluation found that there is a sound basis for articulating the outcome, particularly from the strategic outcome and expectations articulated in the Secretariat's DPR. The need for a final logic model, including the ultimate outcome, is addressed in the Conclusion section of this report.
For the purposes of the evaluation, outcomes of the Treasury Board submission process were based on analysis of the information gathered from the consultation sessions as well as information found in documentation on the submission process. The Guide was examined, and while it is very detailed in terms of how to do Treasury Board submissions, it is less explicit about why they must be done. Nevertheless, it was possible to make inferences about the ultimate outcome based on other statements found in the Guide. The Guide's clearest statement about the ultimate outcome of the submission process is as follows:
The Treasury Board submission transforms policy rationale and objectives into a program that will achieve those objectives. It details how the federal organization will carry out the policy /initiative; why the proposed method of implementation is the best one; how the proposal contributes to government-wide aims such as accountability, transparency, and interoperability of information; what the expected outcomes and deliverables are, as per the organization's Management, Resources, and Results Structure (MRRS);13 and how the federal organization will conduct monitoring and evaluation to ensure the program is meeting its policy objectives. This includes progress reports on outcomes, projected efficiency, timelines, and cost targets.
Although reference may be made to the MC and Cabinet decisions, it is generally unnecessary to repeat the policy rationale in a submission. The submission must nevertheless be consistent with the Cabinet decisions.14
In other words, questions of policy rationale and intended social and economic outcomes (or "objectives") are dealt with in the MC (as was suggested by participants in the ADM Working Session). The Treasury Board submission deals with more detailed questions of program design and implementation. It is therefore reasonable to infer that the ultimate value or outcome of the submission process has to do with ensuring that the initiative in question is well designed in relation to the policy rationale described in the MC and in terms of its implementation arrangements and its reporting, accountability, and control mechanisms. The evaluation team therefore proposed the following as the ultimate outcome for the Treasury Board submission process:
Program implementation serves the policy outcomes defined by Cabinet in the most economical manner possible and satisfies the government's obligations for transparency, accountability, and prudence.
It was not possible to validate this proposed ultimate outcome with an appropriate cross-section of stakeholders. The evaluation team therefore recognizes that it is simply a place holder for the purposes of the evaluation.
On the other hand, the other elements of the logic model proposed by the evaluation team-the immediate and intermediate outcomes and the outputs-do reflect the common view within the Secretariat.
The involvement of the Secretariat in the submission process consists of six major components, as outlined in the logic model developed by the evaluation team for the purposes of the evaluation (Appendix A; see also Figure 1):
- raising awareness about the Treasury Board submission process among both Secretariat analysts and personnel in federal organizations;
- providing guidance to federal organizations before they enter into the submission process;
- reviewing and providing advice or consulting on draft submissions put forward by federal organizations;
- drafting the submission précis and preparing for the Treasury Board meeting;15
- documenting and communicating the Treasury Board decision; and
- tracking implementation of conditions that Treasury Board may have placed on its approval decision.
The activities identified in the middle row of Figure 1 should lead to the outcomes indicated in the logic model developed by the evaluation team:
- Secretariat analysts and the personnel in federal organizations who are involved in preparing Treasury Board submissions having an increased understanding of Treasury Board submissions and related policies;
- content of submissions being consistent with relevant authorities and policies;
- Treasury Board decisions on the submissions put forth being well informed; and
- post-decision procedures being in place to follow through on the Treasury Board decision (e.g. communication to EMS, tracking of conditions, communication to federal organizations).
The foregoing outcomes are expected to result in high-quality Treasury Board submissions. These in turn are expected to contribute, ultimately, to federal organizations' effective implementation of programs that serve the government's desired policy outcomes in the most efficient possible manner while also fulfilling the government's obligations for transparency, accountability, and prudence.
5. Evaluation Issues and Methodology
Table 2 presents the evaluation issues and questions that were derived from the logic model developed by the evaluation team. The detailed evaluation matrix, which outlines the indicators and data sources used to address the evaluation questions, is presented in Appendix B.
| Evaluation Issue | Evaluation Questions |
|---|---|
| Relevance |
|
| Effectiveness |
Short Term
|
| Economy |
|
(a) Data sources and methods
The evaluation framework uses multiple lines of evidence and complementary research methods to ensure the reliability and validity of the data. The following research methods were used:
- document review;
- interviews with stakeholders and experts;
- a working session with assistant secretaries from the Secretariat and ADMs from selected federal organizations (ADM Working Session);
- review of administrative, financial, and statistical data; and
- a survey of analysts at the Secretariat and in federal organizations.
Each of these methods is described in more detail below.
Document review. Three main types of documents were reviewed during the evaluation:
- general background documentation (e.g. documents describing the Treasury Board submission process’s history, rationale, and legislative framework);
- documents specific to the submission process (e.g. the Guide and other related documents, such as the Analyst Survival Guide, and information on relevant committees, branches, and other groups involved in the process); and
- past studies (e.g. research specific to the Treasury Board submission process, international studies).
For the full list of documents reviewed, please see Appendix C. Note that the evaluation did not review or assess Treasury Board submissions and précis for quality.
Interviews. Twenty-six interviews were completed (Table 3 and Appendix D). Interviewees included
- program analysts and COE analysts:18
- program analysts are the main point of contact for a submission; and
- COE analysts represent sectors within the Secretariat (e.g. Chief Information Officer Branch, Corporate Services Sector, EMS) and provide policy advice to program analysts.
- representatives from 12 selected federal organizations that put forward a Treasury Board submission within the last five years. Four interviews were conducted with federal organizations that fall into each of the following categories:
- occasional submitters (less than one submission per month);
- moderate submitters (one to two per month); and
- heavy submitters (three or more per month).
- external stakeholders, meaning individuals who are close to but not directly involved in the Treasury Board submission process (e.g. individuals from the Privy Council Office and from the Department of Finance Canada).
| Interview Group | Number of Interviews |
|---|---|
| Program analysts and COE analysts | 11 |
| Representatives from federal organizations | 12 |
| External stakeholders | 3 |
| Total | 26 |
All interviews were conducted by telephone. Interviewees were sent an interview guide (see Appendix E) before the interviews were conducted.
Survey. A survey was administered over the Internet to program analysts, COE analysts, and representatives of federal organizations that put forward a Treasury Board submission in the last five years. A total of 547 individuals were asked to complete the questionnaire; 220 useable responses were received, for an overall response rate of 40% (see Table 4).19
| Survey Group | Total Sent | Received | Removed | Total Kept | Response Rate | Confidence Interval |
|---|---|---|---|---|---|---|
| Program analysts | 135 | 60 | 0 | 60 | 44.4% | 95% ± 9.5% |
| COE analysts | 181 | 66 | 0 | 66 | 36.5% | 95% ± 9.6% |
| Federal organizations | 231 | 99 | 5 20 | 94 | 40.7% | 95% ± 7.8% |
| Total | 547 | 225 | 5 | 220 | 40.2% |
All of the Secretariat's program analysts were invited to participate in the survey. They were also asked to provide contact information for all COE analysts they had consulted for advice on Treasury Board submissions during the last five years. Furthermore, the program analysts were asked to provide contact information for the individuals in federal organizations (including the Secretariat) that put forward a submission within the last year. Federal organizations were encouraged to forward the survey to any individual within the organization who had been involved in the Treasury Board submission review process.
Survey results are provided in Appendix G.
Working session with ADMs. A two-hour working session was conducted to gather qualitative information on the relevance, effectiveness, and economy of the Treasury Board submission process. Assistant secretaries from the Secretariat and selected ADMs responsible for corporate and/or strategic planning as well as selected departmental chief financial officers (CFO) from federal organizations were invited to participate. The issues to be discussed during the session (see Appendix F) were provided to the participants in advance.
Administrative, financial, and statistical data. Administrative, financial, and statistical data were gathered for the purpose of assessing the effectiveness and efficiency of the submission process. The evaluation team worked with the Treasury Board Submission Centre to review data related to Treasury Board submissions and to gain a greater understanding of the Submission Tracking System (STS). The evaluation team also reviewed data from the Management Accountability Framework (MAF) database.
Costing. Evaluating economy requires an analysis of the costs involved throughout the submission process. A costing exercise was undertaken; however, because direct costs related to Treasury Board submissions are not tracked separately, only the level of effort of some participants involved in the submission process was available. Section 6(c)(i), "Resources allocated to the submission process," therefore does not identify an estimated cost for the process.
Related international practice: The evaluation team undertook a limited review of submission process models used in other international jurisdictions.21 Given the cursory nature of this review, the evaluation team could not draw conclusions on the appropriateness of other models compared to the Canadian context. Findings from this review are therefore not presented in this report, though some interesting information was discovered during the review. For example, the role played by the Secretariat's assistant secretaries-whereby they present a federal organization's submission to Treasury Board-may be unique internationally. As is the case in the Canadian federal model, Secretariat equivalents in other jurisdictions are responsible for logistical and technical functions related to sessions of Cabinet, strategic and work planning, policy advice, legal functions, some monitoring functions, and their own internal management functions.22 They scrutinize material presented to Cabinet, ensuring that legal and policy considerations have been accounted for within structured submissions. In six of ten surveyed Organisation for Economic Co-operation and Development (OECD) countries,23 the Secretariat equivalent prepares a recommendation on how the submission should be handled in the Cabinet-level meeting. In these jurisdictions, however, it is the deputy minister of the submitting organization who presents the submission at the Cabinet-level meeting and not the equivalent of a Secretariat assistant secretary. Assistant secretary equivalents can therefore focus their attention on submissions that are most strategic or sensitive or for which their recommendation runs counter to that of the submitting organization.
Other information. Once the data were collected and analyzed, information gaps were discovered in a few key areas. To fill these gaps, the following methodologies were used:
- Review of additional human resources (HR), financial, and statistical data—The evaluation team reviewed additional HR and financial data to gain further insight into the cost of the Treasury Board submission process. Documentation on program sector boot camps was also reviewed.
- Interview with a senior advisor from the Secretariat—The evaluation team met with a senior advisor who has extensive knowledge of and experience with the submission process and its related management tools and structure.
- Working sessions with program directors—Four sessions were held with the Secretariat’s Program Directors Group (PDG) to validate findings. In addition, assistant secretaries from the program sectors reviewed the final draft of this report.
- Interviews with Expenditure Management Information System (EMIS) business analysts—The intent of the meetings was to gain additional knowledge and a greater understanding of the Secretariat's Budget Office Systems Renewal (BOSR) Project and the change management work that occurred during implementation of EMIS.
(b) Limitations of the evaluation
Time frame. A federal election was called shortly after the evaluation was launched. Not long after the election, Parliament was prorogued. These events delayed approval for the opinion research to be conducted for the evaluation; consequently, the time available to perform the research was limited. Another consequence of the election and subsequent prorogation was that the evaluation team could not interview Treasury Board ministers regarding the support they receive through the Treasury Board submission process.
Logic model. While the evaluation team believes that the ultimate outcome proposed in the logic model it developed is a valid description of the purpose of the Treasury Board submission process and therefore a valid basis for the evaluation, it was not possible to validate the ultimate outcome with an appropriate cross-section of stakeholders.
Review of performance measurement data. Performance data have not been collected on all aspects of the Treasury Board submission process. For instance, data are not collected on the extent to which the Treasury Board submissions officially submitted by federal organizations are actually required, or their compliance with policies and processes, and on the extent to which Treasury Board decisions reflect recommendations in the précis. In the absence of this information, the evaluation team was unable to assess the quality of submissions and précis. The evaluation therefore relied more heavily on survey and interview data to assess the effectiveness of the process.
Administrative data. Limited administrative data were available on the submission process and its results. Furthermore, as noted in the Treasury Board of Canada Secretariat Audit of Leave and Overtime (2008),24 some overtime data are not reliable, thereby limiting the extent to which the evaluation team could use such data to assess the amount (and related costs) of overtime claimed by program analysts in connection with Treasury Board submissions. In addition, other administrative data such as the number of days between the Treasury Board decision and the issuance of the decision letter would have provided additional lines of evidence.
Surveys and interviews. Given the highly variable nature of Treasury Board submissions, it stands to reason that the submission process experience would differ significantly from one submission to the next. More interviews would have provided better data on the effect of variations in a submission's size, scope, value, and complexity. The evaluation methodology attempted to address this limitation by inviting individuals from all federal organizations to participate in the Web-based survey. The results of the survey were cross-validated with the interview responses.
Costing methodology. Cost information to support a complete and accurate costing of the Secretariat's involvement in the Treasury Board submission process was not available.
Single-window approach to service delivery. Because the Secretariat uses a single-window approach for submissions, in which program analysts and their directors or executive directors are the point of contact for representatives from federal organizations, interaction between COE analysts and representatives from federal organizations is extremely limited. As a result, despite COE analysts having a clear role and contribution at the pre-submission and draft stages of the submission process, representatives from federal organizations may not be fully aware of the extent of this role. This may have led to the evaluation's greater focus on program analysts.
While there are some limitations with the evaluation methodology, multiple lines of evidence were used to draw conclusions about the Treasury Board submission process, strengthening the reliability and validity of the evaluation results. Despite the limitations, the methodology meets the requirements of the Treasury Board Policy on Evaluation and associated standards.
6. Findings
(a) Relevance
Does the Treasury Board submission process address a demonstrable need, and is it appropriate to the federal government?
The Treasury Board submission process aligns well with the Secretariat's strategic outcome: "Government is well managed and accountable, and resources are allocated to achieve results."25 This strategic outcome is consistent with the responsibilities of Treasury Board ministers as set out in the FAA and therefore speaks to the role of the Secretariat in providing advice and recommendations to Treasury Board ministers through the Treasury Board submission process. The need for a Treasury Board submission process to support ministers is particularly relevant, given the recent focus on responsible spending and the renewal of the expenditure management system, which includes the strategic review of the direct program spending of all departments and agencies of the Government of Canada to reallocate funds from low-priority and low-performing programs to higher priorities.
The Guide states the following:
Improving the quality of information and accountability for results are key elements of the new approach to managing spending across government. The new approach supports managing for results by establishing clear responsibilities for departments to better define the expected outcomes of new and existing programs. It supports decision making for results by ensuring that all new programs are fully and effectively integrated with existing programs and by reviewing all spending to ensure efficiency, effectiveness, and ongoing value for money. Finally, it supports reporting for results by improving the quality of departmental and government-wide reporting to Parliament.26
This excerpt describes the "demonstrable need" that the Treasury Board submission process is meant to address. Essentially, for good management of public programs, the government requires rules, practices, and processes that:
- establish clear responsibilities for federal organizations to define expected outcomes;
- ensure all new programs are well integrated with existing programs;
- ensure all spending is consistent with the government’s commitment to achieving efficiency, effectiveness, and value for money; and
- improve the quality of departmental and government-wide reporting to Parliament.
In assessing relevance, the key evaluation issue is therefore whether or not the Treasury Board submission process is capable of meeting these requirements. The question of whether it actually meets these requirements is addressed under the assessment of effectiveness. The question of whether, and to what extent, other processes or mechanisms would be capable of meeting these requirements is outside the scope of this evaluation.
The documents reviewed for this evaluation, as well as the interviews conducted, point to the conclusion that the Treasury Board submission process is relevant to the government's "demonstrable needs" for good public management. As noted above, the submission process aims to ensure that minimum standards are met with regard to:
how the federal organization will carry out the policy initiative; why the proposed method of implementation is the best one; how the proposal contributes to government-wide aims such as accountability, transparency, and interoperability of information; what the expected outcomes and deliverables are, as per the organization's Management, Resources, and Results Structure (MRRS); and how the federal organization will conduct monitoring and evaluation to ensure the program is meeting its policy objectives. This includes progress reports on outcomes, projected efficiency, timelines, and cost targets.27
According to the interviews, general perceptions about the purpose of the submission process broadly correspond to this formal description from the Guide.
This formal description of the Treasury Board submission process is well aligned with the government's stated needs for good public management.
(b) Effectiveness
To what extent has the Treasury Board submission process achieved its expected outcomes?
Ideally, this evaluation would assess the effectiveness of the Treasury Board submission process in relation to its impact on the ultimate outcome in the logic model that the evaluation team developed for the purposes of the evaluation, i.e. "program implementation serves the policy outcomes defined by Cabinet in the most economical manner possible and fulfills the government's obligations for transparency, accountability, and prudence." However, attempts to link the Treasury Board submission process directly to this ultimate outcome would face data-gathering and methodological challenges that could not be addressed within the evaluation's scope and its time and resource constraints. The evaluation's assessment of the effectiveness of the Treasury Board submission process is therefore focused on the immediate and intermediate outcomes specified in the logic model developed by the evaluation team. If the evaluation finds that the Treasury Board submission process is generally effective in contributing to these outcomes, then it would be reasonable to conclude that the submission process is also making a significant contribution to the ultimate outcome.
The evaluation therefore focused on the following questions:
- Have federal organizations and Secretariat employees demonstrated, over the years, an increased understanding of the elements of Treasury Board submissions, policies, and processes?
- Does the Secretariat offer services that enable federal organizations to put forth draft submissions that comply with Treasury Board authorities, policies, and directions?
- Does the Secretariat’s review process for Treasury Board submissions ensure that they comply with government authorities and policies?
- Are Treasury Board decisions well-informed and consistent with the advice, guidance, and recommendations of Secretariat analysts?
- Are mechanisms in place to ensure Treasury Board decisions are carried out?
- What is the level of quality of Treasury Board submissions?
(i) Understanding of the Treasury Board submission process (evaluation question 1)
The majority of survey participants in all three categories (83.3% of program analysts, 76.9% of COE analysts, and 83% of representatives from federal organizations) believed they had a strong understanding of the Treasury Board submission process. There was no correlation between the length of time respondents had been in their job and their perception of their own understanding of the process.
Survey respondents rated their counterparts' understanding of the Treasury Board submission process less favourably than they rated their own (Table 5). For example, 83% of federal organization respondents believed they had a strong understanding of the process, whereas less than 50% of the program analysts surveyed agreed that this was true. There was a similar divergence of opinion between program and COE analysts within the Secretariat. Each group identified a need for more training for the other.
| % Agreed | Program Analysts | COE Analysts | Federal Organization Respondents |
|---|---|---|---|
| I have a strong understanding of the Treasury Board submission process. | 83.3 n=60 |
n/a | n/a |
| I have a strong understanding of the elements of Treasury Board submissions, policies, and processes. | n/a | 76.9 n=65 |
83.0 n=94 |
| The federal organizations I work with have demonstrated, over the years, an increased understanding of the elements of Treasury Board submissions, policies, and processes. | 48.3 n=58 |
n/a | n/a |
| Program analysts I work with have demonstrated, over the years, an increased understanding of the elements of Treasury Board submissions, policies, and processes. | n/a | 50.0 n=62 |
n/a |
Notwithstanding the above, it should be noted that among the 12 interviews conducted with federal organizations, all 4 federal organizations with high submission rates believed that their analysts had a strong understanding of the Treasury Board submission process. Results were mixed among federal organizations with moderate and low submission rates; they indicated that the level of understanding varied depending on the analyst assigned to them.
HR and survey data showed that approximately two-thirds of program analysts had been in their position for two years or less, which may at least partially explain the external perceptions about program analysts' understanding of the submission process.
(ii) Quality of Secretariat tools, support, and services (evaluation questions 2 to 5)
Training. To assess the Secretariat's tools, support, and services, the evaluation team examined documentation on the boot camps held for program analysts and its guidance document on Treasury Board submissions, A Guide to Preparing Treasury Board Submissions. Furthermore, the evaluation team asked whether courses on the Treasury Board submission process were taken from the Canada School of Public Service.
The Secretariat offers program sector boot camps, which have received positive overall ratings and are considered useful in terms of providing a general overview of what program analysts should know. However, in the participant feedback forms that are completed following the boot camps28 and in interviews, Secretariat analysts indicated that the boot camps are not offered often enough and their duration (two days) does not allow for sufficient coverage of the Treasury Board submission process. As one boot camp participant noted, "could… have more hands-on training."
Evaluation participants also indicated that:
- The Secretariat's Guide to Preparing Treasury Board Submissions is viewed as a useful tool.
- The Canada School of Public Service offers courses on the Treasury Board submission process. Although only a third or fewer federal organization representatives attended them, the majority found the courses to be useful. Those who did not attend the courses indicated cost and waiting lists as being the main barriers.
- The Secretariat itself offers learning events on the Treasury Board submission process to federal organizations. Some of the Secretariat's sectors offer them as frequently as four times per year. A detailed examination of these was not undertaken.
Relationship between Secretariat analysts and federal organizations. Survey results demonstrated that efforts are being made to foster a positive working relationship between program analysts and federal organizations. Almost all of the program analysts surveyed (91.6%) reported that they maintain regular contact with federal organizations regardless of whether a submission is currently being developed or processed. All of the program analysts interviewed stated that they have positive and productive relationships with federal organizations. It is worth noting, however, that when asked how they would characterize this relationship, Secretariat analysts did not refer to the three roles-enabler, challenger, and champion-defined in the Change Agenda.
Federal organizations, for their part, indicated that their relationships with program analysts were generally good or had improved. Nearly all federal organization respondents said they knew whom to consult at the Secretariat with respect to their submissions.
Appropriateness29 of Treasury Board submissions put forward. Secretariat analysts have a role to play at the pre-submission stage when federal organizations are considering proceeding with a submission. At the outset, federal organizations may want early feedback to confirm the directsion of their intended submission or to confirm that a submission is in fact necessary. According to the interviews, 58% of federal organizations seek pre-submission assistance, especially when the submission is complex or perceived as higher risk.30 Advice provided by the Secretariat at the pre-submission stage is intended (among other things) to help federal organizations make informed decisions about whether or not to put forth a submission. The assumption is that in the absence of such advice, federal organizations would proceed with a greater proportion of unnecessary submissions, creating a burden on the Secretariat and Treasury Board and reducing the efficiency of the submission process. The appropriateness of submissions was therefore regarded as an indicator of the suitability and quality of the services the Secretariat provides at the pre-submission stage.31
Secretariat analysts were asked about the appropriateness of submissions (Table 6 32).33 Approximately 40% of the program analysts surveyed responded that they received draft submissions at the pre-submission stage (for preliminary guidance) for initiatives that did not in fact require a submission. The views of COE analysts were similar.
| Was the Secretariat consulted prior to sharing? | COE analysts | Program analysts |
|---|---|---|
| n = sample size | n=29 | n=24 |
| Yes | 44.8 | 62.5 |
| No | 10.3 | 25.0 |
| Do not know | 44.8 | 12.5 |
| Were any drafts submitted? | COE analysts | Program analysts |
|---|---|---|
| n = sample size | n=65 | n=58 |
| Yes | 10.8 | 19.0 |
| No | 36.9 | 55.2 |
| Do not know | 52.3 | 25.9 |
Secretariat analysts were also asked how often federal organizations put forward draft submissions for formal consideration (as opposed to simply seeking informal feedback on a preliminary draft). At this stage, a much smaller proportion of analysts (19% of program analysts and 10.8% of COE analysts) felt that inappropriate drafts were submitted. The reported drop in unnecessary submissions between the stage at which the Secretariat is providing preliminary guidance and the stage at which a submission is formally put forward suggests that the Secretariat is having a positive impact on reducing the number of inappropriate submissions.
Accuracy, consistency, usefulness, and timeliness of Secretariat advice. Interviewees identified factors they believed had a positive or negative impact on the consistency, accuracy, and timeliness of the advice the Secretariat provides to federal organizations during the submission process.
Factors seen by interviewees as having a positive impact:
- Secretariat analysts having sufficient time to review submissions and understand their context;
- scheduling of weekly teleconferences between the Secretariat and federal organizations;
- availability of both formal and informal validation processes for the submission (i.e., draft submission for formal consideration as opposed to seeking informal Secretariat feedback on a preliminary draft); and
- within federal organizations, availability of a quality assurance process and a database to track advice provided by the Secretariat.
Factors seen by interviewees as having a negative impact:
- high turnover rate among program analysts;
- federal organizations' lack of direct access to COE analysts and lack of information about COE analysts' time requirements to review submissions (as stated in an interview, "The process seems to break down when an analyst needs to consult other analysts....
I received one set of advice and then after the second draft, I received conflicting advice when it went to a different group."); and - program analysts' workload and the consequent limitations on their availability to review submissions (according to one individual, "Capacity...in...TBS, particularly with the new requirements imposed on the TBS analysts such as strategic review, is an important challenge within the Treasury Board submission process. The strategic reviews have created a whole new workload but not been accompanied by new staff.").
Survey respondents were asked whether the submission review process ensures that Treasury Board submissions comply with government authorities and policies. A large majority of respondents from each of the groups agreed that it did (Table 7).
However, when survey respondents were asked whether the services provided by analysts enable federal organizations to put forth draft submissions that comply with Treasury Board authorities, policies and directions, there was a divergence of opinion. The federal organization representatives and program analysts had very similar agreement percentages to the ones for the above question and as shown in Table 7, whereas only 49.2% of COE analysts agreed. This divergence could be a function of program analysts acting as the single point of contact for federal organizations and COE analysts not always being consulted at the pre-submission stage.
Secretariat advice and guidance to federal organizations. Survey results (Table 8) show that the majority of survey respondents from federal organizations agreed that Secretariat analysts were providing consistent, accurate, useful, and timely advice. For those who disagreed, consistency and timeliness of advice had the lowest levels of agreement for both representatives of federal organizations and program analysts. Specifically:
- Slightly over half (55.3%) of the survey respondents from federal organizations agreed that the advice of program analysts is consistent at the submission review stage. Another 27.7% indicated that this is not the case, while the remaining respondents from federal organizations answered neutrally or did not know. The responses of program analysts had a slightly higher level of agreement, with 62.7% indicating that advice was consistent.
- Similarly, 54.3% of representatives of federal organizations agreed that Secretariat analysts were providing timely advice at the review stage, while 22.3% did not agree.
- Almost two-thirds of federal organization representatives agreed that the advice provided by Secretariat analysts is accurate and useful.
It is important to note that the issues around timeliness and consistency of advice are recognized internally by the Secretariat, not only by federal organizations.
Consultations with senior management highlighted the value of the Secretariat's work with federal organizations during the pre-submission stage. However, the following areas were identified as needing improvement:
- Clarity of process requirements—Some federal organizations are unclear as to what is specifically required in their submissions and why this varies from one submission to the next (e.g., at what point a document is considered a "first draft");
- Interpretation of process rules and guidelines—Some federal organizations experienced frustration because of situations where the rules and guidelines of the submission process were being strictly enforced, though not always according to their spirit and intent (e.g. fast-track vs. urgent or late submissions);
- Working relationship between the Secretariat and federal organizations—Pre-submission work could be improved through a strengthened working relationship between program analysts and federal organizations; and
- Accessibility of COE analysts—It was noted that federal organizations' lack of direct access to COE analysts may in some cases inhibit their ability to prepare high-quality draft submissions.
Adequacy of time for input into submission documents. The time required to complete the submission review process can vary greatly depending on a submission's characteristics. The Guide advises federal organizations to allow at least six weeks34 for the submission process and cautions that it is not unusual for the process to last more than six weeks. Survey results (Table 9) indicate that the process normally takes 8 to 10 weeks. This suggests that the Guide sets up unrealistic expectations, which could lead to a frustrating experience for someone who is new to the process.35
Secretariat analysts were asked for their views on whether they had enough time to review draft submissions properly. While not a majority, a significant number of program analysts and COE analysts (42.5% and 49.2%, respectively) said that they did not have enough time.
Participants in the ADM Working Session argued that the fast-track system used by the Secretariat to give priority handling to certain submissions is another factor affecting the time available to Secretariat analysts to review submissions.36 They said that it causes uncontrollable delays for submissions that are not fast-tracked. In discussions with Secretariat senior management, however, it was noted that there is no formal system for fast-tracking submissions. It is nevertheless true that certain submissions may receive priority treatment on an ad hoc basis due to urgent situations that require them to "jump the queue." It is at the discretion of the President of the Treasury Board to determine when a submission should receive priority handling; the decision is normally taken following negotiation with the minister responsible for the submission. Consultations between Secretariat officials and sponsoring federal organizations regarding time sensitivity of their proposals are held regularly; however, decisions regarding priority handling are not normally shared with the federal organizations whose submissions may be displaced as a result.
Extent to which Secretariat analysts' input is reflected in final Treasury Board submissions. Most Secretariat analysts stated that their advice is included in final submissions that go forward to Treasury Board. Some noted that, if their advice was not included, they might recommend that conditions be placed on the submission or a remark be included in the précis. Survey results were similar, with 86.6% of program analysts and 71.2% of COE analysts responding that they felt their advice was reflected in the final submission.
This was confirmed with federal organization representatives, who stated that they include all or almost all of the comments provided by Secretariat analysts. Some noted that if they are in disagreement with the advice, they consult further to resolve the issue. A small minority (18.0%) of federal organization representatives stated that they include Secretariat comments due to time pressures and the perceived power of the Secretariat and not because they are in agreement with them.
Extent to which Treasury Board final decisions reflect Secretariat recommendations. Once the program analyst is satisfied that the submission is complete, he or she prepares a précis that includes recommendations to Treasury Board. The recommendations are discussed and agreed to at the Secretariat's Strategy Committee, which is chaired by the Secretary and includes the participation of assistant secretaries from across the Secretariat. A presentation based on the précis is then made to Treasury Board by the appropriate assistant secretary. Although the Secretariat has no authority over Treasury Board's decisions, the decisions normally reflect its recommendations. Data from the survey of program analysts suggest that Treasury Board decisions are consistent with the Secretariat's recommendations 82.2% of the time.
Information management. Once Treasury Board makes a decision, the Secretariat records the decision and formally advises the deputy minister of the sponsoring organization within fifteen calendar days. Although program analysts have no formal requirement to do so, they will normally advise the federal organization verbally of the Treasury Board decision as a matter of courtesy. This is usually done as soon as possible, generally the day after the Treasury Board meeting.37 While most federal organizations did not have an issue with the timeliness of the communication of Treasury Board decisions, it should be noted that one-third of federal organization representatives did not agree that Treasury Board decisions are communicated to them in a timely manner.
Of the program analysts and COE analysts interviewed, many indicated that they use their own private filing system for Treasury Board submissions. Most keep a hard copy of the decision for a period of time, after which the documents are archived or sent to the Treasury Board Submission Centre. Some analysts did mention using the Records, Documents and Information Management System (RDIMS),38 but most program analysts stated that RDIMS is difficult to use. Senior managers observed that a central repository for submission-related information would greatly facilitate their work.
The interviews and survey results also indicated that there is no formal system in use within the Secretariat for tracking conditions attached to Treasury Board submissions.
It should be noted that the Submission Tracking System (STS) maintained by the Treasury Board Submission Centre produces the agenda for Treasury Board meetings and indicates whether or not a submission has conditions attached to it. One interviewee observed that there are weekly requests to the Treasury Board Submission Centre from program analysts for background information on Treasury Board submissions, which indicates the potential value of a central repository of information on Treasury Board submissions. There is also the potential to use the system to track the implementation of conditions.
The Secretariat had developed the Expenditure Management Information System (EMIS) in part to ensure a single set of timely, complete, consistent, and accurate financial and non-financial data and a standard, automated, end-to-end expenditure management process. This was to allow for increased monitoring, tracing, and tracking of information, which would in turn support improved analysis and decision making. The expected result was to be more effective management of the government's supply and budget process.39
When the Budget Office Systems Renewal (BOSR) Project was launched (second phase of EMIS), features were developed to accommodate program sector needs with respect to Treasury Board submissions, including the tracking of conditions. These features, however, have not as yet been implemented. Program sectors maintain that these features did not meet their operational needs. Subsequent interviews suggested that this may be due in part to the fact that the Secretariat's program sectors have not fully analyzed and mapped their own business processes.
(iii) Quality of submissions (evaluation question 6)
As indicated in the logic model developed by the evaluation team, the following assumption underlies the Secretariat's involvement in the submission process as it currently exists: awareness and education tools combined with sound advice to federal organizations should contribute to the quality of Treasury Board submissions and their quality should steadily improve over time.
Overall changes in quality of submissions. Program analysts were asked how the overall quality of Treasury Board submissions had changed during their time with the Secretariat (Table 10). While half of the respondents indicated that the quality had not changed, a little more than one-third of the program analysts (37.3%) stated that the quality of submissions had improved and about one in ten (11.9%) observed that their quality had declined. About half of the program analysts noticed no significant change. However, the usefulness of this data is limited, given that most of the Secretariat analysts surveyed had been in their job for no more than two years. Also, it should be noted that the evaluation did not review Treasury Board submissions and related précis for quality.
Management Accountability Framework rating. The Management Accountability Framework (MAF) is structured around ten key elements that set out Treasury Board's expectations of senior public service managers for good public service management.40 Under one of these key elements, "Policy and Programs," the Secretariat assesses the "quality of analysis in Treasury Board submissions" according to the following criteria:
- accuracy and reliability of supporting information;
- quality, adequacy, and soundness of analysis;
- timeliness of consultations with the Secretariat; and
- presence of a quality control process and the extent of its rigour and effectiveness.
Federal organizations' MAF ratings were analyzed for MAF Rounds IV and V. MAF assessments indicate that the overall quality of analysis in Treasury Board submissions increased from Round IV to Round V. Although performance improved in relation to items 1, 2, and 4 cited above, performance with respect to the timeliness of consultations with the Secretariat actually decreased, which supports earlier findings on the adequacy of time available for input. Again, however, the usefulness of this data is limited, given that two years of results were available at the time of data collection and that the rating criteria were amended slightly for Round V.41
Secretariat analysts' views on submission quality. For the purposes of this evaluation, Secretariat analysts identified what they believed to be the key criteria of a high-quality Treasury Board submission. They cited the following:
- adherence to the Guide
- identification of risk and risk mitigation strategies
- completion of required internal consultations
- request for appropriate authorities
- presentation of an appropriate level of justification
- accurate information
- timeliness of submission
- clarity
As seen in Table 11, a significant number of Secretariat analysts, though not an overwhelming majority, indicated that submissions did not meet these criteria. This supports earlier findings on the divergence of opinion with respect to Secretariat analysts and federal organizations having a strong understanding of the submission process (see Section 6(b)(i)).
| Average Rating (%) | Program Analysts | COE Analysts | Federal Organization Respondents |
|---|---|---|---|
| n = sample size | n=59 | n=56 | n=85 |
| Appear to have completed internal42 consultations | 57.3 | 55.1 | 83.1 |
| Were submitted with enough time for Secretariat analysts' review | 57.5 | 50.8 | 83.0 |
| Followed the Guide to Preparing Treasury Board Submissions | 56.7 | 62.7 | 83.7 |
| Risk and risk mitigation strategies identified | 59.5 | 52.5 | 76.4 |
| Asked for the right authorities | 60.2 | 61.9 | 90.3 |
| Had an appropriate level of justification | 54.7 | 55.9 | 89.8 |
| Contained accurate information (e.g. accurate financial tables) | 54.2 | 54.3 | 90.8 |
| Were written in a clear manner | 51.4 | 60.6 | 88.3 |
(c) Economy
Does the Treasury Board submission process consume the minimum amount of resources required to achieve its expected outcomes?
As indicated in the Scope section, this evaluation was limited to the current submission process used to support Treasury Board ministers and did not assess this process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation, however this review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore. The question of economy therefore only considered the resources required to achieve the expected outcomes within the context of the current process. A comparison of alternative mechanisms or processes and the estimated cost of those processes was not undertaken.
(i) Resources allocated to the submission process
Cost of the process
As noted earlier in the report, the cost of the Treasury Board submission process within the Secretariat could not be established. Nevertheless, it is clear through the widespread level of effort and interest in submission-related activities that the process represents one of the key functions of the Secretariat. The evaluation can establish through interview evidence that Secretariat senior management sees the Treasury Board submission process as a core function. In line with this finding, the level of effort devoted to the function within the program sectors and Centres of Expertise is significant. There are approximately 135 program analysts within the Secretariat acting as the single window of advice to federal organizations and supporting Treasury Board ministers in their submission-related decision making. According to their survey responses (60 responses), program analysts devote, on average, 73% of their time to Treasury Board submission-related activities. Acting as that single window, program analysts in turn consult with COE analysts to provide coordinated advice to federal organizations and in support of advice to Treasury Board ministers. In the past 5 years, program analysts have consulted 181 COE analysts. According to their survey responses (66 responses), COE analysts estimate that, on average, 60% of their time is spent on Treasury Board submissions.43 A separate study examining the full cost of the submission process may well be warranted.
Use of human resources
The proportion of more senior level analysts involved in the Treasury Board submission process has declined in recent years. In 2004, 60% of program sector ESs were ES-06s or ES-07s, compared to approximately 50% in 2008. While the proportion of ES-01s, 02s, and 03s involved in the process has remained relatively constant, the proportion of ES-04s and 05s has risen from approximately 32% to approximately 43% over the same period (as depicited in Figure 2). Note that these statistics do not take into account senior management efforts throughout the process.
Figure 2: Change in Proportion of Junior to Senior Staff
How analysts spend their time. Program analysts report spending on average 72.6% of their time working on activities related to Treasury Board submissions. As shown in Table 12,44 the reported time spent on pre-submission advice and consultation is relatively modest. This provides some corroboration of senior management's comments that inter- and intradepartmental communication is not adequate at the beginning of the submission process.
COE analysts report spending on average 60.3% of their time on activities related to Treasury Board submissions, with the majority devoted to reviewing and consulting on submissions. It should be noted, however, that program analysts are responsible for reviewing the entire submission document, while COE analysts normally review only specific sections.
(ii) Factors hindering or facilitating the submission process
A significant number of survey respondents believe that the submission process is "inefficient" (Table 13 45).
The following factors were specifically identified as having an impact on the efficiency of the process:
- the cyclical nature of Treasury Board submissions in relation to the Annual Reference Level Update (ARLU), production of the Supplementary Estimates, preparation of the budget, fiscal year-end, etc.;
- requirements related to processing fast-track submissions; and
- a lack of staffing flexibility to deal effectively with workload peaks46 or submissions that require fast-tracking.
Further research into workload peaks would be beneficial, especially as they appear to occur on a regular and cyclical basis that aligns with the government expenditure cycle and reporting cycle. Also, it is an issue over which the Secretariat, as the administrative arm of Treasury Board, may have some influence.
Use of risk assessments
Federal organizations provide material in their Treasury Board submissions that enable the Secretariat to perform risk assessments. Risks are discussed during the development of the Treasury Board submission, but the ratings are established once the submissions are complete. Risk ratings are included in the précis, and they may have an impact on the amount of time the submission is allocated at the Treasury Board meeting.
Secretariat analysts and the staff of federal organizations commonly suggest that treating low-risk submissions differently would reduce the level of resources consumed by the submission process (Table 14). They generally argue that the risk rating should be established earlier in the submission review process and that the amount of time allocated and scrutiny given to a submission should correspond to the amount of risk the submission presents. As it is now, most submissions receive the same level of attention as they go through the submission review process, regardless of their level of risk or materiality.
Secretariat senior management expressed scepticism on this point, arguing that submissions that appear low risk at the beginning of the submission review process are sometimes perceived as high risk by the end of the process. As well, there may be risks that are of concern to Treasury Board ministers, which the Secretariat may or may not be familiar with. These may arise only when a particular submission is heard within the context of others on the Treasury Board agenda.
Though a large proportion of analysts (both at the Secretariat and in federal organizations) suggested that risk assessment be used to reduce the use of resources (see Table 14), Secretariat senior management clarified that risk assessment is intended to allow Treasury Board ministers to spend more time on proposals of higher risk and not to reduce workload. The results point to an opportunity to further explore how risk assessment can be used to enhance the efficiency of the process.
Other proposed measures for improving the efficiency of the current submission process that emerged during the evaluation's working sessions and interviews include the following:
- addressing the apparent overlap between Treasury Board submissions and MCs and clarifying the ambiguity regarding their purpose;
- participants in the ADM Working Session noted that the content of MCs is often repeated in Treasury Board submissions and saw opportunities for efficiency by linking MCs and Treasury Board submissions;
- systematic strengthening of the working relationship and communication between the Secretariat and federal organizations (as a federal organization interviewee stated: "The process could be improved if they had service standards.");
- establishing portfolio teams (which used to exist at the Secretariat) composed of program analysts and COE analysts who would be available to meet with federal organizations preparing Treasury Board submissions;
- more comprehensive and timely feedback from the Secretariat and more regular meetings between federal organizations and the Secretariat;
- deepening Secretariat analysts' knowledge of federal organizations;
- streamlining the annual supply process by eliminating the role of federal organizations in the Supplementary Estimates and ARLUs;
- coordinating the response of all sectors within the Secretariat that have to sign off on a submission;
- improving the skills, knowledge, support, and level of engagement of federal organizations with respect to the Treasury Board submission process (43.8% of the program analysts and 44.8% of the COE analysts surveyed); and
- implementing five-year investment plans.
In addition, evaluation participants proposed increasing the level of authority delegated to federal organizations as a measure for improving efficiency. Since this suggestion would require more fundamental changes to the mechanism used to support Treasury Board ministers, it is beyond the scope of the evaluation. However, it should be noted that the Secretariat, outside the scope of this evaluation, is currently conducting an "earned and risk-based delegation" project that is examining various approaches and options for potentially increasing the level of authority delegated to federal organizations.
7. Conclusion
The current Treasury Board submission process is relevant, appears to be effective, and has key strengths; however, some important opportunities exist to improve and enhance its effectiveness and efficiency, which should be addressed through a combination of action and further investigation.
Strengths:
- The submission process is relevant to the government's good public management needs. Determining whether there are other effective and efficient alternatives to accomplish the same outcomes was outside the scope of the evaluation.
- There is widespread belief among stakeholders that the submission process is successful at ensuring that federal organizations present submissions that comply with legislative authorities and Treasury Board policies.
- There is widespread belief among stakeholders that the submission process is successful at ensuring that federal organizations put forth appropriate submissions.
- Treasury Board decisions appear to reflect the advice of the Secretariat to a high degree, which is indicative of Treasury Board's confidence in the quality of that advice.
Opportunities for improvement:
- There is a need to finalize the logic model for the Treasury Board submission process to support performance measurement, clear communication of the process's ultimate outcome, and further study.
As noted earlier in the report, the evaluation team developed a proposed ultimate outcome to be used for the purposes of the evaluation. Finalizing and validating a logic model that includes a clear ultimate outcome would provide a solid basis for enhanced performance measurement activities and further study and research.
- A majority of federal organizations are satisfied with the support and advice they receive from the Secretariat's program analysts; however, there are specific areas of concern:
- the turnover rate among program analysts affects the depth of analysts' knowledge about federal organizations and makes it necessary for many federal organizations to constantly rebuild their relationship with the Secretariat; and
- more than 60% of federal organization survey respondents favourably rated the usefulness and accuracy of the advice received, though they were less positive about its consistency and timeliness.
- The evidence regarding the quality of draft Treasury Board submissions is mixed.
While final submissions presented to Treasury Board are considered to be of high quality, evidence regarding the quality of federal organizations' submissions at the beginning of the process is mixed. MAF ratings related to Treasury Board submissions improved in MAF Round V (with the exception of the timeliness of consultations with the Secretariat); however, fewer than 40% of the program analysts surveyed found that the overall quality of draft submissions had improved. A significant number of Secretariat analysts surveyed (in excess of 40% for most factors47) felt that draft submissions do not meet minimum quality standards.
- The Secretariat does not account for the costs of the submission process.
This evaluation could not draw conclusions on the efficiency of the submission process given the absence of costing information. However, data show that approximately 40% of Secretariat analysts believe the process to be either somewhat or very efficient. The results were very similar among federal organization representatives.
- The Secretariat's efforts to educate federal organizations and its own staff about the details of the submission process appear to be falling short of their intended results.
Of the program analysts surveyed, 48.3% agreed that federal organizations have, over the years, demonstrated an improved understanding of the policies and processes related to Treasury Board submissions. As for training tools provided by the Secretariat, federal organizations consider the Guide to be useful. According to the interviews conducted and review of the documentation on program sector boot camps, it was found that the boot camps are not offered often enough and their duration (two days) does not allow for sufficient coverage of the Treasury Board submission process. Half of the COE analysts surveyed indicated that program analysts have developed a strong understanding of the submission process over the years.
- Rather than a shared, centralized system, analysts at the Secretariat use their own private filing systems to manage documentation related to Treasury Board submissions.
This practice, compounded by turnover among Secretariat analysts, represents a gap in the management of Treasury Board submissions. Specifically, the Secretariat's ability to efficiently and effectively track workload changes, undertake higher level thematic analysis, forecast future demand, and access archived information when needed is impeded. There is also an increased risk of information loss. Although ensuring the security of the information would present its challenges, improvements to managing information related to Treasury Board submissions would enhance oversight and monitoring as well as the service provided to client departments.
8. Recommendations
Opportunities to improve the Treasury Board submission process fall into three areas:
- finalizing the logic model for the Treasury Board submission process;
- deepening the Secretariat's understanding of certain aspects of the process itself and practices related to it and implementing improvements to the process wherever possible; and
- providing the Secretariat's analysts with more and better tools to enhance the effectiveness and economy of the submission process.
Finalizing the logic model for the Treasury Board submission process
The Secretariat should finalize the logic model for the Treasury Boad submission process to support performance measurement, clear communication of the process's ultimate outcome, and further study.
Since the ultimate outcome should not exist in isolation, the Secretariat should take the lead in refining, in consultation with federal organization representatives, its logic model for the submission process. Such a logic model would become the basis for:
- a common view of the overall purpose of the submission process and its relationship to other processes such as the MC process;
- a common view regarding the role that analysts should play with respect to federal organizations;
- a shared understanding between the Secretariat and federal organizations regarding the ultimate purpose of the submission process and the type of advice and support that federal organizations should expect to receive from Secretariat analysts; and
- performance measurement and further research and study relating to the submission process.
Deeper understanding of the submission process and related practices, leading to improvements
The Secretariat should examine in detail, and address where necessary, survey results relating to advice its analysts provide to federal organizations during the submission process.
A majority of federal organization respondents agreed that the advice received from Secretariat analysts is timely and consistent. However, the number of respondents that did not agree is sufficient enough to warrant further work by the Secretariat to understand factors that may impede the timely and consistent delivery of advice.
The Secretariat should examine how human resources issues, especially the turnover rate among program analysts, affect the Treasury Board submission process and knowledge management.
Turnover among program analysts is likely to be an important factor underlying many of the issues noted in this evaluation. It is reasonable to assume, for example, that turnover would contribute to the reported levels of dissatisfaction among federal organizations with the services provided by program analysts.48
During the conduct of the evaluation, the Secretariat undertook work to further understand the factors contributing to turnover; a recommendation in this regard is therefore not made.
Notwithstanding steps that may be taken by the Secretariat to address turnover among program analysts, it is nevertheless reasonable to expect that turnover will persist for the foreseeable future. Therefore, other than simply seeking to reduce turnover, the Secretariat should explore ways to reduce its negative impact on the Treasury Board submission process. As noted below (Recommendation 6), implementing an effective system for knowledge and information management is an example of how the Secretariat could reduce the negative impact of turnover.
The Secretariat should account for the costs of managing the submission process-at least for the department if not for the entire public administration.
In the absence of complete and reliable cost information, the Secretariat cannot fully assess the extent to which its own resources are being properly and efficiently allocated to manage the submission process.
Development of more and better tools for the submission process
The Secretariat should review, and improve where necessary, its professional development, training, and outreach activities related to the Treasury Board submission process.
Program analysts' concerns about the capacity of federal organizations to produce high-quality submissions, along with the reported level of dissatisfaction from a minority of federal organizations with the advice received, point to a need for the Secretariat to review the professional development that it offers to its own staff and the training and outreach activities for the submission process that it extends to federal organizations.
In seeking to improve its professional development, training, and outreach activities, the Secretariat's course of action should be based on the first three recommendations noted above and the anticipated results that their implementation would bring. Finalization of a logic model for the Treasury Board submission process (Recommendation 1), together with a deeper understanding of federal organizations' dissatisfaction with the Secretariat's service (Recommendation 2), along with an awareness of how human resources issues affect the Treasury Board submission process (Recommendation 3) are likely to help the Secretariat understand how to improve the way it communicates with its own staff and with federal organizations regarding the submission process.
- The Secretariat should explore options for a more systematic approach to information and knowledge management in the Treasury Board submission process.
Given that the submission process has a conventional structure and that there are important similarities between certain submissions, implementation of an effective knowledge management system has the potential to yield significant improvements to the economy of the submission process. A systematic approach to knowledge management would create an archive of institutional memory related to Treasury Board submissions. This would allow Secretariat analysts to easily access and use the lessons learned and successful practices that resulted from the handling of previous submissions.
Creating an easily accessible bank of institutional memory related to the submission process would also help overcome some of the negative effects of the turnover rate among Secretariat analysts. A good knowledge and information management system would shorten the learning curve for new analysts and address some of the causes of federal organizations' complaints about "inconsistent" advice from the Secretariat.
Finally, a systematic approach to recording and retrieving information on the status of present and proposed submissions would facilitate the management of the submission process's workload, helping to smooth out the workload "crunches" that arise during peak submission periods.
The Secretariat should bear in mind two issues if it decides to explore options for a systematic approach to knowledge and information management.
- A knowledge and information management system for the submission process must be tailored to the actual structure of the submission process. While there is no question that the Treasury Board submission process has an identifiable structure, what appears to be missing at the moment is a common understanding within the Secretariat of the details, suggesting a need for a process map
- Any solution proposed to address identified information management gaps must meet the needs of its users, must be consistently applied, and must be feasible from a cost and time perspective. It is important to first understand why the existing electronic tools for managing information during the submission process are not currently being fully used by Secretariat analysts. If enhancing the existing tools proves to be impractical or too costly, user needs should be clearly defined in relation to the Treasury Board submission process and alternatives considered.
Appendix A: Logic Model
Appendix A: Logic Model - Text versions
Appendix B: Treasury Board Submission Process Evaluation Matrix
| Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1.0 What need does the Treasury Board (TB) submission process fulfill? | 1.1 | Rationale for TB submission process (FAA sections 7-11, TBS RPP, Guide to Preparing TB Submissions) | Legal only | 1 | 1 | ||||||
| 1.2 | Alignment with relevant legislative requirements | 1 | |||||||||
| 1.3 | Comments and rationale for any changes in what is required to go to TB (e.g. changes in delegation of authority, changes in number of submissions) | 1 | 1 | 1 | 1 | ||||||
| 1.4 | Appropriateness of what is required to be submitted (TBS reached an appropriate balance, e.g. regarding different types of submissions, different transactions) | 1 | 1 | 1 |
| Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2.0 Do federal organizations have an increased understanding of the elements of TB submissions, policies, and process? | 2.1 | Usefulness of guides, tools, and outreach provided by TBS | 1 | 1 | 1 | ||||||
| 3.0 Do TBS employees have an increased understanding of the elements of TB submissions and policies? | 3.1 | Usefulness of guides, tools, and internal courses | 1 | 1 | 1 | 1 | |||||
| 4.0 Is TBS offering services that enable federal organizations' draft submissions to comply with TB authorities, policies, and directions? | 4.1 | Accuracy/consistency of TBS's pre-submission guidance | 1 | 1 | 1 | 1 | |||||
| 4.2 | Appropriateness of TB submissions being put forward | 1 | 1 | 1 | |||||||
| 4.3 | Comparison of understanding of time required for TBS to review a submission | 1 | 1 | 1 | |||||||
| 4.4 | Timeliness - Number of late submissions | 1 | |||||||||
| 4.5 | Relationship of TBS analysts to federal organizations (Change Agenda: strategic partner and trusted advisor, proactive risk manager, overseer vs. enabler, challenge vs. facilitation) | 1 | 1 | 1 | 1 | 1 | |||||
| 5.0 Does TBS's submission review process ensure that TB submissions comply with government authorities and policies? | 5.1 | Accuracy, consistency, usefulness, and timeliness of TBS advice/consultation during submission process | 1 | 1 | 1 | 1 | |||||
| 5.2 | Adequacy of time provided for input into TB submission documents | 1 | 1 | ||||||||
| 5.3 | Confidence that input from TBS consultations is reflected in final TB submission documents | 1 | 1 | 1 | 1 | 1 | |||||
| 5.4 | Limited legal contestation of the submission | 1 | Legal only | 1 | |||||||
| 6.0 Is the TB submission process risk-based? | 6.1 | Extent of knowledge of existence of risk criteria | 1 | 1 | 1 | 1 | 1 | ||||
| 6.2 | Appropriateness of risk criteria (government-wide perspective, reflect ministers' concerns, reflect financial and non-financial risks) | 1 | 1 | 1 | |||||||
| 6.3 | Extent/consistency of use of criteria | 1 | |||||||||
| 6.4 | Extent that TB submission process, due diligence, précis, and decisions are impacted by risk rating | 1 | 1 | ||||||||
| 6.5 | Appropriateness of level of involvement in risk assessments (TBS vs. federal organizations) | 1 | 1 | ||||||||
| 7. 0 Are TB decisions well-informed and consistent with advice, guidance, and recommendations provided by TBS analysts? | 7.1 | Consistency of decision with recommendations, number and frequency of additional conditions applied by TB, number and frequency of challenges raised | 1 | 1 | 1 | ||||||
| 7.2 | Number of deferrals and requests for more information | 1 | |||||||||
| 7.3 | Confidence in recommendations provided (e.g. with respect to risk, cost implications for government) | 1 | 1 | ||||||||
| 8. 0 Are mechanisms in place to ensure TB decisions are carried out? | 8.1 | Existence/adequacy of condition tracking processes:
|
1 | 1 | 1 | 1 | |||||
| 8.2 | Timeliness/completeness of decision communication | 1 | 1 | 1 | |||||||
| 8.3 | Completeness/usability of TB submission filing system | 1 | 1 | 1 |
| Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 9.0 What is the level of quality of TB submissions? | 9.1 | MAF score element 5 | 1 | ||||||||
| 9.2 | Completeness of TB submission documents:
|
1 | 1 | 1 | 1 | 1 | 1 | ||||
| 10.0 Does the TB submission process contribute to ensuring departmental/government management, programs, and spending are aligned with Government of Canada priorities? | 10.1 | Documented accounts of TB submissions contributing to alignment (e.g. Auditor General reports, MAF, Results for Canadians, TBS DPR) | 1 | 1 | |||||||
| 10.2 | Explanations of whether or not TB submission process contributes to ensuring alignment | 1 | 1 | ||||||||
| 11.0 Are there any factors that are impeding the achievement of results? | 11.1 | Description of factors impeding achievement of results | 1 | 1 | |||||||
| 12.0 Have there been any unexpected outcomes? | 12.1 | Description of unexpected outcomes | 1 | 1 |
| Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 13.0 What is the level of resource allocation to submission process? | 13.1 | Changes in level of TB submission effort 5 years ago relative to today | 1 | 1 | 1 | ||||||
| 13.2 | Trend analysis of number of submissions | 1 | |||||||||
| 13.3 | Estimates of time spent:
|
1 | 1 | ||||||||
| 13.4 | Identify rate, determine step and where resource increases would have most impact on process flow | 1 | 1 | 1 | 1 | 1 | 1 | ||||
| 13.5 | Factors impeding analysts' ability to contribute to submission process | 1 | 1 | ||||||||
| 14.0 Is the process efficient? What could be done to make the process more efficient? | 14.1 | Usefulness of measures taken as a result of the 2003 audit and the 2007 EXCO Retreat (Roadmap for enabling change) | 1 | 1 | 1 | 1 | 1 | 1 | |||
| 14.2 | Identification of any measures taken by TBS to manage workload (e.g. risk-based approach to submission review, delegation of authority) | 1 | 1 | 1 | 1 | 1 | |||||
| 14.3 | Identification of any areas where efficiency can be improved | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |||
| 15.0 Are Canadians getting value for their tax dollars? | 15.1 | Analysis of efficiency, effectiveness, and impact indicators |
Appendix C: Documents Reviewed
| Title | Author | Date |
|---|---|---|
| Risk-Based Approaches to Treasury Board Submissions | TBS | 2008 |
| Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions | TBS | 2007 |
| Role of the Program Analyst | TBS | |
| Analyst Survival Guide | TBS | |
| Program Analyst Tool Checklist | TBS | |
| Departmental Contact List - Template | TBS | |
| System Accounts for New Program Analysts | TBS | |
| Transport and Transmittal of Protected and Classified Information | RCMP | 2006 |
| Submissions: Suggested Review Process | TBS | |
| Submissions: A Guide to Preparing Treasury Board Submissions | TBS | 2007 |
| Submissions: Treasury Board Submissions Process - Chart | TBS | 2005 |
| Submissions: Preferred Wording for a Proposal in 'STS' | TBS | |
| Submissions: Deadline Schedule for Signed Departmental Submissions | TBS | 2008 |
| Submissions: Flow Chart - Important Dates for Submissions and Précis | TBS | 2008 |
| Submissions: Treasury Board and Submission Control Document Centre Requirements | TBS | |
| Memorandum to Cabinet: Memorandum to Cabinet Process Reference Guide | TBS | |
| Memorandum to Cabinet: Reference Material for Briefing Notes | TBS | 2006 |
| Memorandum to Cabinet: What is Treasury Board Looking for in a Memorandum to Cabinet | TBS | |
| Expenditure Management: Expenditure Management Cycle Chart | TBS | |
| Expenditure Management: Program Activity Architecture (PAA) | TBS | 2008 |
| Expenditure Management: Annual Reference Level Update (ARLU) | TBS | 2008 |
| Management Accountability Framework | TBS | 2008 |
| Capital Plans, Projects and Procurement - Policies and publications | TBS | 2002 |
| Contracting - Policies and publications | TBS | 2003 |
| Government of Canada Estimates documents and other supporting documents | TBS | 2007 |
| Guide on Grants, Contributions and Other Transfer Payments | TBS | 2000 |
| Project Approval | TBS | 2005 |
| Project Management | TBS | 1994 |
| Real Property - Policies and procedures | TBS | |
| Guide on Revolving Funds | TBS | |
| Policy on Transfer Payments | TBS | 2008 |
| Crown Corporations Policies and Guidelines | TBS | |
| Submission Checklist Tool | TBS | |
| Preparing Treasury Board Submissions and Cabinet Documents: Participant's Manual | Canada School of Public Service | 2008 |
| TBS Evaluation Template | TBS | 2008 |
| Change Agenda Triangle | TBS | 2007 |
| Budget Office Systems Renewal, Chapter 1: Business Blueprint | TBS | 2007 |
| Budget Office Systems Renewal, Chapter 3: Realization Plan; Appendix A: Risk Statements | TBS | 2007 |
| Budget Office Systems Renewal, Chapter 3: Realization Plan; Appendix D: Change Management | TBS | 2007 |
| EMS BOSR Outcomes Management Project | Fujitsu | 2006 |
| Financial Administration Act | Government of Canada | 1985 |
| Treasury Board of Canada Secretariat 2007-08 Report on Plans and Priorities | TBS | 2008 |
| "Putting it Together," TBS Boot Camp on The Treasury Board Submission Process | TBS | 2005 |
| Draft Analysis of Grants and Contributions Delegations | TBS | 2007 |
| Public Sector Reform: Delegated Authority in Australia and UK | TBS | |
| 2007 Comprehensive Spending Review: Guidance for Departmental Submissions | UK | 2007 |
Appendix D: List of Interviewees
Federal Organization Interviews
Heavy submitters (greater than 2 per month on average in the last 10 years)
- Public Works and Government Services Canada: Rhonda Nadon
- National Defence: Lynne McKenna-Fleming
- Transport Canada: Deloranda Munro
- Human Resources and Skills Development Canada: Danny Brouillette
Moderate submitters (between 1 and 2 per month on average over the last 10 years)
- Health Canada: Sanjiv Sandhu
- Privy Council Office: John Kay
- Royal Canadian Mounted Police: Stephan Aylward
- Canadian International Development Agency: Patrick Hines
Occasional submitters (less than 1 per month on average over the last 10 years)
- Public Safety Canada: Marie Steward
- Science and Engineering Research Canada: Dominique Osterrath
- Business Development Bank of Canada: Karen Kastner
- Canadian Environmental Assessment Agency: Ron Kuzak
- Treasury Board of Canada Secretariat - Corporate Services1: Carlo Beaudoin
TBS Analysts
Program Sectors
- Social and Cultural: Cheri Reddin
- Economic: Jennifer Cavasin
- Government Operations: Richard Meroni
- International Affairs, Security and Justice: Cameron McEwan
Internal Contacts
- Chief Information Officer Branch: Andrea Prosper
- Office of the Comptroller General: Pierre Laflamme
- Legal Services: Cindy Shipton-Mitchell
- Government Operations: Glenn Richardson
- Expenditure Management Sector: Deirdre McIssac
- Labour Relations and Compensation Operations: Bianca Bertrand
External Stakeholder Interviews
- Privy Council Office: Michael Jay
- Department of Finance Canada: Ian MacDonald
- Canada Public Service Agency (now the Office of the Chief Human Resources Officer within TBS): Joan McCoy
Appendix E: Interview Guides
TBS Analyst Interview Guide
(Note: this is an example of the TBS Analyst Interview Guide)
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process have been achieved, and whether it is cost-effective and any improvements could be made.
As part of the evaluation, GCS is conducting interviews with key stakeholders involved in the process. The goal of the interviews is examine how the process is being implemented, collect information to assess the success of the process, and identify possible process improvements.
The following questions will serve as a guide for our interview. In some cases, questions will not be relevant to your particular situation. The interviewers will focus on those questions relevant to you. Please note that the responses you provide will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation provided to the responsible department. If you have any questions regarding the enclosed, please feel free to contact the GCS project manager, Susan Cole, at (613) 943-3401.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps.
- Awareness: TBS provides awareness and education regarding the TB submission process to all stakeholders.
- Guidance: TBS provides initial pre-submission guidance, planning, and strategies to federal organizations.
- Analysis: TBS provides advice, consultation, and due diligence on draft submissions presented by federal organizations.
- Recommendation: TBS writes the submission précis and provides advice to ministers at briefings and meetings of the Treasury Board.
- Decision: TBS documents and communicates the decision and establishes the tracking of conditions.
The scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
Background
1. What is your role and involvement with the TB submission process?
- How long have you been in this role?
- How many submissions have you been involved with in the past year?
- Do you distinguish between standard and complex submissions? If so, how? Are there any other ways that you categorize submissions for review?
Relevance
2. Based on your understanding, what are the TB submissions to be used for?
- Are TB submissions submitted for purposes for which there are other more appropriate vehicles?
- Have you noticed any differences since the implementation of the Change Agenda in the areas for which TB submissions are used? Please explain.
Impacts
3. Do TBS analysts have a strong understanding of the elements of a TB submission, the policies and guidance related to TB submissions, and the TB submission process? If not, what would help improve their level of understanding?
4. In the past year, have you consulted any of the following material regarding the TB submission process? Did it help improve your understanding of the elements of TB submissions, policies, and TB submission process?
- A Guide to Preparing Treasury Board Submissions
- Other TBS tools (please describe the tool)
- Courses (Canada School of Public Service TB submission course, TBS Analyst Boot Camp, etc.)
5. Are TBS analysts providing accurate, consistent, useful, and timely policy advice/interpretation in regards to TB submissions:
- At the pre-submission stage (when federal organizations are trying to decide whether a TB submission is required)?
- During the TB submission review stage?
- Is the advice consistent within your sector? Among program and policy sectors?
- Is the advice consistent across TBS?
6. In general, do TBS analysts have the appropriate training and support to enable them to provide accurate, consistent, useful, and timely advice to federal organizations in regards to the TB submission process? If not, what is required?
7.
- In the past year, have federal organizations shared drafts for consideration/review that did not require a TB submission?
- What percentage of drafts did you receive that fell into this category?
- Did the authors consult TBS prior to sharing?
- Has this been getting more or less frequent?
- In the past year, have federal organizations submitted drafts for approval that did not require a TB submission?
- What percentage of drafts did you receive that fell into this category?
- Did the authors consult TBS prior to submitting?
- Has this been getting more or less frequent?
8. In the past year, how frequently have federal organizations sought exemptions and exclusions with their TB submissions? Please explain.
9. How would you characterize your relationship with the individuals in federal organizations who you interact with in regards to the TB submission process?
10. Does your program area have a review process in place to validate the advice provided by its analysts? If yes, please briefly describe this process.
- Do you participate in this process? If yes, in what role?
11. Do you have confidence that the advice and consultation you provide is reflected in the final TB submission documents? Please explain.
12. Are you aware of the risk criteria applied to TB submissions?
13. Please describe the role of each stakeholder (federal organization and TBS) in the risk assessment process?
14. Do the risk criteria appropriately define the risks associated with TB submissions?
- Are they reflective of a government-wide perspective, ministers' concerns, financial and non-financial risks, etc.?
15. Are the risk criteria being applied consistently across TB submissions? Please explain.
16. In your view, in the past year, how has the risk rating associated with a submission influenced your program or policy division's response to it with regards to the process, due diligence, précis, etc.?
17. Has the quality of the submission documents being put forward to Treasury Board changed over the past five years? If yes, how so?
18. Are there mechanisms in place to ensure that TB decisions are carried out?
19. What is the process for tracking the conditions applied to approved submissions?
- To what extent is this process effective?
20. How are submission approvals filed for future reference?
- To what extent is the filing system useful and complete?
21. Have there been any unexpected benefits or challenges related to the process? Please explain.
Cost-Effectiveness and Potential Alternatives
22. Is the submission process efficient?
- Has TBS undertaken any measures to manage the TB submission workload within TBS or to improve the efficiency of the process?
- What additional measures, if any, could make it more efficient?
- Are there any specific steps along the submission process where the flow of work gets stalled (i.e. bottlenecked)? If so, do you have any suggestions for improvement?
23. Are you aware of the measures proposed by internal studies such as the 2003 internal audit of the TB submission process and the 2007 EXCO Retreat? Did you find that they helped to improve the efficiency or effectiveness of the program?
Some measures undertaken as a result of the 2003 audit include:
- Security measures with respect to transferring a submission - i.e. new procedures, more secure facsimile machines, and more secure couriers
- New timelines - Are the new timelines reasonable and allow for an efficient process?
- Definition of classification - i.e. new definitions have provided clarity and simplified the decision-making process
- Measures to address analyst turnover and corporate memory - i.e. new measures in place to help new analysts adapt to the work and ensure that when analysts leave, the knowledge is transferred to current staff
- Ministerial renewal of terms and conditions - Proposed authorization for ministers to extend expiring terms and conditions for up to three years within certain parameters, which include strategic reviews, evaluations, and audits
- Increasing departmental delegated authorities - Program sectors review delegations for departments to determine if adjustments are warranted
- Chief financial officer (CFO) sign-off - CFO or delegate to sign off on all draft TB submissions before TBS review to signal appropriate quality and completeness
- Managing low-risk submissions differently - Flag and process TB submissions based on combined risk rating
- Altering the submission calendar - Emphasize prerogative of TBS to determine scheduling of submissions, including in Guide to Preparing Treasury Board Submissions
- Streamlining supply processes - Program sector assistant secretaries and Expenditure Management Sector are working together to review Estimates processes, looking at each activity through lenses of accountability, duplication, and value added, including options to reduce the overall burden of preparatory work for Senate briefings
24. Do you have anything else to add?
LEGAL ONLY
25. Has the amount of legal contestation of TB submissions changed over the last few years? If so, how?
Federal Organization Interview Guide
(Note: this is an example of the Federal Organization Interview Guide)
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process have been achieved, and whether it is cost-effective and any improvements could be made.
As part of the evaluation, GCS is conducting interviews with key stakeholders involved in the process. The goal of the interviews is examine how the process is being implemented, collect information to assess the success of the process, and identify possible process improvements.
The following questions will serve as a guide for our interview. In some cases, questions will not be relevant to your particular situation. The interviewers will focus on those questions relevant to you. Please note that the responses you provide will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation provided to the responsible department. If you have any questions regarding the enclosed, please feel free to contact the GCS project manager, Susan Cole, at (613) 943-3401.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps:
- Awareness: TBS provides awareness and education regarding the TB submission process to all stakeholders.
- Guidance: TBS provides initial pre-submission guidance, planning, and strategies to federal organizations.
- Analysis: TBS provides advice, consultation, and due diligence on draft submissions presented by federal organizations.
- Recommendation: TBS writes the submission précis and provides advice to ministers at briefings and meetings of the Treasury Board.
- Decision: TBS documents and communicates the decision and establishes the tracking of conditions.
The scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
Background
1. What is your role and involvement with the TB submission process?
- How long have you been in this role?
- How many submissions have you been involved with in the past year?
Relevance
2. Based on your understanding, what is the TB submission process to be used for? Are there other vehicles that might be more appropriate for these areas?
Impacts
3. In the past year, have you consulted any of the following material regarding the submission process? Did it help improve your understanding of the elements of TB submissions, the policies and guidance related to TB submissions, and the TB submission process?
- A Guide to Preparing Treasury Board Submissions
- Other TBS tools (Canada School of Public Service TB submission course, etc.)
- TBS outreach (please describe)
4. Do the TBS analysts you have been in contact with have a strong understanding of the elements of TB submissions, the policies and guidance related to TB submissions, and the TB submission process? If not, what would help improve their level of understanding?
5. When deciding whether a TB submission is required, do you consult a TBS analyst? Why or why not?
- If so, was the advice helpful?
6. In the past year, how frequently has your organization sought exemptions and exclusions through TB submissions?
7. How would you characterize your relationship with TBS and the analysts that you interact with at TBS?
8. Once you submit a draft to TBS, to what extent is the advice and consultation provided by TBS on the draft accurate, consistent, useful, and timely?
9. Does TBS Legal Services provide advice consistent with your own organization's Legal Counsel?
10. To what extent do you incorporate the input TBS provides into the final version of the submission?
11. Are you aware of the risk criteria applied to TB submissions?
12. Please describe the role of each stakeholder (federal organizations and TBS) in the risk assessment process?
13. Do the risk criteria appropriately define the risks associated with TB submissions?
- Are they reflective of a government-wide perspective, ministers' concerns, financial and non-financial risks, etc.?
14. Has the quality of the submissions your organization puts forward to Treasury Board changed over the past five years?
- If so, what has contributed to the change?
- Does this show in your organization's MAF assessment?
15. Have there been any unexpected benefits or challenges related to the process? Please explain.
Cost-Effectiveness and Potential Alternatives
16. Is the submission process efficient?
- What additional measures, if any, could make it more efficient?
- Are there any specific steps along the submission process where the flow of work gets stalled (i.e. bottlenecked)? If so, do you have any suggestions for improvement?
17. Do you have anything else to add?
External Stakeholder Interview Guide
(Note: this is an example of the External Stakeholder Interview Guide)
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process been achieved, and whether it is cost-effective and any improvements could be made.
As part of the evaluation, GCS is conducting interviews with key stakeholders involved in the process. The goal of the interviews is examine how the process is being implemented, collect information to assess the success of the process, and identify possible process improvements.
The following questions will serve as a guide for our interview. In some cases, questions will not be relevant to your particular situation. The interviewers will focus on those questions relevant to you. Please note that the responses you provide will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation provided to the responsible department. If you have any questions regarding the enclosed, please feel free to contact the GCS project manager, Susan Cole, at (613) 943-3401.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps:
- Awareness: TBS provides awareness and education regarding the TB submission process to all stakeholders.
- Guidance: TBS provides initial pre-submission guidance, planning, and strategies to federal organizations.
- Analysis: TBS provides advice, consultation, and due diligence on draft submissions presented by federal organizations.
- Recommendation: TBS writes the submission précis and provides advice to ministers at briefings and meetings of the Treasury Board.
- Decision: TBS documents and communicates the decision and establishes the tracking of conditions.
The scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
Background
1. What is your familiarity with the TB submission process?
- What kind of involvement have you had with the process (prompt - do they write submissions or have they been consulted on them by federal organizations)?
Relevance
2. Based on your understanding, what is the TB submission process to be used for? Are there other vehicles that might be more appropriate for these areas?
Impacts
3. Does the TB submission process contribute to ensuring that departmental/government management, programs, and spending are aligned with Government of Canada priorities? Please explain.
4. Are there any particular benefits or challenges related to the process?
Cost-Effectiveness and Potential Alternatives
5. Is the submission process efficient?
- What additional measures, if any, could make it more efficient?
- Are there any specific steps along the submission process where the flow of work gets stalled (i.e. bottlenecked)? If so, do you have any suggestions for improvement?
6. Do you have anything else to add?
Appendix F: ADM Working Session Questionnaire
(note: this is an example of the questionnaire used for the ADM Working Session)
The Evaluation of the Treasury Board Submission Process
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process have been achieved, and whether it is cost-effective and any improvements could be made.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps.
- Awareness: TBS provides awareness and education regarding the TB submission process to all stakeholders.
- Guidance: TBS provides initial pre-submission guidance, planning, and strategies to federal organizations.
- Analysis: TBS provides advice, consultation, and due diligence on draft submissions presented by federal organizations.
- Recommendation: TBS writes the submission précis and provides advice to ministers at briefings and meetings of the Treasury Board.
- Decision: TBS documents and communicates the decision and establishes the tracking of conditions.
Please note that the scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
The purpose of this questionnaire
As part of this evaluation, GCS is conducting a working session with assistant deputy ministers (ADMs) from TBS as well as from some federal organizations that have submitted a TB submission within the last year.
To ensure that a maximum amount of time is focused on discussions, a short list of questions has been prepared in survey format to ensure that key details have been addressed.
We thank you for taking the time to answer this short questionnaire and to return it to us at the end of the session.
The questionnaire should not take more than a few minutes to complete. Please note that the responses you provide will not be attributable to you and that the results will be reported in aggregate form only.
1. Please identify which group you are representing:
- TBS
- Federal organizations submitting TB submissions
- Other, please specify: _________
2. Please identify to what extent you agree or disagree with the following statements in terms of the TB submissions you have seen over the last few years:
| Statement | Disagree | Somewhat disagree | Neither agree nor disagree | Somewhat agree | Agree | Do not know / Not applicable |
|---|---|---|---|---|---|---|
| Follow the Guide to Preparing Treasury Board Submissions | • | • | • | • | • | • |
| Have risk and mitigation strategies identified | • | • | • | • | • | • |
| Ask for the right authorities | • | • | • | • | • | • |
| Include the appropriate level of justification | • | • | • | • | • | • |
| Contain accurate information | • | • | • | • | • | • |
| Are written in a clear manner | • | • | • | • | • | • |
| Have the risk criteria consistently applied to them | • | • | • | • | • | • |
| Are correctly rated for risk | • | • | • | • | • | • |
| Have a risk rating aligned with Treasury Board's interpretation of risk | • | • | • | • | • | • |
3. Please identify if the following items have improved, remained unchanged, or worsened over time:
| Item | Worsened | Somewhat worsened | Unchanged | Somewhat improved | Improved | Do not know/ No answer |
|---|---|---|---|---|---|---|
| Completeness of the submissions | • | • | • | • | • | • |
| Your level of confidence in the advice/recommendations provided by TBS analysts | • | • | • | • | • | • |
| Respect of timelines associated with submissions | • | • | • | • | • | • |
| Overall quality of the submissions | • | • | • | • | • | • |
4. Please indicate what impact you believe the following measures could have on the efficiency of the TB submission process:
| Impact | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know / No answer |
|---|---|---|---|---|---|---|
| Ministerial renewal of terms and Conditions - Authorizing ministers to extend expiring terms and conditions for up to three years within certain parameters, which include strategic reviews, evaluations, and audits | • | • | • | • | • | • |
| Increasing departmental delegated authorities - Program sectors review delegations for departments to determine if adjustments are warranted | • | • | • | • | • | • |
| Chief financial officer (CFO) Sign-off - CFO or delegate to sign off on all draft TB submissions before TBS review to signal appropriate quality and completeness | • | • | • | • | • | • |
| Managing low-risk submissions differently - Flag and process TB submissions based on combined risk rating | • | • | • | • | • | • |
| Altering the submission calendar - Emphasize prerogative of TBS to determine scheduling of submissions | • | • | • | • | • | • |
| Streamlining supply processes - Review the Estimates processes, looking at each activity through lenses of accountability, duplication, and value added, including options to reduce the overall burden of preparatory work for Senate briefings | • | • | • | • | • | • |
Appendix G: Survey Results
Program Analyst Survey Results
| Number of years | % |
|---|---|
| n=sample size | n=60 |
| Less then one year | 33.3 |
| 1 to 2 years | 35.0 |
| 2 to 3 years | 15.0 |
| 3 to 4 years | 8.3 |
| 4 to 5 years | 3.3 |
| More then 5 years | 5.0 |
| Sector | % |
|---|---|
| n=sample size | n=59 |
| Social and Cultural | 22.0 |
| Economic | 18.6 |
| International Affairs, Security and Justice | 23.7 |
| Government Operations (program side) | 35.6 |
| Classification and level | % |
|---|---|
| n=sample size | n=60 |
| ES-02 or equivalent | 5.0 |
| ES-03 or equivalent | 6.7 |
| ES-04 or equivalent | 20.0 |
| ES-05 or equivalent | 18.3 |
| ES-06 or equivalent | 40.0 |
| ES-07 or equivalent | 6.7 |
| EX-03 or equivalent | 3.3 |
| Number of departments | % |
|---|---|
| n=sample size | n=59 |
| Mean | 3.9 |
| 1 | 22.0 |
| 2 | 18.6 |
| 3 | 16.9 |
| 4 | 8.5 |
| 5 | 15.3 |
| 6 | 3.4 |
| 8 | 3.4 |
| 9 | 5.1 |
| 10 | 6.8 |
| Federal Department\Agency | n= | % |
|---|---|---|
| Note: Due to the ability to select multiple responses for this question, total percentage values will not equal 100. | ||
| Agriculture and Agri-Food Canada | 2 | 3.3 |
| Canadian Heritage | 6 | 10.0 |
| Canadian International Development Agency | 5 | 8.3 |
| Fisheries and Oceans Canada | 3 | 5.0 |
| Foreign Affairs and International Trade Canada | 8 | 13.3 |
| Health Canada | 5 | 8.3 |
| Human Resources and Skills Development Canada | 7 | 11.7 |
| Indian and Northern Affairs Canada | 8 | 13.3 |
| Industry Canada | 6 | 10.0 |
| Infrastructure Canada | 2 | 3.3 |
| National Capital Commission | 1 | 1.7 |
| National Defence and the Canadian Forces | 7 | 11.7 |
| Natural Resources Canada | 5 | 8.3 |
| Privy Council Office | 10 | 16.7 |
| Public Works and Government Services Canada | 12 | 20.0 |
| Royal Canadian Mounted Police | 6 | 10.0 |
| Transport, Infrastructure and Communities Portfolio | 7 | 11.7 |
| Treasury Board of Canada Secretariat | 16 | 26.7 |
| **Other | 19 | 31.7 |
| ** Other Federal Department\Agency | n= | % |
|---|---|---|
| Note: Due to the ability to select multiple responses for this question, total percentage values will not equal 100. | ||
| Atlantic Canada Opportunities Agency | 2 | 2.9 |
| Assisted Human Reproduction Canada | 1 | 1.4 |
| Canada Border Services Agency | 1 | 1.4 |
| Canada Council for the Arts | 1 | 1.4 |
| Canada Lands Company Limited | 1 | 1.4 |
| Canada Mortgage and Housing Corporation | 1 | 1.4 |
| Canada Public Service Agency (now Office of the Chief Human Resources Officer) | 2 | 2.9 |
| Canada Revenue Agency | 1 | 1.4 |
| Canada School of Public Service | 1 | 1.4 |
| Canadian Artists and Producers Professional Relations Tribunal | 1 | 1.4 |
| Canadian Centre for Occupational Health and Safety | 1 | 1.4 |
| Canadian Food Inspection Agency | 1 | 1.4 |
| Canadian Grain Commission | 1 | 1.4 |
| Canadian Industrial Relations Board | 1 | 1.4 |
| Canadian Institutes of Health Research | 2 | 2.9 |
| Canadian International Trade Tribunal | 1 | 1.4 |
| Canadian Museum for Human Rights | 1 | 1.4 |
| Canadian Museum of Civilization | 2 | 2.9 |
| Canadian Museum of Nature | 2 | 2.9 |
| Canadian Radio-television Telecommunications Commission | 1 | 1.4 |
| Canadian Science and Technology Museum | 2 | 2.9 |
| Canadian Security Intelligence Service | 2 | 2.9 |
| Canadian Space Agency | 1 | 1.4 |
| Canadian Tourism Commission | 1 | 1.4 |
| Citizenship and Immigration Canada | 1 | 1.4 |
| Correctional Service Canada | 1 | 1.4 |
| Department of Finance Canada | 8 | 11.4 |
| Environment Canada | 1 | 1.4 |
| Farm Credit Canada | 1 | 1.4 |
| First Nations Statistical Institute | 1 | 1.4 |
| Registry of the Specific Claims Tribunal of Canada | 2 | 2.9 |
| Indian Residential Schools Resolution Canada | 2 | 2.9 |
| Indian Residential Schools Truth and Reconciliation Commission | 1 | 1.4 |
| Library of Parliamentr | 1 | 1.4 |
| National Gallery of Canada | 1 | 1.4 |
| Office of the Commissioner of Official Languages | 1 | 1.4 |
| Office of the Information Commissioner | 1 | 1.4 |
| Office of the Privacy Commissioner | 1 | 1.4 |
| Parks Canada | 2 | 2.9 |
| Public Health Agency of Canada | 2 | 2.9 |
| Public Safety Canada | 2 | 2.9 |
| Science and Engineering Research Canada | 1 | 1.4 |
| Social Sciences and Humanities Research Council of Canada | 1 | 1.4 |
| Standards Council of Canada | 1 | 1.4 |
| Statistics Canada | 1 | 1.4 |
| Veterans Affairs Canada | 4 | 5.7 |
| Number of Submissions | % |
|---|---|
| n=sample size | n=60 |
| Mean | 12.5 |
| 1 | 1.7 |
| 2 | 1.7 |
| 3 | 10.0 |
| 5 | 15.0 |
| 6 | 11.7 |
| 7 | 1.7 |
| 8 | 10.0 |
| 9 | 6.7 |
| 10 | 11.7 |
| 12 | 8.3 |
| 15 | 5.0 |
| 17 | 1.7 |
| 20 | 3.3 |
| 25 | 3.3 |
| 40 | 3.3 |
| 42 | 1.7 |
| 50 | 1.7 |
| 100 | 1.7 |
| % completed that were: | Standard % | Complex % | Strategic Review % |
|---|---|---|---|
| n= sample size | n=57 | n=54 | n=38 |
| Overall submission distribution | 54.2% | 38.4% | 7.5 |
| 0 | 1.8 | 3.7 | 36.8 |
| 1 | 7.9 | ||
| 5 | 1.9 | 5.3 | |
| 9 | 1.8 | ||
| 10 | 5.3 | 5.6 | 21.1 |
| 11 | 2.6 | ||
| 15 | 1.9 | ||
| 20 | 7.0 | 11.1 | 13.2 |
| 25 | 1.8 | 7.4 | 5.3 |
| 30 | 1.8 | 11.1 | |
| 33 | 1.8 | ||
| 34 | 1.9 | 2.6 | |
| 35 | 1.8 | 1.9 | |
| 40 | 3.5 | 9.3 | |
| 45 | 1.8 | 1.9 | |
| 50 | 14.0 | 14.8 | |
| 55 | 1.8 | 1.9 | |
| 60 | 10.5 | 5.6 | |
| 65 | 1.8 | ||
| 66 | 3.5 | ||
| 67 | 1.9 | ||
| 70 | 8.8 | 1.9 | 2.6 |
| 75 | 7.0 | 1.9 | |
| 79 | 1.8 | ||
| 80 | 10.5 | 7.4 | |
| 89 | 1.8 | ||
| 90 | 3.5 | 5.6 | 2.6 |
| 94 | 1.8 | ||
| 100 | 5.3 | 1.9 |
| % | % |
|---|---|
| n=sample size | n=59 |
| Mean | 14.5 |
| 0 | 49.2 |
| 10 | 16.9 |
| 20 | 11.9 |
| 25 | 3.4 |
| 30 | 8.5 |
| 40 | 1.7 |
| 50 | 3.4 |
| 75 | 1.7 |
| 100 | 3.4 |
| % | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
|---|---|---|---|---|---|---|---|
| I have a strong understanding of the TB submission process. | 5.0 | 10.0 | 36.7 | 46.7 | 1.7 | n=60 | |
| The Guide to Preparing Treasury Board Submissions helps to improve my understanding of the elements of TB submissions, policies, and process. | 5.0 | 13.3 | 51.7 | 30.0 | n=60 | ||
| TBS-developed tools help to improve my understanding of the elements of TB submissions, policies, and process. | 8.3 | 40.0 | 31.7 | 16.7 | 3.3 | n=60 | |
| TBS Analyst Boot Camp helped to improve my understanding of the elements of TB submissions, policies, and process. | 2.5 | 17.5 | 32.5 | 27.5 | 15.0 | 5.0 | n=40 |
| Canada School of Public Service (CSPS) training on TB submissions helped to improve my understanding of the elements of TB submissions, policies, and process. | 11.1 | 11.1 | 27.8 | 5.6 | 44.4 | n=18 | |
| I have a strong understanding of who to consult within different TBS areas. | 3.3 | 5.0 | 10.0 | 35.0 | 46.7 | n=60 | |
| I benefit from ongoing support, mentoring, and training as a program analyst. | 5.4 | 10.7 | 14.3 | 39.3 | 30.4 | n=56 | |
| I have sufficient time to provide input into TB submission documents. | 3.3 | 10.0 | 58.3 | 28.3 | n=60 | ||
| I am confident that the advice I provide is reflected in final TB submission documents. | 3.3 | 10.0 | 58.3 | 28.3 | n=60 | ||
| The federal organizations I work with have demonstrated, over the years, an increased understanding of the elements of TB submissions, policies, and process. | 3.4 | 15.5 | 22.4 | 44.8 | 3.4 | 10.3 | n=58 |
| I maintain ongoing, regular contact with my client departments regardless of the submission process. | 1.7 | 6.7 | 38.3 | 53.3 | n=60 | ||
| It is easy for me to obtain input from other TBS sectors, as required, when working on TB submissions. | 5.0 | 8.3 | 23.3 | 41.7 | 21.7 | n=60 |
| % | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
|---|---|---|---|---|---|---|---|
| In general, program analysts are offering services that enable federal organizations' draft submissions to comply with TB authorities, policies, and directions. | 3.3 | 46.7 | 48.3 | 1.7 | n=60 | ||
| In general, program analysts are providing consistent policy advice/ interpretation regarding TB submissions at the pre-submission stage (when federal organizations are trying to decide whether a TB submission is required). | 5.0 | 20.0 | 56.7 | 5.0 | 13.3 | n=60 | |
| In general, program analysts are providing accurate policy advice/ interpretation regarding TB submissions at the pre-submission stage (when federal organizations are trying to decide whether a TB submission is required). | 1.7 | 15.0 | 68.3 | 6.7 | 8.3 | n=60 | |
| In general, the submission review process within TBS ensures that TB submissions comply with government authorities and policies. | 6.8 | 55.9 | 35.6 | 1.7 | n=59 | ||
| In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing consistent advice regarding TB submissions during the TB submission review stage. | 1.7 | 15.3 | 16.9 | 57.6 | 5.1 | 3.4 | n=59 |
| In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing accurate advice regarding TB submissions during the TB submission review stage. | 1.7 | 16.9 | 71.2 | 8.5 | 1.7 | n=59 | |
| In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing useful advice regarding TB submissions during the TB submission review stage. | 3.4 | 10.2 | 72.9 | 13.6 | n=59 | ||
| In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing timely advice in regards to TB submissions during the TB submission review stage. | 1.7 | 10.0 | 28.3 | 40.0 | 20.0 | n=60 |
| % | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
|---|---|---|---|---|---|---|---|
| TBS has adequate mechanisms in place to track conditions placed on TB submissions. | 23.3 | 45.0 | 18.3 | 1.7 | 1.7 | 10.0 | n=60 |
| TBS communicates the TB decision in a timely manner to those involved. | 3.3 | 11.7 | 51.7 | 30.0 | 3.3 | n=60 | |
| TBS communicates the TB decision to all required parties (e.g. federal organization, Expenditure Management Sector, program analysts, other TBS analysts, etc.). | 1.7 | 13.3 | 11.7 | 45.0 | 13.3 | 15.0 | n=60 |
| TBS's decision filing system is complete. | 10.2 | 10.2 | 13.6 | 23.7 | 0.0 | 40.7 | n=59 |
| TBS's decision filing system is usable. | 8.3 | 11.7 | 13.3 | 25.0 | 0.0 | 40.0 | n=60 |
12. In the past year, have federal organizations shared drafts for consideration/review that did not require a TB submission?
| Were any drafts shared? | Did the authors consult TBS prior to sharing? | Were any drafts submitted? | Did the authors consult TBS prior to submitting? | |
|---|---|---|---|---|
| n=sample size | n=59 | n=24 | n=58 | n=11 |
| Yes | 40.7 | 62.5 | 19.0 | 81.8 |
| No | 40.7 | 25.0 | 55.2 | 9.1 |
| Do not know | 18.6 | 12.5 | 25.9 | 9.1 |
| % | Shared | Submitted |
|---|---|---|
| n=sample size | n=24 | n=11 |
| Mean | 17.5 | 9.1 |
| 0 | 16.7 | 36.4 |
| 10 | 58.3 | 45.5 |
| 20 | 12.5 | 9.1 |
| 30 | 4.2 | 9.1 |
| 90 | 4.2 | |
| 100 | 4.2 |
| Frequency | Shared | Submitted |
|---|---|---|
| n=sample size | n=24 | n=11 |
| Increasing | 4.2 | |
| Decreasing | 8.3 | |
| Remaining the same | 29.2 | 36.4 |
| Do not know | 58.3 | 63.6 |
| Weeks | % |
|---|---|
| n=sample size | n=53 |
| Mean | 8.5 |
| 4 | 1.9 |
| 5 | 3.8 |
| 6 | 22.6 |
| 7 | 5.7 |
| 8 | 32.1 |
| 9 | 5.7 |
| 10 | 5.7 |
| 11 | 1.9 |
| 12 | 15.1 |
| 14 | 3.8 |
| 20 | 1.9 |
| % | Appear to have completed internal consultations? | Were submitted with enough time for TBS analysts to review? | Follow the Guide to Preparing Treasury Board Submissions? | Have risk and mitigation strategies identified? | Ask for the right authorities? | Have the appropriate level of justification? | Contain accurate information? | Are written in a clear manner? |
|---|---|---|---|---|---|---|---|---|
| n=sample size | n=58 | n=59 | n=59 | n=59 | n=58 | n=59 | n=59 | n=59 |
| Ind. (9.2) | Ind. (5.1) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | |
| Mean | 57.3 | 57.5 | 56.7 | 59.5 | 60.2 | 54.7 | 54.2 | 51.4 |
| 0 | 3.4 | 1.7 | 3.4 | 1.7 | 3.4 | 6.8 | 1.7 | |
| 10 | 5.2 | 3.4 | 6.8 | 10.2 | 3.4 | 1.7 | 3.4 | 5.1 |
| 20 | 1.7 | 6.8 | 3.4 | 3.4 | 3.4 | 6.8 | 1.7 | 5.1 |
| 25 | 5.2 | 6.8 | 3.4 | 1.7 | 6.9 | 8.5 | 6.8 | 5.1 |
| 30 | 3.4 | 1.7 | 1.7 | 10.2 | 6.9 | 8.5 | 1.7 | 10.2 |
| 40 | 5.2 | 1.7 | 3.4 | 1.7 | 1.7 | 3.4 | 3.4 | 5.1 |
| 50 | 25.9 | 23.7 | 25.4 | 15.3 | 12.1 | 16.9 | 25.4 | 20.3 |
| 60 | 6.9 | 11.9 | 6.8 | 5.1 | 17.2 | 8.5 | 16.9 | 18.6 |
| 70 | 6.9 | 16.9 | 18.6 | 10.2 | 8.6 | 13.6 | 11.9 | 10.2 |
| 75 | 8.6 | 5.1 | 8.5 | 8.5 | 20.7 | 10.2 | 8.5 | 10.2 |
| 80 | 10.3 | 6.8 | 8.5 | 8.5 | 6.9 | 8.5 | 3.4 | 3.4 |
| 90 | 8.6 | 6.8 | 5.1 | 10.2 | 8.6 | 5.1 | 5.1 | 3.4 |
| 100 | 8.6 | 6.8 | 5.1 | 13.6 | 3.4 | 5.1 | 5.1 | 1.7 |
| % | A TB decision that is consistent with the recommendations you put forward? | A TB decision that has additional conditions to those you proposed? | Challenges to the submission and/or recommendations? | Legal contestations? |
|---|---|---|---|---|
| n=sample size | n=58 | n=58 | n=58 | n=58 |
| Mean | 82.2 | 5.6 | 8.4 | 3.5 |
| 0 | 5.2 | 63.8 | 55.2 | 74.1 |
| 10 | 1.7 | 22.4 | 13.8 | 3.4 |
| 20 | 3.4 | 5.2 | 5.2 | |
| 25 | 5.2 | 3.4 | ||
| 30 | 1.7 | 1.7 | ||
| 50 | 3.4 | 1.7 | ||
| 70 | 5.2 | |||
| 75 | 1.7 | |||
| 80 | 1.7 | |||
| 90 | 22.4 | |||
| 100 | 56.9 | 1.7 | ||
| Do not know | 6.9 | 6.9 | 13.8 | 12.1 |
| Level of awareness | % |
|---|---|
| n=sample size | n=58 |
| Yes, I am aware, but I do not apply them when I analyze and provide advice. | 3.4 |
| Yes, I am aware, and I do apply them when I analyze and provide advice. | 91.4 |
| No, I am not aware. | 5.2 |
| Risk Criteria being applied consistently? | % |
|---|---|
| n=58 | 100.0 |
| Yes | 43.1 |
| No | 25.9 |
| Do not know | 31.0 |
| Level of quality? | % |
|---|---|
| n=sample size | n=59 |
| Significantly worse | 3.4 |
| Somewhat worse | 8.5 |
| No change (neither better nor worse) | 50.8 |
| Somewhat better | 32.2 |
| Significantly better | 5.1 |
| % | Quality of first draft | Not enough time for TBS review, consultation, due diligence | Lack of program analysts' knowledge of department / need of coordination with department | Policy centre advice not appropriate, consistent, coordinated | Federal organizations' poor follow-up to TBS input or needs | Poor TBS condition tracking | Record keeping | TBS requirements unclear, unresponsive to federal organizations' needs, awkward for federal organizations | Political pressure | n=sample size |
|---|---|---|---|---|---|---|---|---|---|---|
| 1st | 32.8 | 43.1 | 3.4 | 10.3 | 1.7 | 1.7 | 0.0 | 3.4 | 1.7 | n=57 |
| 2nd | 15.5 | 31.0 | 3.4 | 8.6 | 19.0 | 0.0 | 3.4 | 10.3 | 3.4 | n=55 |
| 3rd | 8.6 | 20.7 | 5.2 | 12.1 | 27.6 | 5.2 | 0.0 | 8.6 | 0.0 | n=51 |
| % | TBS tools | Provides insight and knowledge of departmental operations | Keeps ministers informed | Strong process | TBS staff support, good consultation process | Recent changes beneficial | Enhances operations of federal organizations | n=sample size |
|---|---|---|---|---|---|---|---|---|
| 1st Strength | 5.5 | 3.6 | 3.6 | 38.2 | 40.0 | 1.8 | 10.9 | n=55 |
| 2nd Strength | 9.6 | 6.8 | 6.8 | 40.9 | 29.5 | 0.0 | 6.8 | n=45 |
| 3rd Strength | 7.9 | 10.5 | 7.9 | 39.5 | 26.3 | 2.6 | 5.3 | n=38 |
| % | Pre-submission advice and consultation? | Reviewing/consulting on submissions? | Briefing/preparing briefing documents (e.g. précis)? | Other TBS submission activities? | Other TBS non-submission activities? |
|---|---|---|---|---|---|
| n=sample size | n=59 | n=59 | n=59 | n=37 | n=55 |
| Average time for each activity | 15.8% | 28.5% | 21.6% | 6.8% | 27.4% |
| 0 | 3.4 | 0.00 | 3.4 | 18.9 | |
| 5 | 20.3 | 3.4 | 3.4 | 29.7 | 3.6 |
| 10 | 32.2 | 8.5 | 13.6 | 18.9 | 16.4 |
| 15 | 8.5 | 13.6 | 13.6 | 5.4 | 5.5 |
| 20 | 15.3 | 16.9 | 27.1 | 18.9 | 7.3 |
| 25 | 3.4 | 8.5 | 10.2 | 2.7 | 10.9 |
| 30 | 6.8 | 18.6 | 15.3 | 0.0 | 18.2 |
| 35 | 0 | 3.4 | 5.1 | 2.7 | 3.6 |
| 40 | 6.8 | 11.9 | 5.1 | 0.0 | 18.2 |
| 45 | 1.7 | 1.7 | 0.0 | 2.7 | 3.6 |
| 50 | 1.7 | 5.1 | 3.4 | 7.3 | |
| 55 | 3.4 | 1.8 | |||
| 60 | 3.4 | 1.8 | |||
| 65 | 1.8 | ||||
| 70 | 1.7 |
| % | Document preparation (administrative) | MAF related | Provision of advice | Strategic reviews | Oversight | Post-approval activities | Financial exercises | ARLU and Supple-mentary Estimates, Memoranda to Cabinet | Policy input |
|---|---|---|---|---|---|---|---|---|---|
| n=65 | 4.6 | 16.9 | 15.4 | 3.1 | 3.1 | 12.3 | 15.4 | 21.5 | 7.7 |
| Frequency | % |
|---|---|
| n=sample size | n=60 |
| Never | 1.7 |
| Rarely | 10.0 |
| Sometimes | 50.0 |
| Often | 23.3 |
| Always | 13.3 |
| Do not know | 1.7 |
| Level | % |
|---|---|
| n=sample size | n=59 |
| Decreased | 8.5 |
| Remained the same | 45.8 |
| Increased | 45.8 |
| Percentage of workload | % |
|---|---|
| n=sample size | n=59 |
| Mean | 49.6 |
| 0 | 1.7 |
| 20 | 1.7 |
| 25 | 3.4 |
| 30 | 5.1 |
| 40 | 35.6 |
| 50 | 22.0 |
| 60 | 6.8 |
| 70 | 10.2 |
| 75 | 8.5 |
| 80 | 5.1 |
| Level of efficiency | % |
|---|---|
| n=sample size | n=59 |
| Very inefficient | 3.4 |
| Somewhat inefficient | 30.5 |
| Neither inefficient nor efficient | 27.1 |
| Somewhat efficient | 37.3 |
| Very efficient | 1.7 |
| Measures taken? | % |
|---|---|
| N=sample size | n=58 |
| Yes | 50.0 |
| No | 50.0 |
| Measures taken? | % |
|---|---|
| N=sample size | n=56 |
| Yes | 51.8 |
| No | 48.2 |
| % | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know | n= |
|---|---|---|---|---|---|---|---|
| Ministerial renewal of terms and conditions | 1.7 | 3.4 | 13.8 | 51.7 | 12.1 | 17.2 | n=58 |
| Increasing department delegated authorities | 1.7 | 5.1 | 20.3 | 50.8 | 16.9 | 5.1 | n=59 |
| Chief financial officer (CFO) sign-off | 6.8 | 5.1 | 25.4 | 42.4 | 20.3 | 0.0 | n=59 |
| Managing low-risk submissions differently | 0.0 | 0.0 | 6.9 | 51.7 | 37.9 | 3.4 | n=58 |
| Altering the submission calendar | 1.7 | 0.0 | 29.3 | 31.0 | 22.4 | 15.5 | n=58 |
| Streamlining supply processes | 0.0 | 1.7 | 13.6 | 40.7 | 18.6 | 25.4 | n=59 |
| Additional measures | Reduce analyst turnover, better stability | Change timelines – internal and external | Have more tools | Consistent message to clients and internally | More regular involvement with Finance unit, better internal communications | Increased knowledge of federal organizations, skills, quality control | More program analyst support (increased tools and training, reallocate workload) | n= |
|---|---|---|---|---|---|---|---|---|
| % | 6.3 | 31.3 | 15.6 | 9.4 | 9.4 | 43.8 | 9.4 | n=32 |
| Steps | Analyst turnover (re-education) | Acceptance of late submissions / supply cycle points | Principal analyst and executive director review stage | Getting EMS sign-off during Estimates period | Federal organization delays after first draft | Input from policy centres | Workload planning not efficient | n= |
|---|---|---|---|---|---|---|---|---|
| % | 2.9 | 11.4 | 11.4 | 2.9 | 28.6 | 31.4 | 25.7 | n=35 |
| Additional comments | Federal organizations doing poor submissions | Better information sharing, knowledge access, and workflow management needed | Need more time for proper due diligence | More training needed for federal organizations | n= |
|---|---|---|---|---|---|
| % | 12.5 | 50 | 31.3 | 6.25 | n=16 |
COE Analyst Survey Results
| Number of years | % |
|---|---|
| n=sample size | n=66 |
| Less then one year | 15.2 |
| 1 to 2 years | 33.3 |
| 2 to 3 years | 19.7 |
| 3 to 4 years | 10.6 |
| 4 to 5 years | 4.5 |
| More then 5 years | 16.7 |
| Sector | % |
|---|---|
| n=sample size | n=66 |
| Policy Sectors: Chief Information Officer Branch | 19.7 |
| Policy Sectors: Government Operations (policy side) | 12.1 |
| Policy Sectors: Labour Relations and Compensation Operations | 1.5 |
| Policy Sectors: Office of the Comptroller General | 16.7 |
| Policy Sectors: Pension and Benefits | 9.1 |
| Policy Sectors: Expenditure Management | 21.2 |
| Policy Sectors: Service | 1.5 |
| Enabling Sectors: Priorities and Planning | 1.5 |
| Enabling Sectors: Corporate Services | 3.0 |
| Enabling Sectors: Legal Services | 7.6 |
| Enabling Sectors: Regulatory Affairs | 4.5 |
| Other | 1.5 |
| Number of Submissions | % |
|---|---|
| n=sample size | n=63 |
| Mean | 38.7 |
| 0 | 1.6 |
| 1 | 1.6 |
| 2 | 9.5 |
| 3 | 1.6 |
| 3 | 4.8 |
| 4 | 1.6 |
| 5 | 1.6 |
| 6 | 6.3 |
| 7 | 1.6 |
| 8 | 1.6 |
| 10 | 9.5 |
| 12 | 1.6 |
| 15 | 1.6 |
| 20 | 3.2 |
| 22 | 1.6 |
| 25 | 4.8 |
| 27 | 1.6 |
| 30 | 4.8 |
| 31 | 3.2 |
| 35 | 1.6 |
| 40 | 4.8 |
| 45 | 1.6 |
| 50 | 3.2 |
| 60 | 6.3 |
| 65 | 1.6 |
| 75 | 3.2 |
| 80 | 1.6 |
| 100 | 6.3 |
| 115 | 1.6 |
| 135 | 1.6 |
| 200 | 1.6 |
| 275 | 1.6 |
| % completed that were: | Standard % | Complex % | Strategic Review % |
|---|---|---|---|
| n= sample size | n=60 | n=57 | n=33 |
| Overall submission distribution | 61.1% | 34.3% | 4.5% |
| 0 | 21.2 | ||
| 1 | 18.2 | ||
| 2 | 1.8 | 3.0 | |
| 3 | 3.0 | ||
| 4 | 1.8 | 3.0 | |
| 5 | 18.2 | ||
| 6 | 3.0 | ||
| 8 | 3.0 | ||
| 10 | 1.7 | 5.3 | 3.0 |
| 13 | 3.0 | ||
| 14 | 3.0 | ||
| 15 | 8.8 | ||
| 19 | 1.7 | 1.8 | |
| 20 | 5.0 | 5.3 | 9.1 |
| 22 | 1.7 | ||
| 23 | 1.7 | ||
| 25 | 1.7 | 19.3 | 6.1 |
| 26 | 1.8 | ||
| 30 | 5.3 | ||
| 33 | 8.8 | ||
| 35 | 1.8 | ||
| 37 | 1.8 | ||
| 40 | 5.0 | 1.8 | |
| 45 | 1.8 | ||
| 50 | 15.0 | 14.0 | |
| 53 | 1.7 | ||
| 55 | 1.7 | ||
| 58 | 1.7 | ||
| 60 | 5.0 | 5.3 | |
| 65 | 1.7 | 1.8 | |
| 67 | 5.0 | ||
| 70 | 6.7 | ||
| 74 | 1.7 | ||
| 75 | 11.7 | 3.5 | |
| 77 | 3.0 | ||
| 79 | 1.7 | ||
| 80 | 8.3 | 3.5 | |
| 84 | 1.7 | ||
| 85 | 1.7 | ||
| 90 | 5.0 | 1.8 | |
| 92 | 1.7 | ||
| 95 | 1.7 | ||
| 98 | 1.7 | ||
| 100 | 6.7 | 3.5 |
| % | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
|---|---|---|---|---|---|---|---|
| I have a strong understanding of the elements of TB submissions, policies, and process. | 9.2 | 12.3 | 33.8 | 43.1 | 1.5 | n=65 | |
| The Guide to Preparing Treasury Board Submissions helps to improve my understanding of the elements of TB submissions, policies, and process. | 6.6 | 11.5 | 57.4 | 21.3 | 3.3 | n=61 | |
| TBS-developed tools help to improve my understanding of the elements of TB submissions, policies, and process. | 1.7 | 6.9 | 22.4 | 41.4 | 15.5 | 12.1 | n=58 |
| TBS Analysts Boot Camp helped to improve my understanding of the elements of TB submissions, policies, and process. | 12.0 | 16.0 | 28.0 | 28.0 | 16.0 | n=25 | |
| Canada School of Public Service (CSPS) training on TB submissions helped to improve my understanding of the elements of TB submissions, policies, and process. | 14.3 | 21.4 | 64.3 | n=14 | |||
| I have sufficient time to provide input into TB submission documents. | 10.8 | 33.8 | 13.8 | 35.4 | 6.2 | n=65 | |
| I am confident that the advice I provide is reflected in final TB submission documents. | 4.5 | 18.2 | 50.0 | 21.2 | 6.1 | n=66 |
| % | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
|---|---|---|---|---|---|---|---|
| The program analysts I work with have demonstrated, over the years, an increased understanding of the elements of TB submissions, policies, and process. | 3.2 | 17.7 | 19.4 | 33.9 | 16.1 | 9.7 | n=62 |
| Program analysts have a strong understanding of the elements of TB submissions, policies, and process as a result of TBS-provided awareness education activities. | 10.3 | 22.4 | 15.5 | 8.6 | 43.1 | n=58 | |
| Program analysts consult our team in a timely manner with respect to submissions. | 32.8 | 23.4 | 32.8 | 9.4 | 1.6 | n=64 | |
| Program analysts provide me with enough information to adequately respond and provide advice with respect to submissions. | 3.1 | 18.5 | 13.8 | 49.2 | 15.4 | n=65 | |
| TBS is offering services that enable federal organizations' draft submissions to comply with TB authorities, policies, and directions. | 1.7 | 15.3 | 37.3 | 11.9 | 33.9 | n=59 | |
| The submission review process within TBS ensures that TB submissions comply with government authorities and policies. | 1.6 | 1.6 | 9.7 | 54.8 | 24.2 | 8.1 | n=62 |
| Statement | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
|---|---|---|---|---|---|---|---|
| TBS has adequate mechanisms in place to track conditions placed on TB submissions. | 14.1 | 25.0 | 7.8 | 17.2 | 1.6 | 34.4 | n=64 |
| TBS communicates the TB decision in a timely manner to those involved. | 9.8 | 6.6 | 9.8 | 27.9 | 6.6 | 39.3 | n=61 |
| TBS communicates the TB decision to all required parties. | 9.8 | 26.2 | 16.4 | 14.8 | 3.3 | 29.5 | n=61 |
| TBS's decision filing system is complete. | 6.5 | 11.3 | 12.9 | 8.1 | 3.2 | 58.1 | n=62 |
| TBS's decision filing system is usable. | 4.8 | 9.7 | 16.1 | 11.3 | 1.6 | 56.5 | n=62 |
8. In the past year, have federal organizations shared drafts for consideration/review that did not require a TB submission?
| % | Were any drafts shared? | Did the authors consult TBS prior to sharing? | Were any drafts submitted? | Did the authors consult TBS prior to submitting? |
|---|---|---|---|---|
| Yes | 45.3 | 44.8 | 10.8 | 42.9 |
| No | 20.3 | 10.3 | 36.9 | |
| Do not know | 34.4 | 44.8 | 52.3 | 57.1 |
| n=sample size | n=64 | n=29 | n=65 | n=7 |
| % | Shared | Submitted |
|---|---|---|
| n=sample size | n=29 | n=7 |
| Mean | 15.3 | 8.6 |
| 0 | 17.2 | 28.6 |
| 10 | 69.0 | 57.1 |
| 20 | 14.3 | |
| 40 | 3.4 | |
| 50 | 3.4 | |
| 75 | 3.4 | |
| 80 | 3.4 |
| % | Shared | Submitted |
|---|---|---|
| Increasing | 21.4 | |
| Decreasing | 7.1 | |
| Remaining the same | 32.1 | 14.3 |
| Do not know | 39.3 | 85.7 |
| n=sample size | n=28 | n=7 |
| Weeks | % |
|---|---|
| n=sample size | n=49 |
| Mean | 10.3 |
| 6 | 6.1 |
| 3 | 4.1 |
| 4 | 4.1 |
| 5 | 2.0 |
| 6 | 10.2 |
| 7 | 2.0 |
| 8 | 24.5 |
| 9 | 4.1 |
| 10 | 8.2 |
| 12 | 14.3 |
| 14 | 4.1 |
| 15 | 4.1 |
| 16 | 6.1 |
| 18 | 2.0 |
| 26 | 2.0 |
| 56 | 2.0 |
| % | Appear to have completed internal consultations? | Were submitted with enough time for your team to adequately review them? | Follow the Guide to Preparing Treasury Board Submissions? | Have risk and mitigation strategies identified relevant to your area? | Ask for the right authorities relevant to your area? | Have an appropriate level of justification relevant to your area? | Contain accurate information relevant to your area? | Are written in a clear manner? |
|---|---|---|---|---|---|---|---|---|
| n=sample size | n=58 | n=60 | n=53 | n=50 | n=55 | n=55 | n=59 | n=61 |
| Ind. (9.2) | Ind. (5.1) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | |
| Mean | 55.1 | 50.8 | 62.7 | 52.5 | 61.9 | 55.9 | 54.3 | 60.6 |
| 0 | 13.8 | 8.3 | 5.7 | 12.0 | 7.3 | 10.9 | 8.5 | 3.3 |
| 10 | 1.7 | 5.0 | 4.0 | 3.4 | 1.6 | |||
| 20 | 3.4 | 1.7 | 3.8 | 6.0 | 3.6 | 3.6 | 1.7 | 1.6 |
| 25 | 6.9 | 15.0 | 1.9 | 4.0 | 3.6 | 5.1 | ||
| 30 | 5.2 | 3.3 | 5.7 | 6.0 | 5.5 | 1.8 | 5.1 | 3.3 |
| 40 | 5.0 | 1.9 | 2.0 | 7.3 | 10.9 | 8.5 | 9.8 | |
| 50 | 22.4 | 18.3 | 18.9 | 16.0 | 14.5 | 16.4 | 16.9 | 24.6 |
| 60 | 3.4 | 5.0 | 7.5 | 12.0 | 9.1 | 7.3 | 13.6 | 4.9 |
| 70 | 5.2 | 5.0 | 3.8 | 2.0 | 5.5 | 12.7 | 3.4 | 16.4 |
| 75 | 8.6 | 15.0 | 22.6 | 14.0 | 9.1 | 9.1 | 10.2 | 11.5 |
| 80 | 5.2 | 6.7 | 7.5 | 8.0 | 21.8 | 7.3 | 10.2 | 11.5 |
| 90 | 12.1 | 6.7 | 13.2 | 2.0 | 5.5 | 10.9 | 8.5 | 8.2 |
| 100 | 12.1 | 5.0 | 7.5 | 12.0 | 10.9 | 5.5 | 5.1 | 3.3 |
| Aware of risk criteria | % |
|---|---|
| n=sample size | n=66 |
| Yes, I am aware, but I do not apply them when I analyze and provide advice. | 33.3 |
| Yes, I am aware, and I do apply them when I analyze and provide advice. | 41.3 |
| No, I am not aware. | 25.4 |
| Applied consistently? | % |
|---|---|
| n=sample size | n=61 |
| Yes | 14.8 |
| No | 21.3 |
| Do not know | 63.9 |
| Level of overall understanding | % |
|---|---|
| n=sample size | n=62 |
| Significantly worse | 3.2 |
| Somewhat worse | 19.4 |
| No change (neither better nor worse) | 43.5 |
| Somewhat better | 27.4 |
| Significantly better | 6.5 |
| % | Quality of submission | Timelines | Document manage-ment processes used throughout TBS | Insufficient feedback from TB decisions, poor TBS condition tracking | Separation of the two TBS roles, as central agency and department, and its affect on roles and responsibi-lities | Limited program analyst under-standing of policy role, program analyst workload and turnover | Inconsis-tency of advice, lack of coordinated approach | Cumber-some process | Disorgani-zation of department, don't understand tools | Problems with financial information | n=sample size |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1st | 29.1 | 40.0 | 3.6 | 0.0 | 0.0 | 10.9 | 5.5 | 5.5 | 3.6 | 1.8 | n=55 |
| 2nd | 21.6 | 19.6 | 13.7 | 0.0 | 2.0 | 7.8 | 3.9 | 13.7 | 11.8 | 5.9 | n=51 |
| 3rd | 7.0 | 11.6 | 20.9 | 11.6 | 2.3 | 9.3 | 2.3 | 16.3 | 11.6 | 7.0 | n=43 |
| % | TBS tools | Strong process, i.e. is comprehensive, has clear results, based on risk assessment, focuses priorities | TBS staff support, good internal consultation process | Recent changes beneficial | Good communications, promotes external relationships | STS repository, submission tracking | n=sample size |
|---|---|---|---|---|---|---|---|
| 1st Strength | 6.5 | 50.0 | 32.6 | 0.0 | 6.5 | 4.3 | n=46 |
| 2nd Strength | 11.8 | 41.2 | 20.6 | 5.9 | 17.6 | 2.9 | n=34 |
| 3rd Strength | 13.0 | 43.5 | 21.7 | 17.4 | 0.0 | 4.3 | n=23 |
| % | Pre-submission advice and consultation? | Reviewing/consulting on submissions? | Briefing/ preparing briefing documents (e.g. précis)? | Other TBS submission activities? | Other TBS non-submission activities? |
|---|---|---|---|---|---|
| Average time for each activity | 15.5% | 29.2% | 6.6% | 9.0% | 39.7% |
| % | Development of policies, management activities | Part B submissions | MAF related | Provision of advice | Strategic reviews | IT oversight | Post-approval activities | Financial exercises | ARLU and Supplementary Estimates |
|---|---|---|---|---|---|---|---|---|---|
| n=38 | 18.4 | 5.3 | 15.8 | 21.1 | 5.3 | 7.9 | 7.9 | 10.5 | 7.9 |
| Workload provided time to provide advice | % |
|---|---|
| n=sample size | n=66 |
| Never | 1.5 |
| Rarely | 15.2 |
| Sometimes | 34.8 |
| Often | 36.4 |
| Always | 6.1 |
| Do not know | 6.1 |
| Level of workload | % |
|---|---|
| n=sample size | n=64 |
| Decreased | 7.8 |
| Remained the same | 37.5 |
| Increased | 48.4 |
| Do not know | 6.3 |
| Level of efficiency | % |
|---|---|
| n=sample size | n=52 |
| Very inefficient | 7.9 |
| Somewhat inefficient | 34.9 |
| Neither inefficient nor efficient | 19.0 |
| Somewhat efficient | 34.9 |
| Very efficientt | 3.2 |
| Measures undertaken? | % |
|---|---|
| n=sample size | n=63 |
| Yes | 38.5 |
| No | 61.5 |
| % | Delegated authorities | New tools | Focus on high-risk, high-value submissions | None taken / Not aware of | More staff | Removed the need to sign off on submission after the Treasury Board date | Change to EMIS |
|---|---|---|---|---|---|---|---|
| n=24 | |||||||
| Awareness of particular measures | 5 | 5 | 4 | 5 | 2 | 2 | 1 |
| Believe measure is better | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| Believe measure is no better | 0 | 1 | 0 | 0 | 0 | 0 | 1 |
| % | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know | n=sample size |
|---|---|---|---|---|---|---|---|
| Ministerial renewal of terms and conditions | 1.5 | 7.6 | 36.4 | 13.6 | 40.9 | n=66 | |
| Increasing department delegated authorities | 1.5 | 4.5 | 4.5 | 37.9 | 18.2 | 33.3 | n=66 |
| Chief financial officer (CFO) sign-off | 4.5 | 9.1 | 28.8 | 28.8 | 28.8 | n=66 | |
| Managing low-risk submissions differently | 1.5 | 3.0 | 6.1 | 39.4 | 33.3 | 16.7 | n=66 |
| Altering the submission calendar | 1.6 | 6.3 | 27.0 | 27.0 | 38.1 | n=63 | |
| Streamlining supply processes | 1.5 | 1.5 | 1.5 | 30.3 | 12.1 | 53.0 | n=66 |
| % | Make the process more objective and less subjective, better aligned and streamlined | Better training on internal process to have more knowledgeable analysts | Better federal organization and TBS engagement, earlier federal organization consultation and support | Better time frames | Better internal tracking and document system | Aligning Part A and Part B submissions | Regular TBS working groups to assess ongoing enhancements and changes |
|---|---|---|---|---|---|---|---|
| n=29 | 20.7 | 27.6 | 44.8 | 6.9 | 27.6 | 3.4 | 6.9 |
| % | Delays with federal organizations, federal organizations wait for last sitting, bad drafts from federal organizations | Lack of internal knowledge of internal processes and internal communications | Sign-offs and time pressures owing to political pressures and waves of submissions | None |
|---|---|---|---|---|
| n=22 | 31.8 | 22.7 | 22.7 | 9.1 |
| % | Need more time to do a better job (CIOB) | Resist unreal departmental time expectations, educate federal organizations on this | More conditions and monitoring thereof | Greater internal information sharing, more precise information available, and reduction in conflicting information | More intensive program analyst training | Introduce more performance measurement, audit, and evaluation evidence at the Memorandum to Cabinet stage | Increase review time for high-risk submissions | Investments in government-wide solutions needed rather than costly one-off solutions by individual departments |
|---|---|---|---|---|---|---|---|---|
| n=18 | 11.1 | 11.1 | 5.6 | 33.3 | 22.2 | 5.6 | 5.6 | 5.6 |
Federal Organization Survey Results
| Number of years | % |
|---|---|
| n=sample size | n=94 |
| Less then one year | 18.1 |
| 1 to 2 years | 28.7 |
| 2 to 3 years | 25.5 |
| 3 to 4 years | 10.6 |
| 4 to 5 years | 3.2 |
| More then 5 years | 13.8 |
| Federal organization | % |
|---|---|
| n=sample size | n=92 |
| Canadian Heritage | 4.3 |
| Foreign Affairs and International Trade Canada | 7.6 |
| Fisheries and Oceans Canada | 1.1 |
| National Defence and the Canadian Forces | 4.3 |
| Health Canada | 1.1 |
| Human Resources and Skills Development Canada | 17.4 |
| Indian and Northern Affairs Canada | 1.1 |
| Industry Canada | 1.1 |
| Infrastructure Canada | 1.1 |
| Natural Resources Canada | 1.1 |
| Privy Council Office | 1.1 |
| Public Works and Government Services Canada | 8.7 |
| Royal Canadian Mounted Police | 1.1 |
| Treasury Board of Canada Secretariat | 7.6 |
| Transport, Infrastructure and Communities Portfolio | 6.5 |
| Other | 34.8 |
| Other organizations | % |
|---|---|
| n=sample size | n=30 |
| Atlantic Canada Opportunities Agency | 3.3 |
| Atomic Energy of Canada Limited | 3.3 |
| Canada Border Services Agency | 3.3 |
| Canada Post Corporation | 3.3 |
| Canada Public Service Agency (now Office of the Chief Human Resources Officer) | 13.3 |
| Canadian Environmental Assessment Agency | 3.3 |
| Canadian Food Inspection Agency | 6.7 |
| Canadian Museum of Nature | 3.3 |
| Canadian Nuclear Safety Commission | 3.3 |
| Canadian Museum of Civilization | 3.3 |
| Correctional Service Canada | 3.3 |
| Courts Administration Service | 3.3 |
| Department of Finance Canada | 3.3 |
| Environment Canada | 3.3 |
| Export Development Canada | 3.3 |
| National Film Board of Canada | 3.3 |
| Office of the Commissioner for Federal Judicial Affairs Canada | 3.3 |
| Office of the Commissioner of Lobbying of Canada | 3.3 |
| Public Safety Canada | 0.0 |
| Public Service Commission of Canada | 10.0 |
| Public Service Labour Relations Board | 3.3 |
| Social Sciences and Humanities Research Council of Canada | 3.3 |
| Status of Women Canada | 3.3 |
| Veterans Affairs Canada | 3.3 |
| Number of analysts | % |
|---|---|
| n=89 | |
| Mean | 3.9 |
| 1 | 29.2 |
| 2 | 18.0 |
| 3 | 21.3 |
| 4 | 11.2 |
| 5 | 5.6 |
| 6 | 2.2 |
| 7 | 4.5 |
| 8 | 1.1 |
| 9 | 1.1 |
| 10 | 4.5 |
| 11 | 1.1 |
| Who to consult? | % |
|---|---|
| n=94 | |
| Yes | 95.7 |
| No | 4.3 |
| Number of Submissions | % |
|---|---|
| n=91 | |
| Mean | 11.1 |
| 0 | 3.3 |
| 1 | 17.6 |
| 2 | 16.5 |
| 3 | 17.6 |
| 4 | 6.6 |
| 5 | 7.7 |
| 8 | 2.2 |
| 9 | 1.1 |
| 10 | 2.2 |
| 12 | 1.1 |
| 16 | 1.1 |
| 17 | 1.1 |
| 20 | 3.3 |
| 25 | 2.2 |
| 27 | 1.1 |
| 30 | 3.3 |
| 35 | 2.2 |
| 40 | 2.2 |
| 50 | 1.1 |
| 51 | 1.1 |
| 75 | 1.1 |
| 100 | 2.2 |
| % completed that were: | Standard % | Complex % | Strategic Review % |
|---|---|---|---|
| n=sample size | n=80 | n=71 | n=41 |
| Overall submission distribution | 56.6% | 39.3% | 4.1% |
| 0 | 2.5 | 5.6 | 53.7 |
| 2 | 2.4 | ||
| 3 | 2.4 | ||
| 6 | 2.4 | ||
| 10 | 1.4 | 12.2 | |
| 17 | 1.4 | 2.4 | |
| 20 | 5.6 | 4.9 | |
| 23 | 2.4 | ||
| 25 | 5.0 | 8.5 | 9.8 |
| 28 | 1.4 | ||
| 30 | 6.3 | 4.2 | 2.4 |
| 33 | 1.3 | 5.6 | |
| 34 | 4.2 | ||
| 37 | 1.3 | ||
| 40 | 8.8 | 5.6 | 2.4 |
| 44 | 1.3 | ||
| 45 | 1.3 | ||
| 47 | 1.3 | 1.4 | |
| 50 | 20.0 | 23.9 | 2.4 |
| 60 | 3.8 | 5.6 | |
| 66 | 5.0 | ||
| 67 | 3.8 | 1.4 | |
| 70 | 2.5 | 7.0 | |
| 75 | 5.0 | 1.4 | |
| 80 | 5.0 | ||
| 90 | 26.3 | 15.5 |
| Number of submissions seeking exemptions and exclusions | % |
|---|---|
| n=sample size | n=87 |
| Mean | 10.5 |
| 0 | 57.5 |
| 10 | 19.5 |
| 20 | 5.7 |
| 25 | 4.6 |
| 30 | 4.6 |
| 50 | 5.7 |
| 70 | 1.1 |
| 100 | 1.1 |
| Attended training and learning opportunities | % |
|---|---|
| n=sample size | n=94 |
| Yes | 34.0 |
| No | 66.0 |
| % | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
|---|---|---|---|---|---|---|---|
| I have a strong understanding of the elements of TB submissions, policies, and process. | 5.3 | 10.6 | 39.4 | 43.6 | 1.1 | n=94 | |
| The Guide to Preparing Treasury Board Submissions helps to improve my understanding of the elements of TB submissions, policies, and process. | 1.1 | 7.4 | 11.7 | 50.0 | 23.4 | 6.4 | n=94 |
| TBS-developed tools help to improve my understanding of the elements of TB submissions, policies, and process. | 13.0 | 20.7 | 40.2 | 9.8 | 16.3 | n=94 | |
| TBS outreach helped to improve my understanding of the elements of TB submissions, policies, and process. | 2.2 | 22.6 | 22.6 | 23.7 | 10.8 | 18.3 | n=94 |
| In general, TBS analysts are offering us services that enable our draft submissions to comply with TB authorities, policies, and directions. | 4.3 | 10.6 | 50.0 | 28.7 | 6.4 | n=94 | |
| In general, TBS analysts are providing consistent policy advice/interpretation regarding TB submissions at the pre-submission stage. | 1.1 | 22.6 | 18.3 | 34.4 | 15.1 | 8.6 | n=94 |
| In general, the submission review process within TBS ensures that TB submissions comply with government authorities and policies. | 1.1 | 2.2 | 9.9 | 47.3 | 31.9 | 7.7 | n=94 |
| In general, TBS analysts are providing consistent advice regarding TB submissions during the TB submission review stage. | 6.4 | 21.3 | 11.7 | 41.5 | 13.8 | 5.3 | n=94 |
| In general, TBS analysts are providing accurate advice regarding TB submissions during the TB submission review stage. | 1.1 | 8.5 | 21.3 | 43.6 | 18.1 | 7.4 | n=94 |
| In general, TBS analysts are providing useful advice regarding TB submissions during the TB submission review stage. | 2.1 | 10.6 | 19.1 | 45.7 | 17.0 | 5.3 | n=94 |
| In general, TBS analysts are providing timely advice regarding TB submissions during the TB submission review stage. | 5.3 | 17.0 | 19.1 | 35.1 | 19.1 | 4.3 | n=93 |
| TBS communicates the TB decision with respect to our submission(s) in a timely manner. | 2.2 | 6.5 | 17.2 | 30.1 | 35.5 | 8.6 | n=94 |
| Our federal organization has adequate mechanisms in place to track conditions placed on our TB submissions. | 4.3 | 14.9 | 12.8 | 30.9 | 24.5 | 12.8 | n=94 |
| Weeks | Valid Percent |
|---|---|
| n=sample size | n=85 |
| Mean | 8.9 |
| 2 | 2.4 |
| 3 | 3.5 |
| 4 | 7.1 |
| 5 | 4.7 |
| 6 | 18.8 |
| 7 | 3.5 |
| 8 | 18.8 |
| 9 | 8.2 |
| 10 | 8.2 |
| 11 | 2.4 |
| 12 | 9.4 |
| 13 | 1.2 |
| 14 | 3.5 |
| 16 | 3.5 |
| 22 | 1.2 |
| 26 | 3.5 |
| % | Were complete with respect to your required internal consultations prior to submitting to TBS? | Were, in your opinion, submitted with enough time for TBS to adequately review them? | Followed the Guide to Preparing Treasury Board Submissions? | Had risk and mitigation strategies identified? | Asked for the right authorities? | Had an appropriate level of justification? | Contained accurate information (e.g. accurate financial tables)? | Were written in a clear manner? |
|---|---|---|---|---|---|---|---|---|
| n=sample size | n=85 | n=87 | n=83 | n=85 | n=85 | n=85 | n=85 | n=84 |
| Ind. (9.2) | Ind. (5.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | |
| Mean | 83.1 | 83.0 | 83.7 | 76.4 | 90.3 | 89.8 | 90.8 | 88.3 |
| 0 | 6.0 | 7.1 | 1.2 | |||||
| 10 | 2.3 | |||||||
| 20 | 1.1 | 2.4 | 1.2 | 1.2 | ||||
| 25 | 2.4 | 1.1 | 2.4 | 1.2 | ||||
| 30 | 4.7 | 1.1 | 1.2 | 1.2 | ||||
| 40 | 1.2 | 1.2 | ||||||
| 50 | 9.4 | 5.7 | 2.4 | 15.3 | 1.2 | 2.4 | 3.5 | 6.0 |
| 60 | 3.5 | 2.3 | 2.4 | 7.1 | 2.4 | 2.4 | 1.2 | 1.2 |
| 70 | 3.5 | 5.7 | 3.6 | 4.7 | 2.4 | 4.7 | 2.4 | 1.2 |
| 75 | 4.7 | 8.0 | 1.2 | 5.9 | 5.9 | 5.9 | 3.5 | 8.3 |
| 80 | 12.9 | 18.4 | 15.7 | 4.7 | 4.7 | 9.4 | 11.8 | 10.7 |
| 90 | 9.4 | 11.5 | 16.9 | 12.9 | 14.1 | 16.5 | 21.2 | 21.4 |
| 100 | 49.4 | 42.5 | 49.4 | 41.2 | 64.7 | 56.5 | 55.3 | 48.8 |
| Level of awareness | % |
|---|---|
| n=sample size | n=94 |
| Yes, we are aware, but we do not apply them when we prepare our TB submissions. | 7.8 |
| Yes, we are aware, and we do apply them when we prepare our TB submissions. | 50.0 |
| No, we are not aware. | 42.2 |
| Own risk criteria? | % |
|---|---|
| n=sample size | n=94 |
| Yes, but we do not apply them when we prepare our TB submissions. | 3.3 |
| Yes, and we do apply them when we prepare our TB submissions. | 52.7 |
| No | 44.0 |
| Risk criteria being applied consistently? | % |
|---|---|
| n=sample size | n=94 |
| Yes | 23.1 |
| No | 19.8 |
| Do not know | 57.1 |
| % | Inconsistent or inadequate advice from TBS analysts, analyst turnover | Complexity of information asked, too much information in submissions | Guide and tools insufficient | Time issues | Process issues (including political pressures) and changing requirements | Internal departmental issues | Receiving TBS communications, TBS capacity | n= |
|---|---|---|---|---|---|---|---|---|
| 1st Challenge | 20.5 | 9.6 | 2.4 | 36.1 | 21.7 | 7.2 | 2.4 | n=83 |
| 2nd Challenge | 26.3 | 13.8 | 1.3 | 18.8 | 21.3 | 13.8 | 5.0 | n=80 |
| 3rd Challenge | 19.2 | 20.5 | 5.5 | 21.9 | 17.8 | 6.8 | 8.2 | n=74 |
| % | Helpful tools (generally) | Support provided by helpful analysts | Ensures strong business case, diligence, accountability, challenge, and standards | Timelines understood | Provides future rigour to departments and Government of Canada | TBS willing to listen to suggestions, cooperative and collaborative process with feedback functions, TBS capacity | Promotes internal departmental strengths | n= |
|---|---|---|---|---|---|---|---|---|
| 1st Strength | 18.3 | 29.6 | 36.6 | 0.0 | 4.2 | 8.5 | 2.8 | n=71 |
| 2nd Strength | 6.7 | 31.7 | 28.3 | 5.0 | 8.3 | 15.0 | 5.0 | n=60 |
| 3rd Strength | 2.7 | 16.2 | 29.7 | 8. | 13.5 | 24.3 | 5.4 | n=37 |
| Level of workload | % |
|---|---|
| n=sample size | n=89 |
| Decreased | 4.5 |
| Remained the same | 28.1 |
| Increased | 67.4 |
| Level of workload | % |
|---|---|
| n=sample size | n=91 |
| Very inefficient | 8.8 |
| Somewhat inefficient | 33.0 |
| Neither inefficient nor efficient | 17.6 |
| Somewhat efficient | 38.5 |
| Very efficient | 2.2 |
| Measures taken to improve? | % |
|---|---|
| n=sample size | n=81 |
| Yes | 45.7 |
| No | 54.3 |
| % | Guide | Outreach | Early role of analyst and role of analyst as trusted advisor | Reduced submissions | New templates, protocols | Clear timelines | Not aware of new measures | Involve Finance unit for financial tables | Better communications |
|---|---|---|---|---|---|---|---|---|---|
| n=45 | |||||||||
| Awareness of particular measures | 26.7 | 8.9 | 13.3 | 2.2 | 17.8 | 8.9 | 4.4 | 8.9 | 8.9 |
| Believe measure is better | 8.9 | 4.4 | 4.4 | 0.0 | 4.4 | 0.0 | 0.0 | 0.0 | 0.0 |
| Believe measure is no better | 2.2 | 0.0 | 2.2 | 0.0 | 2.2 | 2.2 | 0.0 | 4.4 | 0.0 |
| % | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know | n=sample size |
|---|---|---|---|---|---|---|---|
| Ministerial renewal of terms and conditions | 1.1 | 2.2 | 9.0 | 30.3 | 38.2 | 19.1 | n=89 |
| Increasing department delegated authorities | 1.1 | 3.3 | 11.1 | 22.2 | 43.3 | 18.9 | n=90 |
| Chief financial officer (CFO) sign-off | 13.3 | 24.4 | 23.3 | 13.3 | 22.2 | 3.3 | n=90 |
| Managing low-risk submissions differently | 1.1 | 2.2 | 6.6 | 29.7 | 47.3 | 13.2 | n=91 |
| Altering the submission calendar | 6.7 | 15.6 | 20.0 | 18.9 | 11.1 | 27.8 | n=90 |
| Streamlining supply processes | 2.2 | 12.2 | 32.2 | 21.1 | 32.2 | n=90 |
| Better tools, specific tailored templates for recurring submissions, updated Guide to Preparing Treasury Board Submissions | More regular information meetings with TBS staff, earlier discussions, proactive not reactive | More training that includes TBS context and needs | Harder deadlines between Crown corporations and departments, Crown corporations work directly with TBS | TBS analysts with departmental knowledge, analyst stability | Limited review of draft and comprehen-sive, timely, and consistent feedback | An ombudsman or arbitrator to review/ bypass petty changes or difficult individuals | Clear explanation, decision, and rationale for next time | Delegation within TBS for submissions with pre-approved spending | Better calendar | Streamline process, eliminate departments' role for Supplemen-tary Estimates and ARLU | n=sample size | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| % | 6.8 | 13.6 | 4.5 | 6.8 | 13.6 | 18.2 | 4.5 | 4.5 | 4.5 | 4.5 | 18.2 | 44 |
| Good analysts have made the process flow well | Challenges owing to analysts' wide discretion, inconsistent comments, and instability, to inconsistent standards, and to sometimes unclear role | Political considerations undermines the process | Needs to be streamlined for some submissions, to become more efficient | Needs to be a cooperative and consultative process | Taking the time to produce a good quality first draft is essential to the efficiency of the process | n=sample size | |
|---|---|---|---|---|---|---|---|
| % | 16.7 | 26.7 | 10.0 | 13.3 | 30.0 | 3.3 | n=30 |
Appendix H: Description of Tools
Following are more specific observations on each of the tools:
- A Guide to Preparing Treasury Board Submissions (the Guide) – Program analysts appreciated it as a communication tool and felt that it was useful for new analysts. Federal organizations appreciated the enhanced checklists, tables, and specific criteria. Nonetheless, both groups saw areas for improvement, including a better index, better Web functionality for searching the online PDF, and the inclusion of more specific details. They also suggested that the implementation of the Guide had been painful and that there should be a better way to consolidate updates besides the current use of "one pagers."
- Other TBS-developed tools – Federal organization interviewees mentioned the usefulness of training on the new Policy on Transfer Payments, of costing manuals, of the Guide to Costing, and of other guides. Most program analyst and COE analyst interviewees (7 of 10) mentioned using other useful tools, including policies and guides posted on the TBS website, informal networking structures, and internal TBS outreach presentations.
- Program Analysts Boot Camp – Most program analyst interviewees (7 of 10) indicated that the boot camp was useful for new analysts; however, only half have been on the course themselves. One analyst noted that the course is not being offered as often as it used to be, given that some analysts have been in their role for six to eight months without having taken it. Another analyst noted that the course used to last two weeks and is now only two-day course.
- Outreach – Few federal organization survey respondents (34.4%) found TBS outreach to be helpful. Interviewees explained that outreach might be in the form of conference presentations, brown bag lunches, or specialized training. A few interviewees (3 of 12) had attended conferences where TBS had made presentations on topics related to submissions, which they all found useful. Another interviewee mentioned having a TBS director deliver a presentation to his department on the strengths and weaknesses of its submissions, while another interviewee had asked for such a presentation on the Guide and got no response.
- Mentoring and other training – Most interviewees did not note the presence of mentoring; however, a couple of interviewees indicated that mentoring should occur more frequently, especially within program analyst groups.
- The Treasury Board submission course offered by the Canada School of Public Service – Only 34.0% of federal organization survey respondents had attended the course, the majority of whom had been in their role longer than two years. Of the respondents who had been in their role less than two years, only 20.0% had attended the course. Among the federal organization interviewees, only 2 of the 12 had attended the course, and both found it to be useful. Similarly, some interviewees (5 of 12) had colleagues or acquaintances who had attend the course, all of whom reportedly found it useful. They noted that the course was especially good at setting the submission process in the greater government context. Two interviewees cited the following reasons for not taking the course: it was too expensive and was too hard to get in to.
Appendix I: Description of EXCO Measures
In an effort to streamline the Treasury Board submission process, TBS has considered and introduced a number of new measures to eliminate the need for a submission. For instance, during the TBS Executive Committee (EXCO) Retreat held in the fall of 2007, it was noted that delegated authorities have not kept pace with inflation or departmental capacity, resulting in many low-risk, transactional Treasury Board submissions.2 While increasing delegated authorities to be reflective of inflation has not removed the need for many submissions, for a few small agencies that are frequent submitters of many low-risk submissions, the need for more than 30 submissions can be eliminated annually.3 This will grant more time across TBS for high-risk submissions and strategic initiatives.4
Renewal of program terms and conditions (T&Cs) comprises a significant portion of Treasury Board business (700+ submissions from June 2000 to September 2007). It was proposed at the EXCO Retreat that ministers be authorized to extend expiring T&Cs for up to three years. This will allow Treasury Board more time to discuss high-risk or strategic business. Ministers will take responsibility for program T&Cs, giving their respective departments time to align T&Cs with new policy requirements and with the findings of strategic reviews.5
To qualify for increased delegated authorities, departments will have to demonstrate sound management practices. Of the 39 organizations reviewed, only 8 were recommended as potential candidates for increased delegated authorities. The reasons identified for not recommending organizations included the following: absent or poor track record, capacity issues in managing grants and contributions, low volume of submissions to have any workload impact, or existing thresholds or case-by-case review were deemed appropriate.6 To have increased delegated authorities, a federal organization must demonstrate that the first and second purposes of the submission process (oversight and management) can be completed in-house. In other words, the organization can manage itself successfully and oversight is not required.
It was also noted during the EXCO Retreat that TBS analysts spend a large amount of time reviewing poor quality draft submissions from departments. It is proposed that the chief financial officer (CFO) or delegate sign off on all draft Treasury Board submissions before TBS review to signal appropriate quality and completeness.7 This would assist analysts in their quality review and facilitation roles and would allow them to appropriately devote their time to analysis and not quality control. Furthermore, conditions would solely be used for high-risk submissions.
Appendix J: Definitions
| Name | Definition |
|---|---|
| Treasury Board | Is a statutory committee of Cabinet established under the Financial Administration Act. It consists of six ministers, including the President of the Treasury Board (Chairperson) and the Minister of Finance, who meet regularly to consider submissions sponsored by federal organizations. |
| Cabinet | The Cabinet decides the government's priorities and policies, determines the legislation that will be presented to Parliament, and approves departmental spending and allocates funds. The Cabinet consists of all ministers who are appointed on the recommendation of the Prime Minister. |
| Privy Council Office | A formal advisory body to the Crown appointed by the Governor General on the advice of the Prime Minister. The Clerk of the Privy Council is the most senior non-political official in the Government of Canada and provides professional, non-partisan support. |
| Précis | Document prepared by the Secretariat and addressed to Treasury Board. It contains Secretariat recommendations on Treasury Board submissions. |
| Expenditure Management Information System | The Expenditure Management Information System (EMIS) is an integrated and secure budget office system that supports the Treasury Board of Canada Secretariat (TBS) in fulfilling its expenditure management role. |
| Change Agenda | Defined in Context |
| TBS Program Sectors | These sectors, which support Treasury Board in its role as the government's management board and budget office, include the Economic Sector, the International Affairs, Security and Justice Sector, the Social and Cultural Sector, and the Government Operations Sector. They are the "single window" for departments to TBS on a wide range of issues, from Treasury Board submissions to Management Accountability Framework (MAF) assessments. |
| Treasury Board Submission Centre | As part of the Strategic Communications and Ministerial Affairs Branch, the Treasury Board Submission Centre (the Centre) provides the Secretariat, the Treasury Board, and other federal government institutions with support services and expert advice on the processes and procedures for submissions (which includes maintaining and enforcing deadlines). |
| TBS Boot Camps | Two-day orientation sessions that provide participants with a general overview of TBS roles, initiatives, and business. There are currently boot camps on 12 different topics, which range from "The Business of Supply and the Estimates Process" to "Crown Corporations." |
| Management, Resources, and Results Structures | The Policy on Management, Resources, and Results Structures supports the development of a common, government-wide approach to planning and managing the relationship between resource expenditures and results while serving as a consistent and enduring foundation for collecting, managing, and reporting financial and non-financial information to Parliament. |
| Strategic Reviews | Strategic reviews are departmental assessments of all direct program spending to ensure programs are managed effectively and efficiently. These reviews support a more rigorous results-based approach to managing taxpayer dollars responsibly and delivering effective and efficient programs that can better meet the priorities of Canadians. |
References
1. Source: A Guide to Preparing Treasury Board Submissions
2. Refers to all Treasury Board submissions except orders in council requiring Treasury Board recommendation.
3. The study did not assess Treasury Board submissions and associated précis documents for quality. Findings are based on qualitative data gathered from a survey and stakeholder interviews.
4. These issues are derived from the definitions of "relevance," "effectiveness," and "economy" found in Appendix A of the Treasury Board Policy on Evaluation.
5. A Guide to Preparing Treasury Board Submissions.
6. A limited review of submission process models used in other jurisdictions (see Methodology section) suggests that the role played by the Secretariat's assistant secretaries, whereby they present a federal organization's submission to Treasury Board, may be unique.
7. The Secretariat's program sectors are responsible for reviewing and providing advice on the management of federal organizations' financial resources.
8. A Guide to Preparing Treasury Board Submissions.
9. The term "Secretariat analysts" refers to both program analysts and COE analysts.
10. Other than Treasury Board, generally involves a policy committee.
11. It should be noted that when the Secretariat undertakes a Treasury Board submission on its own behalf, it is considered a federal organization like any other.
12. There is no formal definition of "complexity." The evaluation therefore relied on respondents' own criteria of "complex" and "standard." Note that strategic reviews (which can be "standard" or "complex") are not included in the table.
13. The Policy on the Management, Resources and Results Structures supports the development of a common, government-wide approach to planning and managing the relationship between resource expenditures and results while serving as a consistent and enduring foundation for collecting, managing, and reporting financial and non-financial information to Parliament. (http://www.tbs-sct.gc.ca/rma/mrrs-sgrr_e.asp)
14. A Guide to Preparing Treasury Board Submissions, p. 5
15. The Treasury Board submission and précis are then presented to Treasury Board ministers for consideration and approval.
16. While this question was part of the evaluation matrix, it was not covered in this report because (as discussed) it was determined that this was not an outcome of the Treasury Board submission process.
17. This question is not addressed in the findings of the report because the evaluation did not find any unexpected outcomes.
18. Note that when the report refers to "Secretariat analysts," it includes both program analysts and COE analysts if the context in which it is used is the Secretariat but only program analysts if the context is federal organizations (as program analysts are the primary contact for federal organizations). When this distinction is relevant, it is noted in the report.
19. The survey questionnaire was available on the Internet for a period of three weeks. Reminder emails were sent to respondents twice during the three-week period.
20. Responses and comments indicated that these respondents filled out the wrong survey; consequently, their responses were removed to ensure that the data was not contaminated.
21. Models in use in provinces were not reviewed.
22. Co-ordination at the Centre of Government: The Functions and Organization of the Government Office Comparative Analysis of OECD Countries, CEECs and Western Balkan Countries. Michael Ben-Gera. 2004.
23. Ibid. The ten countries are Australia, Austria, France, Germany, Iceland, Ireland, Netherlands, Norway, Spain, and Switzerland.
24. "The audit found that … a more consistent and inclusive approach to overtime costs and overtime forecasts, improvement to management and authorization of leave and overtime, … and development of formal and approved guidelines in the administration of leave and overtime would be beneficial."
25. The Program Activity Architecture of the Treasury Board of Canada Secretariat
26. A Guide to Preparing Treasury Board Submissions, p. 1
27. Ibid., p. 5
28. Feedback reviewed from the following program sector boot camps: February 2009, January 2008, August 2007, September 2004, March 2001, and September 2000.
29. "Appropriateness" refers to whether a submission put forward by a federal organization is necessary (i.e. required).
30. Interviewees also commented on the drawbacks of MCs not being required to undergo an early stage approval process from the Secretariat. One interviewee noted that the absence of Secretariat involvement can lead to problems when MCs include elements that may be contrary to Treasury Board policies.
31. Some federal organizations, however, indicated that they might choose to proceed with a submission even when the program analyst believes that it is not necessary.
32. Numbers may not add up to totals due to rounding.
33. This same question was not asked of federal organizations; therefore, the opinion of whether submissions were necessary is attributable only to Secretariat analysts.
34. From the time the program analyst receives the federal organization's first draft of the Treasury Board submission.
35. It is important to note that the time requirement may be significantly less in some cases, e.g. when the submission is a "top-up" to an existing program or when the submission is deemed to be of low risk and low complexity.
36. Some may argue, however, that flexibility in the Treasury Board submission process is an advantage in that it allows certain submissions, such as those related to Canada's Economic Action Plan, to be given priority handling when special circumstances warrant it.
37. According to guidelines in A Guide to Preparing Treasury Board Submissions
38. RDIMS is a Canadian federal government initiative to manage the full life cycle of any type of electronic document (email, correspondence, reports, etc.).
39. Budget Office Systems Renewal (BOSR) Project, Treasury Board of Canada Secretariat, 2007
40. http://www.tbs-sct.gc.ca/maf-crg/indicators-indicateurs/2008/policies-politiques/policies-politiques-eng.asp
41. Furthermore, scores between MAF Rounds IV and V are not fully comparable. The average MAF scores analyzed for the evaluation are for larger federal organizations.
42. Federal organizations' internal consultations
43. It should be noted that this percentage may not be representative of the level of effort of all COE analysts since only some COE analysts' duties include the review of Treasury Board submissions. Those COE analysts with significant duties relating to Treasury Board submissions may have been more likely to respond to the survey.
44. Numbers may not add to totals due to rounding.
45. Numbers may not add to totals due to rounding.
46. Data on the timing of Treasury Board submissions over the last 10 years show definite peaks in June, October, December, and March.
47. Factors contributing to the quality of Treasury Board submissions (according to survey): submissions appear not to have completed internal consultations (program analysts: 42.7%, COE analysts: 44.9%); submissions not allowing enough time for Secretariat analysts' review (program analysts: 42.5, COE analysts: 49.2%); submissions did not follow the Guide (program analysts: 43.3%, COE analysts: 37.3%); submissions lacking risk and mitigation strategies (program analysts: 40.5%, COE analysts: 47.5%); submissions requesting wrong authorities (program analysts: 39.8%, COE analysts: 38.1%); submissions without appropriate level of justification (program analysts: 45.3%, COE analysts: 44.1%); submissions containing inaccurate information (program analysts: 45.8%, COE analysts: 45.7%); submissions written in an unclear manner (program analysts: 48.6%, COE analysts: 39.4%).
48. While the evaluation team recognizes that turnover rates in federal organizations may also contribute to the quality of Treasury Board submissions and the efficiency of the submission process, this matter is outside the scope of the evaluation and therefore not addressed in the recommendation.
Appendice References
1. The data from this interview was not used, as Corporate Services was not identified as a submitting organization to be interviewed.
2. Treasury Board of Canada Secretariat, Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions, 2007.
3. Treasury Board of Canada Secretariat, Increased Delegated Authorities for Contributions, 2007.
4. Treasury Board of Canada Secretariat, Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions, 2007.
5. Ibid.
6. Treasury Board of Canada Secretariat, Increased Delegated Authorities for Contributions, 2007.
7. Treasury Board of Canada Secretariat, Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions, 2007.