Results - Immediate Impacts
Evaluation Question |
Individual Number |
Indicators |
Focus Group: Assistant Secretaries |
Interviews: TBS Analysts / Internal Contacts (n=10) |
Interviews: Federal Organizations (n=12) |
Interviews: External Stakeholders ( n=5) |
Web-Based Survey: Federal Organizations |
Web-Based Survey: TBS Analysts |
Web-Based Survey: TBS Internal Contacts |
Data Analysis |
Literature Review |
2.0 Do federal organizations have an increased understanding of the elements of TB submissions, policies, and process? |
2.1 |
Usefulness of guides, tools, and outreach provided by TBS |
|
|
1 |
|
1 |
|
|
|
1 |
3.0 Do TBS employees have an increased understanding of the elements of TB submissions and policies? |
3.1 |
Usefulness of guides, tools, and internal courses |
|
1 |
|
|
|
1 |
1 |
|
1 |
4.0 Is TBS offering services that enable federal organizations' draft submissions to comply with TB authorities, policies, and directions? |
4.1 |
Accuracy/consistency of TBS's pre-submission guidance |
|
1 |
1 |
|
1 |
1 |
|
|
|
4.2 |
Appropriateness of TB submissions being put forward |
|
1 |
|
|
|
1 |
1 |
|
|
4.3 |
Comparison of understanding of time required for TBS to review a submission |
|
|
|
|
1 |
1 |
1 |
|
|
4.4 |
Timeliness - Number of late submissions |
|
|
|
|
|
|
|
1 |
|
4.5 |
Relationship of TBS analysts to federal organizations (Change Agenda: strategic partner and trusted advisor, proactive risk manager, overseer vs. enabler, challenge vs. facilitation) |
|
1 |
1 |
|
1 |
1 |
1 |
|
|
5.0 Does TBS's submission review process ensure that TB submissions comply with government authorities and policies? |
5.1 |
Accuracy, consistency, usefulness, and timeliness of TBS advice/consultation during submission process |
|
1 |
1 |
|
1 |
1 |
|
|
|
5.2 |
Adequacy of time provided for input into TB submission documents |
|
|
|
|
|
1 |
1 |
|
|
5.3 |
Confidence that input from TBS consultations is reflected in final TB submission documents |
1 |
1 |
1 |
|
|
1 |
1 |
|
|
5.4 |
Limited legal contestation of the submission |
1 |
Legal only |
|
|
|
1 |
|
|
|
6.0 Is the TB submission process risk-based? |
6.1 |
Extent of knowledge of existence of risk criteria |
|
1 |
1 |
|
1 |
1 |
1 |
|
|
6.2 |
Appropriateness of risk criteria (government-wide perspective, reflect ministers' concerns, reflect financial and non-financial risks) |
1 |
1 |
1 |
|
|
|
|
|
|
6.3 |
Extent/consistency of use of criteria |
|
1 |
|
|
|
|
|
|
|
6.4 |
Extent that TB submission process, due diligence, précis, and decisions are impacted by risk rating |
1 |
1 |
|
|
|
|
|
|
|
6.5 |
Appropriateness of level of involvement in risk assessments (TBS vs. federal organizations) |
|
1 |
1 |
|
|
|
|
|
|
7. 0 Are TB decisions well-informed and consistent with advice, guidance, and recommendations provided by TBS analysts? |
7.1 |
Consistency of decision with recommendations, number and frequency of additional conditions applied by TB, number and frequency of challenges raised |
1 |
|
|
|
|
1 |
|
1 |
|
7.2 |
Number of deferrals and requests for more information |
|
|
|
|
|
|
|
1 |
|
7.3 |
Confidence in recommendations provided (e.g. with respect to risk, cost implications for government) |
1 |
|
|
|
|
|
|
1 |
|
8. 0 Are mechanisms in place to ensure TB decisions are carried out? |
8.1 |
Existence/adequacy of condition tracking processes:
- in federal organizations
- at TBS
|
|
1 |
|
|
1 |
1 |
1 |
|
|
8.2 |
Timeliness/completeness of decision communication |
|
|
|
|
1 |
1 |
1 |
|
|
8.3 |
Completeness/usability of TB submission filing system |
|
1 |
|
|
|
1 |
1 |
|
|
Results - Intermediate/Long-Term Impacts
Evaluation Question |
Individual Number |
Indicators |
Focus Group: Assistant Secretaries |
Interviews: TBS Analysts / Internal Contacts (n=10) |
Interviews: Federal Organizations (n=12) |
Interviews: External Stakeholders ( n=5) |
Web-Based Survey: Federal Organizations |
Web-Based Survey: TBS Analysts |
Web-Based Survey: TBS Internal Contacts |
Data Analysis |
Literature Review |
9.0 What is the level of quality of TB submissions? |
9.1 |
MAF score element 5 |
|
|
|
|
|
|
|
1 |
|
9.2 |
Completeness of TB submission documents:
- follow Guide to Preparing TB Submissions
- risks and mitigation strategies identified with sufficient information to allow for TBS analysis of risk
- asking for the right authorities
- appropriate level of justifications
- internal consultations in federal organizations are complete
- accuracy (tables are correct)
- clarity of TB submission
|
1 |
1 |
1 |
|
1 |
1 |
1 |
|
|
10.0 Does the TB submission process contribute to ensuring departmental/government management, programs, and spending are aligned with Government of Canada priorities? |
10.1 |
Documented accounts of TB submissions contributing to alignment (e.g. Auditor General reports, MAF, Results for Canadians, TBS DPR) |
|
|
|
|
|
|
|
1 |
1 |
10.2 |
Explanations of whether or not TB submission process contributes to ensuring alignment |
1 |
|
|
1 |
|
|
|
|
|
11.0 Are there any factors that are impeding the achievement of results? |
11.1 |
Description of factors impeding achievement of results |
|
1 |
1 |
|
|
|
|
|
|
12.0 Have there been any unexpected outcomes? |
12.1 |
Description of unexpected outcomes |
|
1 |
1 |
|
|
|
|
|
|
Effectiveness
Evaluation Question |
Individual Number |
Indicators |
Focus Group: Assistant Secretaries |
Interviews: TBS Analysts / Internal Contacts (n=10) |
Interviews: Federal Organizations (n=12) |
Interviews: External Stakeholders ( n=5) |
Web-Based Survey: Federal Organizations |
Web-Based Survey: TBS Analysts |
Web-Based Survey: TBS Internal Contacts |
Data Analysis |
Literature Review |
13.0 What is the level of resource allocation to submission process? |
13.1 |
Changes in level of TB submission effort 5 years ago relative to today |
|
|
|
|
1 |
1 |
1 |
|
|
13.2 |
Trend analysis of number of submissions |
|
|
|
|
|
|
|
1 |
|
13.3 |
Estimates of time spent:
- on pre-submission stage
- reviewing/consulting on submissions
- briefing and preparing briefing documents (e.g. précis) and
- other TB submission-related activities
|
|
|
|
|
|
1 |
1 |
|
|
13.4 |
Identify rate, determine step and where resource increases would have most impact on process flow |
|
1 |
1 |
1 |
|
1 |
1 |
1 |
|
13.5 |
Factors impeding analysts' ability to contribute to submission process |
|
|
|
|
|
1 |
1 |
|
|
14.0 Is the process efficient? What could be done to make the process more efficient? |
14.1 |
Usefulness of measures taken as a result of the 2003 audit and the 2007 EXCO Retreat (Roadmap for enabling change) |
|
1 |
1 |
|
1 |
1 |
1 |
|
1 |
14.2 |
Identification of any measures taken by TBS to manage workload (e.g. risk-based approach to submission review, delegation of authority) |
|
|
1 |
1 |
1 |
1 |
1 |
|
|
14.3 |
Identification of any areas where efficiency can be improved |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
|
|
15.0 Are Canadians getting value for their tax dollars? |
15.1 |
Analysis of efficiency, effectiveness, and impact indicators |
|
|
|
|
|
|
|
|
|