Common menu bar links
Breadcrumb Trail
ARCHIVED - Evaluation of the Public Service Modernization Act Strategic Investment Framework
This page has been archived.
Archived Content
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
2.0 Findings
2.1 Relevance
SIF activities were undertaken throughout departments and agencies to support the development of HRM capacity needed to implement the PSMA. The evidence showed that these activities were relevant. In fact, all key informants stated that there remains an ongoing need for the types of activities that the SIF funded.
2.1.1 Continued Need
Is there a continuing need to support departments and agencies in developing human resources management capacity?
The evaluation found evidence to suggest that there is a continuing need to support departments and agencies in developing HRM capacity.
The document review confirmed that having adequate HRM capacity was critical for implementing the HR reforms required by the PSMA and related legislation. The 2005 Status Report of the Auditor General cautioned that a lack of HRM capacity could "hinder the government's efforts to translate the provisions of the PSMA into concrete reforms"[8] and cause departments to "revert to their old systems, stalling progress toward modernized human resources management." PSHRMAC also identified in 2007 that HR capacity is a key challenge (among others) for implementing the PSMA.
The funding that the SIF provided to departments and agencies was meant to support them in addressing these challenges. There is also evidence that even after its sunset, there is a continuing need. For instance, the Prime Minister's Advisory Committee on the Public Service, in the Seventeenth Annual Report to the Prime Minister on the Public Service of Canada (2010), recognizes the "inadequacies of an outdated public service workplace"[9] and concludes that excellence in people management is fundamental and that role renewal is playing an important role.[10]
Almost all key informant interviews also indicated that there is a continuing need to support departments and agencies in developing HRM capacity. When questioned, they cited specific areas that still needed support, including training, human resources IT systems, staffing, and development of HRM tools and best practices. The document review shows that departments and agencies invested an additional $83.6 million of their own funds over the five-year period toward related projects. The evaluator considers this an indicator of their perceived need.
2.1.2 Alignment of Strategic Investment Framework Activities with Public Service Modernization Act objectives
Did the objectives and related activities of the SIF generally align with the objectives and spirit of the PSMA?
The evidence was consistent in indicating that SIF objectives and activities were aligned with the objectives and spirit of the PSMA.
The specific objectives of the PSMA were as follows:
- Create a more flexible staffing framework to manage and support employees and to attract the best people, when and where they were needed;
- Foster more collaborative labour–management relations to ensure a healthy and productive workplace; and
- Clarify accountabilities for deputy heads and managers.
The SIF was specifically created to facilitate the PSMA's implementation. It was divided into notional funding envelopes (see Table 2.1) totalling approximately $200 million.
From the departmental perspective, almost all key informants who were interviewed indicated that the SIF was appropriately aligned with the spirit of the PSMA, demonstrated by the activities it funded to support departments and agencies in implementing PSMA legislation and related changes. A large majority of respondents indicated that the objectives and related activities of the SIF were well aligned with the needs of departments and agencies because they were, in general, broad enough and flexible.
Table 2.1. Strategic Investment Framework Notional Envelopes as of April 2004
| Notional Envelope | Definition/Description | Notional Budget ($ millions) |
|---|---|---|
| Institutional Change | This funding supports the creation of the following:
|
60 |
| PSMA HR IT Systems |
This funding supports adjustments to HR IT systems in relation to the following:
|
55 |
| People Preparation | This funding is invested in the following:
People preparation also includes an amount for labour–management cooperative activities related to learning that are consistent with the intent of the PSMA. |
40* |
| New Functions |
This is the funding to put in place resources for new functions where no base exists, e.g.:
|
36 |
|
Project Management |
This funding supports the horizontal management and related work to be undertaken, including the PSMA Secretariat. It also includes the following:
|
9 |
| Total | $200 |
* The people preparation notional budget later increased to $62 million.
2.2 Implementation
Efficiency of Strategic Investment Framework Implementation
The evaluation results indicate that the SIF was organized and implemented as expected. Although the Master Plan was issued a year later than the SIF, the first progress report indicated that the PSMA Secretariat developed and maintained project implementation timetables to identify key activities and critical milestones. Status and summary reports were used later on to inform the commitments and achievements. Evidence shows that all PSMA legislative requirements were met to a significant degree. There is evidence on the effectiveness of the project plan in guiding horizontal initiatives across departments and agencies.
2.2.1 Effectiveness of the Role of the Project Office[14]
How effective was the role of the Project Office, delivered in concert with other players?
How effective was the role of the Project Office in the context of the functionality of the governance model (i.e., as a secretariat)?
The evaluation focused on two main areas: the extent to which the role of the Project Officewas clear and adequately communicated, and the overall effectiveness of the Project Office.
The role of the Project Office was well articulated and communicated in SIF documentation; however, it is unclear to what extent it was well understood among departments and agencies. The evaluation found that the Project Office effectively managed the SIF's implementation. Evidence shows that the governance framework and management systems and processes in place for the SIF were adequate.
Clarity and Communication of Project Office Roles
The document review found a number of sources that described the role of the Project Office. These included the PSMA Implementation Project Charter (2004), the first progress report on PSMA implementation (2005), the Results-Based Management and Accountability Framework for the PSMA Implementation Project (2005) and the Master Plan for the Implementation of the Public Service Modernization Act (2006).
However, the extent to which these documents were communicated to departments and agencies and other stakeholders is not clear. Although some respondents indicated their understanding of the role of the Project Office, most key informants reported a lack of clarity and/or confusion, citing lack of communication as the reason.
Effectiveness of the Project Office
Progress reports and other assessments provided some evidence of the effectiveness of the Project Office in terms of overall management of the SIF's implementation. The progress reports prepared in 2005 and 2007 by PSHRMAC and the 2007 status report on PSMA deliverables provide sound evidence to confirm that the Project Office put in place the processes and mechanisms needed to effectively manage the SIF's implementation.
A 2009 assessment by the Treasury Board of Canada Secretariat of SIF management controls for adequacy and effectiveness, during the period from May 2004 and March 2009, found that the governance framework and management systems and processes that were put in place for the SIF were adequately designed.[15] It showed the following:
- A formal governance structure was in place;
- HRMAC provided oversight to monitor the SIF's direction, plans and actions;
- Deputy heads were accountable for PSMA implementation; and
- The PSMA Secretariat actively supported the SIF's implementation.
A few key informants indicated that the Project Management Committee was effective and kept everyone informed, although challenges were mentioned, including high turnover and the change in governance (when the Project Office was moved and then downsized).
2.2.2 Project Plan Implementation
How effective was the project plan and its implementation?
The evidence suggests that, for the most part, appropriate mechanisms were in place for effective implementation of the project plan.
Project Management
The PSMA Implementation Project started in 2001 and was expected to sunset in 2006 but was extended until 2010. It encompassed four phases: conception and design; development and implementation; post-implementation; and monitoring, evaluation and reporting.
The PSMA Master Plan consisted of the Project Charter, project path, work stream plans, and project implementation timetables. The Master Plan included objectives of the PSMA's implementation, the governance landscape, governance and stakeholders, PSMA outcomes, risk management, the PSMA Reserve Fund, and the SIF funding life cycle.
The SIF funding life cyclewas a rigorous process used to manage SIF allocations. This process is summarized as follows:
- Call letters: These were sent to departments and agencies to determine their funding requirements and their internal reallocation of financial resources;
- A notice of intent: Departments and agencies submitted this notice to outline requirements and provide a PSMA implementation summary;
- A business case: This was submitted by departments and agencies whose notices of intent had been screened by the PSMA Secretariat and found to be compliant. Business cases outlined specific investments requirements and their planned PSMA implementation activities;
- Treasury Board submission: Business cases were evaluated by an interdepartmental committee and presented for Treasury Board review and approval;
- Decision letters: Departments and agencies were notified by a decision letter of Treasury Board decisions;
- Memorandum of Understanding[16]: Departments and agencies signed a Memorandum of Understanding; and
- Assessment and reporting: PSMAS monitoring and reporting processes are triggered.
Key guiding documents such as the SIF (April 2004), the Results-Based Management Accountability Framework (September 2005) and the PSMA Implementation Project Charter (November 2005) were completed and implemented. Progress reports and status reports subsequently provided summaries of completed activities, milestone dates, achievements, gaps and actions as required in the Master Plan 2006.
Progress reports on PSMA implementation indicated that delays in the implementation occurred as a result of departmental reorganizations and the time needed to manage the change agenda.
Funding Decision-Making Process and Principles
There is documentary evidence[17] that departments and agencies were provided with the PSMA-SIF business case guidelines and template to help prepare their business case(s) to access SIF funding. These clearly outlined the decision-making process for funding. Although some key informants suggested that the tools, templates and advice on costing could have been better, a large majority reported that SIF guidelines and criteria were clear.
Some feedback from key informants suggested that the template could have been simpler and less onerous to complete and that supporting documents could have better explained the criteria and provided advice on costing proposals. Key informants reported the issue of the short timeframe for implementation once the allocation decisions had been made. Key informants also reported issues regarding short timeframes between allocation decisions and implementation. Some departments and agencies were not ready to take advantage of SIF funds or had inadequate time for planning.
The SIF business case guidelines and template outlined five funding principles that were considered during the review of the business cases by the evaluation committee. The five principles were as follows:
- Criteria assessment: To assess how well the business case meets the requirements and funding principles;
- Qualitative assessment: To determine the completeness of the information provided, as per the business case template;
- Funding assessment: To determine the cost efficiency and effectiveness of financial requirements, including the costing methodology;
- Horizontal partnerships: To encourage PSMA initiatives that would lead to service-wide benefits, or enable shared, cumulative benefits across multiple organizations to gain efficiency and meet collective needs. Organizations are encouraged to seek out and pursue horizontal partnerships with other departments and agencies; and
- Holistic assessment: To consider the cohesiveness and overall criticality of the business case."[18]
Key informants were able to comment on only the horizontal partnership principle and generally indicated that the project plan was effective in coordinating collaboration across departments and agencies.
Notional Funding Envelopes
Table 2.2 provides a comparison of total SIF disbursements by notional funding envelope. Overall, $188.7 million, or 94 per cent, of SIF funding was disbursed to departments and agencies, which is an indication of an efficient approach. Over the five fiscal years, departments and agencies requested additional funds for training and learning initiatives through the business case process. As a result, $62.1 million of the $188.7 million was disbursed according to the "people preparation" envelope. When comparing the overall amount disbursed with the amount spent by departments and agencies in all envelopes, 83.4 per cent of the funds approved were invested by departments and agencies, which is again a key indicator of efficiency.
Table 2.2. Strategic Investment Framework Disbursements and Expenditures by Notional Envelope
| Notional Envelope | SIF Envelope ($) | SIF Disbursements ($) | SIF Spending ($) | Percentage of SIF Disbursements | Percentage Spent of Approved Funding |
|---|---|---|---|---|---|
| New Functions | 36,000,000 | 29,611,036 | 32,450,948 | 82.3 | 109.6 |
| HR Systems Changes | 55,000,000 | 46,913,298 | 45,155,097 | 85.3 | 96.3 |
| People Preparation | 40,000,000 | 62,106,164 | 42,284,288 | 155.3 | 68.1 |
| Institutional Change | 60,000,000 | 41,363,744 | 21,311,005 | 68.9 | 51.5 |
| Project Management | 9,000,000 | 8,704,699 | 16,110,476 | 96.7 | 185.1 |
| Total | 200,000,000 | 188,698,940 | 157,311,814 | 94.3 | 83.4 |
Source: PSMA-SIF database
Audit and Evaluation Framework
The audit and evaluation component for the implementation of the PSMA (including the SIF) required the development of an evaluation framework, terms of reference for the evaluation, a results-based management and accountability framework, and reporting. The latter was based on progress reports.
Each fiscal year, a summary of department/agency reports on PSMA implementation (PSMA monitoring and summary reports, including progress and financial data) was presented to PSHRMAC, from 2005–06 to 2008–09. PSMA monitoring and reporting reports for fiscal year 2008–09 noted, however, that the reporting burden on departments and agencies was a concern. To address this, call letters replaced the HR Integrated Reporting Portal and the non-financial data requested of departments and agencies. The revised approach was limited to brief descriptions of SIF activities, which were output-based and therefore did not provide outcome information.
PSHRMAC used the data collected through this reporting process to draft progress reports and a summary of results. Because the results reporting was output-oriented, it could not be used to assess outcomes.
Risk Management
All business cases submitted for SIF funding consideration were required to address risk management and include an assessment of significant risk areas. Risk assessment and risk management were built into the assessment of funding applications. The SIF business case guidelines, short evaluation form and the SIF business-case detailed evaluation template are examples of the tools developed to assess risk in each business case.[19]
2.2.3 Guidance on Horizontal Work Across Departments
How effective was the project plan in guiding horizontal work across departments?
The documentary evidence indicates that the project plan was effective in guiding horizontal work across departments and agencies. Most interviewees, however, were unable to comment on this.
The Management Framework[20] component of the PSMA implementation, established as an objective "the development and implementation of stakeholder engagement throughout the process." Under this framework, the governance landscape component involved "the description of the governance structure which included an interdepartmental committee structure consisting of central agencies and departmental representatives to contribute to the implementation of the PSMA." The support of horizontal management and related work undertaken by either the PSMA Secretariat or the departments and agencies were set out under the notional envelope "project management." Horizontal partnership was also one of the funding principles considered in evaluating business cases.[21]
The document review shows that two interdepartmental committees were created to support horizontal collaboration and partnership. The Interdepartmental Project Management Committee was an advisory body composed of 14 representatives, and the Interdepartmental Project Management Network was a community forum composed of representatives from 90 organizations.
The second progress report provides some evidence on the activities undertaken by the PSMA Secretariat that could be linked to horizontal partnership activities:
- Some organizations collaborated with others on shared products such as the brochure "How to Apply for a Job";
- Some departments and agencies worked together to establish informal conflict management systems training programs;
- Small agencies worked with the SATST and bargaining agents to establish a joint Labour Management Consultation Committee; and
- Some departments and agencies collaborated to obtain economies of scale in delivering training within regions at missions outside of Canada and for communities of interest.
The data review shows that functional communities and federal councils accessed SIF funds through some departments acting as "bankers." Approximately $2.2 million (3.3 per cent) of the $65.4 million received by departments and agencies was used by functional communities and federal councils for online tools and learning events.
Based on information provided by key informants, the project plan was not intended for distribution to departments and agencies. This seems to be reflected by the fact that most respondents could not comment on the effectiveness of the project plan, which was therefore not a conclusive line of evidence. Nonetheless, a few respondents indicated that interdepartmental committees, which included small and large departments and agencies, facilitated coordination across the federal government. Only a few commented that departments and agencies worked collaboratively.
Several respondents pointed out that there was not enough emphasis on assisting smaller departments and agencies, suggesting that the plan may have been too restrictive for smaller agencies and provided limited opportunity for them to collaborate horizontally with larger departments and agencies at the planning stages.
On the other hand, the SATST final report of 2009[22] indicates that 100 per cent of clients surveyed in the SATST 2008 survey agreed with the statement "the support and assistance that SATST provided my organization is of significant assistance in integrating the requirements of the Public Service Modernization Act."
2.2.4 Effectiveness of the Capability and Functionality of Information Technology
How effective was the IT capability and functionality specific to this project?
There appears to have been sufficient IT capability and functionality for SIF implementation as evidenced by the creation of a combined PSMA-SIF database and the HR Integrated Reporting Portal. However, there were no users available to interview, which would have provided another line of evidence to support this conclusion.
The document review indicated that the PSMA-SIF database was developed in 2007 to ensure stewardship of the PSMA Reserve Fund and therefore of SIF funds. The electronic system reconciled and accounted for all funding to ensure the tracking and monitoring of expenditures, and to record information on the internal reallocations made by departments and agencies. It also facilitated in preparing reports that were considered to be timely, accurate, up to date and in line with good financial management practices. In fiscal year 2006–07, the HR Integrated Reporting Portal was created to gather SIF data from departments and agencies. Key informants were not able to speak to the IT capability or functionality related to SIF funds.
2.2.5 Compliance With Public Service Modernization Act Legislative Requirements
To what degree were PSMA legislative requirements met?
The evaluation found that PSMA legislative requirements were met to a significant degree.
The document review found that all legislative components of the PSMA are in force. This includes the Public Service Labour Relations Act (April 1, 2005), amendments to the Financial Administration Act (April 1, 2004), a new Public Service Employment Act (December 31, 2005), and amendments to the Canadian Centre for Management Development via the Canada School of Public Service Act (April 1, 2004).
As well, all institutional changes were in place at the time of the second progress report (2007): mandates were revised for the Public Service Commission of Canada, Treasury Board of Canada Secretariat and the Privy Council Office; the Canada School of Public Service was launched; the Public Service Staffing Tribunal and the Public Service Labour Relations Boardwere operational; and PSHRMAC was created to lead the PSMA Implementation Project.
The evaluation did not assess the degree of compliance by departments and agencies that had PSMA legislative requirements. However, evidence from the document review suggested that additional requirements stemming from the legislative package were met. For instance, all staffing delegation instruments were in place government-wide; departmental staffing policies, programs and processes had been developed; and departments and agencies had reviewed and updated labour relations delegation instruments and grievance procedures to be in line with changes to the Financial Administration Act and Public Service Labour Relations Act.
2.2.6 Appropriate Linkage and Leveraging With Non–Public Service Modernization Act Modernization Activities
Were there appropriate linkages and leveraging with non-PSMA modernization activities that contributed to the successful implementation of the PSMA?
Although leveraging generally took place, there was insufficient evidence to clearly indicate the extent to which linkages and leveraging with non-PSMA modernization activities occurred or were even the objective of SIF activities.
A large majority of respondents indicated that non-PSMA modernization activities occurred parallel to, and supported, SIF implementation. However, there were insufficient details to clarify the extent to which this occurred.
A few respondents reported having made some effort to link non-PSMA and non-SIF implementation activities. Those who did provided examples referred to shared services, the implementation of PeopleSoft, and the development of classification tools. These activities enabled departments and agencies to be more efficient overall and contributed to improved results related to SIF implementation.
2.3 Economy
2.3.1 Value for Money
Was there value for money of the actual investment?
Although results from the document review and from key informant interviews showed that SIF funding did indeed provide value, the evaluation could not determine if this could have been accomplished with fewer resources.
A large majority of key informants indicated that without SIF funds, their departments and agencies would not have been able to achieve the changes required by the PSMA as successfully or as quickly, and most of the smaller departments and agencies would not have had the resources at all. The initiatives reportedly made departments and agencies more effective and efficient in the area of HRM.
SIF investment provided opportunities to undertake activities such as:
- Training for HR managers and staff related to the new PSMA legislation;
- Integrated HR planning;
- Adaptation and/or development of HR IT systems and HR management tools;
- Communications strategies; and
- National learning events and conferences.
The above suggests that SIF funding provided value. However, since alternatives to the SIF were not identified and costed, the evaluation is not able to make a conclusion regarding value for money.
2.3.2 Level of Uptake of Institutions
What was the nature and level of uptake of institutions?
The evidence shows that the level and nature of uptake of SIF funds was appropriate. Eighty-eight per cent of departments and agencies were assisted either directly or indirectly through SATST or parent departments with their PSMA implementation requirements.
Uptake by Institutions
In 2009, there were 115 departments and agencies in the administrative database. Table 2.3 indicates the coverage of departments and agencies by SIF projects.
Table 2.3. Coverage of Department and Agencies by Strategic Investment Framework Projects 2008–09
| Institution | Number of Departments and Agencies | Percentage of Departments |
|---|---|---|
| Direct funding (includes departments, agencies, the Treasury Board Portfolio and Treasury Board Partners) | 32 | 28 |
| Direct funding (also received SATST support) | 9 | 8 |
| SATST support (received no direct funding but received funding through PSHRMAC) | 43 | 37 |
| Indirect support through parent department | 17 | 15 |
| No funding; participated in government-wide or functional events | 14 | 12 |
| Total* | 115 | 100 |
| Functional organizations and Regional Councils | 10 | — |
*In 2005, the database showed 117 organizations. By 2009, the Law Commission and the Canadian Centre for Independent Resolution of First Nations Claims no longer existed.
The administrative data review also found that departments and agencies that received direct funding under the SIF accounted for more than 90 per cent (180,000) of employees in the core federal public service. Departments and agencies that received indirect support through parent departments and support from SATST accounted for 2 per cent of the employees (see Table 2.4).
Table 2.4. Coverage of Employees by Strategic Investment Framework Projects 2008–09
| Institution | Number of Departments and Agencies | Core Public Administration FTEs* | Separate FTEs | Total FTEs |
|---|---|---|---|---|
| Direct funding (includes departments, agencies, the Treasury Board Portfolio and Treasury Board Partners | 32 | 184,158 | 1,639 | 185,797 |
| Direct funding (also received SATST support) | 9 | 1,454 | 115 | 1,569 |
| SATST support (received no direct funding but received funding through PSHRMAC) | 43 | 3,267 | 2,387 | 5,654 |
| Indirect support through parent department | 17 | 640 | 1,595 | 2,235 |
| No funding; participated in government-wide or functional events | 14 | 13,230 | 59,088 | 72,318 |
| Total** | 115 | 202,749 | 64,824 | 267,573 |
| Functional organizations and Regional Councils | 10 | — | — | — |
*Full-time equivalents
**In 2005, the database showed 117 organizations. By 2009, the Law Commission and the Canadian Centre for Independent Resolution of First Nations Claims no longer existed.
Of the 41 organizations that received SIF funding, 33 were line departments or part of the five PSMA Corporate Partners Groups (Justice Canada, the Public Service Staffing Tribunal, the Public Service Labour Relations Board, the Public Service Commission of Canada and the Privy Council Office). Departments and agencies received $65.4 million, with approximately $164 million awarded to Treasury Board Portfolio organizations and PSMA Corporate Delivery Partners.
Leveraging
Economy was demonstrated by the significant leveraging achieved through departmental reallocations to projects funded through the SIF (see Table 2.5). Over $83.5 million was reallocated by departments, agencies and central partners from 2004 to 2009, which amounted to approximately 30 per cent of the total PSMA expenditures.
Table 2.5. Total Approved Funding and Expenditures*
| Year | PSMA Reserve Fund | Internal Reallocations Reported | Total Expenditures | |
|---|---|---|---|---|
| Disbursements | Reported Expenditures | |||
| Before the SIF | ||||
| 2002–03 | $16,725 | $15,418 | 0 | 15,418 |
| 2003–04 | $23,954 | $16,678 | 0 | 16,678 |
| Total Before the SIF | $40,679 | $32,096 | 0 | 32,096 |
| After the SIF | ||||
| 2004–05 | $14,198 | $14,114 | 8,520 | 22,634 |
| 2005–06 | $81,903 | $56,499 | 42,032 | 98,531 |
| 2006–07 | $25,302 | $18,530 | 12,067 | 30,597 |
| 2007–08 | $37,340 | $33,639 | 13,943 | 47,581 |
| 2008–09 | $29,957 | $34,530 | 7,003 | 41,533 |
| Total after the SIF | $188,699 | $157,312 | 83,566 | 240,877 |
| Total PSMA-SIF Fund** | $229,378 | $189,408 | 83,566 | 272,973 |
Note: Of the total $238-million PSMA Reserve Fund, $8.6 million was returned to the fiscal framework by way of the February 2008 federal budget.
* In $000s*
*Totals may not add due to rounding.
Source: SIF, Summary data from database, September 2009
Uptake by Targeted Area for Support
Disbursements were made for activities under each notional envelope (see Table 2.6). Over time, there was substantial use of SIF funding for all types of projects, even within the last years of the funding. The expenditures under the SIF were approximately $15 million per year prior to 2005–06 but peaked at $56.4 million in 2005–06. Although the use of the fund dropped in subsequent years, expenditures still averaged approximately $29 million per year.
Table 2.6. Strategic Investment Framework Expenditures by Envelope (Thousands) and Fiscal Year
| Envelopes | Pre-SIF | Post-SIF | |||||
|---|---|---|---|---|---|---|---|
| 2002–03 | 2003–04 | 2004–05 | 2005–06 | 2006–07 | 2007–08 | 2008–09 | |
| New Functions | — | — | $2,709 | $8,267 | $8,390 | $7,436 | $5,646 |
| HR System Changes | $1,500 | $1,018 | $2,319 | $12,768 | $8,243 | $7,470 | $14,352 |
| People Preparation | — | — | $1,994 | $22,637 | $1,631 | $8,571 | $7,449 |
| Institutional Change | $80 | $1,478 | $3,736 | $6,089 | $100 | $6,026 | $5,358 |
| Project Management | — | — | $3,354 | $6,735 | $164 | $4,134 | $1,721 |
| Not Separated by Envelope | $13,837 | $14,181 | — | — | — | — | — |
| Total* | $15,417 | $16,677 | $14,113 | $56,499 | $18,530 | $33,638 | $34,529 |
*Totals may not add due to rounding.
Source: SIF, Summary data from database, 2009
2.4 Effectiveness
The evaluation could not conclude firmly on the extent to which the SIF investment strategy contributed to achieving the expected medium- and long-term outcomes of the PSMA. Nonetheless, the expected outputs and most immediate and short-term outcomes were achieved.
2.4.1 Effectiveness of the Application of the Investment Strategy in Achieving Public Service Modernization Act Objectives
How effective was the application of the investment strategy in achieving the desired objectives (results and outcomes) of the PSMA?
The application of the SIF investment strategy appears to have been effective in facilitating the achievement of the PSMA objectives. However, due to limited lines of evidence, it is unclear to what extent this was the case.
Assessing the SIF investment strategy's effectiveness in achieving the PSMA objectives can be done by looking at a combination of factors, the key ones being as follows:
- A targeted funding approach;
- Investment in an implementation unit;
- Relevant funding categories;
- The use of leveraging;
- The use of the SATST to represent small departments and agencies; and
- The use of "banker" departments to disburse funds to their functional communities.
At the outset, a targeted rather than broad funding approach was used. Envelopes were dedicated to agreed upon categories that supported key elements of the PSMA (e.g., people preparation, institutional change). The distribution of funding across the notional envelopes was fairly even (with the exception of project management, which represented the central implementation unit), which demonstrates that they all were relevant to departments' and agencies' needs. Using leveraging as a tool in the funding process helped create a more economical means of implementation. The use of "banker" departments helped reach functional communities horizontally. Using the SATST as a mechanism for representing small departments and agencies and funding distribution also assisted in the efficient disbursement of funding.
The oversight mechanisms used by the Project Office helped ensure that funds were being used for the agreed upon activities. However, as previously indicated, results reporting was limited and output-based. The PSMA-SIF database, the HR Integrated Reporting Portal, and call letters to departments and agencies provide evidence that projects were appropriately monitored and reported. Compliance with the call letters among departments and agencies was high for each fiscal year (for which data was available): 100-per-cent compliance in 2005–06, 95-per-cent compliance in 2006–07, and 100-per-cent compliance in 2007–08.[23]
Finally, evidence from the document review, key informant interviews and case studies suggest that the investment strategy of the SIF created to some extent a more flexible staffing framework, fostered more collaborative labour management relations, and clarified accountability for deputy heads and managers.
2.4.2 Achievement of Strategic Investment Framework Outputs and Outcomes
What results were achieved (including both outputs and outcomes)?
Information analyzed from the five case studies and key informant interviews demonstrate that all SIF outputs and most intended immediate and short-term outcomes[24] were to some extent achieved.
Although the evidence was stronger in some areas than in others, key informants mentioned a wide range of projects that resulted from SIF implementation. Importantly, most key informants indicated that several project outcomes could be attributed to the SIF given that the projects could not have gone forward without funding.
Outputs
The document review found evidence to indicate that all expected outputs were achieved:
- Planning and communication: Funding guidelines and eligibility criteria were produced, as was training material for departments and agencies and a related communications strategy.
- Proposal review and project funding: SIF funding arrangements were put in place and, as a result, business cases were funded.
- Monitoring and reporting: The document review found evidence that progress reports were prepared periodically, that the HR Integrated Reporting Portal was developed, and that a database was implemented.
Immediate Outcomes
Of the three immediate outcomes, two were achieved to some extent.
- Awareness among departments and agencies regarding the SIF and funding guidelines
Although some key informants could not comment on awareness of the SIF and its guidelines, it could be inferred that most of the departments and agencies were familiar with these documents because they successfully accessed funding.
- Alignment of investments with PSMA priorities and SIF principles
As previously indicated under "Relevance," the SIF investment strategy included the submission of business case proposals, which were based on specific funding principles and the business case guidelines and template. These procedures ensured that proposals from departments and agencies were aligned with SIF principles and PSMA priorities. Evidence already presented indicates that this outcome was achieved.
- Projects are implemented in a timely manner and co-funded by departments and agencies
The administrative data review found that departments and agencies co-funded SIF projects or that they were funding other PSMA-related activities while in receipt of SIF funding. On average, reallocations made by departments and agencies accounted for $83.5 million, or 30 per cent, of total PSMA expenditures.
Whether projects were implemented in a timely manner is not clear; however, the evidence suggests that timeliness was an issue. The administrative data review found that over three quarters of departments and agencies lapsed SIF funds from one year to the next and occasionally over two or more years. Key informants indicated that delays in project implementation were frequently a result of –09 because of delays in PSMA implementation as a result of the reorganization of the management office.
- Stakeholders are apprised of SIF funding investments and results achieved
The document review showed that Treasury Board ministers were informed periodically on the progress of the PSMA's implementation, including SIF funding investments and the results achieved through progress reports.
Short-Term Outcomes
Although the evidence shows that all six short-term outcomes were achieved, it was not adequate in order to verify the extent of achievement.
- Institutional change: Selected existing institutions have expanded mandates and new institutions are created with new mandates.
Different progress reports confirmed that the institutional changes planned with SIF funds were successfully achieved, such as the following:
- Revision of mandates for the Privy Council Office, the Public Service Commission of Canada, and Treasury Board of Canada Secretariat;
- Launch of the Canadian School of Public Service;
- Establishment of the Public Service Staffing Tribunal and the Public Service Labour Relations Board;
- Creation of the Public Service Human Resources Management Agency of Canada (which became the Canada Public Service Agency and has since been integrated back into the Treasury Board of Canada Secretariat).
- Human Resources Information Technology System: There are examples where the necessary human resources information technology (HR IT) systems are in place to support new and evolving needs. However, it is not known to what extent this is the case or to what extent the needs are being met.
Key informants were generally unable to comment on enhancements made to existing government-wide IT systems (e.g., introduction of PeopleSoft, Human Resources Information System, or HRIS), but they did identify HR IT systems as a top priority area that requires continuing support.
The document review found a number of examples of HR IT systems that were developed to support SIF implementation, including the PSMA-SIF database and the HR Integrated Reporting Portal. At the departmental level, the case studies showed that a number of specific systems were also supported by the SIF. For instance, Infrastructure Canada created an electronic information system (HRIS) to replace a paper-based one to deal more effectively with HR issues. National Defence established an integrated staff log application to manage and report on large collective processes, and Agriculture and Agri-Food Canada made enhancements to its business intelligence tools.
- People preparation:Public service managers, human resources professionals and staff participate in PSMA-related learning events.
The 2006–07 summary report indicated that by 2007, 75 per cent of reporting organizations conducted activities in the area of learning and communication such as the following:
- Training as a prerequisite to staffing sub-delegation;
- Training in support of informal conflict management systems and alternative dispute resolution methods; and
- Communication to promote awareness of new approaches to staffing, conflict management systems, etc.
Results from the document review and case studies found a number of examples of learning events on various PSMA-related topics that took place between 2005 and 2008, such as refresher training on various HR-related activities, policy training for HR professionals, and sub-delegation training for managers. Actual participation rates for all departments and agencies were not available.
- People preparation: Public service managers, HR professionals and staff have the necessary skills and knowledge to implement the PSMA.
Considering that all departments and agencies directly or indirectly used SIF funds and that almost all were engaged in PSMA activities, including learning and communicating, it can be inferred that public service managers, HR professionals and staff have acquired to some degree the skills and knowledge to implement the PSMA. However, the evaluation could not assess the extent to which this is the case.
Findings from interviews and case studies indicate that almost all key informants agreed that public service managers have thenecessary skills and knowledge to implement the PSMA.
- New functions: New functions are in place, operational, and ready to apply for and/or have secured A-base[25] funding.
Evidence gathered from the document review and case study analysis indicates that several departments and agencies established or adopted new functions. However, at the time of data collection there was no clear indication that these systems became operational or had secured A-base funding. The document review shows that in several departments and agencies, HR planning was integrated with business planning, and new integrated HR planning tools were developed and are in use (at Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, Environment Canada, Industry Canada, Aboriginal Affairs and Northern Development Canada, and Public Works and Government Services Canada).[26] Examples from the case studies show that human resources planning at Agriculture and Agri-Food Canada was aligned with the elements of the "people" component of the MAF, whereas at National Defence human resources planning was aligned with the National Defence's Appointment Delegation and Accountability Instrument. Other new functions included the development of instruments for guidance on human resources topics, such as the "Staffing Desk Reference for Managers" at National Defence.
- Project management: Horizontal PSMA implementation, management and governance mechanisms are in place, supported and effective.
Horizontality was one of the SIF's funding principles and, based on the document review, this was achieved in terms of the role of the Project Office and governance structures. Most key informants believed that the project management was somewhat effective in achieving horizontal PSMA implementation.
Medium-Term Outcomes
The outcomes that were expected from SIF funding in the medium-term included the following:
- Institutions that have a new mandate are better able to support public service HR needs;
- Improved ability to support and track new and evolving HR requirements;
- Resourcing processes are better supported;
- Public service managers, HR professionals and staff have the understanding and cultural readiness to manage and behave in the spirit of the PSMA;
- HR policies, accountabilities and processes are developed that are consistent with PSMA principles and objectives;
- Effective stewardship of PSMA implementation reserve fund; and
- Coordinated and leveraged efforts with related modernization activities and stakeholders.
Although these outcomes may have been achieved, the limitations relating to reporting and stakeholder access for interviews resulted in the evaluation being unable to conclude on them.
Long-Term Outcomes
The outcomes that were expected to be achieved in the long-term as a result of SIF funding were as follows:
- The right people are hired where and when needed;
- Labour–management relations that are more collaborative;
- Increased focus on learning and training for employees at all levels; and
- Improved clarity regarding roles and accountability.
As with the intermediate outcomes, the long-term outcomes may have been achieved; however, the evaluation could not conclude that this was the case.