ARCHIVED - Evaluation of the Public Service Modernization Act Strategic Investment Framework

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

1.0 Introduction

This report presents the findings of an evaluation of the Strategic Investment Framework (SIF), which was a fund established to guide investments in implementing the Public Service Modernization Act (PSMA). The evaluation was conducted between December 2009 and March 2010 by Goss Gilroy Inc., and the report was finalized by the Internal Audit and Evaluation Bureau of the Treasury Board of Canada Secretariat.

The legislative review of the PSMA was undertaken after the evaluation of the SIF, during which the environment continued to change. As a result, there may be additional insights or conclusions brought to light with regard to the PSMA itself.

1.1 Context

In 2003, when the PSMA Reserve Fund was established, the elements (questions) to be evaluated were identified that formed the basis of this evaluation. These questions focused on implementation rather than results and influenced the collection of data during implementation. Because this evaluation was conducted between 2009 and 2010, it also included the assessment of relevance in order to meet the evaluation standards of the Policy on Evaluation, published in 2009.

The evaluation could not conclude on the achievement of the PSMA-SIF medium- and long-term outcomes mainly because of the limited information on outcomes, difficulty in attributing SIF outcomes to the PSMA, and limitations on the availability of knowledgeable key informants. This does not suggest that these outcomes were not achieved, but rather that the evaluator was unable to make a judgment on this.

During the PSMA's implementation, the departments in which the SIF coordinating office resided were reorganized three times. The impact of reorganizing three different lead organizations during the implementation of a single government-wide initiative was significant. This also affected the evaluation. The reader should be mindful of the constraints on the evaluation imposed by these circumstances and the resulting data limitations. Because this evaluation took place after the close of the program, lessons learned rather than recommendations are presented.

1.2 Background

The PSMA received royal assent on November 7, 2003, with the overall goal of modernizing employment and labour relations in the public service. The $238‑million PSMA Reserve Fund was establishedwithin the fiscal framework in order to implement the Act; initially, $38 million was used in support of early preparatory work and project management functions related to the PSMA's start-up.

In 2004, the remaining $200 million of the PSMA Reserve Fund was allocated to the SIF, which was established to guide investments in implementing the PSMA. The SIF was to be implemented in a way that:

• Ensured sound and defensible investment decisions;
• Enabled transparency of decision making;
• Strengthened accountability for results in implementing the PSMA; and
• Ensured targeted use of resources.

A transitional government-wide body, the Public Service Modernization Act Secretariat (PSMA Secretariat), formerly the Human Resources Modernization Implementation Secretariat of the Treasury Board of Canada Secretariat, was created to orchestrate the changes brought about by the PSMA, coordinate efforts across departments and agencies, and manage the allocation of, and applications for, funds available for implementing the PSMA. The PSMA Secretariat, as part of the newly created Public Service Human Resources Management Agency of Canada (PSHRMAC), acted as the coordinating body or Project Office, with responsibility for the management of implementation funds, communications, and project oversight in implementing the PSMA.

In addition to PSHRMAC, the Public Service Commission (PSC), TBS, the Canada School of the Public Service (CSPS) and Justice Canada were directly involved in the implementation of the PSMA as Corporate Delivery Partners and had specific responsibilities for results.

1.3 Evaluation Objectives and Scope

The objective of this evaluation was to examine the implementation, relevance and performance (efficiency, effectiveness and economy) of the investments made under the SIF, the extent to which they facilitated achieving outcomes of the PSMA, and the lessons learned from this initiative. The scope of the evaluation focused on the departments that accessed SIF funds, which included central agencies. This evaluation did not assess the effectiveness of the tools or instruments implemented by these departments. Although the SIF was central to the PSMA's implementation, this was not an evaluation of the PSMA but of the use of SIF funds.^[4]

1.4 Evaluation Issues and Methodology

1.4.1 Strategic Investment Framework Logic Model

During the planning phase of the project, the logic model was updated to ensure it reflected the full range of impacts of the SIF. The alignment of SIF activities and outputs with the approved PSMA outcomes was considered essential for developing appropriate indicators for the SIF that were consistent with the outcomes approved and monitored for the PSMA overall (see Figure 1.1). The short-term outcomes reflect the five notional SIF-funding envelopes, each with its own set of medium-term outcomes that link to the long-term outcomes for the SIF and the PSMA.

[Text version]

[See full size image]

1.4.2 Evaluation Matrix

In accordance with the Treasury Board of Canada Secretariat's Policy on Evaluation, the evaluation used multiple lines of evidence and qualitative and quantitative data. The list below identifies the key evaluation issues and questions. These are addressed using four main lines of evidence: key informant interviews, document review, administrative data review, and case studies.

As noted above in 2003, when the PSMA Reserve Fund was established, the elements to be evaluated were identified. These questions focused on implementation rather than results. Because the evaluation was conducted between 2009 and 2010, the evaluation included an assessment of relevance in order to meet the requirements of the Policy on Evaluation and its related standards.

Evaluation Issues and Questions

Issue: Relevance

• Is there a continuing need to support departments in developing HRM capacity?*
• Did the objectives and related activities of the SIF align with the objectives and spirit of the PSMA generally?

Issue: Implementation

Efficiency

• How effective was the project plan and its implementation?
• How effective was the implementation project plan in guiding the horizontal work across departments and agencies?
• How effective was the Project Office's role in the context of the functionality of the governance model (i.e., as a secretariat)?
• How effective was the application of the investment strategy in achieving the desired results and outcomes of the PSMA?
• How effective was the capability and functionality of IT that was specific to this project?
• Were there appropriate linkages and leveraging with non-PSMA modernization activities that contributed to the successful implementation of the PSMA?
• To what degree were PSMA legislative requirements met?

Economy

• How affordable was the actual investment? Was there value for money of the actual investment?
• What was the nature and level of uptake of institutions, employees or target groups?

Effectiveness

• How effective was the application of the investment strategy in achieving the desired results and outcomes of the PSMA?
• What results were achieved (including both outputs and outcomes)?

* This refers to HRM broadly.

1.4.3 Key Informant Interviews

A total of 23 key informant interviews were conducted either in-person or by telephone, with representatives working in the area of human resources management in federal departments and agencies. The original methodology had planned for 50 interviews; however, difficulty in locating potential participants limited the number possible.

1.4.4 Document Review

The document review was intended to address all evaluation questions. Three main types of documents were reviewed: background documents (e.g., the 2004 SIF Treasury Board Submission, the Master Plan for the PSMA Implementation Project); operational documents (e.g., business case guidelines for the SIF, business case evaluation templates, Human Resources Management Advisory Committee [HRMAC] minutes); and reporting and evaluation documents (e.g., progress reports to Treasury Board ministers, PSMA monitoring and reporting summary reports, and relevant program reviews).

1.4.5 Administrative Data Review

The administrative data review built upon a number of separate spreadsheets provided by the project authority. Goss Gilroy Inc. amalgamated a spreadsheet (which included the department, name, number of employees, whether they were subject to the PSMA, whether they received PSMA funding, and whether they received additional support) and merged it with a large data file that contained a separate worksheet for each department. This worksheet included the information on the total approved expenditures under the PSMA Reserve Fund, actual PSMA Reserve Fund expenditures, and project costs funded from departmental reallocations. These items were recorded for each fiscal year for each of the notional funding envelopes of the SIF (as outlined in the preceding).

1.4.6 Case Studies

Five case studies were conducted to provide a more in-depth look at how SIF funds were used within departments and agencies. For each case study, the evaluator conducted at least one interview and reviewed the administrative data and documentary evidence for that department. The evaluation, however, did not evaluate the effectiveness of the tools, instruments or policies adopted or adapted by these departments for the SIF's implementation.

Case study summaries were appended to the Key Informant Interviews Technical Report.^[5] Case studies were conducted with the following:

• National Defence;
• Agriculture and Agri-Food Canada;
• The Public Service Commission of Canada;
• The Public Service Labour Relations Board;
• The SATST^[6]; and
• Four small agencies (interviews).

1.5 Limitations

There were a number of limitations to the evaluation, which are described below. These limitations led to the evaluator being unable to assess the extent to which some intermediate outcomes and the ultimate outcome were achieved, which weakened conclusions on the extent of the achievement of immediate outcomes.

Availability of knowledgeable key informants: One of the main limitations of this evaluation was the low availability of key informants who were knowledgeable about SIF implementation in their departments. Only 23 key informants, or 46 per cent of the intended sample of 50 provided information, some of which was incomplete. Reasons for this included the following:

• As already indicated, the SIF Project Office was subject to three reorganizations, which resulted in a great deal of turnover over the years and made it difficult to locate potential interviewees.
• The evaluation took place after the SIF was "sunsetted." Many of the temporary organizations that had been established to support its implementation had been disbanded (e.g., the SIF Project Office and the SATST).
• Because the support structures for the SIF had been disbanded, it was difficult to locate personnel from central agencies who could answer the various questions related to the Project Office and the SIF's implementation.

Limited information on outcomes: Departmental reporting was largely at the output level. Also, although information such as Management Accountability Framework (MAF) ratings was available for each department, the measures were at such a high level of aggregation that they could not be directly related to any one activity, let alone a specific SIF-funded project.

Departmental versus project impacts: One of the limitations of a horizontal evaluation is that departmental impacts cannot be determined with great accuracy. Although case studies were used as a line of evidence, this was not a departmental-level evaluation.

Attribution: The projects funded under SIF were implemented over several fiscal years, with different types of projects, alone or in combination with other sources of financing. The profiles of SIF projects, in terms of their nature, size and timing, were different for every department. In addition, the close relationship between the outcomes of the SIF and those of the PSMA added another level of complexity to attribution issues.^[7]

The alignment of SIF outputs and activities with PSMA outcomes determined that any attribution of SIF funding to outcomes largely had to rely on case studies that were used to explore linkages between the SIF and outcomes at the departmental level.

Although there are limitations on the availability of knowledgeable key informants and limited information on outcomes, other lines of evidence were used to draw conclusions on the implementation of the SIF, strengthening the reliability of the validity of the results. Despite the limitations, the methodology meets the requirements of the Treasury Board's Policy on Evaluation and associated standards.

• Previous Page
• Table of Contents
• Next Page