ARCHIVED - Evaluation of the Research and Policy Initiatives Assistance Program - Treasury Board of Canada Secretariat

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

3.0 Evaluation Findings

This section of the report presents a summary of the evaluation findings, which are organized by evaluation question.

3.1 Program relevance

Conclusion: RPIA Program projects are aligned with program and departmental objectives. The program is meeting the needs of sectors by providing funding for projects that otherwise may not have been possible. Some sectors would not have had access to the project benefits if the funding had not been provided.

Findings: RPIA Program projects are aligned with program and departmental objectives. According to the RPIA Program's terms and conditions, program objectives include the following:

• To improve, advance, and promote public service management practices by supporting research initiatives that contribute to the advancement of public policy research and public service management; and
• To enhance the effectiveness of public service management practices, reporting, and service delivery to the public. [2]
• The Secretariat's raison d'être, as articulated in its 2008–09 Report on Plans and Priorities (RPP), is to ensure that government is well-managed and accountable and that resources are allocated to achieve results.[3]

RPIA Program projects that were reviewed were related to conducting research and holding events on topics that were relevant to the objectives of Secretariat and the program. The projects reviewed, as part of the file review, consisted of projects producing research reports, symposia, educational events, workshops, seminars, conferences, lectures, and other studies. Often, the projects reviewed brought together a leading group of commentators with relevant experience and expertise on the topic of interest. Topics examined through RPIA Program projects reviewed included the following:

• Public service governance;
• Inter-jurisdictional service delivery;
• The design of public policy, such as the design and administration of programs of grants, contributions, and other forms of financial support; and
• The promotion of approaches to technology, information, and service (a commitment of the Secretariat's RPP and a priority of the Chief Information Officer for the Government of Canada).

As well, RPIA Program project documentation noted that the program supports the introduction of new ideas into the mainstream discourse on public sector management, reporting, and service delivery.

Interviewees also believe that RPIA Program projects are contributing to program objectives. For example, one interviewee noted that the RPIA Program was a catalyst toward government-wide reform and a move toward responsible management. Another interviewee noted the link between the RPIA Program and the Secretariat's strategic objective that the government be well-managed, noting that lectures supported through the program generate ideas on good management and that those ideas are shared between agencies.

The RPIA Program is meeting the needs of sectors by providing funding for projects that otherwise may not have been possible. Some sectors would not have had access to the project benefits if the funding had not been provided.

Interviewees believe that the RPIA Program met the needs of sectors, primarily because it allocated funds to undertake work that would not otherwise have been possible. Interviewees noted that without the RPIA Program, there would not have been funds available to undertake the projects. As noted by one interviewee, there is little discussion forums on public sector management, and it is difficult to generate dialogue. RPIA Program–funded projects have helped to fund activities to generate dialogue on this topic. 

Interviewees said that recipient organizations would not have carried out the activities without RPIA Program funding. In addition, according to interviewees, the department would not have had the same benefits had it not provided the funding. Two of the three interviewees for this evaluation specifically noted the need for an independent, neutral, third-party organization to achieve the objectives of the funded project. These interviewees noted that the Secretariat could not have done the projects themselves, as they would have been viewed as biased. As well, the projects funded under the RPIA Program brought together multiple organizations to work together. Without the RPIA Program, stakeholders would have worked more in isolation.

3.2 Program success

Conclusion: There is a lack of evidence on file regarding the benefits of the projects; however, interviewees believe that the RPIA Program has enhanced policies or programs and helped to increase knowledge and build capacity.  

Findings: Information from the file review shows that there is a lack of evidence on file regarding the benefits of the projects. For the most part, the only information available from the file review, with regard to benefits of the projects, was information about anticipated benefits as opposed to actual benefits realized. These anticipated benefits came from the submission and recommendation form on the file. They include the following:

• To extend and deepen the dialogue across levels of government;
• To advance public policy development around the design and use of the grants and contributions tool;
• To provide an opportunity for exchange on contemporary public service issues and provide an opportunity for exchange on the advancement of public service management; and
• To improve client service delivery.

The RPIA Program does not have a requirement to provide any kind of post-project summary for the project file. While some project reports were gathered during the course of the evaluation, they often did not provide information about the benefits of the project to the Secretariat. Thus, actual benefits of these projects are not known to the RPIA Program staff.

Recommendation No. 1: The program area should implement a mechanism to ensure that the impact of the program can be measured with respect to actual benefits rather than anticipated benefits. For example, a post-project report could be a requirement for all RPIA Program projects. This could be as simple as a one-page template that is filled out for each project.

Interviewees believe that the RPIA Program has contributed to its outcomes, i.e., enhancing policies or programs, increasing knowledge of emerging policy issues, and building capacity for analysis. The interviewee for one project noted that the project was impacting current policy development and the subsequent development of new standards, and that the project was essential to ensure that current policy development efforts align with other jurisdictions. A second interviewee noted that recommendations coming out of the RPIA Program led to a government action plan that was acceptable to stakeholders.  

Interviewees also agreed that the RPIA Program has helped to increase knowledge of emerging policy issues, e.g., public service retention. Interviewees noted that their RPIA Program projects had brought together people from various groups, such as public policy practitioners, academics, voluntary-sector representatives, public service employees, and representatives from organizations that deal with public policy. Bringing together people from various jurisdictions who are involved in discussions and share information increases the knowledge base.

Interviewees also agreed that the RPIA Program has helped to build capacity for analysis. For example, interviewees remarked that, as a result of the RPIA Program projects, sectors have a much better understanding of the needs of other jurisdictions and what they can contribute in federal policy development. One interviewee noted that the sector is in a better position to make analyses and recommendations to Treasury Board ministers as a result of an increased information base and analytical capacity as a result of RPIA Program projects. Another interviewee noted that getting different people together leads to establishing priorities and identifying key items that have the most impact. 

One interviewee found it hard to comment on project success, indicating that the benefits of the project were directed more to other departments rather than to the Secretariat.

3.3 Program cost-effectiveness and alternatives

Conclusion: Interviewees believe that the program is cost-effective because of the benefits of partnering with other organizations, i.e., leveraging, and because few alternatives are available. In most cases, the Secretariat provided a portion of the funding to carry out projects. The program is not seen as being sufficiently visible within the department, and records management could be improved.

Findings: Interviewees believe that the RPIA Program is cost-effective. While interviewees were not able to establish a numeric return on investment, all indicated that the program is cost‑effective. 

There appear to be few options for alternatives to the program. When asked if there are alternative means to achieve the objectives of the RPIA Program, interviewees suggested that work could be done in-house; however, there are insufficient resources (i.e., human and financial) to do so. As mentioned previously, in some cases an independent third party is needed to successfully achieve the objectives of the project. Contracting was also identified as an alternative, but interviewees were not certain that this could be done, given the requirements of the Financial Administration Act. According to interviewees, difficulties would be encountered due to the fact that there are often multiple partners for RPIA Program projects and shared benefits. As well, RPIA Program projects do not necessarily have a clear output or deliverable.

Cost-effectiveness was enhanced by the fact that the Secretariat typically provided only a portion of the funding to carry out projects. The department partnered with other organizations to undertake the projects, thus leveraging other resources. For the files reviewed, the Secretariat contributed anywhere from 8 to 44 per cent of the total cost of the project.

Because of the nature of the program and its relatively small size, program efficiency was not seen as a relevant issue to focus on in this evaluation. However, some efficiency-related information was gathered during interviews. 

Interviewees generally had positive views on program administration. Most interviewees noted that the application process was straightforward and easy and that approvals were timely. However, interviewees identified one improvement that should be made to the program. They believe that the program needs to be marketed better within the department. They reported that it was difficult to find out about the program. As well, one interviewee commented that there did not seem to be any one person who was able to guide him through the process and who could answer all his questions. Another interviewee noted that she had to find out about the program and do the administrative work herself. Interviewees agreed that the program needs to be more visible within the department.

Recommendation No. 2: The Secretariat should increase the visibility of the RPIA Program within the department and ensure that employees are aware of the program and know whom to contact for more information.

In addition, the file review undertaken for this evaluation highlighted a few issues relating to project documentation. First, not all project-related documents, i.e., the submission and recommendation forms, were available through the RPIA Program manager. Second, post‑project reports are not required on file, and the actual benefits of projects are not known to RPIA Program staff.

Recommendation No. 3: Project documentation should be collected and retained more systematically and stored in a central location.

• Previous Page
• Table of Contents
• Next Page