Common menu bar links
Breadcrumb Trail
ARCHIVED - Evaluation of the Research and Policy Initiatives Assistance Program - Treasury Board of Canada Secretariat
This page has been archived.
Archived Content
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
Executive Summary
This report presents the findings and conclusions of the evaluation of the Research and Policy Initiatives Assistance Program (RPIA Program) for the Treasury Board of Canada Secretariat (the Secretariat). The evaluation was undertaken to respond to the requirements for an evaluation of the program for its renewal, in accordance with the Policy on Transfer Payments and Treasury Board requirements. The Secretariat's Internal Audit and Evaluation Bureau undertook a stewardship function over the course of the evaluation. The evaluation findings will also serve program managers by providing information to improve program delivery and performance. The evaluation focussed on three main areas: relevance, success, and cost-effectiveness.
Description of the RPIA Program
The RPIA Program at the Secretariat is a class-contribution program used for small, ad hoc initiatives that contribute to Secretariat objectives but do not deliver a clear good or service. The program can provide up to 50 per cent of eligible costs of individual approved projects, up to a total of $500,000 paid out over the life of the initiative. There is no separate funding base for the program. The RPIA Program functions less as a program in and of itself and more as a mechanism for existing programs to deliver their objectives. As such, funding is provided from existing budgets. Use of this authority requires the recommendation of a branch head, the sign-off of the department's senior full-time financial officer, and the approval of the Secretary of the Treasury Board.
Over the five-year period of the evaluation (2003–04 to 2007–08), a total of $928,674 was approved through the RPIA Program.
Evaluation methodology and limitations
The evaluation methodology was designed to ensure that the cost of the evaluation was commensurate with the relatively small amount of funding provided through this program. As a result, the evaluation integrated the use of only two lines of evidence: file review and key stakeholder interviews.
As with any evaluation, there are limitations to the evaluation methodologies. First, the interviews and file review did not include all projects that received funding through the RPIA Program. However, efforts were made to select those projects that accounted for the largest percentage of program spending over the evaluation period. Government Consulting Services (GCS) considers the files selected to be a good sample and believe they represent Secretariat projects under the RPIA Program.
A second limitation of the evaluation relates to a lack of quantitative information. The interviews and file review conducted for the evaluation are qualitative sources of information. Little quantitative information was available about the RPIA Program, particularly with regard to benefits or outcomes.
Program relevance
RPIA Program projects are aligned with program and departmental objectives. Projects that were reviewed related to conducting research and holding events on topics relevant to the objectives of the Secretariat and the program. Topics examined through RPIA Program projects that were reviewed included public service governance; inter-jurisdictional service delivery; design of public policy, such as the design and administration of programs of grants, contributions, and other forms of financial support; and promotion of approaches to technology, information, and service.
The RPIA Program is meeting needs of sectors by providing funding for projects that otherwise may not have been possible. Interviewees noted that recipient organizations would not have carried out the funded activities in the absence of the RPIA Program; as well, the RPIA Program's funding has generated discussion on areas of public sector management, where traditionally there has been little dialogue. According to interviewees, the Secretariat would not have had access to the benefits of the projects if funding had not been provided. Finally, interviewees noted, that to achieve the objectives of some projects, an independent, neutral, third-party organization was required. These interviewees noted that the Secretariat could not have undertaken the projects themselves, as they would have been viewed as biased. As well, the projects funded under the RPIA Program typically brought multiple organizations to work together. Without the RPIA Program project, stakeholders would have worked more in isolation.
Program success
The file review conducted for this evaluation highlighted a lack of evidence in Secretariat files regarding the benefits of the projects. For the most part, the only information available from the file review, with regard to benefits of the projects, was information about anticipated benefits rather than actual benefits realized. The RPIA Program does not have a requirement to provide any kind of post-project summary for the project file. Thus, actual benefits to the Secretariat as a result of these projects are not routinely documented.
Recommendation No. 1: The program area should implement a mechanism to ensure that the impact of the program can be measured with respect to actual benefits rather than anticipated benefits. For example, a post-project report could be a requirement for all RPIA Program projects. This could be as simple as a one-page template that is filled out for each project.
Interviewees believe that the RPIA Program has contributed to enhancing policies or programs, increasing knowledge of emerging policy issues and building capacity for analysis.
According to interviewees, the RPIA Program has helped to enhance policies or programs for which Secretariat sectors are responsible. Interviewees also agreed that the RPIA Program has helped to increase knowledge of emerging policy issues, e.g., public service retention. They noted that RPIA Program projects brought together people from various groups, and these people were involved in discussions and shared information, which increased the knowledge base. Finally, interviewees agreed that the RPIA Program has helped to build capacity for analysis by increasing the information base of sectors and increasing analytical capacity as a result of RPIA Program projects.
One interviewee found it hard to comment on project success, indicating that the benefits of the project were more directed to other departments rather than to the Secretariat.
Program cost-effectiveness and alternatives
There appear to be few options for alternatives to the RPIA Program. Interviewees pointed to insufficient resources (i.e., human and financial) for the Secretariat to be able to conduct the work in-house. As well, in some cases an independent third party is needed to successfully achieve the objectives of the project. Issues were identified with regard to the only other potential alternative suggested (i.e., contracting the work), mainly due to the fact that there are often multiple partners for RPIA Program projects with shared benefits, and such projects do not necessarily have a clear output or deliverable.
Interviewees believe the program is cost-effective because of the benefits of partnering with other organizations, i.e., leveraging. In most cases, the Secretariat provided only a portion of the funding to carry out projects. For the files reviewed, the Secretariat contributed anywhere from 8 to 44 per cent of the total cost of the projects.
Because of the nature of the program and its relatively small size, program efficiency was not seen as a relevant issue to focus on in this evaluation. However, some information was gathered during interviews that related to efficiency. Interviewees were generally positive in their views on program administration. Most interviewees noted that the application process was straightforward and easy and that approvals were timely.
A few suggestions were identified to improve the program. First, interviewees believe that the program needs to be marketed better within the department.
Recommendation No. 2: The Secretariat should increase the visibility of the RPIA Program within the department and ensure that employees are aware of the program and know whom to contact for more information.
In addition, the file review undertaken for this evaluation highlighted the fact that not all project-related documents, i.e., the submission and recommendation forms, were available through the RPIA Program manager, and, since post-project reports are not required to be on file, the actual benefits of projects are not routinely documented.
Recommendation No. 3: Project documentation should be collected and retained more systematically and stored in a central location.