Evaluation of the Research and Policy Initiatives Assistance Program

Addressing a Continuing Need

The document review confirmed a need for government to work toward more effective, streamlined operations. Specifically, Blueprint 2020 encourages the federal government to reflect on how government does business, in order to pursue continual improvement. Economic Action Plan 2013 asks departments to continue examining ways of streamlining government operations. Given that the ultimate outcome of the RPIA Program is to advance and make public service management more effective, the RPIA Program supported these broad themes.

The review of program documentation found that the RPIA Program addressed a continuing need of government to develop and access ways of streamlining operations and enhancing the effectiveness of public service management practices. The RPIA Program addressed this need by providing a mechanism to support ad hoc research initiatives that aim to improve public policy, service delivery to the public, and public service management practices.

The ongoing need for the RPIA Program was expressed by the majority of interviewees, who agreed that the program allowed sectors to establish ad hoc initiatives with external organizations, to create networks, and to enable critical thinking on policy research that benefits Canadians. Only two sectors used the RPIA Program during the review period; however, this limited use could be attributed to low awareness of the program; in the questionnaire administered to directors, five of nine reported being unaware of its existence.^{Footnote 8}

All beneficiary organizations responding to the questionnaire stated that the program assisted them in meeting their information and research needs to a great extent or to a very great extent.

Responsiveness to Departmental and Beneficiary Needs

RPIA initiatives addressed and responded to beneficiaries’ needs for information on government management practices, regulatory reform, and improved service delivery to the public. The majority of beneficiaries agreed that the RPIA Program was useful in the work they do to shape public policy and that they would continue participating in the program if it were extended.

Evidence from the document review shows that the RPIA Program supported research and policy initiatives aimed at improving public service management practices.

There is evidence that opportunities exist for greater use of the program. A few respondents to the Director’ Coordinating Committee questionnaire stated that they would use the RPIA Program if it were extended.

Conclusion: Relevance

The RPIA Program is relevant. Through third party organizations, the program responds to the government’s continuing need to develop and access ways of streamlining government operations and enhancing the effectiveness of public service management practices. The program addresses this need by supporting ad hoc initiatives to improve public policy research and enhance public service management practices.

Program Design, Eligibility Criteria, and Roles and Responsibilities

Interview groups had mixed views of the appropriateness of the RPIA Program’s design. Three of seven groups favoured the program design for its lack of centralized, formal oversight, believing it provided for a streamlined, flexible process. The other groups were critical of the design, citing heavy administrative burden and steep learning curves. The program’s design requires interaction with select individuals and may involve iterations to ensure that proposals are well drafted. Two groups of sponsors stated that the program operated in a complex fashion and that it was labour intensive to establish an agreement. These groups reported that the program’s approval process was overwhelming for third party organizations and in one case had resulted in a withdrawal of interest.

Similarly, interview groups expressed mixed views on the clarity of the RPIA eligibility criteria and terms and conditions. Both groups of managers stated that the eligibility criteria were clear and readily available. Eligibility criteria were reported to be fair by managers in all interview groups, and these criteria were seen to be broader than the criteria of comparable programs in other departments. A majority of managers said that the broadness of the criteria gave sectors the flexibility to advance their research projects. This point of view was not shared by sponsors, who reported seeking help from other sponsors before approaching program managers for information. One group of sponsors said that the eligibility criteria were neither restrictive nor unreasonable. This group stated that delays occurred because the nature of the initiative involved working with external organizations that had unique processes, approvals, and authorities. Another group stated the eligibility criteria were challenging and unreasonable when dealing with multi-year projects and international organizations.^{Footnote 9}

Interview groups were also divided on the appropriateness and clarity of the roles and responsibilities of financial advisors and sponsors of potential initiatives. Two groups said that the roles and responsibilities were clear; a third group suggested that roles and responsibilities of sponsors and administrators needed to be clarified; a fourth group said that the program was delivered as designed, which was appropriate; and a fifth group stated that the approval process needed to be simplified by removing the requirement for approval by the Deputy Head and the Chief Financial Officer.

When compared with similar programs,^{Footnote 10} the RPIA Program’s design was found to be more flexible and convenient for sponsors to use for small-scale, policy-related projects. Other programs had longer application forms, more detailed project selection procedures, and longer delays in receiving funds.

Communications Strategy

A majority of respondents to the Directors’ Coordinating Committee questionnaire stated that they were unaware of the existence of the RPIA Program. However, they indicated that the RPIA Program appeared to be a viable option for their research initiatives and that they were interested in learning more about it.

The jurisdictional review found that the RPIA Program was fully comparable to programs operating in the Department of Finance Canada, the Atlantic Canada Opportunities Agency, and Industry Canada, as it relies less on promotion and more on sponsors’ awareness of the program. This finding was corroborated by the review of program files, which demonstrated that the majority of initiatives were sponsored by one sector and by the same sponsors.

The document review found that the communications strategy for the program was based on a 2013 presentation by Corporate Services to all financial analysts. This presentation was also posted on the Secretariat’s InfoSite, to ensure that all employees were able to obtain more information about the program. Although managers stated that the program was well communicated, additional probing in the interviews revealed that the presentation had been delivered only once and had not been updated since. One group of managers added that it would be a good practice to have an updated presentation given to financial analysts every two years. Neither the evaluators nor one group of experts could find the RPIA Program’s templates (terms and conditions, contribution agreement template, submission/recommendation form, and project evaluation grid) on the Secretariat’s external website.

Two of the three groups of sponsors indicated that the application forms were not clear and that written instructions would be useful. Evaluators noted that even though information on the RPIA Program was posted on the Secretariat’s InfoSite, no guidelines, samples, or written instructions on how to fill in forms were provided.

There were mixed views among interview groups on the adequacy of the financial management guidelines for RPIA Program initiatives. Although both groups of program managers emphasized that they advise sectors, track resources, and administer funding, most of the sponsor groups identified a need for more guidance and admitted that they were using their own knowledge and guidance materials to create new initiatives. In their view, financial coordination was ad hoc.

A review of the program’s management response to the recommendations from the previous evaluation found no evidence that the Corporate Services Sector had increased the program’s visibility either by posting program materials on the Secretariat’s external website or by preparing a communications strategy that would inform sectors of the program’s existence.

Performance Measurement

Performance measurement is a mandatory component of all programs and is to be implemented at the time of a program’s design. Under the Policy on Transfer Payments, monitoring of performance must take into account the program’s materiality and must reflect risks specific to the program, the value of funding in relation to administrative costs, and the risk profile of the recipient.^{Footnote 11} In their review of RPIA Program documents, evaluators found no evidence that performance measures were in place and minimal evidence of performance measurement data. Although the document review shows that the responsibility for collecting performance measurement rests with sponsors, interviewed sponsors were not aware of this role.

Sponsors stated that performance measurement would create a burden for beneficiaries if it required asking beneficiaries to demonstrate outcomes. One group of sponsors cautioned that if a performance measurement system were implemented, it should not require input from beneficiaries because they were already overwhelmed by the administrative burden the program imposed. Managers also questioned whether formal performance measurement would be needed if sponsors were encouraged to hold back 10 per cent of funds until the end of the planned activity.

When evaluators asked managers to describe the type of information needed to manage the program, one group mentioned the project evaluation grid, which was recommended, but not required, to be administered at the close of the initiative. This grid requires sponsors to provide a description of the initiative and to identify the actual outcomes versus the planned outcomes in the proposal.

In the review of program files, evaluators found only one case in which the project evaluation grid was completed. One group of sponsors reported that they had never been asked to complete the grid and were unaware of it. They stated that they would have filled it in for beneficiary organizations had they been asked.

Alignment with the Policy on Transfer Payments and the Directive on Transfer Payments

The evaluators analyzed the 2000 and the 2008 versions of the Policy on Transfer Payments and associated directive as part of a comparative review of the program’s terms and conditions. The evaluators found that the program’s terms and conditions had not been updated at the time of the program’s renewal in 2009 and, as a result, did not conform to the 2008 policy and directive.

Section 6.5.2 of the Policy on Transfer Payments refers to establishing, at the time of the program design, a performance measurement strategy and maintaining and updating it throughout its life cycle, to effectively support an evaluation or review of relevance and effectiveness of the transfer payment program.

Sections 6.5.7 to 6.5.9 of the Policy on Transfer Payments refer to risks levels, engagement of applicants and recipients, and the implementation of departmental service standards.

Section 6.1.4 of the Directive on Transfer Payments refers to the availability of program information to potential recipients via the Secretariat’s website.

Section 6.5.1 of the Directive on Transfer Payments refers to the implementation of a risk-based assessment regarding the monitoring and reporting of recipients.

Appendix D, item 10 of the Directive on Transfer Payments refers to the description of official languages requirements in order to respect the obligations of the Government of Canada as set out in Part VII of the Official Languages Act and states that services and benefits are made available in both official languages.

One group of managers supported having a formal policy review to assure alignment, now that a new policy has come into force.

Expanding the Secretariat’s Knowledge

A review of project documents such as funded research papers, final reports, contribution agreements, and the evaluation report grid found that the RPIA Program had provided access to some extent to an expanded knowledge base on subjects of interest to the Secretariat. The RPIA Program had funded lectures on adapting public policy in a changing world and had addressed issues such as public service reform, leadership, and innovation. Moreover, RPIA funds had provided the Secretariat with access to research on fields of interest to the government, such as communicating and measuring progress in regulatory reform and exploring good practices for managing emerging risk and corresponding management actions in large, complex public and private sector organizations. The RPIA Program had also helped a sponsoring sector promote dialogue on innovation, fiscal constraint, horizontal governance, and long-term challenges among senior government officials from OECD countries, identifying lessons learned and best practices.

Beneficiaries responding to the questionnaire stated that their projects had contributed to the achievement of the RPIA Program’s immediate outcomes to a great extent or to a very great extent. Similarly, through their development and implementation of initiatives, sponsors reported an increase in their sectors’ capacity to do public policy research. This group stated that without this program some activities would not have taken place.

Interview groups held mixed views on whether this immediate outcome was achieved solely as a result of the RPIA Program. The majority of sponsor groups indicated that lectures and workshops gave the Secretariat access to renowned specialists and subject-matter experts to some extent and strengthened the Secretariat’s partnerships with organizations of stature. One group of managers suggested that the same information and partnerships might be realized for international projects through professional service contracts and memberships in professional organizations.^{Footnote 12}

A Better-Informed Policy Context and an Expanded Community of Interest

The majority of respondents stated that the RPIA Program contributed to the outcomes of a better-informed policy context and an expanded community of interest through non‑governmental partnerships and communication of policy contexts nationally and internationally.

From the file review of contribution agreements and the document review of research papers and final reports, the evaluators concluded that the RPIA Program supported regulatory management and oversight and management accountability policy initiatives, as the funded initiatives fell under these themes. The evaluators found documented evidence that one initiative “improved, advanced, and promoted public service management practices by supporting a dialogue among high-level officials on how government could effectively respond to policy and governance challenges in the context of fiscal constraint.”

Evidence from the file review demonstrated that the RPIA Program expanded the “community of interest” in regulatory management, the government’s role, risk management, policy, and innovation. In one case, the program funded a research initiative that explored the theme of good practices for managing emerging risks, drawing on the expertise of academics, expert practitioners, and representatives from other governments. In another case, the program funded a research initiative that brought together regulatory experts from international governments and academia to promote debate on regulatory policy evaluation, resulting in a practical framework that could be applied in OECD countries.

The majority of beneficiaries confirmed in their questionnaires that the program contributed to their organizational objectives by:

• Furthering their research, expertise, and understanding;
• Being instrumental in the collection of data for analysis across several countries; or
• Allowing them to draw on expert knowledge, lessons learned from other countries, and the expertise of leading academics, to further their public policy mandates.

Increased Leverage of External Partners’ Resources and Other Capacities Toward Achieving Departmental Objectives

All sponsors affirmed that the Secretariat leveraged beneficiaries’ resources to a great extent. These resources were typically financial, expertise, and access to partners’ networks. One beneficiary was able to attract additional financial resources from other countries, as well as to expand its partnerships with other international organizations, to access international expertise and knowledge from academics because of the program.

The document review showed that beneficiary organizations were extremely resourceful in how they organized, planned, networked, and implemented their RPIA initiatives. Partnerships with beneficiaries built sponsor sectors’ capacity in these areas.

The majority of interview groups agreed that the program contributed to a great extent to increasing the Secretariat’s knowledge of emerging policy issues, and to some extent to enhancing departmental capacity to produce thorough and relevant analysis. Interviewees stated that the contribution to these outcomes was indirect or limited because initiatives alone do not make policy. Rather, the results of initiatives are used to influence policy development by bringing awareness of a changing environment.

Similarly, the majority of beneficiaries reported achieving intermediate outcomes to a great extent or to a very great extent.

The document review demonstrated that even though the application forms collected information on anticipated outcomes regarding how the Secretariat’s capacity would be enhanced and how the sectors’ knowledge would increase as a result of the initiatives, there was limited information in the files to demonstrate the actual impacts. Evaluators found that project evaluation grids, which would have provided a narrative for the capture of these impacts, were not systematically completed by sponsors.^{Footnote 13}

The majority of interview groups believed that the program contributed to some extent to the achievement of the effectiveness of public service management by virtue of long-standing research partnerships with international organizations. These respondents were less certain of the extent to which the achievement of final outcomes could be attributed solely to the program. Nevertheless, the majority of beneficiaries responding to the questionnaire reported that their projects contributed to a great extent or to a very great extent to the achievement of the RPIA Program’s final outcomes.

As previously noted, evaluators detected some gaps in the documentation of actual versus planned results, making a verifiable assessment of impacts difficult. However, there was evidence that all 11 initiatives proceeded according to their proposals, that all payments were appropriate and accurate, and that all initiatives were completed.

Conclusion: Effectiveness

Overall, the program achieved the outcomes articulated in its logic model to varying extents. However, the lack of performance measures and the inconsistent use of project evaluation grids created difficulty in readily demonstrating evidence of the program’s impacts. The absence of a formal central administrator or program manager led to challenges for sponsors in understanding roles and responsibilities. Sponsors had an uneven understanding of the program, resulting in incomplete tracking of performance measures and inconsistent use of project evaluation grids. A low level of program awareness contributed to the lack of use by some sectors.

The program’s terms and conditions are based on the 2000 version of the Policy on Transfer Payments rather than the 2008 version. Consequently, there is a lack of alignment with some current policy requirements such as the need to implement performance measurement, service standards, and risk-based assessments for monitoring, as well as to integrate official languages requirements and to post program information on the Secretariat’s external website.

Conclusion: Economy and Efficiency; Design Alternatives

Outcomes were achieved in the most economical manner possible. Although minimal resources were expended to administer the program, a more focused program management approach in line with the program’s low materiality and risk would promote even more efficient operations.

1.1 Continuing Need for the Program

• To what extent does the Research and Policy Initiatives Assistance (RPIA) Program address a continuing need?
• Is the program responsive to departmental and beneficiaries' needs?

1.2 Alignment with Government Priorities and the Secretariat’s Strategic Outcome

• Do RPIA program objectives align with government and the Secretariat’s priorities?
• To what extent are RPIA initiatives consistent with the Secretariat’s priorities?

1.3 Alignment with Federal Roles and Responsibilities

• Is there a role for the federal government in improving, advancing, and promoting public service management practices that contribute to the advancement of public policy research and public service management?

2.1 Effectiveness

a. Program Design

• Are the program design and eligibility criteria appropriate?
• Are the roles and responsibilities of the program components clear? Appropriate?
• Is the program’s communications strategy adequate?
• To what extent does the RPIA Program’s performance measurement system meet management’s information needs?
• To what extent is the RPIA Program aligned with the Policy on Transfer Payments?

b. Achievement of Immediate Outcomes

• To what extent is the RPIA Program achieving its immediate outcomes?

c. Achievement of Intermediate Outcomes

• To what extent is the RPIA Program achieving its intermediate outcomes?

d. Achievement of Final Outcomes

• To what extent is the RPIA Program achieving its final outcomes?

2.2 Economy and Efficiency

• To what extent is the RPIA program economical and efficient?

2.3 Design Alternatives

• What might optimize the performance of the program?