ARCHIVED - Evaluation of Foundations

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

II. Methodology

A. Approach

Our approach to the collection of information for this evaluation relied upon four inter-related lines of enquiry, as shown in Exhibit II-1.

Exhibit II-1

Approach to data collection, analysis and reporting

This approach was designed to provide information from multiple sources to enable the evaluation issues to be assessed from several perspectives and to better understand the positions advanced by participants who are most closely involved with the use of foundations for public policy purposes. The study also had to be completed within a relatively short time period – from September, 2006, to January, 2007 – which necessitated a concentrated approach to data collection.

B. Methodologies

1. Documents review

Our evaluation team reviewed a broad range of documentation on the government's use of foundations to achieve policy goals, the evolution of the terms and conditions under which foundation funding has been provided, and the results achieved by various foundations. The principal sources of information included:

• Government of Canada Budget Plans and Economic Statements.
• Funding agreements between the Government of Canada and foundations.
• Finance Canada information relating to the selection of foundation candidates and use of the foundation model.
• Treasury Board documentation regarding the establishment and administration of funding agreements with foundations.
• Foundations' annual reports and corporate plans.
• Evaluation studies conducted on behalf of foundations.
• Reports of the Office of the Auditor General, and Standing Committee on Public Accounts and Standing Senate Committee on National Finance relating to foundations.

The review of documentation was used to identify information relating to the rationale for establishing foundations as instruments of public policy, governance and accountability requirements, results achieved to date, and characteristics of disbursements and operating costs.

2. Case studies

A series of six case studies of selected foundations were used as one of the two core data collection and analysis methods in the evaluation. The six case study foundations were:

• Canada Foundation for Innovation (CFI)
• Canada Millennium Scholarship Foundation
• Genome Canada
• Aboriginal Healing Foundation (AHF)
• Green Municipal Fund (GMF)
• Pacific Salmon Endowment Fund Society

The case studies were used to obtain insights into the appropriateness, effectiveness and costs of specific foundations, which were used, in conjunction with findings from interviews with other foundations and stakeholders, to identify common characteristics, themes and conclusions applicable to all, or most, foundations.

The methodology for the case studies involved:

• A program of 25 interviews involving 56 participants, composed of:
  • 26 senior managers of the selected foundations and selected board members (15 interviews) plus 3 interviews with 4 stakeholder representatives.
  • 28 officials from the case study foundations' funding departments with liaison and monitoring responsibilities (8 interviews).
• Reviews of supporting documentation related to the performance and effectiveness of foundations, which formed part of the overall review of documentation described in section 1, above.
• A high-level analysis of the case study foundations' costs, which was used to asses their cost structures, trends and cost drivers. To the extent possible, we also compared the foundations' cost structures to those of similar government programs, such as granting councils, Infrastructure Canada and the TEAM program (NRCan).

The case study foundations were selected on the basis of four primary criteria:

• Type of foundation, differentiating between foundations funded using fixed term agreements and foundations with perpetual endowments.
• Relative scale of funding, using three categories:
  • Large—Over $500 million
  • Medium—Between $100 and $500 million
  • Small—less than $100 million
• Sector orientation, differentiating between the various types of sectors/activities supported by foundations. The categories used are:
  • Infrastructure/Technology Demonstration
  • R&D
  • Education
  • Community-based initiatives (AHF, PSEF/PSF)
  • International information exchange and networking.
• Form of funding provided to beneficiaries, differentiating between:
  • Project grants (and loans, in the case of the GMF)
  • Research grants
  • Scholarships and bursaries
  • Funding for internal programs and operations.
• Basis for creation: legislation, Canada Corporations Act or Other.

In addition, the mix and balance of the case study foundations was also checked against three secondary criteria to ensure the six selected provided a reasonable representation of the full range of foundations in operation. These secondary criteria were:

• Funding departments, with the spectrum composed of: Industry, Human Resources and Skills Development (HRSD), Health, Natural Resources, Environment, Indian Residential Schools Resolution Canada, Foreign Affairs, Fisheries and Oceans, and Heritage.
• Years in which the foundations were first announced.
• Number of funding grants received, that is, the number of instances in which government funding was provided.

Exhibit II-1 provides a summary of the characteristics of the six case study foundations against the above criteria. 

Exhibit II-1

Characteristics of the case study foundations

Case Study Foundations	Primary Criteria					Qualifying Criteria
	Type of Funding Agreement	Scale of Funding ($ mill.)	Sector Orientation	Form of Funding	Legal Basis	Depart- ment	Year Created	# of Grants
Canada Foundation for Innovation	Fixed Term	Large ($3,650)	Infra- structure/ Tech. Demo.	Project Grants	Legislation	Industry	96/97	5
Millennium Scholarship Foundation	Fixed Term	Large ($2,500)	Education	Bursaries, Scholar-ships	Legislation	HRSD	97/98	1
Genome Canada	Fixed Term	Large ($600)	R&D	Research Grants	CCA1	Industry	99/00	4
Aboriginal Healing Foundation	Fixed Term	Medium ($390)	Community- Based Initiatives	Project Grants	CCA1	IRSRC	97/98	2
Green Municipal Fund	Perpetual Endowment	Large ($550)	Infra- structure/ Tech. Demo.	Project Grants & Loans	CCA2	NRCan & Environment	99/00	3
Pacific Salmon Endowment Fund Society	Perpetual Endowment	Small ($30)	Community- Based Initiatives	Project Grants	B.C. Society Act	Fisheries & Oceans	00/01	1

1. I corporated under the Canada Corporations Act as a not-for-profit organisation.

2. Parent organisation, Federation of Canadian Municipalities, incorporated under the Canada Corporations Act.

3 Key informant interviews

Additional interviews were conducted with representatives of other foundations, their funding departments, central agencies and selected programs operating in similar fields to foundations. These interviews were used to obtain breadth of coverage regarding the appropriateness of using foundations, their effectiveness, and approaches to the design of governance structures and administration of funding agreements. A total of 26 interviews with 38 participants were conducted, involving:

• 14 interviews with 19 representatives of other foundations:
• 5 interviews with 5 representatives of the other funding departments (plus 3 interviews with 6 representatives of 2 departments who spoke in relation to a case study foundation and a non-case study foundation).
• 7 interviews with 14 representatives from Treasury Board Secretariat, Finance Canada, Natural Sciences and Engineering Council (NSERC), Canadian Institutes of Health Research (CIHR) and the Technology Early Actions Measures (TEAM) program at Natural Resources Canada.

A list of the participants in the interviewing program is presented in Appendix A.

4. Literature review

In preparation for the evaluation, TBS commissioned a study to identify international best practices pertaining to the governance and accountability of organizations that shared many features of the foundation model, relating to what have become known as "semi-autonomous organizations" and "non-departmental public bodies" (or "quangos"). We reviewed the findings from this research and conducted a limited further search of the academic and "grey" literature (non-peer reviewed, usually government-sponsored reviews) to determine any lessons regarding the effectiveness of these approaches and their governance. A list of the key documents reviewed is presented in Appendix B.

C. Limitations of the approach

This evaluation study of the use of foundations draws on the findings from a series of cases studies; interviews with representatives of foundations, funding departments, central agencies and a small number of other stakeholders; and a supporting review of documentation and research literature.

In reviewing the findings from the work a number of limitations of the methodology should be borne in mind. Firstly, the participants in the key informant interviews formed a convenience sample, composed of people with direct roles in the management and monitoring of foundations, and their relationships with, and accountabilities to, the federal government. At a time when a number of the foundations are approaching periods where decisions regarding renewal and refunding, or winding up, will need to be made we had to be cognizant of the different degrees of possible self-interest at play and to seek to balance or cross-validate respective views. 

Our sample of key informants had broad representation from across the spectrum of foundations and funding departments, which provided us with a good balance of perspectives on the appropriateness of foundations as instruments of public policy. However, the limited time within which the data collection and analysis was undertaken precluded us from including a larger pool of representatives of foundations' partners and stakeholders, particularly at other levels of government and within the public research community, who could have added to the richness of the analysis of effectiveness.

With regard to the analysis of what foundations cost, we had originally anticipated developing estimates of the costs of delivering foundations programs using departmental programs. However, we found that the very reasons why foundations have been established meant that undertaking these same activities within the departmental context would require a significantly different program design and delivery structure from that which foundations are able to use. In the time available, we were not able to develop a reliable basis for such comparisons and limited our analysis of costs to the cost structures and trends of foundations, particularly those of the case study foundations, and comparisons to the operating and administration costs of a number of similar government organisations. These comparators were chosen on the basis of the similarity of their programs and activities to those of some foundations, particularly those that research or infrastructure projects. We also supplemented this with a qualitative analysis that drew upon the foundation and departmental representatives' views of the cost differences that would arise if foundation activities were to be delivered using departmental programs.

 

• Previous Page
• Table of Contents
• Next Page